Apr 08, 2026
Infrastructure
Notes on observability for GPU-bound services
The interesting metric is not latency. It is the variance of latency conditioned on batch composition.
· read
01
// domain 01
My Home Data Center
A serious homelab built for serious work: GPU-bound inference, observability across the stack, Kubernetes orchestration, and quiet redundancy. Less rack-photography, more thoughtful capacity planning.
Proxmoxk3sRTX PRO 6000OpenTelemetryClickHousevLLMTailscaleUniFiTrueNAS
// architecture
Topology, network, storage.
A realtime 3D map rendered in three.js and fed by the Proxmox API. The hub at the center is the host (node01); each orbiting node is a guest, a QEMU VM (faceted) or an LXC container (cube), sized by allocated memory and colored by role. Stopped guests read dim with no traffic; running guests stream packets scaled to their live CPU. When the BMC is reachable the host shows live temperature and power. Drag to orbit, tap any guest for its details.
topology · home-data-center · realtime
drag · scroll · tap
// demo data renders when Proxmox is unreachable. Set NUXT_PVE_URL + NUXT_PVE_TOKEN to read live guest status, and NUXT_BMC_URL + NUXT_BMC_API_KEY to overlay the host's telemetry. Only coarse guest state leaves the server.
// overview
Why I built it.
Cloud is wonderful for most things and wrong for a few. I run my own infrastructure for the same reason I cook my own dinner, not because it is cheaper, but because the decisions belong to me.
The goals: a stable platform for the platforms I build, a quiet place to run GPU workloads that would be expensive elsewhere, and a real environment to learn the deeper layers of the stack: power, cooling, networking, observability, storage.
The long-term vision is unglamorous: still here in ten years, still running, still teaching me something every quarter.
// rack & hardware
What it's made of.
A single 25U rack in a sound-treated alcove. The build philosophy is overspec where it matters (memory, networking, cooling) and underspec where it does not (chassis bling, RGB).
Rack
StarTech 25U Enterprise-Grade Server Rack Cabinet
19in enclosed 4-post · 35in mounting depth · 3307lb capacity
Chassis
Gigabyte R283-Z97-ACL1 barebones 2U rackmount server
dual-socket SP5 platform
Compute
2 × AMD EPYC™ 9115 · 16-Core 2.6 GHz · 64 MB Cache
125 W per socket
Memory
256 GB DDR5 5600 MHz ECC Registered DIMM
oversubscribed comfortably
Storage
2 × Samsung PM1743 1.92 TB U.3 NVMe
PCIe 5.0 x4 · 2.5" SSD
Network
Broadcom NetXtreme E-Series Dual-Port 25GbE NIC
PCIe Ethernet
Power
1+1 2700 W 80 PLUS Titanium redundant PSUs
hot-swap redundancy
GPU
NVIDIA RTX PRO 6000 Blackwell Max-Q · 96 GB GDDR7
inference + experiments
UPS
2U Rackmount APC SRT3000RMXLA Smart-UPS On-Line
3 kVA · graceful shutdown via PowerChute API
// photos and full BOM live in the engineering notes. Link forthcoming.
// proxmox
Virtualization strategy.
Proxmox is the substrate. Everything boots from it: the k3s cluster, Home Assistant, observability, the storage gateway. ZFS underneath, cloud-init for fast respins, and an inventory that lives in Git instead of a screenshot.
GPU passthrough is configured per-VM rather than per-cluster, which keeps the GPU available for short experimental runs without disrupting the production cluster.
vm-inventory.yaml
7 active
- k3s-control-01Kubernetes control plane8c16 GB
- k3s-worker-01k8s worker · general16c32 GB
- k3s-worker-gpuk8s worker · GPU passthrough24c96 GB
- home-assistantHAOS · VM with USB passthrough4c8 GB
- observabilityGrafana · InfluxDB · OTel collector8c32 GB
- storage-gwNFS / iSCSI gateway4c16 GB
- devbots-runnerLong-running agent workloads8c32 GB
// kubernetes
Cluster architecture.
k3s, not full k8s. Closer to operating a tool than a fleet. Three nodes: one control, one general worker, one GPU worker with passthrough. Longhorn handles persistent storage; ingress runs through Traefik with cert-manager.
kubectl get deploy -A
8 workloads
| namespace | name | replicas | purpose |
|---|---|---|---|
| inference | vllm-gateway | 1 (GPU) | LLM serving |
| inference | whisper-stt | 2 | speech recognition |
| platform | atlas-collector | 3 | OTel ingest |
| platform | atlas-query | 2 | ClickHouse front-end |
| neurish | web | 3 | Nuxt SSR |
| neurish | api | 2 | core API |
| fwe | web | 3 | Friends With Epilepsy |
| ai | devbots | 4 | autonomous agents |
gpu node
One node, scheduled jealously
Tainted for inference workloads only. Time-sliced when experiments need it; otherwise reserved for vLLM and Whisper.
storage
Longhorn for clusters, NFS for warm tiers
Three-replica volumes on NVMe, automatic snapshots, off-site backups via restic to a colo and to Backblaze.
lessons
Boring is the goal
Fewer custom controllers, more upstream defaults. Every "clever" piece of YAML is a future incident in disguise.
// ai infrastructure
Local inference, agents, RAG.
Most of the AI in my work runs on a single GPU at the back of the rack. Cloud providers fill in where local cannot: frontier models, scaling spikes, anything I'd rather not be on the hook for.
// 01
Local Inference
vLLM on the GPU node serves Llama-class models for offline experiments, RAG, and Friends With Epilepsy moderation pre-screens. CPU-side llama.cpp handles small, latency-tolerant tasks.
// 02
DevBots
Long-running coding agents wired into GitHub, CI, and Temporal. They open PRs against my own repos, run their own tests, and ping me only when they need a decision.
// 03
RAG Pipeline
Document ingest → embeddings (BGE) → pgvector. Retrieval feeds both production assistants and the ongoing-thoughts corpus.
// 04
Vector Database
pgvector inside the existing Postgres clusters: same backups, same observability, same SQL. No second database to babysit.
// observability
What I watch, and how.
OpenTelemetry collectors ingest from every namespace and from the hypervisor itself. Logs land in ClickHouse; metrics in InfluxDB; traces in Tempo. Grafana stitches them together. The dashboards I look at most are not the pretty ones. They are the ones that tell me whether last night's automations behaved.
Project Atlas is the unifying layer, a thin Go service that joins traces with the deploy log, so an incident always has the commit that caused it within arm's reach.
atlas.svg · trace · join · deploy
topology
// live · this is the rack
A read-only window.
Static placeholders today; will refresh from the Atlas API in the next iteration.
cluster nodes
8 / 8
k3s · all healthy
gpu utilization
34%
rtx 4090 · 1× node
AI requests · 1h
1.2k
across 4 models
automations live
142
home assistant
air · indoor
21°C
AQI 38 · humid 47%
power · 24h
14.3 kWh
32% solar
uptime · ingress
99.99%
rolling 30d
storage · pool
412 / 720 GB
longhorn · replicated