Host: inference — jakemon Central and the GPU

Role: the home of jakemon's central stack, the fleet's observability plane, and the fleet's only GPU. LAN address 192.168.5.53, inference.lan; its own hostname is dell-workstation. Specs: 12 cores, 30 GB RAM, an NVIDIA RTX 4060 Ti with 16 GB, Ubuntu 26.04 LTS. It is a GUI workstation, so its sleep targets are masked; otherwise GDM would suspend the monitoring server. The conventions for the network are in Home Lab Fleet and Ingress.

Ports

Checked with ss -ltn and docker ps on 2026-10-11.

Port	Bound to	Owner
`9090`	all interfaces	Prometheus. LAN-bound on purpose, because agents on other hosts `remote_write` here. No auth.
`3100`	all interfaces	Loki. LAN-bound for log pushes. No auth.
`3000`	all interfaces	Grafana. Public as `monitor.jakefear.com`, through the tunnel and Cloudflare Zero Trust.
`9093`	loopback	Alertmanager. Nothing remote talks to it.
`11434`	all interfaces	ollama, a systemd service rather than a container
`8080`	all interfaces	Open WebUI
`9400`	all interfaces	`gpu-exporter` (NVIDIA DCGM)
`9114`	`172.17.0.1`	`ollama-exporter`, which polls ollama's `/api/ps`
`12345`	loopback	jakemon Alloy agent, which runs in the host network namespace
`9488`	loopback	jakemon `container-meta` sidecar

What runs here

jakemon central (compose project jakemon, under /opt/jakemon): Prometheus, Loki, Grafana, Alertmanager, cf-analytics-exporter, visibility-exporter and visibility-shipper. See JakemonHub.
The host's own jakemon agent (compose project agent). It pushes to the central stack on this same host, exactly as every other host's agent does.
ollama and Open WebUI. jakemon tails ollama's journal for inference throughput and polls /api/ps for the loaded model. Both appear on the jakemon-inference-gpu dashboard.

History

Until 2026-07-21: the central stack ran here.
2026-07-21 to 2026-10-09: the host was packed for a move to the US, and the central stack ran on docker2.
2026-10-03: the host was back and monitored again.
2026-10-09: the central stack moved back here, freeing docker2 for applications. The move followed the order that loses no data: re-point the agents first, then stop the old central, copy its data and start the new one. Running agents hold their unsent backlog and flush it when the new central comes up.

Gotchas

Losing this host blinds the fleet. Nothing else collects metrics or logs. The Watchdog alert pings an external dead-man's switch, so a silent central still pages a human through healthchecks.io. Telegram cannot deliver it, because Alertmanager itself runs here.
Agents push to an IP address resolved at deploy time. After a LAN renumbering, or if this host's address changes, fix DNS and then redeploy every agent (bin/deploy-all.sh). Until then they push to the old address.
Prometheus and Loki are open on the LAN. This is fine on the trusted LAN. Never expose these ports any other way. Grafana is the only surface reachable through the tunnel.
The DCGM exporter version is deliberately behind upstream. Moving to DCGM 4.x needs the NVIDIA driver requirements checked on this host first.

How it is managed

bin/deploy-central.sh (jakemon) deploys the central stack to JAKEMON_CENTRAL=inference.
bin/deploy-agent.sh inference deploys this host's agent.
bin/deploy-exporter.sh inference gpu-exporter and bin/deploy-exporter.sh inference ollama-exporter deploy the host exporters.