Part of the Simple Agility stack. jakemon is the stack's observability plane: the data backbone that every Simple Agility product (Wikantik, Roller, Stockroom and those to come) joins its own data against. Its contracts with the rest of the stack are on their own pages. SimpleAgilityObservabilityPlane covers what it owns, push feeds versus pull queries, and the "do not build this in your backend" list. SimpleAgilityTelemetryContract covers what products must emit. Whether each contract item is built is tracked in SimpleAgilityRoadmap. This hub covers how jakemon itself is built and operated.
jakemon is the centralized observability stack for the host fleet: metrics, logs, dashboards and alerting. It lives in its own repository (~/source/jakemon), separate from Wikantik. The old Wikantik observability overlay was retired in May 2026 and must not be re-added there. The repository holds config, compose files, dashboards, stdlib-Python exporters and bash. It has no application code, by rule, with one exception (SA-4, 2026-10-12): an application that serves the whole stack and belongs with the plane. The planned agent-facing query server is that exception, and it will live here. The agent runtime is not covered; it is a separate stack-level component.
Release: v1.0.3 (tagged 2026-10-24) is current. It makes IndexNow verify on wikantik.com, jakefear.com and maiiavorobiova.com, and restores the CrafterCMS proxy's JSON access log, so craftercms-traffic has per-vhost data again. v1.0.2 (2026-10-24) made the MCP tool panels count Wikantik's ECS event.action=mcp_tool_called log lines, and puts deployment on alerts from the central stack's own scrapes. v1.0.1 (2026-10-24) added an estimated human-traffic row and an MCP burst row to wikantik-traffic, and swapped that dashboard's map to OpenStreetMap. v1.0.0 (2026-10-11) was the first tagged release.
| Kind of fact | Canonical home |
|---|---|
| Cross-project facts: jakemon's role, the contracts, what is built, the fleet and ingress | This wiki. Edit it here. The jakemon CLAUDE.md points here. |
| jakemon internals: config, deploy gotchas, pinned images | The jakemon repository's CLAUDE.md and docs/ |
Incident runbooks (Incident<AlertName> pages) | docs/runbooks/ in the jakemon repository, published to this wiki by bin/runbook-publish.py. Edit the file, not the page, or the next publish overwrites your edit. |
| Dashboards | dashboards/ in the repository, provisioned into Grafana. Do not edit them in the UI. |
| Central stack | Prometheus v3.13.2 (LTS line), Loki 3.7.6, Grafana 12.4.8, Alertmanager v0.33.1, on inference (192.168.5.53) since 2026-10-09. It ran on docker2 from 2026-07-21. |
| Agents | Grafana Alloy v1.18.1 plus a container-meta sidecar on all six hosts: inference, docker1, docker2, cloudflare, nas, minisforum |
| Alerting | 46 alert rules across 12 rule files, each bound to a runbook; delivered to Telegram; the Watchdog heartbeat pings an external dead-man's switch |
| Recording rules | 31, including the host_container:*, host_service:* and host_slice:* resource-attribution series |
| Dashboards | 18 provisioned. See JakemonDashboardsInventory. |
| Retention | Prometheus 90 days (50 GiB cap); Loki 30 days |
| Scale | About 77–81k active series and 2.4k metric names (2026-10-24) |
The model is push-based. Every fleet host runs a Grafana Alloy agent: the universal config is agent/config.alloy, and per-host scrape targets and probes live in hosts/<host>/apps.alloy. The agent pulls exporters over loopback and pushes to the central stack: remote_write for metrics, loki.write for logs. Nothing on the central host scrapes the fleet.
network_mode: host, since 2026-10-11), so the node exporter sees the real NICs, sockets and conntrack. Network history before that date describes the agent container, not the host.container-meta sidecar (stdlib Python, 127.0.0.1:9488) does two jobs:
job="container" plus a container label. Selected host files and journals are tailed too: the edge nginx log, cloudflared, Pi-hole and ollama.cf-analytics-exporter: Cloudflare zone analytics.visibility-exporter: Google, Bing and Yandex webmaster data. See SearchVisibilityExporter.visibility-shipper: the nightly push feed into Wikantik's /admin/insights/ingest. This is the reference push feed of the plane contract.bin/deploy-exporter.sh):
crafter-exporter on docker2.nginx-exporter on cloudflare.gpu-exporter (DCGM) and ollama-exporter on inference.| Label | Meaning |
|---|---|
host | The fleet host the data came from. Check it before diagnosing production from a metric, because minisforum (the workstation) runs development copies of production containers. |
deployment | The whole stack deployment (the tenant). Today the only value is jakemon-home. It is on every agent-shipped series and log stream. |
service, service_instance | The product (wikantik) and one running copy of it, named <product>-<env> (wiki-prod, wiki-dev). Set per target in apps.alloy, on metrics and on the product's log streams (SA-5). Live since 2026-10-12: wiki-prod (docker1, metrics and logs), crafter-prod (docker2, metrics and logs), roller-prod (docker2, logs only, because Roller has no /metrics yet), and wiki-dev and crafter-dev (minisforum, logs). service always means the product; the per-systemd-unit host_service:* series use unit. |
job | The scrape job. Alloy's built-in exporters are integrations/unix (host), integrations/cadvisor (containers), integrations/postgres and integrations/blackbox/<target>. Applications use their product name (wikantik). |
instance | Prometheus's own meaning: the scrape target (host.docker.internal:8080, alertmanager:9093). It is not a product-instance name; that is service_instance. |
container | Docker container name, the same on metrics (via container-meta) and logs. Auto-named docker run containers collapse to ephemeral. |
http://inference.lan:9090/api/v1/query. It is open on the LAN, with no auth. For dashboard data, lift the PromQL from the dashboard JSON in dashboards/ rather than going through Grafana.http://inference.lan:3100/loki/api/v1/query_range, also open on the LAN.http://inference.lan:3000 on the LAN, and monitor.jakefear.com through Cloudflare Zero Trust. It needs a login.ssh inference 'curl -s localhost:9093/api/v2/alerts'.Pull queries are for ad-hoc and agent reads only: never from a product's request path, never from a browser. See the plane contract.
Every alert rule carries runbook: Incident<AlertName>, and bin/validate.sh, the commit gate, rejects a rule without one. There are 42 Incident* pages on this wiki, one per alert name. Each is published from docs/runbooks/ with type: runbook, the tag incident-runbook and an alert: field, following the binding convention on SimpleAgilityOnCallAutomation. bin/runbook-check.sh verifies that every label resolves to a live page. Start with IncidentHostNotReporting, IncidentServiceDown or IncidentWatchdog.
bin/validate.sh: the gate before every commit. It checks config, rule unit tests, dashboard JSON, shellcheck, image pins and runbook labels.bin/deploy-all.sh: central, every agent and every host exporter.bin/deploy-central.sh and bin/deploy-agent.sh <host>: one layer at a time.bin/deploy-exporter.sh <host> <exporter>: one host exporter.bin/runbook-publish.py then bin/runbook-check.sh: after editing a runbook.hosts/<host>/.SSH_<host> line to remote.env.JAKEMON_HOSTS./metrics, a health endpoint and ECS JSON logs, per the telemetry contract.apps.alloy, labelled with service and service_instance.webmaster_* metrics.Cross-project decisions are recorded in SimpleAgilityDecisionLog. The ones that change jakemon:
service and service_instance="<product>-<env>", attached per target in apps.alloy, and inherited by alerts on product series. Built 2026-10-12. Every host's apps.alloy defines a discovery.relabel "products" hook, empty where no product runs, and bin/validate.sh enforces it.| json, which turns log.level into log_level and trace.id into trace_id.