Simple Agility Observability Plane (jakemon)

jakemon is the observability plane of the Simple Agility stack: the one place where the fleet's metrics, logs, alerts, and externally-captured signals are collected and retained, and the data backbone every product's optimization loop joins against. This page is the contract; how jakemon is built and operated lives in JakemonHub. Unbuilt pieces named here are sequenced in SimpleAgilityRoadmap.

Today it is Prometheus + Loki + Grafana + Alertmanager on docker2, one Grafana Alloy agent per host pushing to them, two custom exporters (Cloudflare analytics, search visibility), and a nightly shipper that Hypertext Transfer Protocol (HTTP) POSTs retained search-visibility history into Wikantik. It is config, dashboards, Python-standard-library exporters and bash — there is no application code in jakemon and that is a rule, which is why the agent runtime is a separate stack-level component.

What the plane owns

JobExamples todayOwner
1. Fleet telemetry — host, container, database, edge and application metrics; log aggregation; alert rules and routing; dashboardsnode/cAdvisor/postgres exporters, 33 alert rules → Telegram, 17 provisioned dashboardsplane
2. External data capture and retention — polling third-party sources and keeping the historyGoogle Search Console (GSC), Bing and Yandex webmaster Application Programming Interfaces (APIs) → webmaster_* gauges + snapshot files; Cloudflare zone analytics; IndexNow submissions; the URL Inspection sweepplane
3a. Source-generic signal detection — anything derivable from a single external source with no product knowledgevisibility/opportunities.py (content_gap, decay, …), the measured Click-Through Rate (CTR)-by-position curveplane, shipped as evidence
3b. Domain decision-making — ranking, gating, calibration, backlog, actuation, effect measurementWikantik OpportunityEngine, traffic gate, WeightCalibrator, EffectEvaluatorproduct

The rule that decides 3a from 3b: if computing it needs the product's own tables or vocabulary, it belongs in the product. Every product in the suite has its own optimization needs and joins its own data set against what the plane makes visible — whether that is gathered from external sources like search engines or constructed from the product's own logged errors and user behavior. As the stack evolves, opportunities.py-like logic shifts left into the products; the plane keeps the capture and the retention.

The rule that decides where data lives

Read this before the "do not build this" table below, because the table is meaningless without it and reading the table alone will lead you to delete things you should keep.

Telemetry emitted so somebody can observe the system goes to the plane. Data that is a functional input to the product's own behavior — its loop, its ranking, a user-facing feature, a compliance obligation — is a product table.

The same event can legitimately produce both. A retrieval query is logged to stdout (plane: "how much traffic is the retriever taking, is it erroring") and written to retrieval_query_log (product: the agent_gap rule reads it to decide what content is missing). That is not duplication to be cleaned up; the two records answer different questions, have different retention needs, and have different owners.

Product tables that are correct and must not be "consolidated" into Loki: Wikantik's retrieval_query_log, briefing_log, content_change_log, search_visibility_snapshot, content_opportunity_seen, and the /admin/audit log. Each is a functional input or a compliance record, not observability.

The test to apply: if this table vanished, would the product behave differently, or would only a dashboard go blank? Behave differently → product table. Dashboard goes blank → it belonged in the plane.

Out-contract: how products get data from the plane

Two mechanisms, chosen by a rule, not by taste.

Push feeds — for durable datasets

If a product needs to join the data with its own tables, retain it longer than the plane's retention, or show it to its own users, the plane ships it and the product owns a table. Prometheus is a time-series store, not a join engine; Loki retention is the plane's decision, not the product's.

Every push feed has the same shape (the search-visibility feed, ship_visibility.pyPOST /admin/insights/ingestsearch_visibility_snapshot, is the reference implementation):

Canonical feed payload

The ingest body is a JavaScript Object Notation (JSON) envelope: a feed name, a feed_version, and rows. Everything else is feed-specific.

{
  "feed": "search_visibility",
  "feed_version": 1,
  "as_of": "2026-08-17",
  "source": "jakemon",
  "rows": [
    { "engine": "google", "site_host": "wiki.wikantik.com",
      "page_path": "/wiki/HybridRetrieval", "query_text": "hybrid retrieval",
      "impressions": 41, "clicks": 2, "position": 12.4 }
  ]
}

Feed evolution

The envelope evolves like the log envelope, with one extra rule because feeds cross a version boundary between two independently-deployed repositories:

  1. Consumers ignore unknown fields; they never reject on them. The plane must be able to add a column and deploy before the product knows about it. A product that 400s an unrecognized key makes every plane change a lock-step release.
  2. feed_version increments only on a breaking change — a removed field, a renamed key, a changed unit or meaning. Additive changes do not bump it.
  3. A product may refuse a feed_version it does not know, and should say so in the error body rather than ingesting rows it will misread.
  4. Rows the product cannot use are skipped and counted, never silently dropped — the same reasoning as the Knowledge Graph's skippedNonConformantCount.

Feeds and multiple product instances

The stack is instanced per tenant, but one tenant may run several instances of one product (wiki-prod and wiki-support, say) against a single plane. That makes "which instance receives this feed?" a real question, and the answer is deliberately simple:

Requesting a new feed

A product may not poll a third-party source to get around a missing feed. The process:

  1. Check the capture exists. If the plane is not collecting the underlying data at all, that is a new exporter — a plane work item, not a feed.
  2. Define the row shape and the natural key with the plane's owner, along with the settle/lag characteristics and how far back backfill must reach.
  3. The product ships first: the migration, the idempotent upsert, the ingest endpoint, and the scoped credential — so the receiving end exists before anything is sent.
  4. Then the plane adds the shipper target and backfills.

Known gap: there is no generic aggregate-and-ship mechanism today. ship_visibility.py is bespoke to search visibility, and a second feed (Roller's engagement data, for example) currently means writing a second shipper by hand. Generalizing it is a plane work item tracked in SimpleAgilityRoadmap; until it lands, budget for a bespoke shipper per feed and do not assume a feed can be conjured from an existing metric.

Pull queries — for ad-hoc and agent reads

A one-off question ("error rate on /api/bundle since the deploy?", "is the alert still firing?") is a read-only Prometheus Query Language (PromQL) / Loki Query Language (LogQL) query against the plane's HTTP APIs. No table, no feed. Two hard limits:

Today pull is raw HTTP on the trusted Local Area Network (LAN); an agent-facing query surface on the plane (Model Context Protocol (MCP)) is planned — see SimpleAgilityOnCallAutomation.

Roadmap: Operational Data Store and SLO/SLA KPIs

The plane relies today on the default stores behind Prometheus, Loki and the visibility snapshot files. In scope for the plane's evolution — and therefore not for any product to build — is a more accessible, queryable Operational Data Store (ODS) carrying a deeper layer of Key Performance Indicators (KPIs) on Service Level Objectives (SLOs) and Service Level Agreements (SLAs): availability, latency and error budgets per service and per tenant, computed once by the plane and shipped or queried like any other feed. A product that needs an SLO number reads it from the plane; it does not compute one from its own request log.

Do not build this in your backend

Apply The rule that decides where data lives first. This table is about observability infrastructure, and every row assumes the data in question exists only so somebody can watch the system. An agent adding a capability to any Simple Agility product should stop if it finds itself writing any of the following:

You are about to write…Use insteadNot covered by this row
A metrics table, a counter store, a /stats cache of request rates/metrics scraped by the plane (telemetry contract); PromQL for readsa counter the product acts on (rate-limit state, quota)
A log table, or an events table whose only purpose is "user did X" reportingstructured log lines with an event field → Loki; LogQL for readsa table the product's own loop reads (retrieval_query_log), or a compliance audit log (/admin/audit)
A poller for Search Console / Bing / Yandex / Cloudflarethe plane's exporters + a push feed
Alert evaluation, paging, escalation timersPrometheus rules + Alertmanager; the product exposes health and metrics onlyin-product user notifications, which are a feature
An SLO / uptime / error-budget calculatorthe plane's ODS/KPI layer (roadmap)
A cross-service trace storecorrelation_id in the log envelope; the plane correlates
A per-instance dashboarda provisioned Grafana dashboard in the jakemon repositorya product admin screen showing product state

What does belong in the product: the tables that hold shipped feeds, the joins against product data, ranking/gating/calibration, the backlog, actuation surfaces, effect measurement, and any user-facing presentation of the result.

Boundaries the plane will not cross

See also