Simple Agility On-Call Automation

The second loop the Simple Agility stack runs against its observability plane is operational rather than commercial: an alert fires, something diagnoses it, something acts, a human is kept in the loop at the right level. Today that "something" is a human on Telegram. This page documents the intended shape — the on-call agent, the escalation ladder, and above all the interfaces each product must expose to be operable by an agent — so that products being built now grow the right surfaces instead of retrofitting them.

Status (verified 2026-10-11):

Built: the alert-to-runbook binding, on both ends. jakemon routes 46 alert rules to Telegram, with a Watchdog heartbeat to an external dead-man's switch. Every rule carries a runbook label, and jakemon's commit gate rejects one without it. The wiki has 42 incident runbooks, one per alert name, each tagged incident-runbook with an alert: field.
Not built: an on-call agent, an Alertmanager webhook receiver for it, and a jakemon Model Context Protocol (MCP) surface. The MCP surface is cleared to live in jakemon (SA-4). Wikantik's scoped admin:<area> grants and admin MCP are the only product-side pieces already in place. Everything below marked planned is sequenced in SimpleAgilityRoadmap.

The on-call agent is stack-level, not a jakemon component

jakemon observes and does not act. Its only application code is what serves the whole stack and belongs with the plane, such as the read-only query server (SA-4). The on-call agent acts, so it lives in the stack-level agent runtime (not yet built), the same component that will host the Search Engine Optimization (SEO) actuator from the feedback-loop pattern. One runtime gives one place for agent scheduling, action ledgers, circuit breakers, blast-radius limits and the human-oversight policy switch, shared by every agent that acts on the stack.

The loop

Alert — Alertmanager fires; a webhook receiver delivers the alert payload (labels, annotations, firing time) to the agent runtime. Telegram to a human continues in parallel until the agent has a track record.
Runbook — the alert names its runbook; the agent fetches that page from Wikantik via the knowledge MCP. See The alert-to-runbook binding below, which is a convention that has to be built, not one that exists.
Diagnose — the agent reads the plane: current alert state, Prometheus Query Language (PromQL) and Loki Query Language (LogQL) queries, the trace.id trail in the products' ECS logs, per the telemetry contract. Today raw Hypertext Transfer Protocol (HTTP) on the Local Area Network (LAN); the plane is expected to grow an agent-facing read surface (an MCP server exposing alert state, metric queries, log queries and dashboard links) — a jakemon deliverable.
Act — through the product's scoped, audited operational surfaces (see Required product interfaces), climbing the escalation ladder only as needed.
Record and close — action, rationale and outcome go to the runtime's ledger; the alert resolves, or the agent escalates to a human with the ledger attached; every invasive step produces a follow-up item (below).

The alert-to-runbook binding

The hop from "an alert fired" to "here is the procedure" needs to be deterministic — an agent must not be searching for its runbook during an incident. Both ends now exist, built to this convention. jakemon labelled its rules on 2026-08-18. It published the runbooks from docs/runbooks/ in its repository between 2026-10-02 and 2026-10-08, and added the incident-runbook tag and alert: field on 2026-10-11. bin/runbook-check.sh in jakemon verifies that every label resolves to a live page.

On the alert (jakemon): every rule carries a runbook label whose value is the exact Wikantik page slug.

- alert: WikantikDown
  expr: up{job="wikantik"} == 0
  for: 5m
  labels:
    severity: critical
    service: wikantik
    runbook: IncidentWikantikDown
  annotations:
    summary: "Wikantik {{ $labels.service_instance }} on {{ $labels.host }} has been unreachable for 5m"

On the page (Wikantik): type: runbook is already in use for developer procedures ("how to write an MCP tool"), so incident runbooks are distinguished rather than mixed in — a filter on type: runbook alone must not return build instructions to an agent handling an outage.

type: runbook, plus the tag incident-runbook, plus a frontmatter field alert: naming the alert(s) the page serves.
Page slug convention Incident<AlertName>, so the binding is guessable from either direction even if a label is missing.
Lookup is list_pages_by_filter(type=runbook, tag=incident-runbook) filtered on alert:, with the runbook label as the direct path.

Structure each incident runbook the way the agent will use it: what fired and what it means; how to confirm it is real (the exact PromQL/LogQL); the ordered remediation steps annotated with their escalation rung; what to check to confirm the fix; when to stop and page a human. A runbook whose steps do not name their rung forces the agent to guess how invasive it is allowed to be.

Written with the alert rule, not after the first incident: an alert without a runbook is an incomplete change.

How jakemon implements it.

Source and publishing. The runbook source is docs/runbooks/Incident<AlertName>.md in the jakemon repository, committed alongside the rule it documents. bin/runbook-publish.py publishes it here through the Wikantik admin MCP. Edit the file, not the page, or the next publish overwrites the page.
Structured block. The publisher derives the structured runbook: frontmatter block (when_to_use, steps, pitfalls) from the page's What it means, First checks, How to clear and Notes sections.
Escalation rungs. The steps do not yet name their rung on the ladder below. That is a known gap before an agent acts on them.

The escalation ladder

Agents should be willing to escalate to invasive tools — but escalation has obligations, and whether the agent may act invasively before a human looks is a company policy decision, configured in the runtime, not fixed here.

Rung	Means	Obligations
1	Product's scoped admin Application Programming Interface (API) / MCP tools (rebuild an index, reflow a page, rotate a credential, restart a connector)	ledger entry
2	Container-level operations through the deployment tooling (restart a service, redeploy a pinned version)	ledger entry; human notified
3	`ssh` / `docker exec` / direct database access	concurrently page the human on-call so a person oversees the actions; wait for human acknowledgment if company policy says so; ledger entry with the exact commands
—	after any rung ≥ 2	trace what would have prevented the escalation — which scoped surface, runbook step, alert or metric was missing — and file it as a work item on the product or the plane. Escalations are supposed to become rarer because each one improves a surface.

Agent identities and credentials

The runtime hosts several agents with different powers, and they do not share an identity. Getting this wrong once — one AllPermission service account reused by everything — silently removes the ladder, because every rung becomes reachable from rung 1.

One identity per agent role, not per runtime: the on-call agent and the SEO actuator hold different accounts with different grants. The SEO actuator has no business restarting a connector; the on-call agent has no business editing page content.
Grants are per-area and enumerated — Wikantik's admin:<area> model. An agent's account holds admin:index, admin:connectors and so on, never AllPermission.
Rung 3 credentials are separate again and are not held by the runtime at steady state; obtaining them is itself the escalation event and is what makes rung 3 auditable rather than merely policy-bound.
Every action is attributable — the product's audit log records the acting principal, so "which agent did this" is answerable from the product side without consulting the runtime's ledger.
Rotation is routine, and a revoked agent credential must degrade that agent to read-only rather than crash the runtime (the shipper's idle-on-missing-credential behavior is the precedent).

Required product interfaces (the point of this page)

A Simple Agility product is on-call-automatable when it exposes:

Scoped, audited operational actions as an admin API and/or admin MCP tools, gated by a fine-grained permission the agent's credential can hold without holding everything. Wikantik's admin:<area> grants (AdminPermission, AdminAuthFilter) with a dedicated per-integration account are the template.
Incident runbooks in Wikantik, one per failure mode the product knows about, following the binding convention above and referenced by the plane's alert runbook labels.
The telemetry contract — /metrics, ECS JSON logs with event.action and trace.id, a liveness endpoint — because the agent diagnoses from the plane, never by reading the product's files.
Read-your-own-state endpoints an agent can call to confirm an action took effect (index status, connector run history, queue depth) — the product's view, so the agent is not inferring success from a metric lagging a scrape interval.
Idempotent, safe-to-retry actions, so an agent that loses a response can call again without double effect.

And from the plane: alert rules that carry runbook, service and service_instance labels (SA-5), a webhook receiver for the runtime, and the planned read surface.

What the runtime will own (so products and jakemon do not)

Scheduling; the action ledger; circuit breakers ("stop after N actions per hour per service"); blast-radius limits ("one page per run" for the SEO actuator; "one restart per service per day" for on-call); the human-oversight policy switch per rung; agent identity and credential custody; the escalation obligations above. Neither a product nor jakemon should implement these.