The second loop the Simple Agility stack runs against its observability plane is operational rather than commercial: an alert fires, something diagnoses it, something acts, a human is kept in the loop at the right level. Today that "something" is a human on Telegram. This page documents the intended shape — the on-call agent, the escalation ladder, and above all the interfaces each product must expose to be operable by an agent — so that products being built now grow the right surfaces instead of retrofitting them.
Status honesty (verified 2026-08-18): jakemon routes 33 alert rules to Telegram with a Watchdog heartbeat; none of the 33 carries a runbook label; there is no incident runbook page in the wiki (all 16 type: runbook pages are developer procedures); no on-call agent exists; jakemon has no Model Context Protocol (MCP) surface. Wikantik's scoped admin:<area> grants and admin MCP are the only product-side pieces already in place. Everything below marked planned is sequenced in SimpleAgilityRoadmap.
jakemon is config, dashboards and data by rule — no application code. The on-call agent therefore lives in the stack-level agent runtime (not yet built), the same component that will host the Search Engine Optimization (SEO) actuator from the feedback-loop pattern. One runtime gives one place for agent scheduling, action ledgers, circuit breakers, blast-radius limits and the human-oversight policy switch, shared by every agent that acts on the stack.
correlation_id trail from the telemetry contract. Today raw Hypertext Transfer Protocol (HTTP) on the Local Area Network (LAN); the plane is expected to grow an agent-facing read surface (an MCP server exposing alert state, metric queries, log queries and dashboard links) — a jakemon deliverable.The hop from "an alert fired" to "here is the procedure" needs to be deterministic — an agent must not be searching for its runbook during an incident. Neither end of the binding exists yet, so the convention is defined here before anything is written against it.
On the alert (jakemon): every rule carries a runbook label whose value is the exact Wikantik page slug.
- alert: WikantikDown
expr: up{job="wikantik"} == 0
for: 5m
labels:
severity: critical
service: wikantik
runbook: IncidentWikantikDown
annotations:
summary: "Wikantik on {{ $labels.instance }} has been unreachable for 5m"
On the page (Wikantik): type: runbook is already in use for developer procedures ("how to write an MCP tool"), so incident runbooks are distinguished rather than mixed in — a filter on type: runbook alone must not return build instructions to an agent handling an outage.
type: runbook, plus the tag incident-runbook, plus a frontmatter field alert: naming the alert(s) the page serves.Incident<AlertName>, so the binding is guessable from either direction even if a label is missing.list_pages_by_filter(type=runbook, tag=incident-runbook) filtered on alert:, with the runbook label as the direct path.Structure each incident runbook the way the agent will use it: what fired and what it means; how to confirm it is real (the exact PromQL/LogQL); the ordered remediation steps annotated with their escalation rung; what to check to confirm the fix; when to stop and page a human. A runbook whose steps do not name their rung forces the agent to guess how invasive it is allowed to be.
Written with the alert rule, not after the first incident — an alert without a runbook is an incomplete change.
Agents should be willing to escalate to invasive tools — but escalation has obligations, and whether the agent may act invasively before a human looks is a company policy decision, configured in the runtime, not fixed here.
| Rung | Means | Obligations |
|---|---|---|
| 1 | Product's scoped admin Application Programming Interface (API) / MCP tools (rebuild an index, reflow a page, rotate a credential, restart a connector) | ledger entry |
| 2 | Container-level operations through the deployment tooling (restart a service, redeploy a pinned version) | ledger entry; human notified |
| 3 | ssh / docker exec / direct database access | concurrently page the human on-call so a person oversees the actions; wait for human acknowledgment if company policy says so; ledger entry with the exact commands |
| — | after any rung ≥ 2 | trace what would have prevented the escalation — which scoped surface, runbook step, alert or metric was missing — and file it as a work item on the product or the plane. Escalations are supposed to become rarer because each one improves a surface. |
The runtime hosts several agents with different powers, and they do not share an identity. Getting this wrong once — one AllPermission service account reused by everything — silently removes the ladder, because every rung becomes reachable from rung 1.
admin:<area> model. An agent's account holds admin:index, admin:connectors and so on, never AllPermission.A Simple Agility product is on-call-automatable when it exposes:
admin:<area> grants (AdminPermission, AdminAuthFilter) with a dedicated per-integration account are the template.runbook labels./metrics, structured logs with event and correlation_id, a liveness endpoint — because the agent diagnoses from the plane, never by reading the product's files.And from the plane: alert rules that carry runbook, service and instance labels, a webhook receiver for the runtime, and the planned read surface.
Scheduling; the action ledger; circuit breakers ("stop after N actions per hour per service"); blast-radius limits ("one page per run" for the SEO actuator; "one restart per service per day" for on-call); the human-oversight policy switch per rung; agent identity and credential custody; the escalation obligations above. Neither a product nor jakemon should implement these.