Incident Response: SRE On-Call Runbooks, Incident Command Systems, and Forensic Triage

When complex distributed systems experience catastrophic failures (cascading thread pool exhaustion, database connection saturation, corrupted replication logs), standard development practices are insufficient. Ad-hoc debugging during active outages leads to communication breakdowns, conflicting mitigation attempts, and prolonged Time-to-Mitigate (TTTM).

SRE Incident Response adapts proven emergency management frameworks—specifically the Incident Command System (ICS)—to coordinate rapid diagnosis, clear stakeholder communication, and surgical mitigation under pressure.

1. Quick-Reference: The Incident Severity Matrix

+-----------------------------------------------------------------------------------------------------------------------+
|                                           INCIDENT SEVERITY CLASSIFICATION                                            |
+-----------------------------------------------------------------------------------------------------------------------+
| Severity Level         | Customer Impact                        | Page SLA                   | Command Protocol       |
+------------------------+----------------------------------------+----------------------------+------------------------+
| SEV-1 (Critical)       | Core platform down / Data loss         | < 5 minutes (24/7 Page)    | Full ICS Bridge        |
| SEV-2 (Major)          | High latency / Partial feature outage  | < 15 minutes               | Incident Lead + Scribe |
| SEV-3 (Moderate)       | Non-critical internal tool degraded    | Next Business Day          | Ticket Assignment      |
+-----------------------------------------------------------------------------------------------------------------------+

2. Incident Command System (ICS) Roles

                     +-----------------------------------+
                     |      Incident Commander (IC)      |
                     | (Owns process, authority, cadence)|
                     +-----------------+-----------------+
                                       |
              +------------------------+------------------------+
              |                                                 |
              v                                                 v
+---------------------------+                     +---------------------------+
|    Operations Lead        |                     |   Communications Lead     |
| (Directs technical debug) |                     | (Status page & Exec comms)|
+-------------+-------------+                     +---------------------------+
              |
              v
+---------------------------+
|   Investigating Scribe    |
| (Maintains timeline & logs|
+---------------------------+

Incident Commander (IC): Holds total operational command over the incident. The IC does not write code or debug logs; their sole responsibility is maintaining team coordination, assigning technical tasks, and establishing status update cadences.
Operations Lead: Coordinates subject matter experts (database admins, backend engineers) investigating root causes.
Communications Lead: Updates external public status pages (status.wikantik.com) and internal executive leadership, shielding technical responders from interruption.
Scribe: Logs a continuous timeline of events, hypotheses tested, dashboards consulted, and configuration changes deployed.

3. The Golden Hour Forensics Protocol

During an active outage, the immediate priority is Mitigation First, Root Cause Second:

Capture Ephemeral Evidence: Before restarting a frozen JVM or recycling a Kubernetes pod, capture thread dumps (jstack), heap profiling samples, and connection pool states.
Rollback First: If the outage coincides with a recent software deployment or configuration change, execute an immediate rollback before attempting live surgical fixes.
Shed Load: Activate shedding rate-limits (429 Too Many Requests) to allow database backends to recover from connection saturation.

References

Beyer, B., et al. (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media (Chapter 28: Managing Incidents).
FEMA. (2020). National Incident Management System: Incident Command System. Emergency Management Institute.
Allspaw, J. (2016). Post-Mortem Action Items: How to Fix What Broke Without Breaking the Culture.