leadership · high

Production Incidents: On-Call, RCA & Blameless Postmortems

Asked about a production incident, the order of operations is the whole answer. Mitigate before you diagnose: the instinct to root-cause first is wrong during an active incident, so restore service immediately — roll back the recent deploy (the prime suspect), fail over, disable the feature flag, or scale up — to stop user impact, then investigate on a stable system; detection should come from alerting on SLOs and symptoms (error rate, latency) rather than only known causes, and stating severity early right-sizes the response. Incidents are coordination problems, so assign an Incident Commander to drive (kept out of hands-on debugging) with clear ops and comms roles and regular status updates so the organization isn't guessing, and keep a timeline as events unfold. Once stable, do genuine root-cause analysis: reproduce the failure, ask the '5 whys' to get past the surface trigger to the systemic cause, and verify the fix. Finally, run a blameless postmortem whose purpose is prevention, not punishment — assume people acted reasonably with the information they had and fix the system and process (better alerts, guardrails, tests, automation, rollback paths) rather than telling someone to 'be more careful' — documenting the timeline, user impact, root cause, what went well and badly, and action items with named owners and due dates. In an interview this signals maturity: calm under pressure, users first, and a prevention mindset over finger-pointing.

The code

/*  "Tell me about a production incident you handled."
    The order matters:
      1. MITIGATE first  - stop the bleeding (rollback/failover)
      2. COMMUNICATE     - severity, status, an incident commander
      3. ROOT CAUSE      - only after it's stable (5 whys, repro)
      4. POSTMORTEM      - blameless: fix the system, not the person
    Signal: calm under pressure, users first, prevention mindset.   */

What this lesson walks through

1. 01Mitigate before you diagnose
2. 02Coordinate: roles, comms, RCA
3. 03Gotcha — blameful postmortems + ownerless actions
4. 04Blameless postmortem — fix the system

The instinct to find the root cause first is wrong during an active incident. Restore service FIRST — roll back the recent deploy, fail over, disable the feature flag, scale up — stop the user impact, then investigate on a stable system. Detection comes from alerting on SLOs/symptoms (error rate, latency) not just causes. State the severity early so the response matches the blast radius.