staff-engineering · staff
Incident Command for Backend Teams
Quick answer
Incident command is a lightweight operating structure for severe production failures: a commander owns decisions and external comms, operations mitigate, comms update stakeholders, and a scribe records timeline. Staff engineers either command or stabilize technical response without turning the call into a crowd of conflicting heroes.
Why this matters
- Outages get worse under role confusion.
- Customers and executives need clear, honest updates.
- Postmortems need timelines, not memory.
- Staff interviews probe crisis judgment and leadership.
Learning objectives
- Declare severity and start command when needed.
- Assign IC, ops, comms, scribe roles.
- Separate mitigation from root-cause archaeology.
- Communicate without speculation.
- Hand off cleanly and feed the postmortem.
Explain like I am 5
When the kitchen is on fire, one person directs, some throw water, someone calls for help—and nobody debates recipes mid-fire.
Mental model
flowchart TD
Detect[Detect + severity] --> IC[Incident Commander]
IC --> Ops[Ops / Mitigate]
IC --> Comms[Stakeholder comms]
IC --> Scribe[Timeline]
Ops --> Stabilize[Service stable]
Stabilize --> Handoff[Handoff + postmortem]
| Role | Does | Does not |
|---|---|---|
| IC | Priorities, decisions, staffing | Deep-dive every stack trace alone |
| Ops | Mitigate, rollback, failover | Endless debate without action |
| Comms | Customer/exec updates | Unverified root cause claims |
| Scribe | Timeline, actions | Silent note-taking only |
Core concepts
Severity and triggers
Define severities (e.g. SEV1 customer-impacting total outage). Trigger command early—late structure is expensive.
Mitigate first
Restore service (rollback, failover, feature flag, capacity) before perfect diagnosis. Parallelize investigation only after mitigation is underway.
Communication cadence
Fixed intervals (e.g. every 15–30 minutes): impact, status, next update time. Prefer “unknown, investigating” over fiction.
Decision rights
IC decides trade-offs (data risk vs downtime). Escalate for business calls (refunds, public statements) with clear options.
Handoff and learning
When stable: handoff owner, open incident ticket, schedule blameless postmortem with action items that change systems—not only people.
Worked example
Multi-region DNS failure impacting payments:
- SEV1 declared; IC named.
- Ops fails traffic to healthy region; feature-flags noncritical batch.
- Comms posts status page: elevated errors, mitigation in progress.
- Scribe logs times and actions.
- Postmortem produces runbook fix + synthetic DNS check—not “be more careful.”
Trade-offs
| Rigid process | No structure |
|---|---|
| Overhead on small blips | Chaos on big ones |
Staff default: full command for SEV1/SEV2; lightweight bridge for smaller incidents.
Failure modes
| Mode | Mitigation |
|---|---|
| Too many cooks | Mute extras; assigned roles only |
| Hero culture | Rotate IC; celebrate mitigation |
| Premature root cause | Park theories; mitigate |
| Missing customer voice | Comms role mandatory |
| No follow-through | Action owners + due dates |
Interview mode
Skeleton: “In incidents I establish command early—mitigate first, clear roles, honest cadence updates, then a blameless postmortem that changes the system.”
Knowledge check
Mitigating user impact and stabilizing the service
Writing a full root-cause essay mid-outage
Letting everyone debug independently without roles
Speculating root cause in public status updates
Related
By Shubham Jain