Incident Response — SEV1 Production Outage

Why the phases sequence this way

Four of the five phases carry [Due + 0d]. That is the shape of the day, not an oversight — detection, comms, investigation and mitigation all happen inside one shift, and their order is a dependency order, not a calendar. Only Post-Incident Review moves the clock, to +1d, +2d and +3d.

The decision worth arguing about is that Communication & Escalation runs second, ahead of Investigation & Diagnosis. You post the first status page update before you know what broke. Comms blocks only on a declared severity, which triage produces in its second subtask; investigation blocks on nothing and runs as long as it runs. Queue comms behind diagnosis and the customer's first update depends on the slowest task in the incident.

Mitigation blocks on investigation only partly: implement emergency mitigation needs a blast radius, not a root cause. Plenty of incidents are mitigated before anyone knows why — which is why deploy permanent fix if root cause is known is conditional, and why the smoke tests after it belong to @qa_lead.

Who owns which gate

• @incident_commander — declares the incident, sets severity, assigns roles, sends the all-clear. The IC does not debug. The moment the IC opens a terminal, the incident has no commander.
• @sre_lead — dashboards through root cause through mitigation. Hands on keyboard.
• @comms_lead — status page, customer email, the 30-minute cadence, support talking points. One voice outward.
• @scribe — the timeline. Three low-glamour subtasks, and the only artefact you cannot reconstruct from logs afterwards.
• @qa_lead — one subtask, production smoke tests. The independent check between “the graph came back” and “it is over”.

These are role handles, not people. In the template editor @incident_commander matches no Jira member, so it can be promoted to a variable — one picker at apply time fills every row that mentions it.

SEV-1 versus SEV-2, in practice

The template asks you to assess severity level (SEV1/SEV2/SEV3) and leaves the ladder to you. A level is real when it maps to two numbers and one list: a response time, an update cadence, and who gets woken up. “Page on-call, first update inside 15 minutes, VP Engineering notified, 30-minute cadence” is a severity level. “This feels bad” is a label. If SEV-2 carries the same response time as SEV-1 you have one level with two names, and inside a quarter everything is filed SEV-1.

Declare on customer impact, not on how hard the fix looks — and allow re-declaration. A SEV-2 that turns out to be region-wide gets promoted, and the promotion goes in the timeline.

What breaks when a step gets skipped

• Skip the comms phase and your status page and your support team contradict each other in public — support answers from the last thing they saw in the engineering channel, which is usually a hypothesis. The talking-points subtask is the cheapest one here and the one most often dropped.
• Skip the monitoring verification and you send an all-clear on a metric that had not caught up. Error rates fall before queues drain.
• Skip the timeline and the postmortem becomes a memory contest 48 hours later.
• Skip the follow-up action items — !Highest in the final phase — and you have run a blameless postmortem that changed nothing.

On blameless: the word earns its place only if someone owns converting “who deployed it” into “what let it reach production”. The template separates identify contributing factors and systemic gaps (@sre_lead) from the report, and schedules the meeting inside 48 hours — the window where people still remember and nobody has yet built a story they need to defend.

Running this with five people, and with fifty

Five people

Five roles, five people, two of them asleep. There is no separate incident commander; the on-call engineer is IC, SRE and comms at once. The cost is specific — whoever holds the diagnosis is also writing the update every 30 minutes, and that context switch turns a 40-minute incident into a two-hour one.

Do not delete the roles. Collapse five to two and keep the labels: one person on the keyboard as @sre_lead, one holding @incident_commander and @comms_lead. The issue history still records which hat was on, which is what makes the postmortem possible. What a small team should drop is the cadence promise — commit to 60 minutes and hit it.

Fifty people

At fifty the roles are separate people and the failure mode inverts. Nobody is overloaded; everyone assumes someone else has it. Four engineers pile into the investigation, nobody owns the status page, and the first customer update goes out at minute 50. That is what assign incident roles (IC, scribe, comms) prevents.

The fix path changes too. At five people, mitigation and permanent fix are the same deploy. At fifty there is a change process, and the emergency patch that closed the incident still has to land in it — see the monthly patch deployment cycle, where the out-of-band route is the exception you write down before you need it.