Monthly Patch Deployment

Why four weeks, and why in this order

Twenty-four date offsets across 28 days, and the week boundaries are gates, not decoration. Week 1 turns vendor bulletins into a plan. Week 2 answers one question — does this break anything — and needs a full week, because compatibility failures surface in the third integration you forgot you had, not in a smoke test. Week 3 produces CAB approval and the maintenance window, both with lead times measured in days. Week 4 is the shortest stretch: the deploys land +21d to +23d.

The dependency chain is short: test results → rollback plan → staging verification → CAB approval → maintenance window → deploy. Everything else in the 39 issues hangs off that spine in parallel. Teams break the cycle by opening week 3's change request before week 2's test results exist.

Who owns which gate

• @security — opens on the vendor bulletins, closes on the compliance report, and owns assess patch criticality and risk at !Highest. Urgency is a security call, not an ops one.
• @patch_lead — the plan, the rollback requirements, the staging go-ahead, the change request. The only role in all five phases.
• @qa — owns the week 2 phase, plus staging verification in week 3. Gates “does it work”, separately from “are we allowed”.
• @change_manager — CAB approval, and closing the record. The authorisation gate: not whether the patch works, but whether the business accepts the window and the risk.
• @ops_team — the window, the backups, all three batches and the verification of each.
• @comms — the maintenance notice on +18d, three days ahead of batch 1. The three days are the point.

Two are separated on purpose: whoever decides the window is not whoever signs off the rollback. Merge them and the person under pressure to ship owns the call on whether retreat is possible.

The rollback gate sits in week 2

Identify rollback requirements for each patch lands on +10d — inside testing, eleven days before anything touches production. That is the strongest opinion in the template: a rollback plan written in week 4 comes from someone who already has the deploy scheduled and is motivated to conclude that rollback is straightforward.

Some patches cannot be rolled back at all — kernel updates that migrate on-disk state, database patches that rewrite a schema. Learning that in week 2 lets you pick the compensation: snapshot, staged cutover, or a differently staffed window. Learning it at 02:00 means the choice was made for you. Create system backups and snapshots at !Highest on +20d stands in where rollback does not exist.

Why production goes out in three batches

Each batch has its own verify subtask, and the next batch is the following day. That 24 hours is the feature: a regression that only appears under a full business day of traffic gets a day to surface before it reaches the rest of the estate.

Batch 1 being the critical servers reads backwards until you think about staffing: it goes out inside the announced window with the team watching and the rollback plan open, while batch 3 lands when attention has moved on. Canarying the least critical systems first is the better read when your risk is an unknown patch interaction — pick deliberately.

What breaks when a step gets skipped

• Deploying without a tested rollback. It presents as a four-hour outage while someone rebuilds a server from a backup nobody had verified.
• Skipping the staging integration check. Smoke tests pass on the patched system. What breaks is whatever talks to it.
• Skipping the maintenance notice. The support queue fills with people reporting planned downtime as an incident, so on-call triages tickets instead of watching graphs.
• Skipping the asset management update. Nothing breaks this month; next quarter the compliance report is fiction.

A patch failure that ends in an outage ends in the incident response cycle, where investigation checks recent deployments and config changes early. Deployments you did not record cost you the first twenty minutes.

Five people versus fifty

Five people

There is no CAB. @change_manager and @patch_lead are the same person, so the approval gate approves itself. Keep the subtask and change what satisfies it: a written answer to three questions — what breaks, who notices first, how do we get back — reviewed by someone who is not the patch lead. Not a change advisory board, but it catches the same class of problem.

Run the full 28-day cycle for the routine set and document an out-of-band path for the CVE that cannot wait — in advance, because the version you draft at 22:00 with a public exploit in circulation is the one that skips the rollback plan.

Fifty people

The roles are real and full-time, which fixes ownership and creates two problems. Batches grow until verify batch 2 systems are healthy is a day of work, not a checkbox — split it by service owner, not server count. And watch what the cycle is for: with an audit calendar, the compliance report on +26d becomes the deliverable and patching becomes how you produce it. The retrospective on +28d is the only subtask that can change the cycle.