The first 30 minutes after an outage: a plan written in advance

When a critical service goes down, the technical failure is only half the problem. The other half unfolds among people, and usually costs more.

Without decisions made in advance, the first thirty minutes look the same at any company. The manager asks the CTO every two minutes what’s happening — and the CTO spends time answering instead of restoring the system. Marketing keeps running paid traffic to a non-functioning site. Support is silent because they don’t know what to say, and customers learn the news from social media before hearing from the company. Engineers in chat argue about whose commit caused it instead of rolling back the release.

All of this is avoidable waste. An incident response playbook costs incomparably less than a backup site, and during an incident saves a comparable amount of time.

Four decisions made in advance

1. The person who leads is not the person who fixes

During an incident, the technical lead should work on the system, not on status reports. That’s why you introduce an incident coordinator role: it can be a project manager, a team lead, or the on-call — the job title isn’t important, what matters is that it’s assigned in advance.

The coordinator does three things: gathers the right people, keeps a timeline of events, and shields engineers from external questions. All status inquiries go to them, and they publish updates at an agreed rhythm — for example, every fifteen minutes, even if there’s nothing new to say. A regular “working, no changes” update answers more than half of incoming questions.

The timeline of events is not bureaucracy. Two days later no one will remember when the service was restarted and when metrics recovered, and without that the post-incident review turns into an exchange of impressions.

2. What and when we tell customers

The decision about the threshold at which the company publicly notifies customers is made in advance and is not debated during an incident. For example: if a critical function is down for more than ten minutes — publish a notice.

Prepared templates are useful: what exactly is not working, what is being done, when the next update will be. Don’t promise a recovery time unless you’re confident in it — a broken promise annoys more than the outage itself. Promise the time of the next update; that’s under your control.

Separately: the status page must be hosted outside your infrastructure. A status page that fails along with your product is useless exactly when you need it.

Honest and timely communication noticeably reduces load on support and the number of angry contacts — not because customers become more patient, but because they interpret silence as indifference.

3. Where emergency credentials and runbooks are stored

A classic trap: the authorization server is down, the password for the backup site is in the corporate messenger, and the messenger is hosted on the same infrastructure.

From this follow several simple requirements. Recovery instructions and emergency credentials are stored independently of the main infrastructure and are accessible even without it. The team has a backup communication channel — a common external messenger agreed on in advance, not chosen during the incident. Two-factor authentication for critical accounts isn’t tied to a single person who might be on a plane.

Check this with one question: if everything went down right now, from which device and according to which instructions would the on-call person start acting?

4. Rollback is more important than fixing

The most common mistake during an incident is trying to find and fix the cause on the live production system. That stretches a ten-minute outage into several hours.

The playbook should state clearly: if the failure happened after a deployment and the cause is not found within ten minutes — roll back to the previous version and investigate afterwards. The coordinator makes the decision, not the author of the change: the author is psychologically reluctant to roll back their own work and will almost always ask for “five more minutes.”

One caveat that’s often forgotten: rollback is not always possible. If a release included an irreversible data schema change, you can’t go back, and that must be known in advance, not discovered during the incident. This leads to a practical development requirement: database schema changes should be compatible with the previous code version. Then rollback remains available, and this is one of the cheapest investments you can make in resilience.

Post-incident review without blaming

The last part of the plan concerns what happens after restoration, and it determines how the team will behave next time.

If someone loses their bonus for each incident, employees start hiding mistakes. Instead of immediately saying “I deployed a bad migration, we need to roll back,” a person will try to fix it quietly for three hours — and by the time others learn about the problem, its scale has grown far beyond the original.

A blameless postmortem is not softness, it’s calculation. Its purpose is to shift the question from “who messed up” to “what in our processes allowed one person’s action to stop the business.” The answer almost always lies outside the individual: there was no pre-deployment check, no restriction of privileges, no ability to roll back, the change was deployed on a Friday evening.

A practical sign of a healthy culture is how much time passes between when an engineer realizes they made a mistake and when the team learns about it. If it’s minutes — the process works. If it’s hours — people are afraid, and no tooling will compensate for that.

What it looks like on paper

A forty-page runbook won’t be opened during an incident. A one-page document accessible without the main infrastructure works:

• Who is the incident coordinator and who covers them.
• The threshold that declares an incident.
• Backup communication channel.
• Where emergency credentials are stored.
• Rollback rule and who decides.
• Who notifies customers and using which template.
• Where the timeline is recorded.

And most importantly — this document should be tested at least once in drills. Like backups, an untested plan is an assumption, not a plan: a backup that no one restored.

If you need to draft such a runbook for your team and test it in practice — get in touch.