A backup that has never been restored is not a backup

To the question “Do we have backups?” a technical specialist almost always answers affirmatively, and answers honestly: there’s a job in the scheduler, files are being created, space is being used. The manager mentally ticks a box and returns to their tasks.

The problem is that the question asked was the wrong one. The correct question is different: when was the last time we restored from these backups and how long did it take?

What becomes clear on that day

The list is short and repeats from company to company.

The copies were on the same server. Backup was configured to a neighboring partition on the same disk. The disk failed — both were gone.

The files are empty. The backup script broke after a database update six months ago and has been creating zero-length files ever since. No one checked because the job completed without error.

There is a copy, but no key. Archives were encrypted, and the key was held by an employee who left.

Restoration takes longer than expected. Everyone was sure they would be back up in an hour. Deploying a multi-terabyte database from a logical dump took nearly two days, and the business was down the whole time.

Not everything was restored. The database came up, but server configuration, object storage contents, certificates, and external service credentials were not included in backups — nobody thought of them. A working database without the application that knows how to use it does not restore the business.

Replication is not a backup

This common misconception deserves a separate mention.

A replica reproduces changes from the primary database almost instantly — and that is its advantage in hardware failure. It is also its uselessness in any other kind of failure: an accidentally executed delete command, a faulty migration, ransomware, or a malicious insider action will be reproduced on the replica in seconds. Data deleted on the primary server ends up deleted on the replica too.

Replication protects against hardware failure. A backup protects against human error — and human errors happen more often than disks fail.

Practical consequences follow: backups should be stored separately from the production infrastructure and, if possible, in a mode that prevents modification and deletion for the retention period. The account under which the production system runs should not have the right to delete backups — otherwise whoever gains access will delete them first. This is exactly what modern ransomware operators do: first the backups, then everything else.

The classic rule still works: three copies of the data, on two different media or with two providers, one copy off the primary site.

What should be included in backups

Besides the database:

• Server configuration and infrastructure descriptions.
• Object storage contents — files and documents uploaded by users.
• Credentials and access keys for external services, in an exportable form.
• DNS zone settings: you won’t be able to reconstruct them from memory during an incident.
• The recovery runbook itself — if it only exists in the system that has fallen, you don’t have it.

Drills: how to test without breaking anything

The test does not require touching the production environment. Three steps and one working day are sufficient.

1. Clean environment. Take a new server in an isolated environment. Brand new: restoring to a machine that already has an environment configured proves nothing — you will test the backup, but not the completeness of the procedure.

2. Restore only from external storage. Engineers take copies from the regular offsite storage and deploy the system from scratch. No files from production servers, no “I’ll edit the config from memory now” — everything that was missing is recorded as a gap.

3. Timing and verification of the result. Measure the time from the start until the moment the system is up and able to accept an order. You need to test the full working scenario, not just the availability of the landing page.

The first drills almost always yield an unpleasant result: the procedure takes many times longer than expected, and two or three missing elements are discovered. That is the benefit — finding this out in controlled conditions costs one working day; finding out during an incident costs business downtime.

The measured time is your real RTO. Not the one written in a document, but the one that is confirmed. A reasonable cadence is once a quarter, and always after significant architecture changes.

One more check that is often forgotten

Restoration verifies that data comes back. Separately, you should verify that they are correct.

After recovery, compare control metrics: number of orders for the last month, total balances, count of active users. A backup may deploy perfectly technically and still contain data a week older than expected — for example, because the backup job ran before, not after, nightly processing.

Three questions you should ask today

You can ask them in the work chat right now. Abstract answers like “everything is set up” are not acceptable — you need specifics.

1. Where are the backups physically stored? Acceptable answer: with another provider or at least on a different site, in storage from which the production system cannot delete them.

2. When was the last time we restored the system from these backups onto a clean server and how long did it take? Acceptable answer: a specific date within the last quarter and a specific time in hours. The answer “when we set it up, three years ago” means there are no verified backups.

3. What exactly will not be restored if the site disappears entirely? A healthy reaction is a short and honest list. The answer “everything will be restored” without having run drills means the question hasn’t been investigated.

A disaster recovery plan does not exist for auditors. An untested plan in a real incident stops working at the first step where reality diverged from the document — and it always diverges.

Related posts: RTO и RPO — how to set recovery requirements in business terms, and первые 30 минут после падения — what the team does while recovery is in progress.