Four levels of fault tolerance: which one are you at and what do you really need

Fault tolerance is not a binary state. It’s a ladder of four levels, and each solves its own problem at its stage of the product’s life.

Trying to skip a rung and build a complex system too early is classic overengineering. It’s expensive, slows development, and creates a false sense of security: formally everything is redundant, but in practice nobody understands the system as a whole anymore.

A level is determined by two quantities, both set by the business, not the engineers:

• RTO — how long the business can live without a working system.
• RPO — how much data loss is acceptable (time period).

The main law of fault tolerance

Engineers want to build distributed systems because it’s an interesting problem. Business works differently.

The level of fault tolerance is dictated not by the team’s ambitions but by a simple inequality:

Annual cost of the solution < expected annual loss from downtime

The right-hand side is calculated as: cost per hour of downtime × expected number of downtime hours per year. If an hour of downtime for an online store costs 40 000 ₽, and a realistic forecast is eight hours of downtime per year, the expected loss will be about 320 000 ₽. Moving to the fourth level will cost millions per year. The conclusion is obvious: you don’t need the fourth level, you need an honestly completed second level described below — with verified backups and external monitoring.

If an hour of downtime costs several million, automatic failover of the database pays for itself on the first incident.

The answer is almost always one rung lower than the technical team wants, and one rung higher than the CFO is willing to pay for. The manager’s task is to reconcile those two positions with numbers, not beliefs.

Let’s go through the four levels — from a single server to a geographically distributed system. For each: what measures are appropriate, what mistakes are typical, and what signs show it’s time to move to the next rung.

Level 1. MVP: “Just make it work”

RTO: up to a day · RPO: up to a day · Budget: minimal

The product has just appeared. The main task is to validate the hypothesis and see if the market needs it. Users usually either don’t pay or participate in beta testing.

How the infrastructure is arranged

• One server, usually a cheap virtual machine.
• Everything is hosted on it at once: frontend, backend, database.
• Deployment is manual — by script or the git pull command.
• Knowledge about the system exists only in the head of one developer. There is no documentation, and right now it’s not needed: priority is given to speed of validating the hypothesis.

Main risks

The main mistake at this level is lack of backups. You need to copy not only the database but also the code and server configuration: without them recovery becomes archaeology. If the server disappears — and cloud providers do have outages — you’ll have to rebuild the system from scratch, and the first users won’t forgive that.

The second mistake is softer but more common: backups are made but stored on the same server. Such a backup disappears together with the original.

Verdict. One server and configured copying to an external storage is an economically justified solution for an MVP. Nothing more is needed at this stage.

Level 2. Basic reliability: first paying customers

RTO: hours · RPO: tens of minutes · Budget: moderate

The product has grown: paying users have appeared, and with them — obligations. Business starts demanding guarantees.

Transition to this level almost always follows the first major outage. A typical scenario: the company launches its first large marketing campaign. Unusual traffic hits the single server, user-uploaded files fill the disk, memory is exhausted — and the system goes into a cascading failure: first the database, then the backend and frontend. A developer restarts the server manually, users go to competitors, and the investor asks uncomfortable questions.

How the infrastructure is arranged

1. Separate load balancer. Nginx or an equivalent is moved in front of the application. This gives traffic control: if the backend dies, the user sees a clear standby page instead of a 500 error.
2. Static assets moved out. Images and user files move to object storage (S3 and compatible). The server disk no longer fills up.
3. Layer separation. The backend is spread across two or three servers, the database lives separately. User sessions are moved to a shared cache (Redis) so any server can handle any request.
4. Database replication. A primary-replica scheme is set up: the replica continuously follows changes from the primary. Backups are taken from the replica and stored on a different site.
5. Basic monitoring. Disk usage, memory consumption, and — via an external check — availability of key pages and scenarios are monitored: checkout flow, login, payment. The first written recovery runbooks appear.

Main risks

Usually at this stage the first systems engineer is hired, and they have a natural temptation to introduce technology for technology’s sake. Typical traps:

• Premature splitting of the system. Breaking a monolith into forty services when the team has three developers — a decision that will backfire: coupling will increase and system understanding will drop.
• Automating the wrong things. An engineer spends weeks on complex auto-recovery scenarios for an operation performed manually once a year.
• Application not ready for redundancy. It doesn’t reconnect to a restarted Redis, stores uploaded files on local disk, or breaks if multiple instances handle requests simultaneously. The infrastructure is redundant, but the code is still built for a single server.

Also remember: by moving sessions to Redis, you have created a new single point of failure. At this level it’s an acceptable trade-off, but be aware of it.

What’s still manual here

Failover to the replica. Replication works on its own, but the decision “primary server died, promote the replica” is made and executed by a person. That’s why RTO is measured in hours — it includes the time for the engineer to wake up, understand the situation, and switch.

Verdict. Single points of failure at the hardware level are eliminated, but recovery still depends on the on-call engineer.

Level 3. Mature production: automation

RTO: minutes · RPO: seconds · Budget: medium

The project generates revenue, and downtime directly converts to lost money. Formal availability commitments to customers appear. The number of services grows into the dozens, manual management can’t cope, and a new engineer joins the project and spends weeks ramping up because the relationships between systems are nowhere documented.

How the infrastructure is arranged

• Orchestration. Kubernetes or an equivalent is introduced: it distributes load across servers, restarts crashed services without human intervention, and scales them with load.
• Content Delivery Network (CDN). Static assets are served from nodes nearest to the user.
• Automatic database failover. Replication management is handed to a specialized system — for example, Patroni for PostgreSQL: it detects primary failures and promotes a replica. Connections go through a pool (PgBouncer), otherwise during failover the application will hit connection limits.
• Reliable message delivery. Asynchronous tasks go to a broker (Kafka, RabbitMQ). The important part is not just installing it but configuring it: replication factor > 1 and requirements for acknowledgements from multiple replicas — otherwise one node’s failure will destroy in-flight messages.
• Observability. Monitoring ceases to be just hardware graphs: business metrics are added (orders per minute, share of successful payments), historical trends and external checks. Monitoring watches itself too — otherwise the alerting system will quietly die first, and silence will be deceptively peaceful.
• Organizational readiness. A written disaster recovery plan appears, and the team regularly drills it. Postmortems are conducted without blame: an engineer who’s afraid to report a mistake will report it late.

Main risks

• Code not ready for orchestration. Services restart in loops due to memory leaks, the app can’t run in multiple instances or breaks on database migrations incompatible with the previous code version. Kubernetes doesn’t solve these problems — it just makes them more visible.
• Delaying the transition. Trying to stay on manual management when the project has outgrown it leads to rising operational costs and burnout of on-call staff.

Verdict. Automation takes over routine tasks. Loss of a server, database, or entire availability zone is handled without human intervention — there’s no need to wake an engineer at 3 AM.

Level 4. Geographical distribution

RTO: minutes even if a whole region fails · RPO: close to zero · Budget: high

Here what changes is not recovery speed but the scale of failure the system can survive. Level three protects against loss of a server or site within a single region. Level four protects against loss of an entire region: a data center fire, a broken backbone cable during roadworks, or a provider shutdown by a regulator. At this level it’s no longer just about infrastructure but business continuity.

How the infrastructure is arranged

• Multiple regions. Infrastructure is distributed across geographically independent sites with separate power, communication channels, and jurisdictions.
• Network-level traffic management. BGP announcements from your own autonomous system or global load balancing allow you to steer traffic away from a failed region without waiting for DNS propagation by operators.
• Managing multiple clusters. Unified deployments and coordinated configurations across all sites. Note that a standard federation of Kubernetes clusters didn’t mature — in practice this is solved with configuration delivery tools and external load balancing.
• Distributed databases. Moving from classic RDBMS to systems designed to work across multiple regions simultaneously (CockroachDB, YugabyteDB, Spanner).
• Controlled failures. Resilience is verified by regular experiments in production — from turning off individual machines to simulating network partitions between regions. Only scenarios run in practice are considered proven.
• One-click failover procedure. Isolating a failed region is performed with a single command, not a sequence of forty steps that nobody can remember under stress.

Main risks

• You can’t outrun physics. Latency between regions is tens of milliseconds and unavoidable. Synchronous replication adds that time to every write operation — the application becomes noticeably slower. Asynchronous replication preserves speed, but on failover you lose writes that didn’t make it. Choosing between “slower” and “with data loss” is an architectural decision, not a setting.
• Data divergence. If regions are partitioned, each may decide it survives and continue accepting writes. The result is two conflicting datasets that must be reconciled manually.
• Nonlinear complexity. Support costs grow faster than reliability. You need rare and expensive specialists, and any change must be rolled out consistently across all sites.

Verdict. Expensive, complex, and requires a dedicated operations team — but the business survives the loss of an entire data center.

Where to start

1. Determine the cost per hour of downtime. Without this number any discussion about fault tolerance remains a matter of taste.
2. Draw a dependency map. What depends on what: domain, DNS, mail, payment gateway, database, external services. Mark which failures stop sales.
3. Test recovery. Not “are backups made”, but “when was the last time we restored from a backup and how long did it take”. A backup that nobody has restored from is not a backup, it’s an assumption.
4. Align your level with reality. If you find third-level tooling in a first-level problem — you’re paying for reliability you don’t use.

If you’re not sure which level you’re at, start with an external check: the service will check DNS, mail, TLS certificate and site speed and show weak points in plain language.

If you need a deeper analysis — we will determine your level, calculate the cost of downtime and draw up a migration plan without unnecessary expenses.