Signs you're paying for reliability you don't need

People rarely talk about excessive reliability: insufficient reliability announces itself with outages, while excess looks like a virtue. A company pays for complexity it doesn’t use, and calls it prudence.

Meanwhile, complexity introduced prematurely doesn’t just spend the budget. It adds new ways to fail. A system of ten interacting components fails not in ten scenarios, but in a substantially larger number, and not all of them are predictable in advance.

Why this happens

The reasons are rarely malicious, and understanding them is more useful than condemning.

Engineers choose technologies not only by the task, but also by the labor market: experience with a modern stack increases their value, experience supporting a monolith on a single server does not. This is rational behavior, and it doesn’t change by mere exhortation.

A second reason is lack of feedback. An excessive solution doesn’t create immediate problems: everything works, it’s just more expensive and complex. The bill comes later, when people in the team change.

A third is imitation. Engineering talks from large companies describe architectures designed for load you don’t have, and solutions are transplanted along with the context that doesn’t transfer.

Symptoms

Orchestration for a load you don’t have. Kubernetes for an online store with two hundred visits a day. It’s a great tool for dozens of services and multiple teams, but for a single application it adds its own network layer, its own storage, its own failures, and requires someone who understands it all.

Fragmentation that outpaces the team size. Forty services for three developers. Splitting a system into independent parts is primarily to keep teams from stepping on each other’s toes: when three people work on a product, there’s no one to interfere, while you already bear the costs of version coordination, network calls, and debugging interactions. It’s worth recalling Conway’s observation: the structure of a system mirrors the structure of the organization. If the organization is a single team, a system of forty services doesn’t fit it.

Automating rare operations. An engineer spends three weeks on a scenario that performs an action the company does once a year. A one-hour written instruction would solve the same task — except that an instruction won’t silently become outdated, while the automated script will likely break over the year, and no one will notice until it’s needed.

Environments nobody uses. Four copies of the production environment billed monthly by the provider, while two of them haven’t been opened for weeks.

Failover without verification. Formally there is a backup site, but switching to it has never been performed. This is the worst case: costs have been fully incurred, but confidence is false — an untested switch in the moment of an outage works in roughly half of cases.

Monitoring that nobody responds to. Hundreds of configured checks and a notification channel where messages haven’t been read for a long time because something is always on fire there. Such monitoring costs money and gives nothing.

The real cost

The temptation to measure redundancy by the provider bill is strong, but hardware is the cheapest part.

Knowledge concentrates in one person. The complex system is understood entirely by its author. While they are at the company, everything works. When they leave, what’s left is a construct no one dares touch — and any change in it starts taking weeks instead of hours.

Onboarding stretches out. A new engineer becomes productive over months because the relationships between components are nowhere documented, and it’s impossible to understand from the code why the failure of one service breaks an unrelated function.

Change velocity drops. This is the most expensive and the most unnoticed loss. A product hypothesis that could be tested in a week is tested in a month. A competitor will test four in that time.

The likelihood of cascading failures increases. Automation that reacts to a failure can make it worse when misconfigured: it restarts a service that hasn’t warmed up, shifts load to a backup that can’t handle it, and turns a local fault into a total outage.

How to tell justified complexity from excessive

Signs alone prove nothing: there are companies that need Kubernetes, and companies for which forty services are justified. What distinguishes them is not size, but the ability to answer three questions.

First: which specific outage does this prevent and how much does it cost? The answer “this is the right architecture” is not good enough. Acceptable is: “the failure of one server in this scenario costs us 300 thousand, the solution costs 100 thousand per year.”

Second: how many people on the team can support this? If the answer is “one”, the complexity hasn’t been implemented — it’s been taken on credit backed by a specific employee’s personal guarantee.

Third: when did we last use this? A backup site, automatic failover, recovery plan — all of these have either been tested in practice or exist only on paper.

Three confident answers mean the complexity is justified. Lack of answers means you’re paying for reliability you probably don’t have.

If the complexity is already built

The first impulse — to simplify everything back — is usually wrong: migrating back costs money and carries risks, and you shouldn’t touch a working system without need.

It’s wiser to act differently.

Stop growth. Don’t add new components until you can answer the first of the three questions.

Reduce single-person indispensability. Document the system so a second person can understand it. This is cheaper than any migration and removes the main risk.

Remove what’s unused. Idle environments, monitoring nobody looks at, services without traffic — these can be turned off without architectural decisions.

Simplify during the next major change. When a component must be rewritten anyway, decide then whether it needs to remain as it is.

A question to ask regularly

Once a quarter it’s worth asking the team directly: what business problem does what we’re building right now solve, and what will happen if we don’t do it?

The question shouldn’t sound accusatory — in that tone you’ll get defensive answers instead of honest ones. The phrasing “help me understand what we’re paying for” works better than “why did you implement this”.

A good team will answer it in a minute. If no one has an answer, you’ve found not a technical problem, but a managerial one.

Related reads: four levels of fault tolerance — what complexity is appropriate at which stage, and cost of downtime — how to get the figure for the first question.