When a critical service goes down, the technical failure is only half the problem. The other half unfolds among people, and usually costs more.
Without decisions made in advance, the first thirty minutes look the same at …
In a team without automated delivery, a developer performs daily non-coding operations: builds the project manually, checks changes by hand, waits for the staging environment to be free, fixes bugs found a week after …
Trying to make everything equally reliable is the quickest way to exhaust the budget without increasing reliability where it matters. When a team proposes moving the entire infrastructure to a cluster with automatic …
Conversations about fault tolerance usually dive into technical details within the first two minutes: replication, failover, timeouts, quorum. The executive is left to assess the proposal based on the speaker’s …
People rarely talk about excessive reliability: insufficient reliability announces itself with outages, while excess looks like a virtue. A company pays for complexity it doesn’t use, and calls it prudence. …
To the question “Do we have backups?” a technical specialist almost always answers affirmatively, and answers honestly: there’s a job in the scheduler, files are being created, space is being used. The manager mentally …
In most companies the infrastructure diagram exists in two forms. The first is a pretty presentation: clusters, replicas, load balancers, arrows between them. The second is the real one — in a single engineer’s head — …
While infrastructure is working, it looks like an item in the expense report that you want to cut. Once it fails — the tone flips: engineers ask for a budget for “proper reliability”, the CFO looks at the estimate, and …
Fault tolerance is not a binary state. It’s a ladder of four levels, and each solves its own problem at its stage of the product’s life.
Trying to skip a rung and build a complex system too early is classic …
Infrastructure rarely breaks in a single day. Usually it degrades unnoticed — month by month, bit by bit. The team gets used to inconveniences, calls them quirks, and things go on — until they don’t.
People usually …