This often devolves into extremely fragile systems instead. For instance, let's say you failed to load an image on your web site. Would you rather the web site still work with the image broken or just completely fail? What if that image is a tracking pixel? What if you failed to load some experimental module?
Being able to still do something useful in the face of something not going according to plan is essential to being reliable enough to trust.
Systems need to be robust against uncontrollable failures, like a cosmic ray destroying an image as it travels over the internet, because we can never prevent those.
But systems should quickly and reliably surface bugs, which are controllable failures.
A layer of suffering on top of that simple story is that it's not always clear what is and what is not a controllable failure. Is a logic error in a dependency of some infrastructure tooling somewhere in your stack controllable or not? Somebody somewhere could have avoided making that mistake, but it's not clear that you could.
An additional layer of suffering is that we have a habit of allowing this complexity to creep or flood into our work and telling ourselves that it's inevitable. The author writes:
> Once your system is spread across multiple nodes, we face the possibility of one node failing but not another, or the network itself dropping, reordering, and delaying messages between nodes. The vast majority of complexity in distributed systems arises from this simple possibility.
But somehow, the conclusion isn't "so we shouldn't spread the system across multiple nodes". Yo Martin, can we get the First Law of Distributed Object Design a bit louder for the people at the back?
> systems should quickly and reliably surface bugs, which are controllable failures
I was thinking, if the error exists between keyboard and chair, I want the strictest failure mode to both catch it and force me to do things right the first time.
But once the thing is up and running, I want it to be as resilient as possible. Resource corrupted? Try again. Still can't load it? At this point, in "release mode" we want a graceful fallback -- also to prevent eventual bit rot. But during development it should be a red flag of the highest order.
That's an interesting distinction. I think each resource should be self contained. Malformed HTML? HTML error. Malformed or missing image? Browser displays an image error.
The key here is that the web wasn't designed for engineers but for amateurs to slap something together sloppily in the first place.
As an aside it's curious how ridiculously forgiving HTML and JS are while CSS craps itself on a single missing semicolon. As though it were okay for the thing to be semantically and functionally malformed and malfunctioning... as long as it looks good!
Being able to still do something useful in the face of something not going according to plan is essential to being reliable enough to trust.