Essential complexity is exactly that. The entire game is to separate it from the accidental complexity. My career/life was revolutionized by a paper that is 100% focused on this very topic:
The solution to eliminating accidental complexity is ultimately presented as a flavor of relational model, which I simply read as "please just use SQL for most things". This perspective has served me exceptionally well over the last ~4 years now.
A quote I like from Chapter 8:
> The relational model [Cod70] has — despite its origins — nothing intrinsically to
do with databases. Rather it is an elegant approach to structuring data, a means for manipulating such data, and a mechanism for maintaining integrity and consistency of state. These features are applicable to state and data in any context.
Chapter 9 offers an actual proposed solution - Functional Relational Programming.
At the end of the day, suffering in a cesspit of complexity is a choice. The relational model is the answer for untangling highly-complex (aka high-dimensional) problem domains, especially ones that require arbitrary downstream views of the data. The central piece of magic with the relational model (as applied to the real world) is the query planner. You should start thinking of this as an actual code-writing super AI that can answer optimization questions 1000x faster than your best developers. All you have to do is give it a tiny pile of hints to work with, and almost any arbitrary request will be satisfied in a nearly-ideal amount of time.
For others: this is the classic Out of the Tar Pit paper, which I also adore so so much. It promotes very explicit state management and functional programming to reduce accidental complexity.
our friend ChatGPT summarizes the paper thus:
"In summary, "Out of the Tar Pit" argues for a shift in software design principles, advocating for the reduction of accidental complexity through functional programming and dataflow concepts, proper state management, and the use of formal methods. By focusing on these principles, the authors believe that software systems can be made simpler, more robust, and easier to understand and maintain."
Essential complexity is exactly that. The entire game is to separate it from the accidental complexity. My career/life was revolutionized by a paper that is 100% focused on this very topic:
https://curtclifton.net/papers/MoseleyMarks06a.pdf
The solution to eliminating accidental complexity is ultimately presented as a flavor of relational model, which I simply read as "please just use SQL for most things". This perspective has served me exceptionally well over the last ~4 years now.
A quote I like from Chapter 8:
> The relational model [Cod70] has — despite its origins — nothing intrinsically to do with databases. Rather it is an elegant approach to structuring data, a means for manipulating such data, and a mechanism for maintaining integrity and consistency of state. These features are applicable to state and data in any context.
Chapter 9 offers an actual proposed solution - Functional Relational Programming.
At the end of the day, suffering in a cesspit of complexity is a choice. The relational model is the answer for untangling highly-complex (aka high-dimensional) problem domains, especially ones that require arbitrary downstream views of the data. The central piece of magic with the relational model (as applied to the real world) is the query planner. You should start thinking of this as an actual code-writing super AI that can answer optimization questions 1000x faster than your best developers. All you have to do is give it a tiny pile of hints to work with, and almost any arbitrary request will be satisfied in a nearly-ideal amount of time.