One heuristic I've adopted: when faced with a design decision, I ask the question "what's the simplest possible way to do this (that meets the requirements)?" So, do you need a fully event driven Kafka based system, or would a cron job running a python script be sufficient?
The followup is, if we have to change/replace said system, how difficult would that be? If scaling the simplest solution would be difficult/painful, then one can start look to higher complexity solutions. The idea isn't necessarily to always choose the simplest solution, but having it in mind can be helpful. It crystalizes one of the endpoints of the simple/complex spectrum and helps weigh the pros/cons of various approaches.
As a side note, it's sort of amusing that a lot of design focused interview questions are around things like "design a twitter/instagram like system that'll scale to billions of requests per day." I've never had to do anything like that IRL, but no interviewer has ever asked me to design, say, an invoicing system that gets called rarely. So perhaps one of the reasons the arc of software engineering bends towards complexity is that we're continuously rewarding a "build massive scalable systems" mindset?
The followup is, if we have to change/replace said system, how difficult would that be? If scaling the simplest solution would be difficult/painful, then one can start look to higher complexity solutions. The idea isn't necessarily to always choose the simplest solution, but having it in mind can be helpful. It crystalizes one of the endpoints of the simple/complex spectrum and helps weigh the pros/cons of various approaches.
As a side note, it's sort of amusing that a lot of design focused interview questions are around things like "design a twitter/instagram like system that'll scale to billions of requests per day." I've never had to do anything like that IRL, but no interviewer has ever asked me to design, say, an invoicing system that gets called rarely. So perhaps one of the reasons the arc of software engineering bends towards complexity is that we're continuously rewarding a "build massive scalable systems" mindset?