I’ve started doing something like this more frequently, but it is in the interest of job security / cover-my-butt rather than pettiness.
I build something that depends on many systems that others have created and that uses data from probably about every dataset we have. What I build also has very high customer visibility. So whenever something breaks upstream, the first place customers notice the problem is in the system I maintain. Psychologically, people begin to develop a mental association around your system being a “problem” if it keeps getting brought up as the starting point of SEV discussions and customer tickets.
As a result, a lot of what I do now is defensive. I investigate and reverse engineer upstream codebases to identify likely failure modes, and I spend hours analyzing datasets of questionable origin for data quality issues and inconsistencies. I document and date stamp all of the problems that I find, file bug reports and assign them to the relevant teams, and write proposals describing potential solutions to what I see as large architectural design flaws that will come back to haunt us at some point.
All of this work is promptly ignored with “not enough bandwidth right now” or “not a priority compared to feature development”. Which is fine. I document all of those responses too.
Then eventually, something breaks in a big way that is again first noticed by customers within the system I am responsible for. In the past, I would immediately drop what I was doing and scramble into investigation mode for a few days to prove I wasn’t the root cause of the X hundred thousand dollar issue, er... I mean the blameless SEV review & postmortem... but more recently my preemption seems to be paying off, and lately I just reply to the panic with a bunch of links to old Slack threads (where we already discussed the issue), the documents and proposals I created (that no one read), or the bug reports I filed (that were never followed up on).
Perhaps it does come across as a bit petty, but I try to be as polite as I can, and from my perspective it’s an improvement over the previous situation. The only downside is that all of this preventative work takes away time from the primary work that I was hired to do.
This is absolutely not petty, you are doing your job (to provide value to the organization) and doing it extremely well. The work you are doing provides more value to your organization than you seem to realize. If I were your manager I would be pushing to get you a raise/promotion with this as the reason.
Not petty at all... pretty smart imho. Does the metaphorical front-end catch back-end errors with a warning and link to the issue? That would light fires under asses. :-D
This can be framed constructively, do an analysis of what was fixed during the outages from the originally identified and now lobby for funding to fix the rest.
Actually any organisation worthy of the label "engineer" should do a thorough, no-blame, and collaborative post-mortem of such events so they don't happen again, and that would include warnings given.
Feature on time + risk, feature late + no risk (and anything in between), it's in the end an engineering and business decision and either choice might be the right one depending on the circumstances.
Absolutely, this is a often ignored or missed aspect of building systems. Quantifying risk in a fashion that all parties have the same understanding of said risk can be very difficult. There has to be a common language and agreed upon thresholds of risk. After developing shared understanding of risk, it's much easier to highlight and change the behaviors of individuals and the teams.
