How do the conversations typically go when asked "did you know this could happen"/"did we see this coming" or has no one bothered to ask that question yet during the RCA process?
If you haven’t been shot down by an expendable in the past, the best course of action is to find the most complementary position historically taken by leadership, twist to your objective and then credit them publicly with the vision.
You missed that part. "Here is the email chain it was discussed in, including the bit where I raised the concern of what could happen" is generally pretty solid CYA.
I actually did not miss that part, but thanks for calling it out regardless. The question was instead borne out of curiosity as to know how the follow up and subsequent discussions fare even with CYA in pocket.
What has your experience been? Have you had success reprioritizing necessary and critical fixes that were previously not in scope postmortem?
Oh yes. There is nothing quite like "it cost us money" to suddenly change leaderships' priorities. I've watched as product priorities went from "keep bolting on features" to suddenly "we should rewrite this problematic monolith" when thigns fell over and money was lost.
See that makes sense, perhaps it make too much sense. My last two jobs I fought the same battles of "I discovered x, we really should do something about it because it can have really bad outcomes" and having the concern depri'd and ultimately blow up in all of our faces.
And in both situations even when said thing exploded in a bloody mess I struggled to get the buy in to give the fix the full attention it needed because "we're losing money not having these features" (I mean no, you're not, but ok)
Hence my incredulity but honest curiosity, being an ops guy who over-prepares and tries to keep his leaders as informed and prepared to make a decision as possible, but still feels like he's often left holding the bag is dang exhausting, you know?
Maybe I'm overthinking it (this happens sometimes, I'm working on it), but waiting around for a security crisis to happen when a solution is in sight, even when facing difficulty prioritizing the work seems dangerous to one's career.
But I admit not knowing a lot of things most engineers just assume are a given, lately.
It's the difference between working in a push vs a pull shop.
In a push shop (e.g. most non-tech businesses / legacy tech), one is forbidden from working on a thing without management signing off on it and pushing the task to you.
In a pull shop, management communicates priorities (increase reliability) and engineering teams pull tasks to fulfill that.
Believe parent is talking about the former, which is always a cluster&#+@ of technically clueless middle management.
I like that dichotomy. If you or anyone else reading this have any thoughts on it, what are some good questions I ought to be asking during interviews to determine what kind of shop I'm talking to might be?
That's been a real struggle lately, and I want to get better at this to avoid these kinds of push shops you're describing
IMHE, people who work in push shops typically don't realize there's an alternative, which makes identifying them somewhat difficult.
On a side note, I've found push shops tend to be highly correlated with length of position (i.e. "What's the longest someone has ever been on this specific team without transferring or pursuing new opportunities?"). But that's probably more accurately identifying heavily regulated industries, which tend to be push shops do to the Byzantine sign-offs required.
I would say don't ask, "Tell me about a time your team originated an idea, took it to production, and the net impact." Because even push shops have the odd exception that can be rattled off. It's more talking about the typical workflow.
The best I could probably come up with is "Take me through a normal task for your team, from idea origination to prod deployment."
If they start with "We were told to do X", then "By whom? And what did they tell you?" (To the latter, did they specify exactly how or just the end goal?)
You’re not overthinking it. Those of us who wait for catastrophe aren’t silent until it happens. We’re just ready and willing to reiterate what’s wrong when it shows up.
Everything about that situation is dangerous to our careers: saying something, saying nothing, saying a lot, stomping our feet, saying we told them so.
But don’t second guess yourself. You’re expressing exactly the right attitude toward addressing things before they’re an emergency. The people who are using those emergencies to get things done agree, we’re just accustomed to getting shut down unless there’s no other option.
