While the phenomenon of promo projects is absolutely real, “not my favorite technology choice” is bad evidence that work is wasted. When you have a platform org that offers tooling and support for a golden path, it matters a lot whether you’re on it. It is actually easier to do gRPC APIs at my work than to cowboy RESTy stuff. And the golden path changes over time. Python was a supported language years ago, now Go is. So of course things are being migrated to Go, not because that’s objectively best, but because if you don’t then your Python builds and deploys on our infrastructure could stop working at any moment and no new library versions will ever be released. We also had a team operating Mattermost at one point because that team’s wages cost less than Slack.
I do think the platform org’s choices can cause a lot of low quality churn for the rest of engineering, and everybody resents having to switch out a perfectly fine dependency for somebody’s half baked promo project, but even the platform group when it does these things is trying to save itself the headcount involved in maintaining legacy. As things get leaner, support for older stuff gets worse, and we have to do even more migration work of dubious value to tread water.
If I were starting a Big Tech tomorrow I would put a lot of value on choosing a stable, long term supported tech stack and laying it out so that platform teams can iterate without distributed migration efforts. But no one ever starts a Big Tech. All that is awkward in Big Tech is due to path dependence flowing from the understandably odd choices made by tiny startups.
The GP wasn't referring to "not my favorite technology choice", but instead for the tendency to over-engineer and invent complexity in order to maintain job security. That is, both a character flaw in an engineer as well as systemic problem in the organization that rewards the expression of that flaw.
Things certainly get overcomplicated! But when you encounter a system you think is overcomplicated, it's rarely as simple as that system's designer doing something wrong. More likely their hand was forced by context. Context and path dependence. It's an emergent phenomenon. Big companies have particular infrastructure, platforms, libraries, standards, expectations. Lots of it. Most of it reasonable when considered in its own context. But the way it interacts with any one systems designer's "simple" task can get weird.
At the benign end of the scale, you might need to use a more heavyweight tool than you'd like, because the company also has more intense problems and prefers to standardize on one tool. At the other end there's cruelty and farce. Elaborate workarounds for things that ought to be easy. Mind-bending debug sessions for what never should have been possible.
No one wants this to happen! The career incentive is to ship. We're tearing our hair out over it - honestly a big reason people have side projects is the catharsis of just doing everything the sane way. Overcomplication is a dysfunction that plagues engineering organizations. But the solutions are more subtle than "just say no" - it's about the quality of the company's platforms, degrees of NIH syndrome in the platform group, tradeoffs made between freedom and consistency in system design culture, wisdom and foresight in anticipating how different things will interact, etc.
Also underrated: at the time the company needed an X, Y was not around yet, so we adopted X. Yes everyone now agrees Y is better. We are migrating to it, but that takes time. No you can't have your own instance ahead of schedule. so you need to make do with X, even though it is worse and more complicated.
There is some of that for sure but fundamental things like software dependencies change so frequently in this industry such that any code you ship has a long and unpredictable tail of maintenance work, so you end up spending a lot of your time doing things like ensuring your code doesn't call broken/deprecated APIs or migrating away from EOL'd database versions just to keep the lights on. I think we SWEs end up creating lots of work for each other.
I do think the platform org’s choices can cause a lot of low quality churn for the rest of engineering, and everybody resents having to switch out a perfectly fine dependency for somebody’s half baked promo project, but even the platform group when it does these things is trying to save itself the headcount involved in maintaining legacy. As things get leaner, support for older stuff gets worse, and we have to do even more migration work of dubious value to tread water.
If I were starting a Big Tech tomorrow I would put a lot of value on choosing a stable, long term supported tech stack and laying it out so that platform teams can iterate without distributed migration efforts. But no one ever starts a Big Tech. All that is awkward in Big Tech is due to path dependence flowing from the understandably odd choices made by tiny startups.