While the phenomenon of promo projects is absolutely real, “not my favorite technology choice” is bad evidence that work is wasted. When you have a platform org that offers tooling and support for a golden path, it matters a lot whether you’re on it. It is actually easier to do gRPC APIs at my work than to cowboy RESTy stuff. And the golden path changes over time. Python was a supported language years ago, now Go is. So of course things are being migrated to Go, not because that’s objectively best, but because if you don’t then your Python builds and deploys on our infrastructure could stop working at any moment and no new library versions will ever be released. We also had a team operating Mattermost at one point because that team’s wages cost less than Slack.

I do think the platform org’s choices can cause a lot of low quality churn for the rest of engineering, and everybody resents having to switch out a perfectly fine dependency for somebody’s half baked promo project, but even the platform group when it does these things is trying to save itself the headcount involved in maintaining legacy. As things get leaner, support for older stuff gets worse, and we have to do even more migration work of dubious value to tread water.

If I were starting a Big Tech tomorrow I would put a lot of value on choosing a stable, long term supported tech stack and laying it out so that platform teams can iterate without distributed migration efforts. But no one ever starts a Big Tech. All that is awkward in Big Tech is due to path dependence flowing from the understandably odd choices made by tiny startups.

The GP wasn't referring to "not my favorite technology choice", but instead for the tendency to over-engineer and invent complexity in order to maintain job security. That is, both a character flaw in an engineer as well as systemic problem in the organization that rewards the expression of that flaw.

Things certainly get overcomplicated! But when you encounter a system you think is overcomplicated, it's rarely as simple as that system's designer doing something wrong. More likely their hand was forced by context. Context and path dependence. It's an emergent phenomenon. Big companies have particular infrastructure, platforms, libraries, standards, expectations. Lots of it. Most of it reasonable when considered in its own context. But the way it interacts with any one systems designer's "simple" task can get weird.

At the benign end of the scale, you might need to use a more heavyweight tool than you'd like, because the company also has more intense problems and prefers to standardize on one tool. At the other end there's cruelty and farce. Elaborate workarounds for things that ought to be easy. Mind-bending debug sessions for what never should have been possible.

No one wants this to happen! The career incentive is to ship. We're tearing our hair out over it - honestly a big reason people have side projects is the catharsis of just doing everything the sane way. Overcomplication is a dysfunction that plagues engineering organizations. But the solutions are more subtle than "just say no" - it's about the quality of the company's platforms, degrees of NIH syndrome in the platform group, tradeoffs made between freedom and consistency in system design culture, wisdom and foresight in anticipating how different things will interact, etc.

Also underrated: at the time the company needed an X, Y was not around yet, so we adopted X. Yes everyone now agrees Y is better. We are migrating to it, but that takes time. No you can't have your own instance ahead of schedule. so you need to make do with X, even though it is worse and more complicated.

There is some of that for sure but fundamental things like software dependencies change so frequently in this industry such that any code you ship has a long and unpredictable tail of maintenance work, so you end up spending a lot of your time doing things like ensuring your code doesn't call broken/deprecated APIs or migrating away from EOL'd database versions just to keep the lights on. I think we SWEs end up creating lots of work for each other.

If you think this point further though: does that make companies overstaffed or just mismanaged? Apparently cash flow supports companies to have these dev teams on staff, what if they worked on something that was a good investment of their time? Being overstaffed would mean that there was no way for companies to create more revenue by investing more in the right areas and I think this is a fallacy. The world is still so mindblowingly analog and inefficient in so many places that I don't think software is done eating the world yet.

> does that make companies overstaffed or just mismanaged?

Good point. The end result is the same: inefficient allocation of resources that could pass for productivity. Weird incentives at the management level.

The results may look similar, but they call for very different responses.

Firing productive engineers isn't going to fix managers whose vision is not aligned with business needs.

This is the part the really got to me once I started to realize that my contributions started really impact a team in a positive way, but the vast benefit of that impact was bestowed upon the manager. More annoyingly, even when the manager was poor ( and, I do not understand why, there seem to be a lot of those around ), the blame was always that of underlings. I was under clearly deluded impression that leaders gets blame and glory.

In my corporate life, I saw total of two exceptions to that rule ( now and my previous boss ).

That's the exact opposite of what a good manager would do. They'd take the blame for things that go wrong and credit the team for everything good that was accomplished. They'd be an advocate for their team and their reports individually. I've been fortunate to work with a few of those managers and they've been the ones that have been really successful long term. Keep searching or take the management route yourself and show them. It's very rewarding!

Managers gets promoted by getting credit for accomplishments, any manager who got promoted a few times will be good at getting credit for what you do in some way or another. Some deserve the credit, others don't.

An third possibility: Management and/or engineering leadership sees real problems but is investing in them poorly. For example, resourcing multiple teams to rearchitect your backend when the real issues lie in test coverage and observability, or something like that.

> * does that make companies overstaffed or just mismanaged?*

My guess is "undercompeted", though I'd hope there is a better word.

If an organization has a large and secure revenue stream whether is works on improving output or not, it will focus inward, on making life better for the insiders, not the customers.

> what if they worked on something that was a good investment of their time?

It's not always obviously true that there is such a project. Neither companies nor teams are fungible enough to make this necessarily true, at least not to the extent that ROI is expected to be higher than a smaller more focused team.

work expands to fill the person-hours allotted

I was at a Series C startup that suffered from this. The CTO was an SRE person at heart so wanted to build out our own infrastructure that could run on any platform. While this was a great idea in theory (basically everything was kubernetes and terraform) in practice it caused us to hire a ton of SRE people that were not contributing to the features that actually made us money. Not one customer cared that our entire service ran on AWS.

I think a lot of companies suffer from premature optimization. I myself even fell victim to it in one of the projects I worked on for that company by rewriting a large portion of the code instead of hacking the new feature in.

It is a hard problem to solve for. How much tech debt do I take on now to deliver customer value. I guess that is something that just comes with experience though. I have noticed that has I gain more experience I am not enamored by the new shiny framework as much anymore and focus more of my effort on delivering whatever feature is needed with the smallest change possible. Yeah this creates tech debt, but at least it keeps the lights on especially when every service seems to get rebuilt in 5-10 years anyways.

I wonder why FAANGs don't direct their employees to spend non-business-critical time on open source projects (dependencies etc) instead of the projects you are lamenting.

Yes

>migrating to Go because, using Protobuffs for a simple eCommerce API that does not need it

I would imagine improving throughput a fractional percentage point could have multiple millions of dollars in return for any company close to FAANG size. I fail to see how these are good examples of 'wasted' engineering effort.

Building your own chat system is a good one though, definitely seems like a vanity project.