The only fix is quality forward thinking details oriented leadership.
Given that corporate America has decided that all employees are meant to be replaceable and therefore transient, it is not rational for any individual to orient themselves to the long term or make near term concessions. Likewise if employee compensation is a function of hours worked more than it is a function of the company's success, it is irrational for an employee to make decisions with respect to the long term, especially if they will be rewarded for speed over complexity tradeoffs and they can leave the company with a great resume when the debt becomes overwhelming.
Admiral Rickover's speech Doing a Job speaks quite a bit to technical debt and the forces that promote it: https://govleaders.org/rickover.htm
> The manager, of course, remains ultimately responsible and must accept the blame if subordinates make mistakes.
> Complex jobs cannot be accomplished effectively with transients. Therefore, a manager must make the work challenging and rewarding so that his people will remain with the organization for many years.
> The ideas I have mentioned are not new—previous generations recognized the value of hard work, attention to detail, personal responsibility, and determination.
Engineering is not a new discipline. Software engineering is, but humans have thousands of years of prior art, and it is being actively ignored by today's tech industry. I was privileged to work for a 100-year-old engineering company. It had a lot of frustrations, but they made damn good hardware. I learned a great deal from them (and also banged my head bloody, trying to get them to let me deal with software differently from hardware).
I think that this should be on a billboard on I-280S:
> ... when the details are ignored, the project fails. No infusion of policy or lofty ideals can then correct the situation.
They treated software in exactly the same manner as hardware: measure twice/cut once, waterfall. All QC was after the fact, and done manually.
We did a great job of delivering yesterday's technology, tomorrow.
I wanted them to add flexibility. Things like a different QC system, iterative development, and faster releases of software.
I won't get into details, though. Much as I wasn't happy with things, I had (still have) tremendous respect for the company, and wish them nothing but good luck (they'll need it).
I will say this, though:
I worked with some of the top engineers and scientists in the world. The company had been producing precision optical equipment for 100 years, and people paid a lot of money for it.
It was extremely hard to suggest that they might want to consider doing stuff differently. They had a formula that worked quite well, and not changing on a whim was a big part of that. I think most folks, hereabouts, would go nuts, working that way, but you can't argue with the results.
I learned to write software of a pretty high Quality, as a direct result of working with these folks.
I violently agree with this comment, but don’t believe the will exists to put people in a position of power or leadership in today’s corporations to set this culture at the top (which is needed to attract, retain, and cultivate talent who grows with the org and cares about their work and its contribution to the org’s success). Short term thinking will reign until forced out at great cost (Boeing, for example).
I agree. Moreover plenty of jobs where they measure just the time to completion. Even small code fix ups as you go will be treated like an afternoon slacking off on Reddit and punishment or firing accordingly. At most places of work.
Another good fix is legacy companies getting replaced by newer, more nimble companies.
It's good for the economy for new startups to tackle old problems in new and novel ways, throwing out decades of legacy cruft and ossification.
It also puts more comp into the hands of the founders and frontier employees that pave the way rather than funneling it to institutional shareholders of an old business - pension funds, etc.
Sure, but for companies that fail their code ceases to be run by them. If their code is used by other groups then they will have to deal with maintenance themselves, if that's even possible.
This risk is somewhat mitigated by demanding only open source software be used. At least then if a software's supplier goes bankrupt then updated versions can still be produced. That's not the case with closed source software.
> This risk is somewhat mitigated by demanding only open source software be used. At least then if a software's supplier goes bankrupt then updated versions can still be produced. That's not the case with closed source software.
If your solution starts with "for this to work all software used will have to be open source" you have a completely nom-solution.
The problem is that all business incentives are based on short-term outcomes. Sometimes there is a culture of being deliberate about tech-debt but that culture eventually gets eroded by the MBA/bean counters, poor hiring, or scaling too fast. The end result is you end up pouring ever increasing amounts into engineering for smaller and smaller deltas of change. Eventually to the point it can become 'terminal' where you literally cannot change the system in one place without breaking it in another. At that point you just sell the company if you can to some PE firm that will load it up with debt, loot it, and dump the corpse to decompose in the nearest body of water.
Tech debt that fails fast or allows the company runway to fund a better design down the road, is a market advantage. Short term thinking is more effective, which is why it's so common. Skip some considerations to figure out the specific target audience or functionality sooner.
I don't think the name fits at all. It's tech centric, which is self serving of the tech community, by the tech community.
Tech debt isn't always a bad thing if it is managed properly. Developing for the short term can help a company to succeed in the market until a better solution can be produced, where more careful work may have prevented the survival of the company.
The problem comes when technical debt is never paid off and continually accumulates, even when it's not necessary for it to.
> Years ago, companies owned their own servers, and the penalties for inefficient systems weren’t that great—just buy some more racks of computers and throw those at the problem. Now that everything has shifted, or is shifting, to the cloud, and companies are paying for computing a la carte, those inefficiencies mean bills that are, on average, double what they could be if old systems were optimized, says Ben DeBow, chief executive of Fortified, a consulting firm that helps companies manage and update their database systems.
I've found good success in the past with associating my optimizations directly into the cloud bill. This looks like "Improved efficiency of {MyService} by optimizing {cache hit rate}, reducing the number of pods required of the service by 25%, saving {$300k/year} in AWS costs." I look for these types of wins every time I am on a new team.
Yeah, this is why tech debt is a $1.52T problem. A few points.
* Reducing the number of pods used by a micro service doesn't reduce cost by 25%. For most micro services data storage, data transfer, and other required dependencies will be a very large chunk of the total bill.
* In the organizations I've been in the director is always looking for ways to reduce the cloud bill. This is always a huge line item for them, and the modern microservice architecture is only making the cloud bill higher.
* How is this anything but objectively good for the company?
* Avoiding stirring the pot won't make you any enemies in the workplace, but it won't make you any friends either. Deliver value where you can and the workplace politics will be what they are.
* I've be a part of, and have seen, cloud bills cut by as much as 80%. When things get that bad you won't lose reputation by fixing it because people in many departments are feeling the negative impact of the microservice.
The other week I saved my company a grand a year by reusing an existing service for some contractors rather than some new cloud service. Then promptly expensed a big chunk of that for repairs on my office chair :)
Have you considered whether your optimizations are too specific to the current needs of the org and may end up making the org less flexible in the future? Optimization is great, but it also often has costs of its own, particularly in the area of complexity.
The term 'technical debt' is conversationally bankrupt. Every possible programming anxiety is technical debt. Don't understand the system, the historical constraints, or the business objectives when it was written? Slap a 'technical debt' sticker on it and now you're the smartest person in the room again.
If you want to be an effective roadmap participant, learn to phrase technical problems in terms of immediate benefit to customers. It can then at least be prioritized against the reams of other immediate customer benefit ideas in the backlog.
If you want to marshal an impressive development cadence, focus far upstream of 'technical debt'. Whatever that term means to you, it's a lagging indicator. The solution is procedural, not technical.
This, 1000%. I’ve never had any luck with figuring out a way to get management on board with tech debt, and this is a nimble startup with smart people… solution is elsewhere, not “write more quality software” near the symptoms.
This dismissal of 'tech debt' as a phrase/concept based on weak strawman examples is just a contrarian rant followed by LinkedIn-esque advice that seems ignorant of the fact that hyper focus on 'immediate benefit' is almost always the cause of.. the technical problems that should be phrased in terms of their immediate benefit, or something.
The biggest problem with the term "technical debt" is that it's thrown about by too many people for too many reasons. Mostly well meaning reasons, but often highly personalized and with literally no intrinsic value or thought attached. More often than not it's used to express a personal preference, rather than a guiding principle.
I've seldom seen it used positively, to a point I don't actually believe it has any benefit as a term and given that I don't think it actually exists.
It's not an actionable term. Talk about and make cases for system performance, operability, functionality, security, cost, support knowledge and similar and you'll have much easier value based conversations that have a chance of having continued buy in long term from your organisation.
One thing I’ve noticed in my career is that it is often quite difficult to distinguish “good” technical infrastructure improvements (meaning ones that will actually help a company achieve its goals) from “bad” ones (stuff that is unnecessary, risky or simply not worth the investment). I've seen these decisions made more using the political capital of high level engineers than by anything data-driven. That’s not to say this isn’t a problem, just maybe more of an art than a science.
A lot of things I've seen engineers do in the name of cleaning up technical debt actively make things worse
Left to their own devices a lot of engineers will go to enormous lengths for either marginal improvements or changes that actively hurt. I can understand why leadership is skeptical of these investments because even for someone extremely technical, if you're not in the weeds everyday of that particular problem, it's very hard to know what's worth it
I really wish businesses thought more about the butterfly effect.
Putting small tickets that help reduce technical debt, increase developer happiness or help make progress towards a longer term technical goal at the bottom of the backlog because something larger and product focused always gets higher priority is such a great way to ensure you end up with lots of technical debt and unhappy teams of developers.
It is demoralizing to continue to identify problems and outline solutions but they never see the light of day. Also when you encounter the problem again and again and again it's reliving this negative memory of "well this is never going to change". Eventually this happens long enough where it becomes a culture at the company and most folks stop investing their energy into identifying them and things get worse as time goes as you cycle through new developers because people quit and new hires come in. The company loses so much with the loss of domain knowledge and hiring that the product greatly suffers in the end.
Never under estimate how much of a difference a motivated and happy team can make.
I’ve been in the software industry for over 16 years now. From my experiences, it seems like it’s always more difficult to get investment in addressing technical debt. First, many developers, me included, aren’t always the best at communicating the value investing in scoped improvements projects brings to the business. I’ve improved at communicating the value upwards over time. Executives don’t see software the same way engineers do and you need to learn to speak their language. Second, investing in a new feature or project can easily become a more desirable investment because it’s new, exciting, and potentially a differentiator. The best way to get investment to address tech debt seems to have a crisis unless organizations have a deliberate, systematic process the helps avoid accumulating too much debt.
You’ll never get a chance to refactor code if you ask for it. The business folks and users themselves don’t care what the code looks like. They don’t care if there are 11 different micro services and a hodgepodge of pudding holding it together. They never will.
Don’t waste time trying to write a better pitch. You won’t get to halt development to essentially deliver nothing as far as they’re concerned.
Write code that is meant to be deleted. Refactor mercilessly. If code looks like crap but had sufficient tests, rewrite the code. If the tests are crap write better tests. Then rewrite the code.
Don’t ship the first thing that works on your PR/MR. Take some time to compress that code down, get rid of any cruft, and make sure it’s easily replaceable. You never know what new requirements or changes in specs will bring you back to this code in the future. Make it easy to replace (in the worst case scenario) if you needed to.
Wrap code that isn’t under test in a good API. Make the API call into the bad code. Write good tests for the API. Replace the bad code paths as you go.
Part of our job is writing maintainable, stable, and reliable code.
But if they’re deliberately asking you to cut corners, start documenting things.
There’s a time and place for “good enough,” and then there’s straight up neglect. Avoid the latter.
I've been trying to push the phrase "Design for deletion", as a contrast to designing for a flexibility or extensibility.
I used to try to predict far-ahead feature or architectural needs, but more often than not they either never materialized or appeared with some unforseen incompatible quirk.
Instead, I want to ensure that whatever I write can be ruthlessly ripped out and replaced with a minimum of pain.
It's not quite the same as a blanket priority on decoupling code, since statically-checkable compile-time coupling Isn't the big problem, and might even be preferred over runtime plugin architecture astronautics.
I'm a huge fan of this "stay in your lane and keep your house clean" philosophy. IME I just obsess over the details of whatever job I have at any time. People don't even really care what you're doing, all they see is someone that gives a damn in a sea of apathy. They'll promote you the first chance they get. I promise.
This has worked for me since I was taking out the trash at a fast food restaurant all the way to leadership positions at Fortune 50.
> many developers, me included, aren’t always the best at communicating the value investing in scoped improvements projects brings to the business
While that is true, I don’t think it could be a communication-from-developers problem. Tech debt impairs development speed and causes small irritations in the every-day use of the product.
In my experience, what works is extreme ownership. Someone with agency has to care.
“The only lasting fix for technical debt is for management to empower the IT side of an organization to start saying no to all the requests for shiny new things, and focus on making what an organization already has more robust, more secure and futureproof.”
One could argue that technical debt is really just regular debt. If can see that, then you also will see that federal fiscal and monetary policy are really the long term fix, not IT departments saying “No” which of course they already do.
The whole problem is that it isn't. If it were regular debt, it would be visible on balance sheets instead of just ruining everything behind the scenes.
Like cost per user for infra? operating cost per hour, by hour? like that...
You need an accountant embedded in your department to just watch and track spend and map it to something. Preferably they are in tech and told to hold product teams feet to the fire.
I've seen plenty of environments where the technical debt stares you in the face wherever you look, but no reports make it to decision makers (and the ones to make decisions are generally not the ones in the position to see all that debt).
If you're in an org pretending everything is fine, that would probably apply to many more problem topics and not just technical debt.
Does that boil down to organization culture and whether lower-level employees feel comfortable raising issues to leadership? I believe Boeing would be in a much better situation if it encouraged raising concerns instead of suppressing them.
I suppose it is often a combination of company culture, local culture and KPIs and then a bunch of smaller things.
I'm not up to speed on the latest and greatest research, but last time I went into that rabbit hole it boiled down to hiding problems or moving them around so the KPIs look good (be it KPIs in the company, or in the case of publicly traded companies, the 'virtual' KPIs of news and press releases), and the degree to which hierarchy forces everything down.
That directionality gets exponentially worse when you have a local culture of strong hierarchy, and a company culture of making leadership look good (or honour culture?). I've only seen that personally from a slight distance where it really looks like a kind of dictatorship where management comes up with an idea, and the subordinates have to make due with it, and only positive results may be shared or they get fired. The ideas are usually bad or really hard to execute so you see people use all available resources to present 'success' with not much left for the actual thing that needed doing.
Business people continue to collect large paychecks while technical debt ruins their companies ability to deliver innovative product.
There are no repercussions, and no mechanism to correct this. There's also no easy visibility into the fact that this is occurring, aside from the mental strain on junior/mid level engineers.
Even at the principle engineer level, the incentive is to continue collecting paychecks and not rock the boat.
There are many exceptions among technically sophisticated companies, but your average old school company is really a vessel for corporate careerists to build personal wealth.
But all of the above assume that technical debt should be fixed, alot of the time its a huge waste of resources to fix with minimal business value. Most money flowing into a company comes from a few systems, everything else is not worth it.
What's wrong with me viewing the business as a way for me to personally build wealth? I want to do a good job and I want to do the best of my abilities but I cannot care more than the management infrastructure does. The entire purpose of the corporation is to build wealth for their shareholders and the board so what incentive do I have that they do not have? How can I be more invested in the company than the owners and leadership of that company?
Maybe it wasn't your intention but it seems like your post is blaming the engineers for the problem rather than the people who allocate the engineers time. The engineering staff is just operating according to the demonstrated core company principles. Of which those poor principles are usually greed.
Sigh… if you want to communicate the issue with business minded individuals in a position of leadership it helps to use the correct accounting terminology. “Technical debt” is a misnomer. Software is an asset. If it is made quickly and shoddily it amortizes/depreciates at a faster rate than otherwise. Weighing maintenance costs against recapitalization is the conversation to have. There is clearly a way to make measurements that can help answer the question of what direction to take.
Technical debt is like financial debt: it's not a problem if it is kept under control, and it may even be a very smart leverage; it has a limited liability in many ways.
Also innovation and debt are often intermingled.
SpaceX cutting corners, it could be seen as debt by slow-paced companies, but at the end of the day, the result is here.
I believe it is in a way. Everywhere I've ever worked in tech leadership has a dataset of applications marked as decom, maintain, or invest. They never plan on doing anything with "maintain" until a replacement comes along. There are hundreds of thousands of saints toiling daily on maintain and decom apps. We need all types to make the world work.
There are trucking companies driving old trucks that break down all the time that are total accounting write-offs. They have competitors with shiny new leased fleets. Both situations are known to accounting and reported on. It's a trade off of what leadership does with that info. Is it better to have no expenses and still bring in occasional revenue or is it better to have reliable revenue with high expenses and low margin? Not always an easy decision but definitely not one that resides in accounting.
- Developers are vastly overrated. They keep developing new frameworks when they’d better spend their time solving business problems. Given their cost, no-one dares telling them anything.
- Can’t motivate them to do tech debt.
- They always suggest to refactor everything.
It’s basically impossible to give business sense to a developer. Otherwise they wouldn’t be developers! But they want to be free.
It’s not a “My boss told me not to do tech debt.”
It’s “Every time I’m given free time to tackle tech debt, I open Draw.io to create charts of what I suggest as a new infrastructure with microkubes.”
This looks like a description of a team or department that has no product owners and no requirements elicitation process.
Quality aspects can be measured, and resulting requirements will mitigate technical debt, unless you put that analysis so far down the backlog that it falls off. But all of that is management or 'team overhead' work.
As an engineer manager, I do agree with the sentiment though. Keeping engineers focused is tough. Allowing them time to burn down tech debt might mean they go off and work on interesting projects instead of something of business value. I’ve also been burned by the “let’s start over and do it right” refactor that resulted in no customer value. Making software is full of tradeoffs.
