I haven't seen a broken build in at least nine years, not since I left the company with a merge process built out of bash scripts that took three hours and required manual hand-holding.
I am genuinely curious what situations you are seeing where builds are making it through CI and then don't compile.
It isn't always worth investing in quality, but when it is it is entirely possible to write essentially bug-free software. I've gone seven months without a bug in production and the one we saw we had a signed letter from product saying "I am okay if this feature breaks, because I think writing the tests that can verify this integration was going to take too long."
FAANG companies aren't prioritizing writing software well: they are prioritizing managing 50,000 engineers. Which is a much harder problem, but the management solutions that work for that preclude the techniques that let us write bug-free software.
One of the great things about startups is that it is trivial to manage five engineers, so there is no reason we have to write software badly.
Of course, if people wrote bug-free code, then there would be no bug !
Bug-free code in the actual code, or bug-free code in the test code, this is the same story.
If you write stuff and never have any bug, then either:
- you are lying
- you do not write much
- you only write really simple things
- you are Jesus, came back from heaven to shine his light on us, poor souls
The more complicated, intricate stuff you have, the more bugs you'll get (and only time will allow you to fix that).
Tests are great do define how you think it should work, and to ensure it keeps doing that way. Take the time to think about the third point on the bullet list above.
In the DVCS era, we have inexpensive branching. Do as thou wilt on your topic or epic branches. Rebase them against main/master before merging upwards. Fix what must be fixed first.
Main/master branch should never fail CI. If it does, there is something seriously wrong with your branch lifecycle and/or deployment process.
If you test defensively[0], CI will catch the vast majority of functional bugs. Anything that is missed suggests at least two bugs: one in tests, and one in business logic.[0]
A reliable deployment pipeline is outside of CI, but can be kept straightforward and minimal to constrain the scope of failures.
Bugs happen, and systems complexify. It is possible to manage both of those risks down to near-zero by the time code reaches production release candidate stage.
In some industries, this is more important than others, though -- obviously the goal is to match quality to business needs.
But I agree with our thread predecessor: I haven't seen a broken build make it anywhere near production in many years, and it's not because of the snarky dismissal that provoked my original response.
[0] In some situations, proper tests are not possible, and in others they are not practical. And I acknowledge that I'm omitting things like visual design/layout bugs, which is probably not reasonable.
I disagree. I write complex code, and it is essentially bug-free.
And no, I'm not Jesus, I just care a lot about quality and have spent the last 20 years finding ways and strategies to improve it.
Reducing the number of bugs does not mean being a god that writes bug-free code on the first draft. It means being able to detect and fix issues as early as possible. In my case I aim to always do that before letting myself push any code to git.
IMO, it only comes down to how much someone really cares about the quality, but here are some examples of what can be done and is very effective:
- Plan ahead your functional and technical design
- Carefully research existing code to confirm the feasibility of the design
- Use a statically typed language
- Use advanced static-analysis tools
- Avoid magic, write explicit code. Especially avoid runtime checks such as reflection. Ideally, everything should be checked statically one way or another.
- Never let a code path/branch/corner case be unhandled, however unlikely it is (and go back to step one to refine the design if a code path has been forgotten in the current design)
- Always have automated testing. The bare minimum is to unit-test all business logic, including all possible code paths. Ideally e2e tests are nice, but not always a good investment. Tests must be 100% independent and never depend on an external environment, otherwise it's going to be flaky at some point.
- Always manually test every feature and path related to my changes (especially don't skip testing the ones that I think are going to be ok) before pushing anything to git.
- Warnings and "optional" notices are unacceptable and must always be fixed (or disabled), otherwise the list will just keep growing, which reduces the visibility of any issue and normalizes having problems.
- Have a CI integration that applies all the automated checks mentioned in this list and make everything mandatory.
Each one of those actions does on it's own significantly reduce the number of bugs. If you combine them all, you can effectively reduce the number of bugs to pretty-much zero. And since the earlier you find a bug, the cheaper is it overall to fix, I've also found-out that in terms of productivity it's always worth the investment (despite many people pretending the opposite).
I am genuinely curious what situations you are seeing where builds are making it through CI and then don't compile.
It isn't always worth investing in quality, but when it is it is entirely possible to write essentially bug-free software. I've gone seven months without a bug in production and the one we saw we had a signed letter from product saying "I am okay if this feature breaks, because I think writing the tests that can verify this integration was going to take too long."
FAANG companies aren't prioritizing writing software well: they are prioritizing managing 50,000 engineers. Which is a much harder problem, but the management solutions that work for that preclude the techniques that let us write bug-free software.
One of the great things about startups is that it is trivial to manage five engineers, so there is no reason we have to write software badly.