One of the nice things about Bazel that the article didn't get a chance to go into is it uses SHA hashes of files, rather than file timestamps, to determine when an artifact has changed and therefore needs to be updated. It's slightly more costly to compute hashes, but it's necessary if you want to have something like a global cache of build artifacts, since synchronizing time across machines is hard.
What I'd recommend for anyone really, is to just do what Google did. For the first six years of Google's lifecycle, they got along just fine with GNU Make. Then they switched to the huge scalable thing once they actually reached that inflection point. I'm obviously not there since I'm just a scrappy open source coder. So for me I'm quite happy to be working with GNU Make and I can foresee myself getting many additional years of use out of it.
For the first decade and a half of Google's company lifecycle, they got along just fine with GNU Make
??? Google was started in 1998, and Bazel was created ~2006 as a replacement for Python + GNU Make ("gconfig"). I was on that team, though I only worked on Blaze a tiny bit. The "google3" build migration was sometime around 2003 or 2004.
So at most there were 6 years of using Make only, i.e. "google2".
Importantly, pre-Blaze google3 wasn't just GNU make -- Python was a huge part of it, which is why the Bazel build language Starlark looks like Python. It used to literally be Python, and now it's a Python-like language with parallel evaluation.
---
If you want to do what "scrappy Google" did these days, then you should use Python + Ninja. Ninja is meant to be generated, just like GNU Make was generated by Python. (A big difference is that GNU make has a big database of built-in rules that basically do nothing but slow down incremental rebuilds.)
This work with Landlock looks very cool, and it would make a lot of sense for Ninja to have optional support for it. Some of the caveats are a bit scary but hopefully that can be worked out over time.
The way I was thinking of doing it was just to have a ./NINJA_config.py --slow-sandbox mode. So you can use any sandbox to warn you about missing dependencies, including something container-based like bubblewrap, symlink farms, or Landlock. I think that would work, though I haven't tried it. The shared library issue is tricky, etc.
It's very useful to have the build config / generator split for this reason, and many others (e.g. build variants go only in the first stage, not the second).
I wrote 3 substantial GNU makefiles from scratch and regretted it largely because it lacks this split -- it has a very tortured way of doing build "metaprogramming". IIRC one dimension of variants was OK, but 2 got you into the "write a Lisp in Make" territory. Might as well use Python (or Lua, etc.)
I wouldn't say it's scary. There's always been full transparency with the caveats and they're being lifted incrementally. https://twitter.com/l0kod/status/1556378406983458818 The workarounds are perfectly reasonable. Also take into consideration that Landlock is so simple as a security tool, that it really opens itself up to people like us being able to focus in on the opportunities for improvement. A lot of security stuff, like containers, is so byzantine that no one would enjoy having open discussions about their design. Landlock has felt different to me, simply because we are having these conversations, and it's been so engaging that it really makes me want to believe in Torvald's whole mythology about the many eyeballs.
Email me if there's anything I can do to help Ninja adopt Landlock. The Cosmopolitan Libc pledge() and unveil() implementations are written to be relatively easy to transplant into other codebases. I'd love to see a broader audience benefiting from our work.
Yeah maybe I shouldn't say "scary", but I think the shared libraries and hard-coded paths would have to be worked out / generalized for something like this to land in upstream build tools.
And obviously having a working demo like this is huge progress in that direction, so I think it's great work.
But I'm also wondering if there's a way to do it without OS-specific support patched in? That is, with a wrapper analogous to bubblewrap, sandbox-exec, etc. I know pledge() should make use of app-specific knowledge, so it can't be a CLI wrapper, but I think that isn't true for the build tool case?
It would be more portable, and reduce an O(M x N) code explosion, if there was a standard access-dropping CLI interface that every OS could implement, and that every build tool could use without custom OS-specific code. I've been meaning to experiment with that for a long time, since I agree this problem is worth solving! (for correctness, caching, distribution, etc.) And I think Ninja is a good start and you can do some creative things with it, like
I also think having the generator split is nice because you could do ./NINJA_config.py --with-sandbox on Linux, and get all the dependencies correct. And then on OS X if there is no good sandboxing tool, you can just do it without the sandbox, assuming that the deps were tested on Linux. (And I agree the most practical answer on Windows is to "use WSL" :) )
There's an element here you may have overlooked that's important to understand. The purpose of a sandbox is to control things. The purpose of dynamic shared objects is to delegate control to other developers so you can leverage their labor at minimal cost. That means giving up control. These two concepts can't be reconciled. You can't properly sandbox dynamic shared objects existing outside your dominion because they're not yours to control.
I solve this problem for myself by using static binaries. I don't need dynamic shared objects. I ship support for it in my project releases, since I know other people do. That weakens the safety my tools can offer dso users, but it doesn't concern me, since I'm offering them incremental value; a weakened sandbox is better than no sandbox at all.
Cosmopolitan Libc is what I have for anyone wanting the stronger model. You can't use it to leverage an ecosystem of third party packages. What it can offer you is a statically linked hermetic environment with less complexity that's more conducive to control, provided you're ok with some assembly being required. Worth considering for your next greenfield project.
I understand that and don't really disagree with any of it, but what I'd say I'd say is that there's a third possibility of doing what containers already do -- use dynamic linking against fixed versions you control.
That is, the way I think of it is more along the lines of whether an executable is a "value" (in the Hickey sense), not whether it's statically linked. So the whole container can be a value, but it's not statically ed linked.
That has bearing on both incremental builds and distribution. This relates to my comments on the article about size optimization:
i.e. I think it makes sense for executables to retain structure for differential compression (they can be a tree, not a flat file.)
---
Also I'm wondering if the sandboxing can be done in a child process, without the cooperation of the build tool. I don't see why not, but I'll have to try it (and especially with the container model, which changes things a bit). That could actually make the dynamic library issue easier to deploy because you've punted some policy (hard-coded paths) into a separate tool, rather than having it in the build tool.
I wrote 3 substantial GNU makefiles
from scratch and regretted it largely
because it lacks this split -- it has
a very tortured way of doing build "metaprogramming".
GNU make definitely still has a lot of rough edges, but parts of your comment seems as though you're making claims about it that haven't been true in a decade or more.
There's quite a few small things about modern GNU make that make it easier to use for meta-programming than it was even just a decade ago. It still has a lot of rough edges that I wish it didn't have though.
Can you point me to a widely used Makefile that uses Guile in GNU make? AFAIK it's not available on most distros, so people don't use it. It's an optional extension
I don't disagree with anything you said, but I think it's important to point out that your experience is unique in that you're talking about build infrastructure for a massive codebase: most of Google's internal stuff, right?
In my experience plain GNU Make works great until you start working with massive projects. Similarly, the ninja speed improvement: you just don't see it unless you have a truly massive project, or you're using low-quality Makefiles, like what CMake produces. I've measured; I can't see it. I have many medium-sized projects, all with simple Makefiles, and it works really really well. I'm pretty convinced at this point that this should be the way to go for most projects out there.
The 3 makefiles I wrote were actually for a medium-size open source project https://www.oilshell.org/ (medium measured by lines of code)
I could probably write a blog post about this, but for two of the Makefiles I needed to enumerate the rules dynamically (inputs weren't fixed, they were "globbed"). And my memory is that this interacted very poorly with other GNU make features build variants
Whereas that pattern is extremely simple with Python/Ninja. Just write a loop or a nested loop and generate a build rule on each iteration. It's done in 5 minutes, whereas GNU make was a constant struggle.
I will chime in in agreement with you, since the replies here are mostly to the contrary. Writing Makefiles by hand sucks eggs.
It's really worse than this because this probably assumed a specific Linux setup at least, but everything goes out the window once Windows is in the mix. If having to deal with potentially multiple shells was a problem with Make, having to deal with multiple shells on Windows (where it could be CMD, PowerShell, or a UNIX like shell under MSys or Cygwin...) is untenable.
Today the only obviously valid reason to use Make is because you have to or already are. Most Linux distros ship Ninja. Hell, MSVC ships Ninja.
There are many examples of Makefiles for open source projects that validate how ugly things can get without needing to be Google scale. One of the most elegant Makefile setups I can think of is Near's setup for bsnes/higan,and honest to goodness, it's still a pretty horrific mess.
I don't want to be hyperbolic, but also it's irresponsible for people to suggest you should just use Make. You shouldn't just use Make. Not even as a beginner. If you wanted a nice turnkey solution for a beginner, I'd suggest CMake or Meson, both of which happily generate Ninja files and also have the bonus of supporting Windows well. CMake has broad ecosystem support, despite it's many shortcomings. It's far from elegant, but definitely practical.
That's Windows' problem and it's not even a real problem anymore because make runs fine in WSL. Microsoft has pretty much gotten their act together in the last four years in supporting open developer tools. They've got bash and ANSI support. It's great. Give them credit where credit is due. It's time to say goodbye to shoehorning unix devops into win32. Doing that gets sillier each year. Especially since, as Microsoft has been generous in supporting tools, they've certainly been boiling the frog with their win32 environment. The last time I extracted a zip file containing dev work on Windows, it extracted at 64 kilobytes per second to the local hard drive, because their virus technology was scanning and uploading everything I was doing. How can you build software in that kind of environment? And why is it that it's always the voices coming from our own community who insist that we should. People shouldn't fall that far in love with an operating system because even with perfect support, builds are still going to go slow on Windows. Use WSL.
This only makes the problem much worse. (Note that I literally reference MSys and Cygwin in my original reply. However, these options are neither ideal for making good Windows binaries nor are they great facsimiles for UNIX. Almost any Makefile needs to be carefully constructed so that MinGW works, especially if you want to support more than just very specific MinGW setups. And there's other downsides I didn't get into, such as licensing problems with winpthreads, and silent dubious behavior like pseudo relocs. I don't recommend using MinGW.)
I like the speed of Ninja, and the flexibility of having a stage that generates the ninja (or make) files. But it's a bit unclear to me what the best practice is for when to generate the ninja files. It feels like the programmer is expected to do this stage manually and there's no great automatic way to generate the ninja files when they need to be generated. How do good build systems solve this?
Yeah unfortunately there probably isn't a "best practice". I think there are just many types of projects and their needs vary. I think a big issue is what your dependencies are.
This person mentioned Ruby + Ninja, and I similarly used Python + Ninja (borrowing the ninja_syntax.py from the Ninja repo itself).
I'd say if you can get away with it, using a real language like that is straightforward and good. If you have a low level project that doesn't depend on anything (like a shell or VM :) ), then it will probably work.
I admitted I probably had to rewrite a small portion of the "120,000 lines of CMake" that ships with CMake, in Python. And that took some time, but it ended up working well for my needs.
If you have a big project with lots of dependencies (especially optional / detected ones), I think that is probably where CMake or Meson is better. They both use Ninja, but I haven't used them myself.
---
But I would also say that there aren't many best practices around GNU make either. The usage spans a very wide range... And lots of people generate it too, like Ninja -- kconfig for the Linux kernel generates GNU make, etc.
autotools and CMake both generate GNU make too. It's a big mess, but I think Ninja is a good foundation for at least not "starting with a mess" :)
> What I'd recommend for anyone really, is to just do what Google did. For the first decade and a half of Google's company lifecycle, they got along just fine with GNU Make. Then they switched to the huge scalable thing once they actually reached that inflection point.
Hopefully as a community we can build things that are scalable and as-simple-as Make. I think please.build is a step in the right direction but still too complicated.
GNU make is great for many things, used correctly. The problem is that POSIX make is extremely impoverished, so sticking to just the POSIX subset is often a bad idea. In many cases using "make" should really mean using "GNU make" so you can use conditionals, automated dependendency generation (via reloads of dependency info), etc.
However I also cringe when people say to write GNU makefiles from scratch -- because those ALSO don't work for everybody, and are generated more often than not (e.g. by autotools, CMake, kconfig, etc.)
So I'd say that GNU make gives you the illusion that you can just use that one tool, but you often end up needing to generate it anyway. And then you should have used Ninja with whatever generator you ended up with :)
1. Getting started is difficult: There is a lot of impedance mismatch across various language toolchains. This can be fixed over time by improving the tooling/libraries/docs. If JetBrains had a "New Bazel Project" and had a way to 1-click-add Java/Python/Ruby/golang/Rust/etc source into //third_party with BUILD files and a version of build_cleaner Bazel would win. Just making it easy to vendor libraries and build software from the IDE is I think all it would take to get popular.
2. The JVM: I am a huge fan of Java (I've build multiple company's backends in it) but it is not a great choice for CLI tooling (even with a daemon). A no-dependency, small, build system executable would go a long way to making it easy to get people started.
3. The cruft: A lot of things in Bazel are the way they are because someone had to get a feature out ASAP to support $BIG_USER to do $BIG_THING and it would be too difficult to migrate away/to something at Google. If we drop the cruft and redesign things from the ground up we can get a nice abstraction for the world. For example, please.build's proto_library is VERY easy to use (way easier than even the Google internal versions imo).
4. The tooling: You can get massive CI improvements, free mutation testing, frameworks for build (integration tests, e2e tests, etc), and much more by building things against the Bazel API. Unfortunately not much outside of Google supports this API. Example of things you can do with this API: https://kythe.io/examples/#extracting-compilations-using-baz...
We could live in a world where people can build langauge-agnostic tooling that automatically works so long as you pass it a `*_binary` target that transparently builds for all major platforms (zig's or APE's as crosstool) which would allow platform vendors to define macros to completely automate deployment and development for their systems. For example we could have:
```
from aws import lambda
lambda.package(
name = "my_api",
deps = [":api_handler"],
)
```
And just by running `build :my_api` you could have a zip file which packages your code (from any language) into a lambda.
> I spend a lot of time advocating for Bazel for this reason.
Bazel works pretty well for large scale projects. There are definitely things I don't like about, but I would agree that for large projects it is better than make.
But for small to medium size projects, bazel adds a lot of complexity and has the problems you mentioned, for not that much benefit. Especially if you are using a language that doesn't have built in support in bazel,or do something that doesn't match up with bazel's way of doing things.
SCons [0] uses something similar (MD5 instead of SHA) since the end of the 90es, so at least that aspect is not a Google invention. The cache there is local though. We never had flaky builds with it and could extend it very nicely, unfortunately it was not very fast.
(GNU) Make by default uses the file change timestamp to trigger actions. But this is definitely not the only way, and you can code your Makefile so that rebuilds happen when a file's checksum changes.
IIRC, the GNU Make Book has the code ready for you to study...
Or, you might get more clever and say "when only a comment is changed, I don't want to rebuild"; file checksums are not the correct solution for this, so you can code another trigger.
What I'd recommend for anyone really, is to just do what Google did. For the first six years of Google's lifecycle, they got along just fine with GNU Make. Then they switched to the huge scalable thing once they actually reached that inflection point. I'm obviously not there since I'm just a scrappy open source coder. So for me I'm quite happy to be working with GNU Make and I can foresee myself getting many additional years of use out of it.