Nix is advanced alien technology that was badly damaged when it fell from the sky. In particular the docs/blogs/forums are net negative some ridiculously high amount of the time, like start composing an email: “I have a very special set of skills, and I will find you…” amount of the time.
But once it starts working: how does anyone do this any other way?
- The language is reasonably well-documented and those docs are rarely wrong.
- Learn bash, like really really learn bash.
- Check out nixpkgs and use that to figure out how to do stuff.
- Only do this if paying the entire cost in time and frustration in one aneurism-inducing spasm upfront is a worthwhile tradeoff for having everything work perfectly every time thereafter.
It’s kinda like being Paul Atriedes: you get magic powers but first you put your hand in the fucking box. What’s in the box?
Something Nix seriously needs is a searchable index of people's nix configs. When I attempted to use it, I had tabs open for around 10 configs that I was cross referencing but I can't find that list of configs anymore and I have more obscure packages I'm trying to figure out how to use and don't want to manually search.
This is on top of a GUI (syncs with git repos, at minimum about:config like with the documentation inline and ability to edit free-form entries), a Nix-style installer (pulls your config from a git host, inline machine profile editor, formatting help and saves formatting to repo), maybe some languages improvements, etc
I spent like ~6 hours going through the first ~10 chapters of "how to learn nix" and the corresponding docs, and I've built up a decent model of how nix works under the hood but have no knowledge of how to do the things I want to do (reproducible dev environment per project)
Hopefully that'll come in the next ~10 hours, or maybe I need to switch to one of these other learning resources
Since "How to Learn Nix" mirrors the (bonkers) presentation order from the official documentation, I am afraid to say that you will not learn how to do this until part 32, deep into the Nixpkgs manual.
But Nix Pills is not really a "bite-sized guide" so much as a "let's motivate and invent the Nixpkgs genericBuilder infrastructure from scratch." So it doesn't really show you an example of how to write shell.nix files.
Writing reproducible dev environments per project ("shell.nix files") is the main reason I use Nix. But Nix does not... make it very easy, out of the box. There are two more posts (separate from the documentation-reading sections) that will explain how to make this more ergonomic:
(This is slightly unfair because if you're using Nix to build software, not just create a development environment, you get nix-shell for free. mkShell is sort of a special case of "I don't want the final result in the /nix/store; I just want you to put me in a shell with these packages installed." It's not a case that Nix optimizes very well for, even though it is my primary use case.)
Finally, obligatory caveat that everything is completely different now in the "new" (unstable, flakes-based) Nix interface:
First, I want to say thank you for documenting your learning journey, because it's an excellent resource. [0]
> Since "How to Learn Nix" mirrors the (bonkers) presentation order from the official documentation
Yes, I should have clarified that the complaint is directed at nix docs and not your blog.
> I am afraid to say that you will not learn how to do this until part 32, deep into the Nixpkgs manual.
I was actually feeling a little dejected after my first sprint at learning this. I had skipped ahead to your summary posts ("so I read the manual") and saw that I wasn't even close to learning how to use nix as I wanted.
Some of the comments here actually gave me hope that the juice will be worth the squeeze, so I'll jump back in tonight. I might switch to `nix.dev` tutorial that will hopefully get me moving a bit, and then jump back to your blog and the official docs.
[0] Really, having your stream of conscious helps a lot. Sometimes you explain something a little more in depth or from a different angle, other times your confusion reassures me that I'm not missing something.
I saw you say elsewhere that you feel like these are "low effort" posts compared to your other writing, but it also feels like it's filling a whole in the online-learning landscape. So again, thanks!
Yeah, I think that nix.dev is a much better resource for getting started with Nix quickly.
I wrote How to Learn Nix mostly out of curiosity: I was interested in Nix as a concept, and I didn't really know what it could do going into it. I picked the title long before it became clear that reading the official documentation was not a good way to actually learn how to use Nix. "The Nix Diaries" or something would be a lot more accurate.
That said I don't know what a better approach would look like. nix.dev and tutorials like that are very useful, but my (possibly outdated) understanding is that they never take a step back and explain how or why things work. So you can follow along with what they're doing, following a carefully lain road, but as soon as the road ends you find yourself in the middle of nowhere without a map of your surroundings. (This was how I initially tried to "learn Nix" in 2016, and basically gave up on it -- nix.dev didn't exist back then, but I followed various random online tutorials.)
I think it's a good idea to come at it from both sides, and How to Learn Nix is much more in the "build intuition" camp than the "get something done" camp. I'm glad you're finding it useful, though!
> I wasn't even close to learning how to use nix as I wanted.
> Some of the comments here actually gave me hope that the juice will be worth the squeeze, so I'll jump back in tonight. I might switch to `nix.dev` tutorial that will hopefully get me moving a bit, and then jump back to your blog and the official docs.
FWIW, my learning journey with Nix was copy-pasting snippets and casually getting a deeper understanding as I needed it over months.
The experience with Nix is 95% wonderful, 5% painful. With many other tools, you can use them without really needing to know what's going on. With Nix.. you'll likely end up wanting to do something which will demand you understand stuff. -- Time spent reading the manual isn't wasted.
I think it's really important to have realistic expectations about Nix/NixOS when deciding how much effort to spend on it.
It's ridiculously hard to get started. The online resources are simply inadequate: incorrect and/or out-of-date information is basically equally common as useful information. There isn't a good book you can get on Amazon. The "Learn" boards thing is full of "you're doing it wrong" bullshit. The Discord is so-so at best for getting help.
For me personally it was somewhere between 1-3 months of being frustrated basically every day and I still want to throw things sometimes. It's basically clone `nixpkgs`, figure out how to get the REPL going, painstakingly reading the code of the stuff that does work, and pain.
It says absolutely nothing about how smart/capable/motivated/senior someone is that a week or two in it's like "fuck this".
The payoff is outrageous, but the barrier to entry is nowhere near something that can be called "acceptable", let alone "batteries included". I completely understand why a lot of people are all-but-hostile to it.
I would budget a minimum of a highly frustrating month working with it every day to start to having the lightbulb moments, and frankly your use case needs to be pretty extreme in one way or another for that to be worth it. A lot of people don't have a friggin month to screw around with a fairly niche infrastructure if what they're doing now works for them.
I really, really hope that at some point someone gets serious about making this fairly magical thing even remotely learnable, but right now it's a fucking mess.
I think this comment applies to nearly all technology today. If it doesn't have a `man` page, good `-h` notes, or a manual released with each version, I basically assume I'm going to have a hellish time Googling around for anything. This doesn't really solve the issue though, since a lot of manuals written are outdated by sources changes that were rushed and forgotten.
We need to come to terms with the fact that the way we as programmers acquire information about our tools is in a dire need of help. Search indices are just not valid references for us. We should never have learned this method of learning in the first place, but it was easy and painless in the beginning.
People need to just slow down a bit and become more methodical one thinks.
Even with StackOverflow, try finding the current canonical means of building Python applications and deploying artifacts. It's a maelstrom of outdated advice.
> I really, really hope that at some point someone gets serious about making this fairly magical thing even remotely learnable, but right now it's a fucking mess.
That's because all the ways you learned how to do things on linux you need to throw out of window and start from scratch.
I've found the Nix language to be a really useful configuration language, even "outside" of the NixOS ecosystem. I reuse nixpkgs's module system, but to describe things that are used outside of the NixOS system (for example in an Ansible playbook).
For example, I'm using it to describe a set of containers and services in my infrastructure, and it allows me to properly describe the interfaces between the different components of the infrastructure, using several modules :
- a "base" module defines the options used for describing the services
- one defines the services themselves
- one handles the reverse proxy configuration for these services (for multiple frontends), as well as the SSL certificates and DNS zones
- one handles the containers themselves (through Docker or LXD)
- one handles the networking overlay
- one handles the "shared services" (databases, caches, etc)
Each of these modules is basically a function that takes the resulting configuration as argument, allowing it to reuse values defined by the other modules, and the configuration itself is the fixpoint of the composition of all of these functions.
It all composes quite well, and the nixpkgs module system allows me to document the options provided by each module. I can even reuse some of the existing NixOS to create a web interface where other admins can search for configuration options and lookup their documentation. Some things are exported in JSON, some are used directly as NixOS modules, some things source the inventory for Ansible playbooks, etc.
It allowed me to create a well-defined source of truth for a very heteregenous architecture, while preserving some of the existing workflows and tooling.
It's not ideal of course : Nix itself is not typed, the nixpkgs module system provides some typing, but the integration is not perfect, and this makes error handling quite tedious. Also, the interpreter is quite slow, there is no documented way to embed it into your own program, and I
I've tried for a couple of days now to set up nix for reproducible iOS builds at work and I'm close to tears.
- Examples that just don't work.
- Commands that run and no output is given, even though the text I'm following says it should output information I need.
- Do I need nix-darwin? Well, the readme explains what it is: "Nix modules for darwin, /etc/nixos/configuration.nix for macOS." What? (Finally I found an answer at Stack overflow, and it seems I don't need it. Seems to be kind of NixOS on top of macOS)
- Can't build with nix-build. After long time searching it seems like you're supposed to run nix-build -E 'with import <nixpkgs> {}; callPackage ./default.nix {}' - shouldn't there be a simpler way for this common case?
- An issue open since 2018 that the manuals simple example doesn't build: https://github.com/NixOS/nix/issues/2259 People chiming in years later that they only could build the example after reading the suggestions in that issue.
- And now I can't start my nix-shell because it says there's a problem with SSL certificates. I'm giving up.
> - Can't build with nix-build. After long time searching it seems like you're supposed to run nix-build -E 'with import <nixpkgs> {}; callPackage ./default.nix {}' - shouldn't there be a simpler way for this common case?
This is because the default.nix you have is exporting a lambda that expects arguments to be saturated, `callPackage` does this for you automagically hence why you have to use it.
Instead you probably want the nixpkgs import line _inside_ default.nix, either in a let-in binding or using the `?` operator to default a lambda argument to something, usually nixpkgs itself bound to a `pkgs` argument so you can use `pkgs`[0].
Even better, you can use Niv to pin nixpkgs to a specific nixpkgs commit so that it doesn't change as you update your system's nixpkgs channel with the `nix-channel` command because `<nixpkgs>` is special syntax referring to what's stored in your $NIX_PATH[1].
You provided what is probably a logical explanation but didn't answer the question in the way he meant it (I think), which is: Addressing common use-cases should be a simple matter. Addressing complex use-cases should ALSO be a simple matter ideally, but is permitted to be a complex matter. Here we have a case where a common use-case is addressed by a complex-seeming solution (unless you are essentially "degreed" in Nix, apparently, and understand exactly why partial function application won't work here or whatever). Or are you saying he's actually doing it wrong and built the nix file wrong?
I'm about to dive headfirst into this nix business and this is intimidating, lol
You're right but just to explain where I came from there were a lot of "just use this default.nix" that had arguments and could not "just be used".
I understood that an argument was needed, that error was clear and when I googled it I found the solution that I wrote.
I supposed it's meant to plug in to something bigger maybe and that's why there are arguments without import everywhere.. maybe? But I've given up now, so I'll never know, we're just gonna suck it up and keep using brew and try to make it idiot-proof somehow.
It just feels like a shame that nix is so unapproachable. It's just as much apple's fault though - if xcode has some open command line tools I'm sure there would have been ready-made formulas for everything we needed. At least I got to vent.
You might want to read "Nix Pills" if you ever want to look into Nix again. It's the best introductory material for Nix that I'm aware of, and it gave me a better picture of the basics required to get comfortable with Nix.
It's on the NixOS website, but IMO it's worth making it more visible to newcomers:
From my experience, it became considerably more easier to figure out or search for solutions once I've grasped the basics from "Nix Pills." Once I became comfortable enough, I also relied heavily on the Nixpkgs repository[1] as a great source of information. It has an approachable codebase and some quick grepping gave me many examples when trying to figure out how to write packages for a particular language.
> You're right but just to explain where I came from there were a lot of "just use this default.nix" that had arguments and could not "just be used".
> I understood that an argument was needed, that error was clear and when I googled it I found the solution that I wrote.
So in this case the advise was worst than useless? Had you been figuring things out on your own, would it have been less bad?
> It's just as much apple's fault though - if xcode has some open command line tools I'm sure there would have been ready-made formulas for everything we needed.
Yes that is too true. Keeping things working with apple is so draining it's hard to have energy left over to do other things right.
> Can't build with nix-build. After long time searching it seems like you're supposed to run nix-build -E 'with import <nixpkgs> {}; callPackage ./default.nix {}' - shouldn't there be a simpler way for this common case?
With other languages, e.g. with Haskell, you can run "stack build" and it will build the project.. or with C, you can run "cc *.c" and it will build an executable with main. -- Whereas, the `default.nix` example given in the tutorial is supposed to be given to callPackage. (The tutorial which provides the example instructs you to modify the all-packages.nix file in a clone of the nixpkgs repository).
...Which is unhelpful with the mindset of "I just want to write a Nix package for my thing", or just weird compared to what you'd expect from "Hello World". (But makes more sense in terms of "how to get started with nixpkgs").
The UX is much better with Nix Flakes. -- With flakes, there's a standard file format where you can run "nix build" and expect it to work.
The problem is that default.nix is like setup.py for Python. It allowed people do all kinds of nonstandard things and custom code. The callPackage pattern was developed later on, and people who use it as some kind of standard, but it was just one more "standard", also cli tools never were update to accommodate is, so that's why the complicated string is used.
The flakes are essentially supposed to standardize it: https://www.tweag.io/blog/2020-05-25-flakes/ (well their goal is to fix other issues, but the standard format of the file fixes that problem too and CLI supports it as well)
I’m an intermediate Nix user - aren’t pkgs usually passed in as an argument so that the caller can choose them? I sort of accepted that verbose command line as a necessary evil of good design.
yes, the nixpkgs changes and some old articles might no longer be accurate :/
> - Commands that run and no output is given, even though the text I'm following says it should output information I need.
could you give an example?
> - Do I need nix-darwin? Well, the readme explains what it is: "Nix modules for darwin, /etc/nixos/configuration.nix for macOS." What? (Finally I found an answer at Stack overflow, and it seems I don't need it. Seems to be kind of NixOS on top of macOS)
yes, you don't need it for what you described. nix-darwin is basically to provide configuration functionality of NixOS to your Mac. Nix-darwin is 3rd party product that's written in Nix for Nix. Similarly Home Manager.
> - Can't build with nix-build. After long time searching it seems like you're supposed to run nix-build -E 'with import <nixpkgs> {}; callPackage ./default.nix {}' - shouldn't there be a simpler way for this common case?
a lot of people wrote about it. This is a convention that evolved from nixpkgs, it's still not officially supported by CLI tool (that's why the command looks so complicated, it's similar to calling `python -c "<whatever>`), seems like the default.nix file which is very similar in concept to setup.py (as you can put whatever you want inside) is being replaced by more stricter flakes.nix file. Which in a way brings some similarities to the callPackage convention.
> - An issue open since 2018 that the manuals simple example doesn't build: https://github.com/NixOS/nix/issues/2259 People chiming in years later that they only could build the example after reading the suggestions in that issue.
Because software evolves and old versions of file like to disappear with time. I run into this as well, although it was nix pills documentation. My solution was to simply get the new version of hello and use that. Although I agree, that's still unnecessarily confusing. I think a proper solution would be to host the file on the own server. Or even better, make a basic hello.c file and list its content in the doc.
> - And now I can't start my nix-shell because it says there's a problem with SSL certificates. I'm giving up.
You most likely uninstalled nss-cacert package. Perhaps `nix-env --rollback` would help.
The Nix hype machine comes up on HN every few weeks.
As someone who tried Nix-- never understood what the big selling point is-- what can Nix do my existing tools can't? (without me spending 6 months)
Nix can create isolated environments, but as far as I see, you can only install the tools Nix repo supports. And the official examples only give toy examples --here's a simple python script you can run with Nix-- but I can just run this simple script in a virtual env, or docker if I have 2-3 different scripts.
What problem is Nix solving? What pain does it fix that is so important it would justify me spending 6 months on it?
And then there is NixOS, which again, I have no idea-- I can create my own Linux distribution that is what, reproducible? Again, is this a real problem anyone faces (and that isnt fixed by docker or ansible or any of the other hundreds of tools)?
My point is: Why is Nix getting so much hype when I dont see them solving any real problems? Sure, they promise a lot, but I have yet to see any real solutions.
- everything from kernel and drivers up is specified declaratively in a uniform, expressive way. You version control your entire infrastructure. Not in the sense of a Dockerfile. The Ansible stuff, the Dockerfile, the build of your library, the build of your own software is in one language that can happily and easily reference each other as a unified system.
- with a few caveats, you are cryptographically guaranteed to get the same result, every time, no matter what. if it worked once, it will never break again.
- it has mechanisms reminiscent of git or something: you can checkout a different computer or set of computers, and if you don’t like the result, revert to what you had before.
- it treats patching of arbitrary software as a completely typical and first-class activity. if anything doesn’t work how you want, you write a patch, check it in, and it works now and always will. upstream at your leisure or not at all.
- you get exactly what you want on your system, no more, no less. no gnome-keyring-daemon bullshit on your headless server Ubuntu 20.04 LTS.
- caching of artifacts both locally and via cache servers run by you or others or both is very granular. no giant slabs of docker composition towering ever higher.
I use Nix in a very simple way - reproducible dev environments. This is very similar to what you might do with Docker - have the right version of the right dev tools. The result is much more user friendly though because it's layered "over the top" of my normal environment rather than being it's own world (but if I want to run things in their own world I can just run nix shell with --pure).
I can then use that same environment to do CI without even creating a docker image, e.g. this for Gitlab CI:
image: nixos/nix:latest
before_script:
- nix-env -f shell.nix -i -A buildInputs
If the amount of stuff is big, or you need to build things rather than get them out of Nix cache, then you can build the docker image using Nix tooling instead.
You can also use Nix to control what is on a machine. E.g. for our lab setup we had a gateway machine behind which was all the test equipment. NixOS was great for having the exact config of that machine checked into version control so we could recreate it. It's far more reliable than Ansible or something like that.
Nothing; even CSS is Turing-complete. It's all a question of how easy, how secure, how reproducible, etc.
> as far as I see, you can only install the tools Nix repo supports
Lol what? Nix literally just executes a given command, with given arguments and given environment variables. This comment is like saying "Makefiles can only build C projects".
Just like Make, if you can build something with some shell commands, then you can build it with Nix. It's often as simple as:
runCommand "my-program" { myEnvVar = myValue; } ''
put your shell commands here
mv my-build-product "$out"
''
('runCommand' tells Nix to run a Bash executable, with the given env vars (including the given commands), and a "default builder" script as argument (that "builder" will eval the given commands))
> Again, is this a real problem anyone faces (and that isnt fixed by docker or ansible or any of the other hundreds of tools)?
Nix has been around for two decades. It's more appropriate to ask what was the point of creating Docker, Ansible, virtualenv, etc. when Nix already existed.
> And then there is NixOS, which again, I have no idea-- I can create my own Linux distribution that is what, reproducible?
Maybe you're getting hung up on terminology; if you would consider "Ubuntu, but with my SSH config file" as "my own Linux distribution", then sure, NixOS creates Linux distributions. If you would call that "configuring a Linux system", then NixOS is for configuring Linux systems.
> Nix has been around for two decades. It's more appropriate to ask what was the point of creating Docker, Ansible, virtualenv, etc. when Nix already existed.
Nix, after two decades, still has an incredibly sharp learning curve, especially to build your own packages. Docker has almost no learning curve, including creating your own images and sharing them with the world.
I disagree. Defining packages is pretty easy, e.g.
with import <nixpkgs> {};
runCommand "my-thing"
{
someSortOfSource = fetchurl {
url = "https://example.com/thing.tar.gz";
sha256 = "0000000000000000000000000000000000000000000000000000000000000000";
};
buildInputs = [ gcc python3 whatever-else-you-like ];
}
''
tar xzf "$someSortOfSource"
cd whateverWasInThatTarball
run whatever commands you like
mv whatever-the-result-should-be "$out"
''
The actual commands can usually be copy/pasted from the README or INSTALL file of whatever project you're "packaging".
> Docker has almost no learning curve, including creating your own images and sharing them with the world.
It took me several attempts to learn Docker, and I'm still very much a newbie with it. To be fair, most of the difficulty I had was due to my misunderstandings; e.g. "surely nobody actually does that?" (e.g. "download entire OS just to run a command"; "ignore all the hashing and just fetch 'latest'"; "run 'apt-get -y update' without having any idea what it's going to do a week from now"; etc.)
Once Docker 'clicked' for me (i.e. I realised that indeed, people are doing that sort of stuff), it became much easier. It's also when I completely lost interest in Docker, and started using OCI containers instead (you can get remarkably far with just 'tar', 'sha256sum' and 'jq'!).
These days my only interaction with Docker is when other people's containers cause an outage at work, due to Docker blowing through all the disk space on our cloud servers. (Fun fact: you can resize AWS disk partitions on-the-fly, but not if they're 100% full, since the partition-editing commands try to allocate temp files)
> Nix has been around for two decades. It's more appropriate to ask what was the point of creating Docker, Ansible, virtualenv, etc. when Nix already existed.
I'd say that the relative popularity of the tools is an indication of how well suited they were for the things that people actually wanted to do.
Nobody (exaggerating here for dramatic effect, but it probably applies to a large part of the unseen 99%: https://www.hanselman.com/blog/dark-matter-developers-the-un...) wants to have fully reproducible builds and a unified language for configuring everything like Nix attempts to do, or even have an OS that attempts to centralize those concepts, like NixOS does because a lot of it is alien to them.
They just want to run their PHP CRUD app in a vaguely predictable way that is less painful than what they've historically done (ship scripts through FTP, experience pain due to php.ini being different, or a system package not being present, or the system package being of a different version). Docker and other solutions like it are the closest that they can get to shipping their machine to prod, given that shipping VMs has more challenges, than just telling their clients or even their ops to just run a Docker container with some configuration.
Nor do they want rootless containers, or distroless containers or Podman or any of the other fancy solutions - they just want their Ubuntu userland and their app to sort of work, most of the time.
We had containers in one way or another long before Docker, for example, FreeBSD jails were pretty good. And yet, they never caught on, nor did FreeBSD outside of large corporations, because for the most part the developer experience just wasn't there. Then came along Docker and changed everything (with plenty of bumps along the way), now building and running your app is pretty simple:
1. nano Dockerfile
2. docker build -t some_image_tag .
3. docker run some_image_tag
(or, you know, push the image somewhere)
Now, the layers system is a mess, permissions management is a mess (hence lots of containers run as root), volumes aren't easily browsable but bind mounts are also problematic, orchestration is all over the place (Compose is nice for a single server but doesn't handle clusters, Swarm is nice for multiple servers but mostly dead, Nomad is weird because of HCL, Kubernetes is nice if it's someone else's problem but there be dragons if you can't just pay a cloud vendor to handle it for you) and it got a large amount of things wrong (e.g. "latest" tags), but it was still good enough and aligned with actually getting things done.
Additionally, most developers don't get paid for ensuring that their application will run in a stable and predictable manner 5 years from now. Most developers get paid for shipping that mediocre piece of software that their boss wants them to before the end of the week and having it work well enough so the company can earn some money, before they're told to move on and do something else the next week. I'm not saying that it's how things should be, but just the way they are, at least for a lot of people who don't tackle interesting problems yet still put food on the table.
That is the "tragedy of good enough" and why Nix, NixOS, jails, FreeBSD or anything else won't be as popular until vaguely low skilled developers can get easily started with them. Maybe things just need to cool down and become more stable, even Kubernetes has gotten less horrible thanks to Rancher/K3s in the last years, maybe in a decade package management and the way OSes are will be more approachable.
> That is the "tragedy of good enough" and why Nix, NixOS, jails, FreeBSD or anything else won't be as popular until vaguely low skilled developers can get easily started with them. Maybe things just need to cool down and become more stable
Right. The Dockerfile format is much less elegant than Nix, but it's easier to get started with, and can be used for most cases.
I don't necessarily see everyone _writing_ Nix. But I think one way Nix can gain in adoption with developers is Nix makes it easy to make dependencies available for working on a project.
I'd also point to git: many developers struggle to use git; and might use it for some time without an intuitive understanding. The network effects of GitHub have somehow been enough to make git a prominent tool, despite it being difficult to use.
I think as more Nix stuff becomes copy-pasteable, there are enough use cases which don't require deep understanding of Nix.
I would say the following for NixOS (which is a distro itself not a way to make a distro):
- I have machine configurations that if say a HDD dies means I can just replace it and be pretty much where I was an hour later from scratch.
- I can update a configuration (even doing the equivalent of a major OS update) without fear as it's possible to just use the old configuration if something is broken.
For Nix in general:
- Can define versioned development environments which aren't awkwardly sandboxed as they are in Docker.
- It's possible to make multi language/ecosystem builds.
- Builds only build what needs building.
- You can build on one machine and ship the build to another as if that machine had built it.
> As someone who tried Nix-- never understood what the big selling point is-- what can Nix do my existing tools can't? (without me spending 6 months)
Right.
The benefits are clearer for DevOps than for developer workstations. (e.g. Nix makes it easy to declare the set of packages installed, with atomic rollbacks).
For developers, I think it's still worth keeping an eye on, since Nix provides a programmatic way of dealing with packages of software; and dealing with packages of software is pretty much what a developer will need to do when driving their system.
e.g. project READMEs typically list some commands for setting up a developer environment; Nix promises a consistent, single command for that (if projects provide Nix files).
For most things that you can do with Nix, there's a quicker + dirtier approach already. e.g. VSCode allows isolated/ephemeral dev environments with remote containers; Docker provides an easy way to be confident software will run a certain way in other system; ZFS/btrfs snapshots allow for rolling back changes; nvm/rvm/asdf allow for project-specific versions of packages; apt-get/brew provide a way of installing packages to a system...
Nix is an elegant and coherent solution to these kinds of problems.
> Nix can create isolated environments, but as far as I see, you can only install the tools Nix repo supports.
Not at all! -- It's comparable to e.g. Docker: With Docker, you can run containers using images served from DockerHub.. but you also have the option of building Docker images and using those.
With Nix, you describe the packages/libraries/tools you want with Nix. This varies from trivially easy, to "have to really understand what you're doing" difficult.
What problem is Debian solving? There was already a poor binary packaging system for GNU/Linux, so why did they create another one? Sure it's easy to switch between similar systems but it is also marginal or useless.
NixOS and Guix present alternative systems that reuse functional developer skills. If you don't have or want those skills then they aren't going to make any more sense than choosing Smalltalk or a lisp machine while being resistant to leveraging their respective paradigms.
From the bottom-up, the core problem Nix solves is that of programmatically expressing package composition, in terms of reproducible builds [0]. For instance, it does away with the common notion of ambient "system libraries", and instead makes it so every single dependency is specified by the build file. If you've ever compiled a compiler and followed the folk advice to do three stages, or did cross compilation and had to juggle build/host/target triplets, Nix takes those patterns and expresses them executably.
Nixos extends this reproducibility up to the level of the OS. Rather than a machine being a giant ball of mutable state, some of it pushed around by configuration management, Nix pulls the entire system state up to a single top value which is set by the bootloader (or similar high-level mutable cell). Like switching to functional programming, at first it just appears different, difficult, and not compelling. But after you start to understand the paradigm, a lot of nice capabilities just fall out, that would otherwise be effectively out of reach.
[0] Not deterministic builds per se, which is an ongoing effort.
I can confidently compile and run multiple incompatible versions of software simultaneously. I can build projects from years ago. I can package large projects from different ecosystems (python 2/3, c/c++, go, Javascript, Octave, Rust) and be confident they will not interfere with each other. I can try bleeding edge software with no risk to it interfering with my system. It is faster and less hassle than juggling various Docker containers and VMs. It protects me from dependency hell. Using Nix to manage an operating system (ie: NixOS) then gives me those benefits for my entire machine. Then I get even more benefits when using Nix on any other (not even necessarily NixOS) machine by pushing around package closures and managing a simple binary cache.
But overall: it makes me more productive. It is my secret weapon to manage the complexity of software development. The benefit is not on day 1, but on day 100 and day 1000.
> Nix can create isolated environments, but as far as I see, you can only install the tools Nix repo supports. And the official examples only give toy examples --here's a simple python script you can run with Nix-- but I can just run this simple script in a virtual env, or docker if I have 2-3 different scripts.
Well, for one, someone send me a random Python script with a dependency on a PostgreSQL driver (that probably needs to compile some C library). Just the script, no `requirements.txt` or `Pipfile.lock` or `pyproject.toml` files declaring the dependencies. I could have just created a virtualenv and tried to install it using pip, that would probably fail because I don't have a proper C compiler setup. Or maybe I have but it is missing headers. Maybe it will work because there is already a wheel for my system, but maybe it will not because some other random reason.
And them `chmod +x ./script.py && ./script.py`, and everything is working.
> What problem is Nix solving? What pain does it fix that is so important it would justify me spending 6 months on it?
Really the meme that you need to spend months just to use Nix properly needs to die. I literally skimped the docs and got a working NixOS install at my first try on the system [1]. The default configuration is really well documented with examples, and I think most people can get what they want quickly.
Eventually I wanted to use something that was not packaged on nixpkgs, or I started to want to upgrade some specific packages without using unstable. At that point it was when I started to think Nix was kinda of painful (not because it was difficult to do, however the documentation is not good and you end up searching a lot until you find some answers). At that point yes, it is kinda a journey, but it is very similar to learning a new language: you eventually can get very proficient at it, but you never stop learning.
> And then there is NixOS, which again, I have no idea-- I can create my own Linux distribution that is what, reproducible? Again, is this a real problem anyone faces (and that isnt fixed by docker or ansible or any of the other hundreds of tools)?
Try to have reproducible system with Ansible. Yeah, I tried. It simply doesn't work well. Ansible is always a pain because steps randomly fail in a new machine, and you can waste hours of debugging for simply issues. Not NixOS: if your configuration builds there is a great chance that everything will work.
NixOS is the only system that I am sure that I have the exactly same configuration as my other systems, as long I am running the same commit from my nix-configs repository. Can't say the same for Ansible: if I setup a machine using Ansible right now and 2 days from now, pretty sure they will have different package versions (since the repository will have changed them), for example.
Also, you don't have rollbacks with Ansible. Yeah, you can have ZFS/Btrfs taking periodic snapshots and use this for rollbacks. This is not the same though. The UX in NixOS is simply so much better (reboot your system and select which generation you want to boot).
> My point is: Why is Nix getting so much hype when I dont see them solving any real problems? Sure, they promise a lot, but I have yet to see any real solutions.
Well, my point is: Nix solves real problems. Maybe you are not bitten as much as I am with them, maybe you think that spend half a hour trying to get a random Python script to work is fine. And I completely concur with you. It doesn't mean that Nix is not worth your time though.
[1]: Now, I think the fact that I used Arch Linux before really helped, since both NixOS and Arch Linux have an installation process that is completely terminal based instead of a proper installer.
Wrt to 1), how often do you reckon a beginner is faced with:
> By default, `args` is a set of derivation names denoting derivations in the active Nix expression. These are realised, and the resulting output paths are installed.
?? - I think the author really hits the nail on the head with this point.
This is how I like to learn most technologies. If e.g. a language does not have proper documentation but rather a bunch of almost-blog-post tutorials, then I have no interest in learning it.
That said, some tech goes too far. You'll never learn C++ by reading the standards.
Written by the same author if you didn't notice. In fact, reading over ianthehenry's blog is an interesting adventure in one mans experience learning this new tool.
IMO, nix-env is a horrible, no-good, very-bad thing. It was also what the manual suggested using for installing packages when I learned nix. I have ditched it on all of my systems and have been so much happier...
but there doesn't seem to be a single agreed upon replacement for it. I wrote my own, I'm sure others have well. It seems home-manager is popular, but it's certainly not even close to universally adopted and I've never gotten around to learning it myself.
Maybe nix-env -irf does the Right Thing; certainly nobody pointed me at it when I was battling against the evil that is nix-env...
I disagree, I used it a number of times. It's practical. Sometimes you just want to install software the old-fashioned way. You can argue that isn't the right way to do things, and anyone running Nix will probably agree, but still I found myself wanting this and it served the purpose well enough.
aidenn0, could you explain what `-irf` does and what it should do? I'm curious now, even though I don't currently use Nix.
> I disagree, I used it a number of times. It's practical. Sometimes you just want to install software the old-fashioned way. You can argue that isn't the right way to do things, and anyone running Nix will probably agree, but still I found myself wanting this and it served the purpose well enough.
The problem is that it lulls you into thinking it serves the purpose well enough and then eventually things break. Then they get more broken. Then you can't figure out which thing you installed is breaking things. Then you finally do and can't figure out how to uninstall the thing that broke things.
> aidenn0, could you explain what `-irf` does and what it should do? I'm curious now, even though I don't currently use Nix.
I'm pretty sure it sets up the environment from a file; I found it one link away from the link in the comment I replied to here[1]
> The problem is that it lulls you into thinking it serves the purpose well enough and then eventually things break. Then they get more broken. Then you can't figure out which thing you installed is breaking things. Then you finally do and can't figure out how to uninstall the thing that broke things.
I'd rather phrase this as: the 'problem' with `nix-env -i` is "I forgot I did that" or "I didn't know I did that".
I believe since the command is the easiest thing to do with Nix, and many people will try it and maybe got back to non-Nix package management, it disproportionately leads to newbies asking for help; thus, the general mood against it.
I think there are two distinct arguments against nix-env:
1. Imperative package management is evil
2. The package name-based interface used by nix-env is evil
I don't ascribe to (1); I quite like installing things "the old-fashioned way." Saying "install this thing" is a very nice, familiar UX. (Even if under the hood you implement that as "add this to my declared list of packages and then rebuild the world," I would still rather run a single command than edit a text file and then run a command most of the time.)
But I think that (2) is true, for the reasons outlined in that post. I have never really heard anyone defending name-based operations; the universal consensus is that it was a bad decision made decades ago. It's something that you can avoid if you are aware of the problem, but it is still, for backwards compatibility, the "default" way that nix-env works. (And although you can install packages the "right" way, you have to resort to name-based operations if you use nix-env's built-in uninstall or upgrade commands, which is also very confusing.)
nix-env -irf is neither (1) imperative nor (2) name-based; I use it as a way to implement my home-rolled declarative package management solution linked in the sibling, taking advantage of the fact that nix-env is okay at building profiles already. The flags literally mean "uninstall everything, then install everything that is listed in this file," but what that translates to is "create a new profile with the contents of this file, disregarding the contents of the current profile."
It's a list of "overlays" (functions of two arguments, which return a key/value set). Overlays make it easy to overrides and replace things.
If you prefer imperative programming, then think of it being used like this:
const result = defaultNixpkgsSet;
for (overlay in overlays) {
// Merge the overlay's return value into the results value
result := result.merge(overlay(
result, // Give each overlay a reference to result
result.copy() // Also give a snapshot of the contents so-far
))
}
return result;
Merging this into the 'result' will replace the 'maven' definition. That new definition is the same as the old one ('snapshot.maven'), but it will be using 'jre8' instead of the default. Two things to note:
- We're using 'result.jre8', which will include any overrides made to 'jre8' (by other overlays, even those being applied after this one!)
- We're using 'snapshot.maven', since using 'result.maven' would cause an infinite loop (since 'result.maven' depends on the output of this override!)
In that link, they're calling their arguments 'final' and 'prev' instead of 'result' and 'snapshot', but the idea's the same. Most overlays call their arguments 'self' and 'super', but I avoided them in this explanation since those names have specific meanings in other languages (they're literally just variable names in Nix).
the result infinite loop issue is easily the most confusing part of nix. It works because nix is a lazy functional language. But I'm not particularly sold on that laziness being useful since it leads to awkward hacks like the two arguments to an overlay function. It has the one benefit of allowing you to not care about the order that overlays get called in. However it does that by giving your code magical come from semantics that make debugging a problem harder than it needs to be. I think the nix lazy semantics would actually have been better left out in favor of clearer dependency semantics.
In a way the clearer dependency semantics of flakes are are driving the reason they are increasing in adoption so much. In essence they work around the problems introduced by overly clever usage of laziness in a nix package.
> But I'm not particularly sold on that laziness being useful since it leads to awkward hacks like the two arguments to an overlay function. It has the one benefit of allowing you to not care about the order that overlays get called in. However it does that by giving your code magical come from semantics that make debugging a problem harder than it needs to be.
I don't consider it an awkward hack; it's a very standard least-fixed-point calculation. Developers these days seem comfortable-enough with 'self' and 'super' in OOP; that fixed-point pattern is essentially the same, but without shoe-horning magic keywords into the language.
I think you could get 99% of the value of an overlay as a simple fold or map over the result. Introducing the second argument feels very much like a workaround to allow you to avoid the infinite loop.
My only experience with Nix was one job, where we hired a new guy who had just done his PHD in Computer Science but never written production code.
Someone on the team was mentioning that our build/deploy pipeline was a pain in the butt, and it would really be nice to have a proper CI process implemented.
PHD spoke up saying he could help with that, and then spent the next three months wrestling with Nix. I left the company before I saw how long it actually took him, though.
I saw enough to realize it would likely never be worth my time to use. Sorry Nix.
Really enjoyed this and feel like this could be an excellent format for learning all kinds of things. I really hope it pushes on to the logical conclusion. Asking a lot, I know.
I don't think of Nix as a package manager; I think of it like Make.
The problem with Make (other than the awful syntax!) is that it's not composable. For example, GCC has a Makefile, with an "install" rule; yet my C project's Makefile can't use that as a dependency; e.g.
hello: hello.o installGCC
gcc -o hello hello.o
hello.o: hello.c installGCC
gcc -c hello.c
# Pseudocode to import the 'install' rule (renaming as installGCC), from the
# Makefile in GCC's source (depends on gccSource to fetch it)
@import: gccSource
import(gcc-10.1.0/Makefile, install -> installGCC)
gccSource:
wget 'http://mirror.0xem.ma/gnu/gcc/gcc-10.1.0/gcc-10.1.0.tar.gz'
tar xzf gcc-10.1.0.tar.gz
Nix solves that problem: we can write a rule (Nix calls them "derivations") which run wget/curl/git/whatever, and we can import files from the result.
Nixpkgs (a big collection of definitions for Nix) takes this to its logical conclusion: C programs not only depend on their source, but also on the compiler, the shell used to run the commands, etc. In turn, the compiler depends on its source, and a shell, and a compiler (if it's bootstrapped). The shell depends on its source, and a compiler, and a shell to run the commands, etc. The downloader (usually curl) depends on its source, and a compiler, and a shell, etc.
The recursion is cut-off by some "bootstrap binaries" (which the bootstrappable-builds project is trying to shrink even further http://bootstrappable.org )
With this "composability" problem solved, it turns out there's actually no need for "package managers" at all; or indeed for "installing" anything: each derivation just says what other derivations it depends on. Nix can act like a package manager if you want it to; e.g. its 'nix-env' command manages a "user profile" (a symlink at ~/.nix-profile, which point to the output of a derivation, which itself symlinks to the output of whatever derivations we want "installed"). NixOS works in a similar way, but with a system-wide /run/current-system symlink (plus a massive pile of options to configure things more easily).
Hence the slight complexity of Nix (in fact, most of the complexity is in the libraries, like Nixpkgs) lets us do away with build tools (Make, Ninja, Ant, etc.), language package managers (maven, pip, cargo, npm, cabal, etc.), OS package managers (apt, yum, etc.), configuration managers (chef, ansible, puppet, etc.). Nix's "binary cache" mechanism also lets us do away with binary formats like rpm, dpkg, wheel, jar, etc. in a transparent manner (e.g. we can still patch sources, override compiler flags, etc.; we just get a cache-miss)
Of course, that's the ideal. There's so much code written with Makefiles, pom.xml files, setup.py files, cargo.toml files, etc. that most Nix definitions just add those tools as dependencies, and let them do the building!
Love the idea of a blog post like this. I tried something very similar a couple weeks ago on a c++ project, both in hopes it might lead to an improved user experience, and also for others struggling to figure it out.
Can someone explain to me why I'd use this over say docker, ansible, or even a make file?
BTW I'm no expert in any of the aforementioned, I've had to use docker, was thinking of learning about make files, and wanted to look at ansible, maybe.
Nix allows to have a "declarative" approach to configuration management, whereas Ansible uses a more "imperative" approach.
For example, applying a Nix configuration is more-or-less guaranteed to be an idempotent operation, while this is not guaranteed at all using Ansible : many Ansible modules are not idempotent at all.
This brings many useful features, for example the ability to rollback to a previous state, or more generally the ability to have some determinism in configuration management of complex systems.
Nix and Docker can be combined together, I have many hosts where some services run in Docker containers, where the Dockerfile is defined in the Nix configuration itself, or generated by composing the upstream Dockerfile with my own configuration values.
> Can someone explain to me why I'd use this over say docker, ansible, or even a make file?
Better reproducability by default.
Docker containers usually have drift from `apt update` and that can let the "works on my machine creep in".
You can avoid this in your own Docker containers, but you can't fix the whole ecosystem using that approach.
The nix ecosystem avoids this drift and other sources of repriducability issues like the plague.
If he is very popular Technologies with very common workflows, Nix might not seem worth it since the sharp edges from the lack of reproducibility have been filed down by brute force.
You might find as soon as you need to do something slightly different, or need to get around a bug right now, you'll want Nix.
Reading this was a bit maddening at times, because, for instance, the author always seemed to give up after 15 seconds of a command executing with not much explanation. A bit meandering.
> for instance, the author always seemed to give up after 15 seconds of a command executing with not much explanation
Because he recreates the experience of an average user who comes into nix.
To quote,
--- start quote ---
nix-env invocations are too slow for anyone to reasonably use Nix as a package manager. Full stop. Every. Single. Command. 30 seconds. It was unbearable. I mean, it was nice for writing this blog post – plenty of time to get my thoughts down – but if I were an actual, normal user? Trying to use Nix? I would just stop. My Nix adventure would end before I finished running the first command. Because I would assume it was broken, ctrl-C outta there, and go back to using Homebrew (or pacman, or apt, or whatever).
> Or you just memorize the “right way to do it,” and every time you type it, a part of you wonders why, until eventually your mind crystallizes around this idea that Nix is confusing and weird and too hard for you to figure out.
This stands true for most of the things in universe for most of the people.
to preface, I'm not saying that you don't know this, but maybe for others:
gentoo is not a linux distro either (from a certain point of view), more a meta-distribution, where you use the package manager (portage) to build everything from source.
fun fact: docker uses alpine linux, which for a while was a gentoo build, until it got to a point where it could bootstrap itself.
> Nix is like a kind of Docker from an alternative universe.
Nix is often compared to Docker, but Docker is essentially just a way to run programs (in a sandbox). Saying "I'll build it with Docker" says how you're going to run it, but not what you're going to run. Unfortunately, the "what" usually turns out to be a shell script, which runs one package manager to fetch another package manager, downloads a bunch of files from some random URLs, etc.
Nix is all about keeping careful track of the dependencies between things, such that all the fetching has been done by the time our scripts start (access to the network, and any paths outside the build sandbox, is disabled by default).
Also, the way Docker tends to be used involves downloading an entire Linux distro, just to run those scripts. Nix does the opposite: scripts only have access to precisely what they've asked for.
But once it starts working: how does anyone do this any other way?
- The language is reasonably well-documented and those docs are rarely wrong.
- Learn bash, like really really learn bash.
- Check out nixpkgs and use that to figure out how to do stuff.
- Only do this if paying the entire cost in time and frustration in one aneurism-inducing spasm upfront is a worthwhile tradeoff for having everything work perfectly every time thereafter.
It’s kinda like being Paul Atriedes: you get magic powers but first you put your hand in the fucking box. What’s in the box?
“Nix!”