> I don’t know why I don’t hear this point get more airtime among people when discussing the pros and cons of Python/Perl as scripting languages.
But it is.
Perl and awk one liners are very common. Bash is used everywhere you have the (controversial) `sudo curl .. | bash -` install method.
Ansible, if I'm not mistaken, requires python3 installed on remote hosts.
If your environments are homogeneous enough, and you don't have a large matrix of operating systems, you should definitely use this advantage.
The troubles start when you have very heterogeneous environment, custom or hardened Linux distros and etc.
It might seem like this is not being utilised as much nowadays, but I suspect it's because the "classical" Linux sysadmin roles are rare and you don't hear about it as much.
> Ansible, if I'm not mistaken, requires python3 installed on remote hosts.
It is not a requirement. There is the raw module that just sends commands over ssh . But if you want to do anything beyond basic most people use raw just to install python so they can use other modules which require python.
Having recently started to experiment a lot with the BSDs (mostly Free, a little Open), which often don't come with even Bash out of the box (usually still Perl though), I can definitely relate to this. The more complex your workplace's OS ecosystem, the harder it gets to write anything which will truly run everywhere, unless you go all the way down to POSIX shell.
I assume everyone already knows that, but shellcheck is great help if you need to go all the way down to POSIX since it will detect /bin/sh shebang and warn about any non-compatible bash-ism.
Where does the expectation come from that Bash should come ootb with BSD? Or anywhere else outside of mainstream Linux distributions with GNU userland? Bash is the standard Bourne/Korn-like shell for the GNU project, which was envisioned as a FSF Unix clone/replacement. BSD and other Unix-like systems pretty much always had their own separate userland, with often different syntax than the GNU tools.
I think the odd one out was really Apple, which ran Bash as the standard shell for many years (a choice they probably regret), while most the rest of the userland was largely from BSD.
No one says it's an expectation or an obligation. It's a matter of fact: there's no bash and it is obviously inconvenient for people used to bash, that's it.
> Knowing that a language is installed by default on the most popular Linux distribution can simplify certain concerns considerably
The author laments the decline of such knowledge, but the proliferation of containers, combined with the marketing efforts of the large cloud providers has reduced the number of people who work with servers directly.
I should say that this knowledge can be useful in certain niche situations, though. For example, I recently learned that GitHub Actions pulls in a basic set of utilities by using complicated distro detection[1], and Gitlab runners similarly pulls in a Docker image with git just to clone a repository!
Instead, the proper way to address this situation would be to compile static binaries, that, by their very nature, become distribution agnostic. The only reason I assume this wasn't pursued, I assume, would be that the developers working on this weren't even aware this was an option.
It's the reason I've been maintaining a small set of packages[2] that I mount into every CI container for my personal projects. I'm also quite hopeful about the cosmopolitan project[3] that builds fat binaries which can bridge this gap, although as I discovered, some containers even lack tools like gzip (which cosmopolitan depends on), so maybe static binaries it is.
> Instead, the proper way to address this situation would be to compile static binaries, that, by their very nature, become distribution agnostic. The only reason I assume this wasn't pursued, I assume, would be that the developers working on this weren't even aware this was an option.
Nowadays you even have younger developers praising static linking, as if this wasn't what we had around since the invention of the first linker, and dynamic linking only became mainstream in the 1990's.
Naturally only knowing Linux, GCC and glibc doesn't help.
> Nowadays you even have younger developers praising static linking, as if this wasn't what we had around since the invention of the first linker, and dynamic linking only became mainstream in the 1990's.
> Naturally only knowing Linux, GCC and glibc doesn't help.
It's amazing how low-brow condescension passes for wisdom. Let me give you a hint: ontogeny does not actually recapitulate phylogeny. So just because you have some vague memories of previous iterations of a technology does not mean that it's the same thing being discussed today
I'm not too young and I saw various methods of deploying code in production: deb packages, tarballs, docker images, static binaires.
I have to say that in my experience for a specific use case of a highload c++ daemon, nothing beats the simplicity and reliability of the static binary with musl. You only want to deploy one thing, you have nothing to share, furthermore you wouldn't want to share anything for the sake of control. So many things to worry about and to waste time on just disappear. When things are on fire having less variables to check is gold.
> Instead, the proper way to address this situation would be to compile static binaries, that, by their very nature, become distribution agnostic.
Thats only true if you do not depend on glibc, which some statically compiled binaries still do. And even then, you’d have to rule out anything that requires newer sysalls or platforms that don’t offer stable ABI guarantees (Linux is actually the edge case here in that most kernels offer no guarantees).
The glibc dependency can usually be eliminated through the same process of static compilation; the binaries I linked are also built this way with no dependencies on glibc, for example.
> platforms that don’t offer stable ABI guarantees
If we're talking about Windows, the Win32 ABI is considered stable, which would also suffice for what I'm looking for, which are portable binaries that can be run on almost any version of a system.
statically linking against glibc does break some things (it's not an officially supported option because of that), though musl is an option and for most apps a reasonable replacement.
At work we have this C++ that needs to be distributed as, in essence, a shared lib, and that needs to work with a lot of variability.
So it has C++ and so needs a C++ stdlib. Dynamically linking is annoying so what if we statically link it? Oh but GNU's C++ stdlib wants glibc and has versioned BS and so even statically linking the C++ bits isn't ideal. And as mentioned statically linking glibc is a big no no.
So:
a) build a musl and install it in a prefix
b) add a few bits (some headers and C runtime objects) and turn it into a sysroot
c) build llvm's libc++ libunwind and a couple others, linking against that sysroot, and install it in the sysroot
d) build our shared lib, statically linking against that sysroot
The result is a shared lib that is self-contained except for the C stdlib, and any reference to the C stdlib has no GNUism nor glibc version. The trick is that musl is essentially a subset of glibc + there is no such thing as a musl-ism per musl's design tenets + the C ABI is stable enough. I don't recall us statically linking to musl - IIRC the trick is just to not have any GNUism at any level, in a way it also working in musl is kind of a happy side effect - but maybe I'm wrong.
As a consequence it works on all glibc versions and musl as well, and is entirely independent of the system's preferred C++ stdlib.
A requirement though is to load the shared library with RTLD_LOCAL (to not accidentally make internal symbols available to the outside) and RTLD_DEEPBIND (to not accidentally pick external symbols instead of internal ones)
> The glibc dependency can usually be eliminated through the same process of static compilation; the binaries I linked are also built this way with no dependencies on glibc, for example.
Cosmopolitan is an extreme outlier because it deliberately replaces the need for glibc. Your typical C or C++ program isn’t going to do that.
> If we're talking about Windows, the Win32 ABI is considered stable,
We are talking about most Unixes. macOS, OpenBSD, etc.
Replacing glibc with musl is usually unproblematic. That's what I did for my own portable toolbox which builds to a single self-extracting binary; https://github.com/9001/lxc
Most operating systems don’t provide guarantees about ABI stability between versions, instead mandating that libc (or other user space APIs) are the intended interface to syscalls.
Because if this, you can often come across statically compiled executables will still depend on glibc while inlining stuff like libpng.
> if you depend on glibc... things can still break, e.g. when compiled on too new glibc it fails to run on older versions.
Setting up a machine or build system for older glibc is not trivial . If you know of any DISTRO which allows you to downgrade glibc out of the box please let me know. Keep in mind that if you go back enough you will also have trouble with kernel headers and compiler interactions. Even in yocto which is a build system that builds custom distributions from scratch does not have a great experience for different glibc versions than what is shipped.
Have a look at the nightmare recipes for libc. On the other hand they do provide the basics of how to ship a custom libc loader across heterogeneous machines but it gets complicated fast and not something any non specialist can do in an hour.
In my experience anybody saying to compile statically does not know about the libc limitations. When they find them they hand wave, I smile and say be my guest.
By the way 2.17 is an appropriate version to target. RHEL6 much? :D
RHEL 7 (Centos 7) still has 5 days before end of support. You can get recent gcc with official developer environment update to deal with new code. Whoosh, 10 years of API compatibility suitable even for corporate environments.
Of course, new code may actually need features from newer libc, and often you won't be able to frankenstein your way out of it with no-op plug functions and copypasting successfully.
There's also a random page on the internet with a barebones cross-distribution compatibility guide:
You don't need a distro with ancient glibc; you can instead create a cross-compilation toolchain.
We used https://crosstool-ng.github.io/docs/ to create a cross-compilation toolchain that runs on modern debian, but produces executables that only need glibc 2.17.
The only downside is that you no longer can simply apt-get a library you need -- all dependencies shipped along with your application also need to be built with this cross-compiler.
> Setting up a machine or build system for older glibc is not trivial .
God, I hate glibc so much. It’s an absolutely terrible design from the 80s. It can and should be trivial to target any arbitrary version of glibc. The way Linux performs linking is so bad and so much worse than Windows. (Headers + thin import lib. Linking expecting a full and complete so is stupid and bad.)
Zig solved this problem. It can trivially target any version of glibc and cross-compile. It’d be nice if the Linux community demanded this level of functionality from their toolchains. Alas. https://andrewkelley.me/post/zig-cc-powerful-drop-in-replace...
Linking does not expect a full and complete so. That's what it's currently fed, but it's not a requirement. You could just as well have "import libs" in ELF too.
Neither GCC nor Clang produce "import libs" on Linux. The entire ecosystem is built around linking whatever full .so happen to be in the global search path.
What Zig does is goes way way way out of it's way to compile import libs which contains all of the exported functions but the implementations are empty.
So you aren't wrong. But the point is that the monumentally large Linux ecosystem does not work that way. Which is a real shame.
From the linker viewpoint glibc is four files on disk. You can have them in ~/.local/glibc FWIW, linker doesn't care. I imagine it should be enough to download libc-dev package and unpack it in ~/.local/glibc
The actual dependency is 2.14, when a faster memcpy was added, 2.17 is just whatever I happened to have around back when I compiled my stuff, I just didn't change anything since then, bits don't rot.
I don't think Nix solves this use case, the linked binaries would still have dependencies stored at /nix/store/... and is hardly a solution for obtaining binaries that work anywhere.
It's not about obtaining binaries that work anywhere, it's about being able to ask for a predictable environment to run in. You don't have to care about the target environment if you can control the derivation that is built to run your program.
I think we also need to separate "Nix" from "The General Approach that is implemented by Nix (and Guix etc.)". I think the "The Nix Approach" (as I'm now going to call it) is fantastic and really is the solution to a lot of problems that we have. I think that the actual implementation of Nix isn't great and painful to use in real life.
> You don't have to care about the target environment if you can control the derivation that is built to run your program.
This is not always possible, since people may have workflows based on different containers or OSes. I (and the author of the article) want to have tools that work across different environments.
> The General Approach that is implemented by Nix (and Guix etc.)
"Reproducible builds" or "hermetic environments" are commonly used adjectives for this, although they are not limited to these two tools either; Bazel from Google does it too, for example.
I think Snap/Flatpak could probably be extended to cover most of the nix use cases.
But some kind of repeatable environment is needed, traditional package management is just pure insanity where everything only works on one specific version.
I don't really see anything broken about them besides the lack of a FOSS snap backend, and the fact that the security isolation still breaks apps sometimes.
They don't have very good deduplication but that's a fixable issue.
What's the problem with storing dependencies in /nix/store? Nix removes the downside of having dependencies. Manually finding and installing them, conflicting dependencies, multiple versions of the same software/libraries, dependencies that are no longer needed but you don't know it, different installation method and instructions for different platforms, temporary installation - all are solved problems with Nix. Every package in nixpkgs, world's largest software collection, is a `nix-shell -p <package-name>` away.
Dependency may not be available on target system and cannot be installed because:
1) no external network access
2) not enough privileges to install
3) different architecture that doesn't have nix dependency available
4) target does not / cannot trust nix external dependencies
There is no such thing as a "nix dependency", Nix builds everything from scratch. You yourself control the whole software supply chain.
[It's still not a completely fixed problem - Nix needs to finish their content-addressable scheme, and for that you need reproducible builds; this is a detail, though.]
This is perfectly OK for your own work, but if you want to distribute stuff to others, statically linking increases your workload a lot. Because you are essentially distributing an increased number of components and your obligations increase multi-fold.
I totally forgot about `bc`! That should certainly count, I remember it having way more under the hood than a "mere" calculator program normally would.
This is the reason why I mastered enough vi to be dangerous, back in the day, Emacs / XEmacs were usually an additional package, while any sort of commercial UNIX had vi on its default install.
Being sent down into the field, rescuing whatever random server used by a customer, meant being able to use whatever was there, and between ed and vi, better vi.
That's probably only true on a full install. Something like Gnome or KDE will obviously drag in Python. But a minimal install with something like fluxbox would not
Curiously, I get why Go doesnt include the source code in its binaries — presumably a hangover from the long standing tradition of source for me but not for thee, as well as sever constraints on resources meaning source-included would be a frivolity — but are there compiled languages where it is an option to include the source?
The only one I can think of is TypeScript where it builds source maps to help debug the underlying JavaScript without having to see it.
Had the same thought recently. But it would be easy enough to make a "go script" frontend that checks for a compiled version, checks the source against a hash stored with that compiled version and shortcuts to the binary if everything matches, (re)creating it if not.
My opinion is that source code should be packaged up with debug symbols. I'm also a fan of breaking out debug symbols from the exe although that's a separate topic.
macOS and Linux too though? As far as I'm aware, most Linux distros need the build-esssentials (or equivalent) package installed, and on macOS the Apple Clang toolchain also isn't pre-installed. Both is quite similar to installing the "MSVC Build Tools" (MSVC command line toolchain without the Visual Studio IDE).
Unfortunately the Python 3 debacle kind of ruined standalone Python scripts. At least for a while. Have distros gone back to having "/usr/bin/python" available (as opposed to "python3" or something else)? I bet there's still loads of system in the wild without a "/usr/bin/python", though.
Scripts should be using `/usr/bin/env python3` anyway, so they work inside virtualenvs and other scenarios where `/usr/bin/python` isn't the Python you want.
What's wrong with explicitly using "python3"? I guess maybe in theory you could maybe write a script that was compatible with both 2 and 3 but that sounds needlessly masochistic, and if there were ever going to be a python4 you wouldn't want to accidentally switch to it automatically either.
(This page will briefly be down, as I add my new X profile at https://x.com/hiAndrewQuinn to its frontpage. Give it maybe 2-3 minutes from the time of this comment. God bless Hugo for being so fast to recompile. EDIT: Nevermind it was more like 30 seconds!)
Python's standard library covers most scripting use cases, even HTTP networkingand it's stable you can write script and forget. No need to "control" stuff. For example I am using minimal python script + cron for updating letsencrypt certificates in misc linux distros. Everywhere it's system python out of the box without any additional dependencies or virtualenvs.
At least on macOS, Apple strongly discourages depending on the builtin scripting languages, and AFAIK python is no longer preinstalled (instead installing Xcode for the first time will slip a python3.9.x and pip3 into your /usr/bin).
Of course you can depend on zsh/bash, but then the Apple UNIX tools are BSD flavoured, not GNU flavoured like on Linux, so Linux shell scripts will also quite often not work out of the box on macOS.
That's why I learned REXX in ye-olde days, because it came installed as default on IBM systems (and not Perl, which was my de-facto prior to). Then python. Nowadays Go(lang).
Still trying to decide if it's worth going to Rust for script'ish type work, I have a feeling it's not quite the right choice.
Python versioning is BS enough and with enough systems problematic enough that for scripts that have to assume widest range? I torture them with POSIX sh.
Yes and no. In a way a system interpreter is just like a compiler. It's there to enable the user to run programs written in a particular language.
The trouble usually starts when a program needs some third party library. So maybe the rule is more like: the system libraries are for the system, not for you. Unless, of course, they are -dev libraries. I do believe modern Ubuntu won't even let users pip install stuff, which should improve things (Gentoo has been configured like this for years).
The post is about out of the box functionality. Requiring a user to also install some system package kinda defeats the purpose here. Plus it's usually not the same version of the library that was expected by the developer of the script.
Even used in a venv pip creates a mess as people start creating ad hoc environments and don't really know how to recreate them (cause of many "works for me" type problems). I always advise people to take the extra step of adding a new dependency to pyproject.toml or requirements.txt and then doing `pip install -e .` or `pip install -r requirements.txt`.
But it is.
Perl and awk one liners are very common. Bash is used everywhere you have the (controversial) `sudo curl .. | bash -` install method.
Ansible, if I'm not mistaken, requires python3 installed on remote hosts.
If your environments are homogeneous enough, and you don't have a large matrix of operating systems, you should definitely use this advantage.
The troubles start when you have very heterogeneous environment, custom or hardened Linux distros and etc.
It might seem like this is not being utilised as much nowadays, but I suspect it's because the "classical" Linux sysadmin roles are rare and you don't hear about it as much.