Config files and flags should be equivalent and substitutable. There's no data difference between text-at-command-line and text-at-command-time.

At my current job I wrote a simple Python class to abstract settings being read from either command line args, environment variables or a TOML-formatted config file. The class first reads settings from the config file if it is available, then environment variables if found, and finally from the command line args if set by the user. The last context to provide a setting value is treated as the actual setting to propagate to the app. If desired the app can then save its current settings state back to a file, either the original config or to a new config file.

I arrived at this solution after getting tired of both having to track down where the value for a particular setting originated at runtime, as well as having multiple implementations of the same kinds of argparse/os.environ/file-based settings systems across our internal tools and scripts.

I just put configuration settings in config.py and import them.

Problem solved.

Plus I have the full programming language available, not some textual subset with Yet Another Dependency to read it.

Agreed. Tried to reconcile this for some of my projects a while back and spent much more time I anticipated. This is an unexpected rabbit hole, between type casting, scoping of flags to subcommands and flags > configuration file > environment variables > default values precedence. And TOML / YAML / JSON / XML / INI multi-format support.

I made my hacks open-source, so if you have a Click-based Python CLI, try click-extra, contribute back, and help me solves this problem once and for all: https://kdeldycke.github.io/click-extra/config.html

What about when an option is present in both the file and the command line? Which one takes precedent? If an option occurs 2x on the command line typically the last one wins. If an option appears 2x in the file does the last one win too?

With restraint and thought you can make the 2 equivalent from a data perspective. That's a ton of effort and if you get it wrong it is pretty hard to change it later on because someone is going to start relying on previous semantics.

This problem has been solved for a long time now: http://www.catb.org/~esr/writings/taoup/html/ch10s02.html

Multiple decades of unix software configuration and pointless differences between distro configuration would seem to indicate otherwise.

Esr even contradicts himself at the end of that link when he says that site wide configs may be immutably set in the system directory.

I believe that the problem has been solved, it has just been poorly implemented in some cases. The general gist that more local configs takes precedence over more global configs is a simple and powerful mechanism which has worked. Yes, distros can be a mess where there are five different config directories but I believe that specific problem is orthogonal to how we solve config precedence.

It is even poorly implemented in such ubiquitous tools as ls. For example if I always want output to be sorted by last modified the _only_ point of configuration is the command line. But I can control colors via either the command line or an environment variable.

I've lost count of the numbers of times I've straced commands to see where they pull configuration from or grepped thru codebases looking for getenv calls. Any unix consensus for configuration is in theory only not practice.

File location is actually pretty germane to precedence. A more "global" configuration cannot be applied if the program is looking for it in the wrong location in the first place. Let's not even discuss the horrible state of where to store local configurations in a users home directory. As long as it starts with a . it's fine :).

I've used Linux exclusively for nearly 20 years - but not because I loved its configuration management story.

ls is a very poor example as it was a program built in v1 Unix over 50 years ago which has remained the same due to backwards compatibility and being standardized in POSIX.

We are arguing over whether the glass is half empty or half full. I can also come up with plenty of examples of poorly designed programs, but I can also bring up examples of programs which are not perfect but much more usable than ls. I’m sure you could have also picked a program which is better designed than ls as well.

Do you have any examples that aren’t confusing and super complicated?

Kind of related but not: if you are interested in cli design I recommend this guide as well: https://clig.dev/

I think people are overly focused on how crappy Unix tools from the previous millennia are (ls, sendmail, dd, etc…) but I do truly believe that cli tools can be ergonomic because I have used and built tools which feel ergonomic.

I think the aws cli is a decent example. It isn’t perfect by any means but it demonstrates how to handle config files, environment variables, and command line arguments in a usable way: https://docs.aws.amazon.com/cli/latest/userguide/cli-configu...

In fact, some programs (mpv comes to mind) make them literally the same, with the config file as a text file containing command line options in the exact same format that you would provide from a shell, with the only difference being that newlines are permitted noop characters.

Command line flags are miserably at representing anything beyond key=value. A config file will easily handle a hierarchical tree 5 levels deep.

These are solvable problems, no?

Are they worth solving though?

Flags at command line time can be equivalent to flags in file from a data perspective.

If you work on it they can behave with similar semantics too.

If you work even harder you can ensure you deploy the config file atomically with the binary. You can also make sure that the file is properly secured, and config values from it are observable and auditable.

Or you can just mandate that programs are configured by the command line flags passed to them and not have to do all that extra work.

IMO, the advantage of flags is in the fact that a human can copy/paste the command line invocation and get all they need to reproduce an execution. That’s great.

But when the command line invocation is so big that you can barely even copy/paste it any more, and it exceeds the max command line length on a stock Linux kernel, you’re way past the point of convenience being the important factor. All the normal places you’d want to see the flags (set +e in a bash script echoing them back to you, etc) are suddenly overloaded with megabytes of text for a command line invocation… who’s gonna look at all that? Who is this serving any more?

“Flags for everything” would seem like a good idea taken way too far to me.