A follow-up recommendation I give (which I suspect might be unpopular with many around here) is to use Python for all but the most trivial one-liner scripts, instead of shell.
In my experience most engineers don't have deep fluency with Unix tools, so as soon as you start doing things like `if` branches in shell, it gets hard for many to follow.
The equivalent Python for a script is seldom harder to understand, and as soon as you start doing any nontrivial logic it is (in my experience) always easier to understand.
For example:
subprocess.run("exit 1", shell=True, check=True)
Traceback (most recent call last):
...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status
Combine this with `docopt` and you can very quickly and easily write helptext/arg parsing wrappers for your scripts; e.g.
"""Create a backup from a Google Cloud SQL instance, storing the backup in a
Storage bucket.
Usage: db_backup.py INSTANCE
"""
if __name__ == '__main__':
args = docopt.docopt(__doc__)
make_backup(instance=args['INSTANCE'])
Which to my eyes is much easier to grok than the equivalent bash for providing help text and requiring args.
There's an argument to be made that "shell is more universal", but my claim here is that this is actually false, and simple Python is going to be more widely-understood and less error-prone these days.
As someone who hasn't used Python in ages, and finds it quite unreadable, I would be saddened and dismayed if instead of shell (which is the lingua franca) a project used Python everywhere.
The exception being if it's a Python app in the first place, then it's fine since it can be assumed you need to be familiar with Python to hack on it anyway. For example I write a lot of Ruby scripts to include with Sinatra and Rails apps.
For a general purpose approach however, everybody should learn basic shell. It's not that hard and it is universal.
It's not a matter of familiarity with linux shell. It's a matter of wasting time debugging/implementing stuff in shell that is trivial to implement in any modern scripting language.
Haven't parsed XML in quite a number of years but I parse JSON to extract values from a curl command all the time with jq and it's really not that bad. If there's anything more than simple extraction required then I agree it's not good for the shell. For that I turn to the primary language of the project preferably (which these days is Elixir and sometimes Ruby for me).
But even JSON and (X|HT)ML you'd be amazed how often a simple grep can snag what you want, and regular expressions are also mostly universal so nearly everybody can read them.
We (2-person team doing embedded QT app on linux) did it the exact way you're arguing for - starting with shell, struggling with it for several months every time something went wrong or we had to change it, and finally giving up and rewriting it in python.
Python wasn't our main language (the app was in C++ and QML which is basically java script). But we both knew Python and it's the easiest language for these sort of things that we both knew and that comes with the system.
My main conclussion is - I'm starting with python the next time.
Life's too short to check for the number of arguments in each function.
I think there is a big difference between this and the parent. If you are working on projects where the "primary language" is Ruby or Python or Elixr than sure, use that, but if your project's primary language is C++, like most embedded applications, you do NOT want to use that.
Any complex or cross-platform C++ projects will need a scripting language in addition to shell and a build system.
Fortunately for me a 10 whole lines of sed, awk and some trimming and a for loop turned my json into a CSV to pass back to a legacy system.
No curl or json modules necessary.
That being said, if I needed anything more complex Python would be my default.
I can understand finding the Python unreadable but where I disagree is that "everybody should learn basic shell" -- no it's not that hard, yes it's universal, but there's more to finding a sustainable solution than that. Try to pivot that shell into something slightly more sophisticated and you run into muddy waters.
I really like Ruby as a replacement shell language because its syntax for shelling out is very succinct and elegant. Python is a bit more verbose but a lot more explicit. Either one is a step function better than shell. You get a real programming language.
Let's extend your analogy to a logical conclusion. Should you write something in a shell because it's "not that hard and universal" or should you insist on using a programming language that lends itself to writing maintainably? If not, we should have no problems with PHP and COBOL, no? But we do.
Use the right tool for the right job. If you're not sure what that is, don't hesitate to pull out a glue language. Python, Ruby, JS -- whatever you need to get the job done. Your shell should be just that -- a shell, not the core.
Python is far more of a lingua franca than shell is. Learning basic Python is a much better use of your time than learning basic shell - it's easier and more widely useful.
Maybe shell makes sense to you, but I recently wrote my first real shell script and it was nothing but pain. Brackets don't mean what you think they mean. 'If' does not work sometimes? and I am not sure why. Sometimes I am supposed to quote variables but not always?
And after all that it broke once it moved to a AIX machine since it was not POSIX compliant.
Maybe it's obvious to you, but when I cannot even figure out if an if statement will work, something is horribly wrong.
I don't even use python and I can just about guarantee it's easier for someone who knows neither.
I'm +1 about the use of Python, but I'd say ONLY when complexity gets higher than simply calling commands (e.g. cat/sed/etc). For basic commands, I'd say Makefile is the way to go, or scripts in a package.json. I find Python becomes a must when you want to reuse commands and pass data around, or support the same kind of commands locally and on remote hosts.
Fabric is the best for this: you put a `fabfile.py` (name inspired by Makefile) in the project root and add @task commands in there. Specifically the `fab-classic` fork of the Fabric project https://github.com/ploxiln/fab-classic#usage-introduction (the mainline Fabric changed it's API significantly, see `invoke` package).
Fabric can run local or remote commands (via ssh). Each task is self-documenting and the best part it that it's super easy to read... almost like literate programming (as opposed to more magical tools like ansible).
Agreed on this point, it's the `public void static main` of Python.
Perhaps I shouldn't have tried to make two points in one post; I wouldn't advocate using docopt or a main function for a simple script, I was more making the case for how easy it is to add proper parameter parsing when you need it; that is easier to remember than what you'd end up writing in bash, which is something like:
PARAMS=""
while (( "$#" )); do
case "$1" in
-a|--my-boolean-flag)
MY_FLAG=0
shift
;;
-b|--my-flag-with-argument)
if [ -n "$2" ] && [ ${2:0:1} != "-" ]; then
MY_FLAG_ARG=$2
shift 2
else
echo "Error: Argument for $1 is missing" >&2
exit 1
fi
;;
-*|--*=) # unsupported flags
echo "Error: Unsupported flag $1" >&2
exit 1
;;
*) # preserve positional arguments
PARAMS="$PARAMS $1"
shift
;;
esac
done
# set positional arguments in their proper place
eval set -- "$PARAMS"
I think you've got to write a lot of bash before you can remember how to write `while (( "$#" )); do` off the top of your head; the double-brackets and [ vs ( are particularly error-prone pieces of syntax.
while getopts "ab:" OPTC; do
case "$OPTC" in
a)
MY_FLAG=0
;;
b)
MY_FLAG_ARG="$OPTARG"
;;
*)
# shell already printed diagnostic to stderr
exit 1
;;
esac
done
shift $((OPTIND - 1))
Yes, I realize it's more difficult to support long options (though not that difficult), but the best tool for the job will rarely check all the boxes. Anyhow, the arguments for long options are weakest in the case of simple shell scripts. (Above code doesn't handle mixed arguments either but GNU-style permuted arguments are evil. But unlike the Bash example the above code does support bundling, which is something I and many Unix users would expect to always be supported.)
Also, I realize there's a ton of Bash code that looks exactly like you wrote. But that's on Google--in promoting Bash to the exclusion of learning good shell programming style they've created an army of Bash zombies who try to write Bash code like they would Python or JavaScript code, with predictable results.
But the point of the article is not to make complicated scripts readable again, but to put even simpler commands (maybe even one long mysqldump command) into very short scripts.
I agree that a cli framework is often easier to use than bash, but it also is a dependency. I think everyone should use what he’s familiar with.
> No language has a simpler api to executing a cli command than bash itself. By definition.
Things that are true "by definition" are not usually useful.
"No language has a simpler api [...] [b]y definition" only if by "executing a cli command" we mean literally interpreting bash (or passing the string through to a bash process). But that's never the terminal[0] goal.
The useful question is whether, for the task you might want to achieve, for which you might usually reach for the shell, is there in fact a simpler way.
It may very well be that the answer is "no", but support for that is not "by definition".
[0]: Edited to add: ugh. Believe it or not, this was not intended.
Every project I work on uses the `invoke`[1] Python package to create a project CLI. It's super powerful, comes with batteries included, and allows for some better code sharing and compartmentalization than a bunch of bash scripts.
Thanks for the recommendation, I used Fabric for local scripting ages ago and haven’t revisited in some time.
Is it mostly the Make-style task decorators for managing the project task list that you consider a win vs. native subprocess.run? Are there other features that you find valuable too?
(Back when the original fabric was written subprocess was much less friendly to work with, of course).
I remember the memfault blog post that introduced me to this package. Great find, and I’ve started doing the same on all my embedded dev projects. Thanks!
# Option 1 (no pipes)
import os
for line in open("/etc/passwd").read().splitlines():
fields = line.split(":")
if fields[0] == os.getlogin():
print(fields[-1])
# Option 2 (Cheating, but not really. For most problems worth solving, there exists a library to do the work with little code.)
import os
print(os.environ['SHELL'])
# Option 3 (pipes, not sure if the utf-8 thing can be done nicer somehow)
import subprocess
username = subprocess.check_output("whoami", encoding="utf-8").rstrip()
p1 = subprocess.Popen(["grep", "^" + username, "/etc/passwd"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["awk", "-F", ":", "{print $NF}"], stdin=p1.stdout, stdout=subprocess.PIPE)
print(p2.communicate()[0].decode("utf-8"))
Thanks, that does answer my question quite thoroughly:) I intended the question not as "how would I get my shell" but as a "how would I pipe commands together", which now that I think about it is perhaps the real issue: I think about solving many problems from the perspective of "how would I do this in /bin/sh?", but perhaps the real answer is that if you're doing it in Python (or whatever) then you should be writing a solution that's idiomatic in that language. Or if you like, perhaps one doesn't need to "standard library" of coreutils if one has the Python standard library, which means that many of the thing's I'd miss in Python are hard because they aren't the right solution there.
:) I agree that that works, of course, but if we have to use shell=True and just shove the whole thing like that then it loses a rather lot of the "Python" appeal. Still valuable if you need to feed the output into Python, of course.
The answer to" how I pipe shell commands in Python" is that you can do it (e.g., using plumbum, fabric, pexpect, subprocess), but you shouldn't in most cases.
In bash, you have to use commands for most things. In Python, you don't.
For example, `curl ... | jq ...` shell pipeline would be converted into requests/json API in Python.
https://news.ycombinator.com/item?id=24558719 does it in 5, which is, I think, proof that you're both right. (Of course, perhaps shell is cheating, since I chained 3 commands into that one-liner, and 2 of those are really their own languages with further subcommands.)
One nice thing about it is the overloading of pipe operator to allow easy creation of pipelines and/or redirection (plus it is cross platform, so you don't depend on the system having a certain shell).
>In my experience most engineers don't have deep fluency with Unix tools, so as soon as you start doing things like `if` branches in shell, it gets hard for many to follow.
Shell is definitely not universal on Windows machines. If you work with mechanical engineers, at least half of them will be using Windows. That being said, Python is gross on Windows, too.
subprocess is far to complicated for casual levels of scripting. We have a handful python-devs here, and everyone is avoiding it for scripts, because it's just so poor designend for this job.
Shell is simple and straight forward, and can bring you quite far for most stuff. Only if you start using complex datastructures it makes more sense to use a mature language like python.
This kind of conversation is never productive in a team setting. The strongest criteria is "what would the majority of people on the team prefer to use?". That'll yield the highest productivity, regardless of whatever syntactic sugar it has.
That might just come down to the usual differences between Python and Ruby. A Python programmer would likely appreciate an explicit call to `subprocess.run`.
As an experienced Ruby programmer I imagine the code you provided looks like a no-brainer. To a Ruby neophyte the backticks, $? sigil, and .to_i method don't strike me as intuitive (but maybe they would be do someone else). We may just have to disagree about the nicety of syntax.
As cle mentioned[1], the strongest criteria is
> what would the majority of people on the team prefer to use?
and I'd agree that ultimately that's what makes the most sense.
The back-ticks and $? are shell standards so I would expect those to be familiar to shell scripters. I agree that if one knows nothing about any of the languages then neither is much better.
Since 3.5 added `subprocess.run` (https://docs.python.org/3/library/subprocess.html#subprocess...) it's really easy to write CLI-style scripts in Python.
In my experience most engineers don't have deep fluency with Unix tools, so as soon as you start doing things like `if` branches in shell, it gets hard for many to follow.
The equivalent Python for a script is seldom harder to understand, and as soon as you start doing any nontrivial logic it is (in my experience) always easier to understand.
For example:
Combine this with `docopt` and you can very quickly and easily write helptext/arg parsing wrappers for your scripts; e.g. Which to my eyes is much easier to grok than the equivalent bash for providing help text and requiring args.There's an argument to be made that "shell is more universal", but my claim here is that this is actually false, and simple Python is going to be more widely-understood and less error-prone these days.