Nit: setuptools does not do this. Proper setuptools packages use the console_scripts entry point, which points to a function will never specify a shebang. A good example is Paste Script [PS]. "paster" is linked the function "paste.script.command:run". When this is pip-installed into a virtualenv, you can `cat` the "paster" executable. It'll look like something like this [pr]; sure enough, the entire thing will have been automatically generated, taking into account the changed Python binary. Slag on setuptools all you want; PEAK got entry points nailed down perfectly.
On the other hand, distutils has a `scripts=` keyword that points directly to a file, which then gets copied and pasted into the bin directory. Since the file has be word-for-word the script, the shebang is written directly. It depends on env, which should normally function; but, if it doesn't, as in the OP's case, there's no fallback. A good example of this is Mercurial [hg]. (Note: I actually haven't used raw distutils all that much, so there might be a work-around for this. A few fruitless minutes of Google Code Search hasn't turned up anything interesting.)
"The only clever feature is that if the first line of the script starts with #! and contains the word “python”, the Distutils will adjust the first line to refer to the current interpreter location. By default, it is replaced with the current interpreter location. The --executable (or -e) option will allow the interpreter path to be explicitly overridden."
This reminds me of a discussion here a few days ago in which people were happily bashing java's classpath. I don't know anything about python, but my feeling at their comparison of java and python was that something that is way too simple to use can't be at the same time easy to customize.
In the light of this article classpaths sound a lot like what they really are: a working compromise everybody can live with.
Fixing it is good, but a nicer to start an interpreter, which handles whichever is the preferred location to Python, is to call env, which is always in usr bin.
#!/usr/bin/env python
Will dynamically start whatever Python is in your $PATH.
Right, I'm saying that he was faced with a group of files he couldn't tar up, because the #! were all fixed. I'm wondering why he couldn't just replace '#!.*/python' with '#!/usr/bin/env python' for everything in the tree.
And I can/could, and will probably still end up doing that as much as I might not like it - and any time you install something else which includes a script in a binpath, you need to rerun the sed command.
You'd think /usr/bin/env would always work, but I've seen this break in production systems where /usr/bin/env is not in the $PATH in situations like cron scripts. It's retarded I know but it does happen.
I don't think you mean this. When you specify an absolute name, $PATH is not even in the picture. (I do know that some systems put env in /bin. Those systems are misconfigured.)
Anyway, shebangs are flaky by design. I just run the correct interpreter directly, as in "/path/to/perl -Ilib bin/script.pl". There is no confusion that way.
You're right - I think what he means is that env is not in /usr/bin. This was the case on very old (Red Hat Linux 5.2 back in the late 90s) systems where there was no /usr/bin/env -> /bin/env symlink.
I was bit confused in my description here, sorry. /usr/bin/env existed but when it was run under the cron user's environment it could not locate the interpreter we required and failed silently. The shebang line worked fine under all other users' accounts. Very frustrating. I assume this problem was due to the cron account's $PATH being set up differently somehow. In any case we just went back to explicitly choosing our site-specific interpreter path.
Related pet peeve: people using #!/bin/sh when they really want #!/bin/bash. They are not interchangeable!
I did exactly that at first; the problem is that it shouldn't need to be done, and additionally, other things (such as .pth files) can also include hardcoded paths.
No it's not. You missed the fact that python packages that include stuff like scripts hardcode the full, explicit prefix in the #! lines. This means you can compile with --prefix all you want, but if you move it, everything breaks in unexpected ways.
I agree, all you should need is a --prefix, and that scripts should obey whatever python interpreter is in /usr/bin/env - but that's not the way it works, sadly.
python packages that include stuff like scripts hardcode the full, explicit prefix in the #! lines.
Such scripts are broken. One should fix the scripts instead of mangling the environment. It's a one-liner:
find /opt/myproject -type f -exec sed -i 's/#!\/usr\/bin\/python/#!\/usr\/bin\/env python/' {} \;
but if you move it, everything breaks in unexpected ways.
If you move it then PYTHONPATH is your friend.
Once you have a python build for each project (which you want to have anyways) you'll notice that you don't need virtualenv or other kludges anymore either. Yep, it's really that easy. 4 lines of code. How many lines was yours again?
Did you read the comment above this? Sed is the initial hack to fix it, but it's still a hack; not a fix. Eventually I punted on it for the Linux dist, but I'm probably going to be stuck with the sed fix for OS/X.
As for pythonpath - that has nothing to do with moving the entire directory path, including the interpreter, /bin dir, etc.
Sorry, I just can't follow where your exact problem is.
I have 9 python projects under /opt/foo* right now. Each of them has their own python in /opt/foo*/sys/python. I su to the respective user who has PATH and PYTHONPATH set appropiately in its profile - and everything works as expected.
I also don't follow why you consider sed a "hack" here. A hardcoded hashbang line is clearly a Bug - but gladly one that is trivially fixed with that one-liner. On the other side we have your "fix" that involves permanently mangling the environment in strange ways. In my book you are the one proposing the hack here.
I'm not proposing mangling anything here; in fact I dropped the mangled environment (as it was a hack, and added too much space) and swapped to a simpler (and cleaner) virtualenv built in /opt.
I still haven't fixed the build something and upload it for people, because of the hardcoded shebang, and yes - sed could fix that.
Also, you have to run the sed command over the tree every time you install a new package which installs a binary
How often do your projects depend on something else, written in python, that installs a binary? For me that number is zero. iirc all of my python dependencies are libraries. And if I stumbled into something that comes with a binary, with a broken hashbang line, then I'd be rather motivated to not only fix it, but to also feed the fix upstream. If the project is useful enough for me to use it then those 10 minutes of my time for a brown-paper-bag micropatch are a no-brainer.
Well, to each their own. - But I still think this is a solution looking for a problem, with the potential to create more problems than it solves.
Not exactly - for example, I depend on fabric, PIP, easy_install, nose, pylint and others. In this case, it's not a matter of fixing upstream on a per-project basis, it's fixing distutils as this is "magical" behavior I (and others) have seen, and that's not a pile of gators I (or I suspect you) would want to wade into. For example read the following threads:
I just checked and interestingly I also depend on easy_install (which I don't even use, but put it in there someday), nose and pylint. None of them gave me problems. I say if fabric and PIP are broken then fix them.
easy_install is irrelevant anyways, nobody in their right mind would use that on a production system.
I'm surprised having the hardcoded paths didn't bite you - of course, how frequently are you moving the interpreter, or installing multiple versions of the interpreter (with the same dependencies) with a shared bin dir?
In Perl, the default build systems change the shebang on installed scripts to point to the perl that "compiled" them. So if you run "/my/weird/perl Makefile.PL && make install", your installed scripts are run by /my/weird/perl.
(We also have a generic way to get at installed data files without hard-coded paths; see File::ShareDir.)
[PS]: http://svn.pythonpaste.org/Paste/Script/trunk/setup.py
[pr]: http://pastie.org/550423
On the other hand, distutils has a `scripts=` keyword that points directly to a file, which then gets copied and pasted into the bin directory. Since the file has be word-for-word the script, the shebang is written directly. It depends on env, which should normally function; but, if it doesn't, as in the OP's case, there's no fallback. A good example of this is Mercurial [hg]. (Note: I actually haven't used raw distutils all that much, so there might be a work-around for this. A few fruitless minutes of Google Code Search hasn't turned up anything interesting.)
[hg]: http://selenic.com/repo/hg/file/47ce7a3a1fb0/hg