I have a problem which is caused by our encapsulated design. Up till now lots of our scripts were written in bash and as a result the #!/bin/bash was always simple.
However now that we are rewriting our scripts in python that is a bit more difficult. We deliver a specific version of python (to avoid version differences in client installed environments from breaking our implementation). Because the specific version of python lives in a installed directory structure I need to route to it.
However I don't think the #! statement can accept environment variables from the shell that executes the file(tried and got a bad interpreter).
eg:
in foo.py I have #!$dirloc/wherepythonlives/python
In the bash shell I executed the file and got bad interpreter.
Is there a way of sneaking an environment variable into that #! line?
Or will I have to depend on an explicit path? We want to support multiple versions of our software (which may mean multiple python versions) on one environment so I was hoping to somehow keep Python's !# statement inside the directory level we install into.
A common way to do this is to use the env program:
#!/usr/bin/env python
This will cause env to look along the PATH environment for a binary called python.
I'm not aware of being able to use environment variable in the shebang. You can use relative paths though
#! ../../usr/bin/python
edit:
You could always use env to specify to use a specific version. Then if that version can be found in $PATH it will be used otherwise the script will fail
#! /usr/bin/env python2.7
Or you could make the entry point a generic script instead.
eg
#! /usr/bin/env bash
if [[ $MYPYTHON ]]
then
$MYPYTHON main.py
else
echo error message
fi
The optimal solution to this dilemma is using distutils (setup.py, which creates correct stubs for you automatically, for a number of given "console entry points") and virtualenv (handling the "isolated multiple installations" part).
I suppose it all depends on how and in what environment your scripts will be invoked. You could call you scripts using #!/usr/bin/env python, which would allow you to control which python is used by manipulating the environment's PATH.
You could always specify a wrapper script as the interpreter, which runs a python executable relative to the script's location:
foo.py:
#!/bin/pyselector
import sys
sys.exit(0)
pyselector:
#!/bin/sh
SCRIPT_PATH="$(readlink -f $1)"
SCRIPT_DIR="$(dirname "$SCRIPT_PATH")"
"${SCRIPT_DIR}/my/local/python" "$#"
Related
I have a script which uses the shebang #!/usr/bin/env python. It works great on machines where Python 3 is the only version available, but on the machines which have both Python 2 and Python 3, it runs the script with Python 2.
If I modify the shebang to be #!/usr/bin/env python3, it would work on the machines with Python 2 and Python 3, but on the machines which have only Python 3, it would fail with “No such file or directory” error.
One solution is to create an alias alias python=python3.
Are there other solutions to have a same shebang working uniformly on every machine?
Unfortunately, there is no universally working way of doing this that would work across any and all up front unknown Linux hosts and you are largely left at the mercy of distro maintainers and local host configuration.
alias won't help, because interpreter specified by #! handled by the kernel and /usr/bin/env it will exec in this case does not know about aliases of your shell.
When using env, you could make sure that the name following env is first found and means what you want it to mean by:
making sure all hosts are setup the same way in that respect (expected packaged are installed or at least symlinks are created)
having user specific construct for execution of your script, such as:
mkdir /tmp/bin
ln -s /usr/bin/python /tmp/bin/python3
PATH="/tmp/bin:${PATH}" ./myscript.py
But none of this is really great and ultimately what you've asked for.
Your interpreter (this is harder than it may sound though, interpreter resolution code is quite simple; where to put it, how to call it for kernel to find and use it) could also be a simple shell script that tries to figure it out that you pack with your python code, but any option you look at isn't really great I am afraid.
There is a PEP-394 for that which suggested / expected on U*X-like system:
you get python for python2
and python3 for python3
But it recognizes this has never been entirely consistently applied... and also not as useful in 2020:
However, these recommendations implicitly assumed that Python 2 would always be available. As Python 2 is nearing its end of life in 2020 (PEP 373, PEP 404), distributions are making Python 2 optional or removing it entirely. This means either removing the python command or switching it to invoke Python 3. Some distributors also decided that their users were better served by ignoring the PEP's original recommendations, and provided system administrators with the freedom to configure their systems based on the needs of their particular environment.
TL;DR unfortunately there is no way that universally works and compensates for decisions of various distro and even individual host maintainers. :(
I would most likely opt to stick with #!/usr/bin/env python3 (which has so far been the recommended naming) and add a README that explains the prerequisites and how to setup the host just to be sure.
For the sake of completeness I should add, the PEP does make a recommendation in this regard: setup and use virtual environment or use a (third party) environment manager. However the way I read the question: "Portable interpreter specification that does not make any assumptions about nor poses any (additional) requirements to target host configuration", this would then not fit the bill and would not mean a substantial improvement over saying: make sure you have python3 executable in the search path and create a symlink if not.
#!/usr/bin/env python3 is the most correct general solution. On systems where Python 3 is installed, this should work, regardless of whether Python 2 is installed or not.
That said, there’s still no guarantee that the right version of Python 3 is installed, and hard-coding, say, python3.7 isn’t viable since most users won’t have several versions installed.
Luckily there’s a better solution: If you distribute your application as a Python package with Setuptools and specify entry_points for your binary/binaries, the installation process will pick the correct interpreter and hard-code its path into your application scripts. This will work regardless of environment, as long as the installer (e.g. pip) can find Python at all.
As a very simple example, say your project example has the following folder structure:
example
├── example
│ ╰── __init__.py
├── pyproject.toml
╰── setup.cfg
And let’s assume your __init__.py has a function main that we want to use as an entry point for an executable script.
Then pyproject.toml would look like this:1
[build-system]
requires = ["setuptools", "wheel"]
And setup.cfg looks as follows:1,2
[metadata]
name = example
version = 0.0.1
[options]
packages = find:
[options.entry_points]
console_scripts =
example-bin = example:main
Now you can build your project using e.g. pip3 bdist path-to-project, which will generate the Wheel installation bundle. When you install that (either after uploading it to PyPI or locally (via pip3 install wheel-filename.whl), pip3 will install the binary example-bin.
example-bin will launch your package’s entry point function, and will work regardless of what the Python 3 binary is called on your system, because pip3 install … creates this file with an absolute path in the shebang line. For example, on my system the first line of that file is
#!/usr/local/opt/python/bin/python3.7
1 Rather than use pyproject.toml and setup.cfg, the same works using the legacy setup.py file — but using the above is simpler.
2 Note that, at the time of writing, there’s a typo in the Setuptools quickstart: instead of [entry_point], it should read [entry_points].
Here's what I've settled on for now, though I'm still trying to write my code such that it can work with Python 2 in the cases where that's necessary. You can modify this to your needs if you have a few known paths, for instance (it doesn't all need to be on one line):
#!/bin/bash
_='''' # Prefer python3 if it exists
command -v python3 &> /dev/null && exec python3 $0 "$#" || exec python $0 "$#"
'''
# Python code here
print ("Hello")
That's based on an example from this page about hybrid scripts: https://gist.github.com/andyneff/fafba17b748bba6d7cd5
The script is initially loaded in bash, which sees the first line beginning with _='''', which is interpreted by bash as setting the variable _ to two null strings ('') in a row.
The second line uses command -v python3 to try and determine if python3 is in the path or defined as an alias, with stdout and stderr redirected to the null device. If it has a successful return code, then exec python3 $0 "$#" is run, where $0 is the name of the script and "$#" expands to the list of arguments. If it fails, the equivalent python command is run.
Since the command is exec'd, the python3/python command replaces the current bash process, and bash never sees the following lines.
When Python reads the file, it sees _='''' and interprets it as the beginning of a '''...''' multi-line triple-quoted string, even though it begins with four apostrophes rather than just three, so the script ends up with a harmless extra _ variable set.
I'm developing a set of script in python3, as shebang I use this:
#!/usr/bin/env python3
Everything goes ok, but in some virtual machines where are executed the name of interpreter is python3.5. I will like to be able to execute my scripts in both enviroment but I can't change the filesystem of virtual machine (so I discard solutions like make a link from python3.5 to python3 )
I look at man of env but I don't find any way to specify a searching pattern or something like that.
I try to set an alias at begining of my sessions pointing to right python interpreter but env don't use it.
My unique solution is call my scripts saying which interpreter must use but is very anoying:
python3.5 myscript.py
Any idea is welcome!, thanks!
No need to bring in separate shell and python scripts, a single file can be both!
Replace your shebang line with this sequence:
#!/bin/sh
# Shell commands follow
# Next line is bilingual: it starts a comment in Python, and is a no-op in shell
""":"
# Find a suitable python interpreter (adapt for your specific needs)
for cmd in python3.5 python3 /opt/myspecialpython/bin/python3.5.99 ; do
command -v > /dev/null $cmd && exec $cmd $0 "$#"
done
echo "OMG Python not found, exiting!!!!!11!!eleven" >2
exit 2
":"""
# Previous line is bilingual: it ends a comment in Python, and is a no-op in shell
# Shell commands end here
# Python script follows (example commands shown)
import sys
print ("running Python!")
print (sys.argv)
If you can install scripts, you can also install a wrapper called python3.5 which simply dispatches python3.
#!/bin/sh
exec env python3 "$#"
You'll obviously need to chmod a+x this script just like the others you install.
You'll have to add the script's directory to your PATH after the system python3.5 directory to avoid having this go into an endless loop, and only use this script as a fallback when the system doesn't already provide python3.5.
As you noted, env doesn't know or care about your personal shell aliases or functions, and doesn't provide for any dynamic calculation of the binary to run by itself; but you have the shell at your disposal (and Python of course, once you find it!) -- it simply uses the PATH so if you can install your other scripts in a directory which is in your PATH (which must be the case for the #!/usr/bin/env shebang to make sense in the first place) you can store this script there, too.
As noted in comments, it's weird and user-hostile to only install python3.5 and not at least optionally make python3 a symlink to it, so perhaps you could eventually persuade whoever maintains the image you are installing into to provide this.
You could create a shell script that uses python 3.5 if it is installed, otherwise uses python 3 and executes your script with the correct version.
No need for python shebang.
In your shell script you may test if which python3.5 returns something; if it does, then python3.5 is installed, otherwise you'd have to use python3
I have a problem on OSX that #!/usr/bin/env python3 is causing the environment variables, such as library paths (LD_LIBRARY_PATH for example), to be lost. On Linux it works correctly.
Reading the man page on env it doesn't appear it should be doing this. It should only be modifying the environment if I request it, and I am not.
What is a portable shebang line that preserves the environment?
Note: I'm detecting the problem since a call to subprocess.open on one of my programs fails since it can't find one of the libraries. Yet if I start python interactively the same call works fine.
This is the OS X System Integrity Protection at work; certain 'dangerous' environment variables such as DYLD_LIBRARY_PATH, and, as you discovered, LD_LIBRARY_PATH, are not passed to any binary with a /usr/bin or /bin path.
As such, /usr/bin/env never sees any of several environment variables and can't pass those on to the python3 child process it spawns.
See several other posts on the subject:
Why doesn't lldb forward my environment variable anymore?
https://apple.stackexchange.com/questions/212945/unable-to-set-dyld-fallback-library-path-in-shell-on-osx-10-11-1
https://apple.stackexchange.com/questions/215030/el-capitan-make-check-dyld-library-path
If you are using subprocess, I'd check for the shebang line, extract the binary named after env, prefix your command with that binary, and bypass env altogether.
In many python scripts do I read the shebang directive #!/usr/bin/env python
I understand it tells which interpreter to use, like in a bash script: #!/bin/bash , but I fail to understand how the python interpreter is specified. If I simply run
$ /usr/bin/env
I get a list of variable path such as SHELL=/bin/bash or JAVA_HOME==/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home.
The thing is there is no information about python when running this command. So I would like to better understand what does /usr/bin/env do exactly, and in which way #!/usr/bin/env python tells where my current python interpreter is.
Try running /usr/bin/env python and see what happens.
When given an argument, env runs the executable it finds.
BTW I also use it with bash scripts: /usr/bin/env bash because distros don't agree on its location (/bin/bash vs /usr/bin/bash)
See also: https://unix.stackexchange.com/questions/29608/why-is-it-better-to-use-usr-bin-env-name-instead-of-path-to-name-as-my
EDIT- extra explanations:
When given an argument, e.g. python, env behaves exactly as any shell would do when trying to find an executable: look at the PATH environment variable, split it at :, and for each directory, try to find an executable named python. The first matching executable is launched.
Typical content of the PATH variable: /bin:/usr/bin:/usr/local/bin
The typical header should be
#!/usr/bin/env python
But I found below also works when executing the script like $python ./my_script.py
#!/usr/bin/python
#!python
What's difference between these 2 headers? What could be the problem for 2nd one? Please also discussing the case for python interpreter is in PATH or not. Thanks.
First, any time you run a script using the interpreter explicitly, as in
$ python ./my_script.py
$ ksh ~/bin/redouble.sh
$ lua5.1 /usr/local/bin/osbf3
the #! line is always ignored. The #! line is a Unix feature of executable scripts only, and you can see it documented in full on the man page for execve(2). There you will find that the word following #! must be the pathname of a valid executable. So
#!/usr/bin/env python
executes whatever python is on the users $PATH. This form is resilient to the Python interpreter being moved around, which makes it somewhat more portable, but it also means that the user can override the standard Python interpreter by putting something ahead of it in $PATH. Depending on your goals, this behavior may or may not be OK.
Next,
#!/usr/bin/python
deals with the common case that a Python interpreter is installed in /usr/bin. If it's installed somewhere else, you lose. But this is a good way to ensure you get exactly the version you want or else nothing at all ("fail-stop" behavior), as in
#!/usr/bin/python2.5
Finally,
#!python
works only if there is a python executable in the current directory when the script is run. Not recommended.
I'd suggest 3 things in the beginning of your script:
First, as already being said use environment:
#!/usr/bin/env python
Second, set your encoding:
# -*- coding: utf-8 -*-
Third, set some doc string:
"""This is a awesome
python script!"""
And for sure I would use " " (4 spaces) for ident.
Final header will look like:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""This is a awesome
python script!"""
Best wishes and happy coding.
The Python executable might be installed at a location other than /usr/bin, but env is nearly always present in that location so using /usr/bin/envis more portable.
From the manpage for env (GNU coreutils 6.10):
env - run a program in a modified environment
In theory you could use env to reset the environment (removing many of the existing environment variables) or add additional environment variables in the script header. Practically speaking, the two versions you mentioned are identical. (Though others have mentioned a good point: specifying python through env lets you abstractly specify python without knowing its path.)
Yes, there is - python may not be in /usr/bin, but for example in /usr/local/bin (BSD).
When using virtualenv, it may even be something like ~/projects/env/bin/python
The /usr/bin/env python becomes very useful when your scripts depend on environment settings for example using scripts which rely on python virtualenv. Each virtualenv has its own version of python binary which is required for adding packages installed in virtualenv to python path (without touching PYTHONPATH env).
As more and more people have started to used virtualenv for python development prefer to use /usr/bin/env python unless you don't want people to use their custom python binary.
Note: You should also understand that there are potential security issues (in multiuser environments) when you let people run your scripts in their custom environments. You can get some ideas from here.