In python the first line of the script should be
#!/usr/bin/env/python{3}
Is there a way to do:
if python:
#!/usr/bin/env/python
else:
#!/usr/bin/env/python3
EDIT:
Running python in RHEL7 bring up python-2.7.5 (default).
Only running python3 will execute python3-3.6.8 on my RHEL7.
I can't say I recommend this at all, but it does work:
#!/bin/bash
_=""""
if [[ -x '/usr/bin/python3' ]]
then
exec /usr/bin/python3 "$0" "$#"
elif [[ -x '/usr/bin/python' ]]
then
exec /usr/bin/python "$0" "$#"
else
echo "No python available"
exit 1
fi
"""
import sys
sys.stdout.write("you made it\n")
Can't do from __future__ import print_function because it has to be the first line of python.
Principles here are to use bash to do the executable detection but also make it valid python so you can just re-execute with the python interpreter on the same file.
None of those are correct. The correct command is /usr/bin/env and the argument to that, after a space, is the name of the actual interpreter you want it to look up.
Your question is a bit of a "turtles all the way down" problem. You have to know what command you want to run. You could of course create yet another tool and call it something like py3orpy but then you need to know that that exists in the PATH before you try to ask env to find it.
I would expect us to eventually converge on python; if you can't rely on that, perhaps the least error-prone solution is to figure out what's correct for the target system at installation time, and have the installer write a correct shebang. Or just install a symlink /usr/local/bin/python3 if it doesn't exist already, and hardcode #!/usr/bin/env python3 everywhere.
Related
So I know there are plenty of answers on stack overflow for ignoring errors in a bash script. None of them seem to work for the source command though.
I have tried the tried and true source ../bin/activate || true
I have tried setting set -e before running the command
I have tried source ../bin/activate 2>&1 /dev/null
I have tried setting set +e before running the command. This did not work either.
but on every run-through of this code, I receive
run.01: line 12: ../bin/activate: No such file or directory
The context for this problem is that I'm creating a simple bash script that runs some python code. The user is instructed for how to create a specific virtual environment and this line will automatically activate it if they set it up correctly, otherwise, this should ignore failing to activate running and continue running the commands in whatever environment is currently activated.
# Try virtual environment
source ../bin/activate || true
## Run.
code="../src-01/driver.v01.py"
## --------------------
graphInputFile="undirected_graph_01.inp"
graphType="undirected"
srcColId="0"
desColId="1"
degreeFind="2"
outFile="count.undirected.num.nodes.out"
python $code -inpGraphFile $graphInputFile -graphFormat $graphType -colSrcId $srcColId -colDesId $desColId -degreeFind $degreeFind -output_file $outFile
The python command should execute regardless of whether or not the source ../bin/activate command succeeds or not. I'm a little a loss for why none of these solutions are working and am currently under the assumption that source might do something different than a normal command given the circumstances.
EDIT:
I added the shebang #!/bin/bash -x to my file as requested but this did not do anything.
I this is my exact terminal output when I run this script.
Lucas-Macbook:test-01 lucasmachi$ sh run.01
run.01: line 14: ../bin/activate: No such file or directory
Lucas-Macbook:test-01 lucasmachi$
Where run.01 is the name of the bash script.
Also to clarify, the code I showed is not censored. that is the entire script (except now with the mentioned shebang at the top.)
This is a bug in Bash versions before 4.0 (macOS is stuck on 3.2).
Given this script:
#!/bin/bash
set -e
echo "Running $BASH_VERSION"
source "does not exist" || true
echo "Continuing"
Running on macOS will say:
macos$ ./myscript
Running 3.2.57(1)-release
./myscript: line 3: does not exist: No such file or directory
While on a modern system with an updated bash version, you get the expected behavior:
debian$ ./myscript
Running 5.0.11(1)-release
./myscript: line 3: does not exist: No such file or directory
Continuing
If you need to support macOS and Bash 3.2, run set +e first to disable errexit, and optionally re-enable it afterwards.
You could add a validation in your code to check whether the file exists before sourcing it :
[[ -f "../bin/activate" ]] && source ../bin/activate
Using the -f flag before a path will return true if the path file exists, and false if it doesn't.
The one liner syntax for bash if statements is as follow :
[[ Condition that returns true or false ]] && Exec if true
I ran into this issue with gitlab pipelines. None of the above worked for me, but combining the answers did:
[[ -f "${ENVIRONMENT}_test" ]] && source "${ENVIRONMENT}_test" || true
This gets me to where I need to go, hopefully it will help someone in the future.
I'm trying to execute a command for each file in a directory but while using their absolute path (such as /home/richi/mydir/myfile.py) instead of their relative path (such as myfile.py).
In other words, I want to execute a command on files in a directory based on their absolute path - similar to for file in *.py; do thecommand -a "$file"; done but not quite.
I'm asking this because I'm trying to implement a Travis CI script running in an Ubuntu 14.04 environment which will install and use pyminifier to recursively minify all the Python code files in a directory.
Please note that I'm asking may be similar to this post but it's not.
Since you're on a standard Linux distro with a full userland, you can just use the realpath command:
Print the resolved absolute file name…
For example:
$ pwd
/home/abarnert/src/test
$ touch 1
$ realpath 1
/home/abarnert/src/test/1
That's it.
If you don't know how to use that from within bash, you can call a subcommand using $(…) syntax:
$ echo $(realpath 1)
/home/abarnert/src/test/1
Of course you want to pass it the value of the variable file, but that's just as easy:
$ file=1
$ echo $(realpath "$file")
/home/abarnert/src/test/1
I'm assuming you're using bash here. With a different sh-style shell, things will be different; with tcsh or zsh or fish or something, it may be even more different.
A really old userland, or a really stripped down one (e.g., for an embedded system) might not include realpath. In that case, you can use readlink, since the GNU version, as usually, adds everything including a couple kitchen sinks, and can be used as a realpath substitute.
Or, if worst comes to worst, Python has come with a realpath function since 2.2:
$(python -c 'import os,sys; print(os.path.realpath(sys.argv[1]))' "$file")
When a Python script is supposed to be run from a pyenv virtualenv, what is the correct shebang for the file?
As an example test case, the default Python on my system (OS X) does not have pandas installed. The pyenv virtualenv venv_name does. I tried getting the path of the Python executable from the virtualenv.
pyenv activate venv_name
which python
Output:
/Users/username/.pyenv/shims/python
So I made my example script.py:
#!/Users/username/.pyenv/shims/python
import pandas as pd
print 'success'
But when I tried running the script (from within 'venv_name'), I got an error:
./script.py
Output:
./script.py: line 2: import: command not found
./script.py: line 3: print: command not found
Although running that path directly on the command line (from within 'venv_name') works fine:
/Users/username/.pyenv/shims/python script.py
Output:
success
And:
python script.py # Also works
Output:
success
What is the proper shebang for this? Ideally, I want something generic so that it will point at the Python of whatever my current venv is.
I don't really know why calling the interpreter with the full path wouldn't work for you. I use it all the time. But if you want to use the Python interpreter that is in your environment, you should do:
#!/usr/bin/env python
That way you search your environment for the Python interpreter to use.
As you expected, you should be able to use the full path to the virtual environment's Python executable in the shebang to choose/control the environment the script runs in regardless of the environment of the controlling script.
In the comments on your question, VPfB & you find that the /Users/username/.pyenv/shims/python is a shell script that does an exec $pyenv_python. You should be able to echo $pyenv_python to determine the real python and use that as your shebang.
See also: https://unix.stackexchange.com/questions/209646/how-to-activate-virtualenv-when-a-python-script-starts
Try pyenv virtualenvs to find a list of virtual environment directories.
And then you might find a using shebang something like this:
#!/Users/username/.pyenv/python/versions/venv_name/bin/python
import pandas as pd
print 'success'
... will enable the script to work using the chosen virtual environment in other (virtual or not) environments:
(venv_name) $ ./script.py
success
(venv_name) $ pyenv activate non_pandas_venv
(non_pandas_venv) $ ./script.py
success
(non_pandas_venv) $ . deactivate
$ ./script.py
success
The trick is that if you call out the virtual environment's Python binary specifically, the Python interpreter looks around that binary's path location for the supporting files and ends up using the surrounding virtual environment. (See per *How does virtualenv work?)
If you need to use more shell than you can put in the #! shebang line, you can start the file with a simple shell script which launches Python on the same file.
#!/bin/bash
"exec" "pyenv" "exec" "python" "$0" "$#"
# the rest of your Python script can be written below
Because of the quoting, Python doesn't execute the first line, and instead joins the strings together for the module docstring... which effectively ignores it.
You can see more here.
To expand this to an answer, yes, in 99% of the cases if you have a Python executable in your environment, you can just use:
#!/usr/bin/env python
However, for a custom venv on Linux following the same syntax did not work for me since the venv created a link to the Python interpreter which the venv was created from, so I had to do the following:
#!/path/to/the/venv/bin/python
Essentially, however, you are able to call the Python interpreter in your terminal. This is what you would put after #!.
It's not exactly answering the question, but this suggestion by ephiement I think is a much better way to do what you want. I've elaborated a bit and added some more of an explanation as to how this works and how you can dynamically select the Python executable to use:
#!/bin/sh
#
# Choose the Python executable we need. Explanation:
# a) '''\' translates to \ in shell, and starts a python multi-line string
# b) "" strings are treated as string concatenation by Python; the shell ignores them
# c) "true" command ignores its arguments
# c) exit before the ending ''' so the shell reads no further
# d) reset set docstrings to ignore the multiline comment code
#
"true" '''\'
PREFERRED_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python
ALTERNATIVE_PYTHON=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
FALLBACK_PYTHON=python3
if [ -x $PREFERRED_PYTHON ]; then
echo Using preferred python $ALTERNATIVE_PYTHON
exec $PREFERRED_PYTHON "$0" "$#"
elif [ -x $ALTERNATIVE_PYTHON ]; then
echo Using alternative python $ALTERNATIVE_PYTHON
exec $ALTERNATIVE_PYTHON "$0" "$#"
else
echo Using fallback python $FALLBACK_PYTHON
exec python3 "$0" "$#"
fi
exit 127
'''
__doc__ = """What this file does"""
print(__doc__)
import platform
print(platform.python_version())
If you want just a single script with a simple selection of your pyenv virtualenv, you may use a Bash script with your source as a heredoc as follows:
#!/bin/bash
PYENV_VERSION=<your_pyenv_virtualenv_name> python - $# <<EOF
import sys
print(sys.argv)
exit
EOF
I did some additional testing. The following works too:
#!/usr/bin/env -S PYENV_VERSION=<virtual_env_name> python
/usr/bin/env python won't work, since it doesn't know about the virtual environment.
Assuming that you have main.py living next to a ./venv directory, you need to use Python from the venv directory. Or in other words, use this shebang:
#!venv/bin/python
Now you can do:
./main.py
Maybe you need to check the file privileges:
sudo chmod +x script.py
If I have a basic Python script, with it's hashbang and what-not in place, so that from the terminal on Linux I can run
/path/to/file/MyScript [args]
without executing through the interpreter or any file extensions, and it will execute the program.
So would I install this script so that I can type simply
MyScript [args]
anywhere in the system and it will run? Can this be implemented for all users on the system, or must it be redone for each one? Do I simply place the script in a specific directory, or are other things necessary?
The best place to put things like this is /usr/local/bin.
This is the normal place to put custom installed binaries, and should be early in your PATH.
Simply copy the script there (probably using sudo), and it should work for any user.
Walkthrough of making a python script available anywhere:
Make a python script:
cd /home/el/bin
touch stuff.py
chmod +x stuff.py
Find out where your python is:
which python
/usr/bin/python
Put this code in there:
#!/usr/bin/python
print "hi"
Run in it the same directory:
python stuff.py
Go up a directory and it's not available:
cd ..
stuff.py
-bash: stuff.py: command not found
Not found! It's as we expect, add the file path of the python file to the $PATH
vi ~/.bashrc
Add the file:
export PATH=$PATH:/home/el/bin
Save it out, re apply the .bashrc, and retry
source ~/.bashrc
Try again:
cd /home/el
stuff.py
Prints:
hi
The trick is that the bash shell knows the language of the file via the shebang.
you can also use setuptools (https://pypi.org/project/setuptools/)
your script will be:
def hi():
print("hi")
(suppose the file name is hello.py)
also add __init__.py file next to your script (with nothing in it).
add setup.py script, with the content:
#!/usr/bin/env python3
import setuptools
install_requires = [
'WHATEVER PACKAGES YOU NEED GOES HERE'
]
setuptools.setup(
name="some_utils",
version="1.1",
packages=setuptools.find_packages(),
install_requires=install_requires,
entry_points={
'console_scripts': [
'cool_script = hello:hi',
],
},
include_package_data=True,
)
you can now run python setup.py develop in this folder
then from anywhere, run cool_script and your script will run.
Just create ~/bin and put export PATH=$PATH:$HOME/bin in your bashrc/profile. Don't mess with the system, it will bite you back, trust me.
Few more things (relevant to the question but not part of the answer):
The other way export PATH=$HOME/bin:$PATH is NOT safe, for bash will will look into your ~/bin folder for executables, and if their name matches with other executables in your original $PATH you will be surprised by unexpected/non working command execution.
Don't forget to chmod+x when you save your script in ~/bin.
Be aware of what you are putting in your ~/bin folder, if you are just testing something or working on unfinished script, its always better to use ./$SCRIPT_NAME from your CWD to execute the script than putting it under ~/bin.
The quick answer is to symlink your script to any directory included in your system $PATH.
The long answer is described below with a walk through example, (this is what I normally do):
a) Create the script e.g. $HOME/Desktop/myscript.py:
#!/usr/bin/python
print("Hello Pythonista!")
b) Change the permission of the script file to make it executable:
$ chmod +x myscript.py
c) Add a customized directory to the $PATH (see why in the notes below) to use it for the user's scripts:
$ export PATH="$PATH:$HOME/bin"
d) Create a symbolic link to the script as follows:
$ ln -s $HOME/Desktop/myscript.py $HOME/bin/hello
Notice that hello (can be anything) is the name of the command that you will use to invoke your script.
Note:
i) The reason to use $HOME/bin instead of the /usr/local/bin is to separate the local scripts from those of other users (if you wish to) and other installed stuff.
ii) To create a symlink you should use the complete correct path, i.e.
$HOME/bin GOOD ~/bin NO GOOD!
Here is a complete example:
$ pwd
~/Desktop
$ cat > myscript.py << EOF
> #!/usr/bin/python
> print("Hello Pythonista!")
> EOF
$ export PATH="$PATH:$HOME/bin"
$ ln -s $HOME/Desktop/myscript.py $HOME/bin/hello
$ chmod +x myscript.py
$ hello
Hello Pythonista!
Just create symbolic link to your script in /usr/local/bin/:
sudo ln -s /path/to/your/script.py /usr/local/bin/script
Putting the script somewhere in the PATH (like /usr/local/bin) is a good solution, but this forces all the users of your system to use/see your script.
Adding an alias in /etc/profile could be a way to do what you want allowing the users of your system to undo this using the unalias command. The line to be added would be:
alias MyScript=/path/to/file/MyScript
i find a simple alias in my ~/.bash_profile or ~/.zshrc is the easiest:
alias myscript="python path/to/my/script.py"
Type echo $PATH in a shell. Those are the directories searched when you type command, so put it in one of those.
Edit: Apparently don't use /usr/bin, use /usr/local/bin
Acording to FHS, the /usr/local/bin/ is the good place for custom scripts.
I prefer to make them 755 root:root, after copying them there.
I was hoping to write a python script to create some appropriate environmental variables by running the script in whatever directory I'll be executing some simulation code, and I've read that I can't write a script to make these env vars persist in the mac os terminal. So two things:
Is this true?
and
It seems like it would be a useful things to do; why isn't it possible in general?
You can't do it from python, but some clever bash tricks can do something similar. The basic reasoning is this: environment variables exist in a per-process memory space. When a new process is created with fork() it inherits its parent's environment variables. When you set an environment variable in your shell (e.g. bash) like this:
export VAR="foo"
What you're doing is telling bash to set the variable VAR in its process space to "foo". When you run a program, bash uses fork() and then exec() to run the program, so anything you run from bash inherits the bash environment variables.
Now, suppose you want to create a bash command that sets some environment variable DATA with content from a file in your current directory called ".data". First, you need to have a command to get the data out of the file:
cat .data
That prints the data. Now, we want to create a bash command to set that data in an environment variable:
export DATA=`cat .data`
That command takes the contents of .data and puts it in the environment variable DATA. Now, if you put that inside an alias command, you have a bash command that sets your environment variable:
alias set-data="export DATA=`cat .data`"
You can put that alias command inside the .bashrc or .bash_profile files in your home directory to have that command available in any new bash shell you start.
One workaround is to output export commands, and have the parent shell evaluate this..
thescript.py:
import pipes
import random
r = random.randint(1,100)
print("export BLAHBLAH=%s" % (pipes.quote(str(r))))
..and the bash alias (the same can be done in most shells.. even tcsh!):
alias setblahblahenv="eval $(python thescript.py)"
Usage:
$ echo $BLAHBLAH
$ setblahblahenv
$ echo $BLAHBLAH
72
You can output any arbitrary shell code, including multiple commands like:
export BLAHBLAH=23 SECONDENVVAR='something else' && echo 'everything worked'
Just remember to be careful about escaping any dynamically created output (the pipes.quote module is good for this)
If you set environment variables within a python script (or any other script or program), it won't affect the parent shell.
Edit clarification:
So the answer to your question is yes, it is true.
You can however export from within a shell script and source it by using the dot invocation
in fooexport.sh
export FOO="bar"
at the command prompt
$ . ./fooexport.sh
$ echo $FOO
bar
It's not generally possible. The new process created for python cannot affect its parent process' environment. Neither can the parent affect the child, but the parent gets to setup the child's environment as part of new process creation.
Perhaps you can set them in .bashrc, .profile or the equivalent "runs on login" or "runs on every new terminal session" script in MacOS.
You can also have python start the simulation program with the desired environment. (use the env parameter to subprocess.Popen (http://docs.python.org/library/subprocess.html) )
import subprocess, os
os.chdir('/home/you/desired/directory')
subprocess.Popen(['desired_program_cmd', 'args', ...], env=dict(SOMEVAR='a_value') )
Or you could have python write out a shell script like this to a file with a .sh extension:
export SOMEVAR=a_value
cd /home/you/desired/directory
./desired_program_cmd
and then chmod +x it and run it from anywhere.
What I like to do is use /usr/bin/env in a shell script to "wrap" my command line when I find myself in similar situations:
#!/bin/bash
/usr/bin/env NAME1="VALUE1" NAME2="VALUE2" ${*}
So let's call this script "myappenv". I put it in my $HOME/bin directory which I have in my $PATH.
Now I can invoke any command using that environment by simply prepending "myappenv" as such:
myappenv dosometask -xyz
Other posted solutions work too, but this is my personal preference. One advantage is that the environment is transient, so if I'm working in the shell only the command I invoke is affected by the altered environment.
Modified version based on new comments
#!/bin/bash
/usr/bin/env G4WORKDIR=$PWD ${*}
You could wrap this all up in an alias too. I prefer the wrapper script approach since I tend to have other environment prep in there too, which makes it easier for me to maintain.
As answered by Benson, but the best hack-around is to create a simple bash function to preserve arguments:
upsert-env-var (){ eval $(python upsert_env_var.py $*); }
Your can do whatever you want in your python script with the arguments. To simply add a variable use something like:
var = sys.argv[1]
val = sys.argv[2]
if os.environ.get(var, None):
print "export %s=%s:%s" % (var, val, os.environ[var])
else:
print "export %s=%s" % (var, val)
Usage:
upsert-env-var VAR VAL
As others have pointed out, the reason this doesn't work is that environment variables live in a per-process memory spaces and thus die when the Python process exits.
They point out that a solution to this is to define an alias in .bashrc to do what you want such as this:
alias export_my_program="export MY_VAR=`my_program`"
However, there's another (a tad hacky) method which does not require you to modify .bachrc, nor requires you to have my_program in $PATH (or specify the full path to it in the alias). The idea is to run the program in Python if it is invoked normally (./my_program), but in Bash if it is sourced (source my_program). (Using source on a script does not spawn a new process and thus does not kill environment variables created within.) You can do that as follows:
my_program.py:
#!/usr/bin/env python3
_UNUSED_VAR=0
_UNUSED_VAR=0 \
<< _UNUSED_VAR
#=======================
# Bash code starts here
#=======================
'''
_UNUSED_VAR
export MY_VAR=`$(dirname $0)/my_program.py`
echo $MY_VAR
return
'''
#=========================
# Python code starts here
#=========================
print('Hello environment!')
Running this in Python (./my_program.py), the first 3 lines will not do anything useful and the triple-quotes will comment out the Bash code, allowing Python to run normally without any syntax errors from Bash.
Sourcing this in bash (source my_program.py), the heredoc (<< _UNUSED_VAR) is a hack used to "comment out" the first-triple quote, which would otherwise be a syntax error. The script returns before reaching the second triple-quote, avoiding another syntax error. The export assigns the result of running my_program.py in Python from the correct directory (given by $(dirname $0)) to the environment variable MY_VAR. echo $MY_VAR prints the result on the command-line.
Example usage:
$ source my_program.py
Hello environment!
$ echo $MY_VAR
Hello environment!
However, the script will still do everything it did before except exporting, the environment variable if run normally:
$ ./my_program.py
Hello environment!
$ echo $MY_VAR
<-- Empty line
As noted by other authors, the memory is thrown away when the Python process exits. But during the python process, you can edit the running environment. For example:
>>> os.environ["foo"] = "bar"
>>> import subprocess
>>> subprocess.call(["printenv", "foo"])
bar
0
>>> os.environ["foo"] = "foo"
>>> subprocess.call(["printenv", "foo"])
foo
0