Develop Reference

How to run bash commands from Python preferably with the os library?

I am trying to run the following python script named test.py. It contains multiple bash commands which I would like to execute in a Linux terminal (unix). This is the content of the file:
import os
os.system('echo install virtualenv')
os.system('sudo pip install virtualenv')
os.system('echo create virtual environment')
os.system('virtualenv my_virtualenvironment')
os.system('echo activate virtual environment')
os.system('source my_virtualenvironment/bin/activate')
I am running the Python script using the following in the terminal:
python3 test.py
The problem that I have is that the commands do not run the same way as they would on a Linux terminal. The output is the following error when trying to execute the last line of the Python script:
sh: 1: source: not found
The last command source my_virtualenvironment/bin/activate normally runs fine if I execute it directly in the terminal (without my Python script). Now, what does sh: 1: mean and why does it not work with my code? I would expect to get something starting with bash: .
Also I have found this solution, but I would like not to use lists for executing commands and maybe even to stick with the os library (if there is a simpler solution without os, I am also open for that):
https://stackoverflow.com/a/62355400/11535508

source is a bash built-in command, not an executable.
Use the full path to the python interpreter in your commands instead of venv activation, e.g. os.system('<venv>/bin/python ...').
The second option is to write your commands into a separate bash script and call it from python:
os.system('bash script.sh')

How to get python via virtual environment recognized in bash file via shebang

I have a script file:
#!/usr/bin/env python3.9
print("python is working")
However when I try and run it:
(karl-env) karl#Karls-MBP scripts (karl/test) $ . test.sh
bash: test.sh: line 3: syntax error near unexpected token `"python is working"'
bash: test.sh: line 3: `print("python is working")'
Following info:
(karl-env) karl#Karls-MBP scripts (karl/test) $ type -a python
python is /Users/karl/.pyenv/shims/python
python is /Users/karl/.pyenv/shims/python
python is /usr/bin/python
I'm in a virtual environment but I fail to understand how to get my environments python recognized via the shebang #!/usr/bin/env python3.9. I do not use Python often hence my noobiness!

This has a little to do with Python and a lot to do with the shell.
You're doing . test.sh – . is an alias for source, which has your shell attempt to interpret the given script as shell commands you'd enter. You want ./test.sh to execute the script.
Your shebang line is explicitly looking for a python3.9 executable, and your environment might not be Python 3.9, so you fall back to something else. Do python (or python3) instead: #!/usr/bin/env python
For the sake of sanity, rename your script to .py; it's not a .shellscript.

Shell/Terminal: Execute a command for all files in a directory using their absolute path

I'm trying to execute a command for each file in a directory but while using their absolute path (such as /home/richi/mydir/myfile.py) instead of their relative path (such as myfile.py).
In other words, I want to execute a command on files in a directory based on their absolute path - similar to for file in *.py; do thecommand -a "$file"; done but not quite.
I'm asking this because I'm trying to implement a Travis CI script running in an Ubuntu 14.04 environment which will install and use pyminifier to recursively minify all the Python code files in a directory.
Please note that I'm asking may be similar to this post but it's not.

Since you're on a standard Linux distro with a full userland, you can just use the realpath command:
Print the resolved absolute file name…
For example:
$ pwd
/home/abarnert/src/test
$ touch 1
$ realpath 1
/home/abarnert/src/test/1
That's it.
If you don't know how to use that from within bash, you can call a subcommand using $(…) syntax:
$ echo $(realpath 1)
/home/abarnert/src/test/1
Of course you want to pass it the value of the variable file, but that's just as easy:
$ file=1
$ echo $(realpath "$file")
/home/abarnert/src/test/1
I'm assuming you're using bash here. With a different sh-style shell, things will be different; with tcsh or zsh or fish or something, it may be even more different.
A really old userland, or a really stripped down one (e.g., for an embedded system) might not include realpath. In that case, you can use readlink, since the GNU version, as usually, adds everything including a couple kitchen sinks, and can be used as a realpath substitute.
Or, if worst comes to worst, Python has come with a realpath function since 2.2:
$(python -c 'import os,sys; print(os.path.realpath(sys.argv[1]))' "$file")

What shebang to use for Python scripts run under a pyenv virtualenv

When a Python script is supposed to be run from a pyenv virtualenv, what is the correct shebang for the file?
As an example test case, the default Python on my system (OS X) does not have pandas installed. The pyenv virtualenv venv_name does. I tried getting the path of the Python executable from the virtualenv.
pyenv activate venv_name
which python
Output:
/Users/username/.pyenv/shims/python
So I made my example script.py:
#!/Users/username/.pyenv/shims/python
import pandas as pd
print 'success'
But when I tried running the script (from within 'venv_name'), I got an error:
./script.py
Output:
./script.py: line 2: import: command not found
./script.py: line 3: print: command not found
Although running that path directly on the command line (from within 'venv_name') works fine:
/Users/username/.pyenv/shims/python script.py
Output:
success
And:
python script.py # Also works
Output:
success
What is the proper shebang for this? Ideally, I want something generic so that it will point at the Python of whatever my current venv is.

I don't really know why calling the interpreter with the full path wouldn't work for you. I use it all the time. But if you want to use the Python interpreter that is in your environment, you should do:
#!/usr/bin/env python
That way you search your environment for the Python interpreter to use.

As you expected, you should be able to use the full path to the virtual environment's Python executable in the shebang to choose/control the environment the script runs in regardless of the environment of the controlling script.
In the comments on your question, VPfB & you find that the /Users/username/.pyenv/shims/python is a shell script that does an exec $pyenv_python. You should be able to echo $pyenv_python to determine the real python and use that as your shebang.
See also: https://unix.stackexchange.com/questions/209646/how-to-activate-virtualenv-when-a-python-script-starts
Try pyenv virtualenvs to find a list of virtual environment directories.
And then you might find a using shebang something like this:
#!/Users/username/.pyenv/python/versions/venv_name/bin/python
import pandas as pd
print 'success'
... will enable the script to work using the chosen virtual environment in other (virtual or not) environments:
(venv_name) $ ./script.py
success
(venv_name) $ pyenv activate non_pandas_venv
(non_pandas_venv) $ ./script.py
success
(non_pandas_venv) $ . deactivate
$ ./script.py
success
The trick is that if you call out the virtual environment's Python binary specifically, the Python interpreter looks around that binary's path location for the supporting files and ends up using the surrounding virtual environment. (See per *How does virtualenv work?)

If you need to use more shell than you can put in the #! shebang line, you can start the file with a simple shell script which launches Python on the same file.
#!/bin/bash
"exec" "pyenv" "exec" "python" "$0" "$#"
# the rest of your Python script can be written below
Because of the quoting, Python doesn't execute the first line, and instead joins the strings together for the module docstring... which effectively ignores it.
You can see more here.

To expand this to an answer, yes, in 99% of the cases if you have a Python executable in your environment, you can just use:
#!/usr/bin/env python
However, for a custom venv on Linux following the same syntax did not work for me since the venv created a link to the Python interpreter which the venv was created from, so I had to do the following:
#!/path/to/the/venv/bin/python
Essentially, however, you are able to call the Python interpreter in your terminal. This is what you would put after #!.

It's not exactly answering the question, but this suggestion by ephiement I think is a much better way to do what you want. I've elaborated a bit and added some more of an explanation as to how this works and how you can dynamically select the Python executable to use:
#!/bin/sh
#
# Choose the Python executable we need. Explanation:
# a) '''\' translates to \ in shell, and starts a python multi-line string
# b) "" strings are treated as string concatenation by Python; the shell ignores them
# c) "true" command ignores its arguments
# c) exit before the ending ''' so the shell reads no further
# d) reset set docstrings to ignore the multiline comment code
#
"true" '''\'
PREFERRED_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python
ALTERNATIVE_PYTHON=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
FALLBACK_PYTHON=python3
if [ -x $PREFERRED_PYTHON ]; then
echo Using preferred python $ALTERNATIVE_PYTHON
exec $PREFERRED_PYTHON "$0" "$#"
elif [ -x $ALTERNATIVE_PYTHON ]; then
echo Using alternative python $ALTERNATIVE_PYTHON
exec $ALTERNATIVE_PYTHON "$0" "$#"
else
echo Using fallback python $FALLBACK_PYTHON
exec python3 "$0" "$#"
fi
exit 127
'''
__doc__ = """What this file does"""
print(__doc__)
import platform
print(platform.python_version())

If you want just a single script with a simple selection of your pyenv virtualenv, you may use a Bash script with your source as a heredoc as follows:
#!/bin/bash
PYENV_VERSION=<your_pyenv_virtualenv_name> python - $# <<EOF
import sys
print(sys.argv)
exit
EOF
I did some additional testing. The following works too:
#!/usr/bin/env -S PYENV_VERSION=<virtual_env_name> python

/usr/bin/env python won't work, since it doesn't know about the virtual environment.
Assuming that you have main.py living next to a ./venv directory, you need to use Python from the venv directory. Or in other words, use this shebang:
#!venv/bin/python
Now you can do:
./main.py

Maybe you need to check the file privileges:
sudo chmod +x script.py

Execute a python script as a command

I'm writing a python script which will be placed in a location. I want to execute it just like a command. for ex.
$ find_branch test
where find_branch is a script placed in anywhere in the system.
I would like to know how to achieve this. I can run it on the place where the script is present by chmod u+x on the script and removing the .py from the script

sudo nano /usr/bin/testpyscript
Then inside the script:
#!/usr/bin/python
print("I'm a python script")
Give it x permission:
sudo chmod +x /usr/bin/testpyscript
Now you can use it as a regular command:
bash-4.2$ testpyscript
I'm a python script
It doesn't have to be exactly at /usr/bin, any location that is inside your $PATH will do. Let's say you want it to be located at some folder inside your home directory, you could do something like this:
pwd
/home/brunorb
mkdir somedir
sudo mv /usr/bin/testpyscript somedir/
export PATH=$PATH:/home/brunorb/somedir/
testpyscript # from any folder in the system
I'm a python script

Make sure python has been added to your path and #!/usr/bin/python is located at the top of your script.
Note You could just try adding your script to your /usr/local/bin/ directory and give it the proper permissions.
sudo cp <your script> /usr/local/bin/

You have a number of options on how to achieve this.
Add the location where you put the script to your PATH environment variable, for example in your ~/.bashrc script:
export PATH="${PATH}:/folder/where/you/put/the/script"
Install the script to a location that is already on your path. It does not have to be a system folder like /usr/bin. Many default Bash setups will include ~/bin in your PATH.
Give the full path to your script on the command line:
/folder/where/you/put/the/script/find_branch test
Run the script through Python. This is very similar to option #2:
python /folder/where/you/put/the/script/find_branch test
Create an alias for the script in your environment. In bash you would do something like the following in your ~/.bashrc:
alias find_branch='/folder/where/you/put/the/script/find_branch'
OR
alias find_branch='python /folder/where/you/put/the/script/find_branch'
For options #1, #2, #3 and #5a to work properly, you should have a shebang with the version of python as the first line of the script. Any of the following will do, depending on how you have/want your environment set up:
#!/usr/bin/python
#!/usr/bin/python2
#!/usr/bin/python3
#!/usr/bin/env python
#!/usr/bin/env python2
#!/usr/bin/env python3
Finally, you do not have to remove the .py extension from the script if you do not want to. Many bash scripts have a .sh extension, for example, which does not prevent them from running as-is. You just have to include the extension in the name of the script when you run it.