How to run bash commands from Python preferably with the os library? - python

I am trying to run the following python script named test.py. It contains multiple bash commands which I would like to execute in a Linux terminal (unix). This is the content of the file:
import os
os.system('echo install virtualenv')
os.system('sudo pip install virtualenv')
os.system('echo create virtual environment')
os.system('virtualenv my_virtualenvironment')
os.system('echo activate virtual environment')
os.system('source my_virtualenvironment/bin/activate')
I am running the Python script using the following in the terminal:
python3 test.py
The problem that I have is that the commands do not run the same way as they would on a Linux terminal. The output is the following error when trying to execute the last line of the Python script:
sh: 1: source: not found
The last command source my_virtualenvironment/bin/activate normally runs fine if I execute it directly in the terminal (without my Python script). Now, what does sh: 1: mean and why does it not work with my code? I would expect to get something starting with bash: .
Also I have found this solution, but I would like not to use lists for executing commands and maybe even to stick with the os library (if there is a simpler solution without os, I am also open for that):
https://stackoverflow.com/a/62355400/11535508

source is a bash built-in command, not an executable.
Use the full path to the python interpreter in your commands instead of venv activation, e.g. os.system('<venv>/bin/python ...').
The second option is to write your commands into a separate bash script and call it from python:
os.system('bash script.sh')

Related

Vscode not interactive with python. Mac M1 chip

I am on a Mac with M1 chip and I have a problem with my VScode and python. It stays stuck on the ZSH shell even when I type the command to switch to bash (chsh -s /bin/bash). Lets say I run a simple code:
import cowsay
import sys
if len(sys.argv) == 2:
cowsay.cow("Hello, " + sys.argv[1])
I am supposed to be able to type my name in the shell after my python file name and it should print the cow saying Hello, Noah.
When I do so ((base) noahhaitas#Noahs-Mac-mini ~ % python3 itunes.py Noah Haitas), this is what I get as a message in my shell:
(base) noahhaitas#Noahs-Mac-mini ~ % python3 itunes.py Noah Haitas
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3: can't open file '/Users/noahhaitas/itunes.py': [Errno 2] No such file or directory
I am trying to figure out how I can switch fully to bash and have the $ in front instead of seeing the % of ZSH.
What can be done as this is frustrating and I looked everywhere online and tried pretty much every solution.
Vscode is just an editor, and the system terminal is still applied.
Use the following command in the terminal to switch bash and restart the terminal:
chsh -s /bin/bash
At the same time, you can also set manually in the vscode terminal. Take Windows as an example:
Read the docs for more details about terminal settings.
By the way, there is an error when you run the file. It seems that the python file is not run in the correct directory.

How to get python via virtual environment recognized in bash file via shebang

I have a script file:
#!/usr/bin/env python3.9
print("python is working")
However when I try and run it:
(karl-env) karl#Karls-MBP scripts (karl/test) $ . test.sh
bash: test.sh: line 3: syntax error near unexpected token `"python is working"'
bash: test.sh: line 3: `print("python is working")'
Following info:
(karl-env) karl#Karls-MBP scripts (karl/test) $ type -a python
python is /Users/karl/.pyenv/shims/python
python is /Users/karl/.pyenv/shims/python
python is /usr/bin/python
I'm in a virtual environment but I fail to understand how to get my environments python recognized via the shebang #!/usr/bin/env python3.9. I do not use Python often hence my noobiness!
This has a little to do with Python and a lot to do with the shell.
You're doing . test.sh – . is an alias for source, which has your shell attempt to interpret the given script as shell commands you'd enter. You want ./test.sh to execute the script.
Your shebang line is explicitly looking for a python3.9 executable, and your environment might not be Python 3.9, so you fall back to something else. Do python (or python3) instead: #!/usr/bin/env python
For the sake of sanity, rename your script to .py; it's not a .shellscript.

Activating a Python virtual environment and calling python script inside another python script

I am using pipenv for managing my packages. I want to write a python script that calls another python script that uses a different Virtual Environment(VE).
How can I run python script 1 that uses VE1 and call another python script (script2 that uses VE2).
I found this code for the cases where there is no need for changing the virtual environment.
import os
os.system("python myOtherScript.py arg1 arg2 arg3")
The only idea that I had was simply navigating to the target project and activate shell:
os.system("cd /home/mmoradi2/pgrastertime/")
os.system("pipenv shell")
os.system("python test.py")
but it says:
Shell for /home/..........-GdKCBK2j already activated.
No action taken to avoid nested environments.
What should I do now?
in fact my own code needs VE1 and the subprocess (second script) needs VE2. How can I call the second script inside my code?
In addition, the second script is used as a command line tool that accepts the inputs with flags:
python3 pgrastertime.py -s ./sql/postprocess.sql -t brasdor_c_07_0150
-p xml -f -r ../data/brasdor_c_07_0150.object.xml
How can I call it using the solution of #tzaman
Each virtualenv has its own python executable which you can use directly to execute the script.
Using subprocess (more versatile than os.system):
import subprocess
venv_python = '/path/to/other/venv/bin/python'
args = [venv_python, 'my_script.py', 'arg1', 'arg2', 'arg3']
subprocess.run(args)

What shebang to use for Python scripts run under a pyenv virtualenv

When a Python script is supposed to be run from a pyenv virtualenv, what is the correct shebang for the file?
As an example test case, the default Python on my system (OS X) does not have pandas installed. The pyenv virtualenv venv_name does. I tried getting the path of the Python executable from the virtualenv.
pyenv activate venv_name
which python
Output:
/Users/username/.pyenv/shims/python
So I made my example script.py:
#!/Users/username/.pyenv/shims/python
import pandas as pd
print 'success'
But when I tried running the script (from within 'venv_name'), I got an error:
./script.py
Output:
./script.py: line 2: import: command not found
./script.py: line 3: print: command not found
Although running that path directly on the command line (from within 'venv_name') works fine:
/Users/username/.pyenv/shims/python script.py
Output:
success
And:
python script.py # Also works
Output:
success
What is the proper shebang for this? Ideally, I want something generic so that it will point at the Python of whatever my current venv is.
I don't really know why calling the interpreter with the full path wouldn't work for you. I use it all the time. But if you want to use the Python interpreter that is in your environment, you should do:
#!/usr/bin/env python
That way you search your environment for the Python interpreter to use.
As you expected, you should be able to use the full path to the virtual environment's Python executable in the shebang to choose/control the environment the script runs in regardless of the environment of the controlling script.
In the comments on your question, VPfB & you find that the /Users/username/.pyenv/shims/python is a shell script that does an exec $pyenv_python. You should be able to echo $pyenv_python to determine the real python and use that as your shebang.
See also: https://unix.stackexchange.com/questions/209646/how-to-activate-virtualenv-when-a-python-script-starts
Try pyenv virtualenvs to find a list of virtual environment directories.
And then you might find a using shebang something like this:
#!/Users/username/.pyenv/python/versions/venv_name/bin/python
import pandas as pd
print 'success'
... will enable the script to work using the chosen virtual environment in other (virtual or not) environments:
(venv_name) $ ./script.py
success
(venv_name) $ pyenv activate non_pandas_venv
(non_pandas_venv) $ ./script.py
success
(non_pandas_venv) $ . deactivate
$ ./script.py
success
The trick is that if you call out the virtual environment's Python binary specifically, the Python interpreter looks around that binary's path location for the supporting files and ends up using the surrounding virtual environment. (See per *How does virtualenv work?)
If you need to use more shell than you can put in the #! shebang line, you can start the file with a simple shell script which launches Python on the same file.
#!/bin/bash
"exec" "pyenv" "exec" "python" "$0" "$#"
# the rest of your Python script can be written below
Because of the quoting, Python doesn't execute the first line, and instead joins the strings together for the module docstring... which effectively ignores it.
You can see more here.
To expand this to an answer, yes, in 99% of the cases if you have a Python executable in your environment, you can just use:
#!/usr/bin/env python
However, for a custom venv on Linux following the same syntax did not work for me since the venv created a link to the Python interpreter which the venv was created from, so I had to do the following:
#!/path/to/the/venv/bin/python
Essentially, however, you are able to call the Python interpreter in your terminal. This is what you would put after #!.
It's not exactly answering the question, but this suggestion by ephiement I think is a much better way to do what you want. I've elaborated a bit and added some more of an explanation as to how this works and how you can dynamically select the Python executable to use:
#!/bin/sh
#
# Choose the Python executable we need. Explanation:
# a) '''\' translates to \ in shell, and starts a python multi-line string
# b) "" strings are treated as string concatenation by Python; the shell ignores them
# c) "true" command ignores its arguments
# c) exit before the ending ''' so the shell reads no further
# d) reset set docstrings to ignore the multiline comment code
#
"true" '''\'
PREFERRED_PYTHON=/Library/Frameworks/Python.framework/Versions/2.7/bin/python
ALTERNATIVE_PYTHON=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
FALLBACK_PYTHON=python3
if [ -x $PREFERRED_PYTHON ]; then
echo Using preferred python $ALTERNATIVE_PYTHON
exec $PREFERRED_PYTHON "$0" "$#"
elif [ -x $ALTERNATIVE_PYTHON ]; then
echo Using alternative python $ALTERNATIVE_PYTHON
exec $ALTERNATIVE_PYTHON "$0" "$#"
else
echo Using fallback python $FALLBACK_PYTHON
exec python3 "$0" "$#"
fi
exit 127
'''
__doc__ = """What this file does"""
print(__doc__)
import platform
print(platform.python_version())
If you want just a single script with a simple selection of your pyenv virtualenv, you may use a Bash script with your source as a heredoc as follows:
#!/bin/bash
PYENV_VERSION=<your_pyenv_virtualenv_name> python - $# <<EOF
import sys
print(sys.argv)
exit
EOF
I did some additional testing. The following works too:
#!/usr/bin/env -S PYENV_VERSION=<virtual_env_name> python
/usr/bin/env python won't work, since it doesn't know about the virtual environment.
Assuming that you have main.py living next to a ./venv directory, you need to use Python from the venv directory. Or in other words, use this shebang:
#!venv/bin/python
Now you can do:
./main.py
Maybe you need to check the file privileges:
sudo chmod +x script.py

R equivalent to `python -i`

Typing python -i file.py at the command line runs file.py and then drops into the python terminal preserving the run environment.
https://docs.python.org/3/using/cmdline.html
Is there an equivalent in R?
I may be misinterpreting what python -i file.py does, but try:
From inside R, at the terminal, you can do:
source('file.R')
and it will run file.R, with the global environment reflecting what was done in file.R
If you're trying to run from the command line, review this post

Categories

Resources