I'm using a virtualenv to execute a script, in this script I call:
os.system('python anotherScript.py')
My question is whether the script is executed in the same virtualenv as the caller script?
It's hard to tell, but if you are running this script under an activated virtualenv, you should be under that virutla environment. You can verify your thought by doing
#script.py
import os
os.system('which python')
and from command-line
virtualenv newvirtualenv
source newvirtualenv/bin/activate
(newvirtualenv) user#ubuntu: python script.py
you should see it is under newvirtualenv/bin/python
Usually, you want to put an exectuable header to use the current environment:
#!/usr/bin/env python
import os
os.system('which python')
This does not say use newvirtualenv, but gives you a little more confident if the script is executed under newvirtualenv, it will definitely be newvirtualenv.
If you use /usr/bin/python this is still okay under virtualenv. But for advanced programmers, they tend to have multiple virtual environments and multiple python version. So depending on where they are, they can execute the script based on the environment variable. Just a small gain.
If you run newvirtualenv/bin/python script.py it will be under virtualenv regardless.
As long as the python binary is pointing at the virtualenv's version, you are good.
e.g. use anaconda to manage virtual envs, and in Pycharm IDE:
os.system('which python') # /usr/bin/python
command = 'python3 xxxx.py'
os.system(command) # make /usr/bin/python as interpreter
If I want to use some modules (e.g. cv2) installed in certain virtual env,
command = '/path/to/anaconda3/envs/your_env_name/bin/python3 xxxx.py'
os.system(command)
Related
I have two python executable, one in Anaconda env and one in WSL(Obvisouly, I'm using Windows).
And there are two files, one is test.sh and the other is test.py.
Both files are simple (I made those for troubleshooting)
# test.sh
python test.py
# test.py
import sys
print(sys.executable)
When I execute test.py, the path is like this.
C:\Users\user\anaconda3\envs\dassl\python.exe
However, when I execute test.sh, the path is like this (Obviously it points to the python in WSL)
/usr/bin/python
Even though I don't know much about WSL, I guess that by using bash command in WSL, python in WSL rather than the one in the anaconda is pointed. I'm struggling with this problem because I can't use pytorch package installed in conda env with a bash command.
Is it impossible to use conda with base command of WSL?
Consider the following script run in a Python 3.x virtual environment on Windows 10:
#!/usr/bin/env python
import sys, os, subprocess
print(sys.executable)
args = ["python", "-c", "import sys; print(sys.executable)"]
p = subprocess.Popen(args, stdout=None, env=os.environ)
p.wait()
On most systems, the two calls to print(sys.executable) return a string showing the location of python.exe within the virtual environment, e.g.
C:> python test.py
C:\venv\Scripts\python.exe
C:\venv\Scripts\python.exe
But occasionally, we'll encounter a machine where the first call finds python.exe from the virtual environment, but the second call finds python.exe from the machine's global installation, e.g.
C:> python test.py
C:\venv\Scripts\python.exe
C:\Users\me\AppData\Local\Programs\Python\Python38\python.exe
The program above is simplified to make it easy to read. The real program uses subprocess.popen to call another python script whose imports fail because the global install doesn't have the same packages as the virtual install.
I suspected the PATH environment variable, but it appears to be the same when printed before and during the subprocess.popen call. In both cases, the venv/Scripts folder exists ahead of the global Python install folder.
I've also tried deleting the virtual environment and recreating it. No luck.
I'm developing a set of script in python3, as shebang I use this:
#!/usr/bin/env python3
Everything goes ok, but in some virtual machines where are executed the name of interpreter is python3.5. I will like to be able to execute my scripts in both enviroment but I can't change the filesystem of virtual machine (so I discard solutions like make a link from python3.5 to python3 )
I look at man of env but I don't find any way to specify a searching pattern or something like that.
I try to set an alias at begining of my sessions pointing to right python interpreter but env don't use it.
My unique solution is call my scripts saying which interpreter must use but is very anoying:
python3.5 myscript.py
Any idea is welcome!, thanks!
No need to bring in separate shell and python scripts, a single file can be both!
Replace your shebang line with this sequence:
#!/bin/sh
# Shell commands follow
# Next line is bilingual: it starts a comment in Python, and is a no-op in shell
""":"
# Find a suitable python interpreter (adapt for your specific needs)
for cmd in python3.5 python3 /opt/myspecialpython/bin/python3.5.99 ; do
command -v > /dev/null $cmd && exec $cmd $0 "$#"
done
echo "OMG Python not found, exiting!!!!!11!!eleven" >2
exit 2
":"""
# Previous line is bilingual: it ends a comment in Python, and is a no-op in shell
# Shell commands end here
# Python script follows (example commands shown)
import sys
print ("running Python!")
print (sys.argv)
If you can install scripts, you can also install a wrapper called python3.5 which simply dispatches python3.
#!/bin/sh
exec env python3 "$#"
You'll obviously need to chmod a+x this script just like the others you install.
You'll have to add the script's directory to your PATH after the system python3.5 directory to avoid having this go into an endless loop, and only use this script as a fallback when the system doesn't already provide python3.5.
As you noted, env doesn't know or care about your personal shell aliases or functions, and doesn't provide for any dynamic calculation of the binary to run by itself; but you have the shell at your disposal (and Python of course, once you find it!) -- it simply uses the PATH so if you can install your other scripts in a directory which is in your PATH (which must be the case for the #!/usr/bin/env shebang to make sense in the first place) you can store this script there, too.
As noted in comments, it's weird and user-hostile to only install python3.5 and not at least optionally make python3 a symlink to it, so perhaps you could eventually persuade whoever maintains the image you are installing into to provide this.
You could create a shell script that uses python 3.5 if it is installed, otherwise uses python 3 and executes your script with the correct version.
No need for python shebang.
In your shell script you may test if which python3.5 returns something; if it does, then python3.5 is installed, otherwise you'd have to use python3
I am running a program using virtualenv. But the multiprocessing.Process spawned here uses system python by default. How do I force it to use virtualenv python.
import os
from multiprocessing import Process
def function1():
# do_something_here
p = Process(func2(), args=(param,))
p.start()
return something
def func2(param):
os.system("which python")
Here it prints "/usr/bin/python". But I need it to use virtualenv python instead.
With sudo venv/bin/python, you effectively activated virtualenv by using python executable in virtualenv directly.
multiprocessing.Process spawn child process with fork(), without exec(), it uses exactly the same python executable as the parent process.
You could confirm python executable in-use by:
>>> import sys
>>> print(sys.executable)
/Users/georgexsh/workspace/tmp/venv/bin/python
>>> print(sys.exec_prefix)
/Users/georgexsh/workspace/tmp/venv/bin/..
Do not use which to determine running python executable path. which, as a Bash command, searches each element of $PATH, for a directory containing an executable file named "python", as you use virtualenv's python directly, not by run its shell activate script first, $PATH not get patched with virtualenv's, as a result, shell command which python would output path of the system python executable.
In fact, at python layer, $PATH is irrelevant, patching $PATH is for the convenience at the Bash layer, to invoke python executable in the virtualenv path by just typing "python", rather than typing full path, What matters most is which python executable is invoked, but not how it is get invoked.
Your problem is here (copied your comment):
#georgexsh I was running it using sudo. By default if you use sudo
then it will use system python. So, I have used "sudo venv/bin/python
main.py" to run the program. Even though I am using venv's python here
it returns "/usr/bin/python" for "os.system('which python')". I don't
understand this behaviour
Basically, what you explain here is something where your virtualenv is not active.
When you activate a virtualenv (. venv/bin/activate), the activation script will change your environment so that your PYTHONPATH is correct and Python executable is searched (and found) first in the virtual env directory. This is what virtualenv does.
By just executing the Python binary from virtualenv directories, your environment is not set for the virtual environment, so any subsequent calls to Python use your default path - as virtualenv is not there to override it.
When you execute sudo, a new process/shell is created and it does not inherit your virtual environment. You might be able to use sudo -E to pass environment but it depends on your sudo. The bulletproof version that should work in every environment is to execute a shell that first activates virtualenv and then executes your script. Something like this:
sudo -- bash -c ". /home/test/mytest/bin/activate; which python"
This executes a bash shell as root, then activates the virtual environment and finally tells you which python it uses. Just modify the above command with your virtual environment path and it might even work.
If your system is shared, just keep in mind that this is a horrible thing to allow your regular users do from security perspective. If you create a passwordless sudo for your regular users to do this, it would give them root access with little tweaking. If it is your own system and the requirement is the knowledge of root password anyway, it does not matter.
I have 2 versions of python installed on windows, 2.7.3 and 3.3. Some of my scripts are 2.x and some 3.x. Is there an easy way when executing these scripts from a command line to direct them to the appropriate interpreter?
Note: For Windows use the new Windows Python launcher (available with Python 3.3 and downloadable here for earlier releases) which recognizes Unix shell shebangs. You can read about it here.
Most Linux distributions will create python2 and python3 aliases for the installed Python 2.x and Python 3.x interpreter (if not you can just create the symbolic links yourself anywhere on your $PATH, the env command will take care of finding them), so you should just need to set the appropriate interpreter as the first line of your script:
#!/usr/bin/env python2
or
#!/usr/bin/env python3
This will direct the shell to use the appropriate interpreter, if you set the script files to be executable and just invoke them directly on the shell. E.g.:
$ chmod +x script.py
$ ./script.py
Try this first: I am on OS X, but when I want to use Python 2.6 instead of Python 2.7 (its a numpy/scipy thing), I simply run python2.6 whatever.py to run whatever.py in Python 2.6. Try that first.
If that doesn't work, then you can use virtualenv - the virtual environment builder for Python.
http://pypi.python.org/pypi/virtualenv
I am sure there are similar alternatives, too.
Pedro Romano's answer is the most elegant way of doing this.
But if you don't want to download and install the Python launcher, create a batch file as described below. You can also create a shortcut, copy C:\Python27\python.exe to C:\Python27\python27.exe, etc.
I'm guessing C:\Python27 and C:\Python33 are already on your system path. If so, you can create a batch file called python2.7.bat in C:\Python27\ which contains:
C:\Python27\python.exe %1
and a similar file (e.g. python3.3.bat) in C:\Python33\
Now, you can run python2.7 script.py from anywhere in command prompt and it should work :)