Environment:
Windows: 10
Python: 2.7.13 and 3.8.1
default Python launcher: py -3
Default python: 2.7.13
> python -V
> Python 2.7.13
> py -3 -V
> Python 3.8.1
launcher.py:
import subprocess
subprocess.call(['python', '-V'])
Test 1: py -3 launcher.py
Output: python 3.8.1 (HOW!)
Test 2: py -2 launcher.py
Output: Python 2.7.13
The output should be Python 2.7.13 only, even the launcher runs under py 3!
note that, adding shell=True will work, but the idea is not to use it, and if I ran
subprocess.call(['python', 'script_under_py_2.py']) # Will run python 3 with script python 2!
Thanks
Adam
This behavior is due to the inconsistent handling of the PATH environment variable when calling Popen on Windows and Unix.
Windows creates its sub-processes using CreateProcess function. The search path for CreateProcess includes parent process directory, which may be the reason why you are having different binaries executed.
On Windows, PATH is only considered when shell=True is also passed.
You can learn more about the issue here:
subprocess PATH semantics and portability (notice the "needs patch" stage)
If the problem is that subprocess.call is failing to honour os.environ['PATH'] then you could always explicitly search that path for the executable you want to call, and then call it by its absolute path, as follows:
# Manually search the operating system's PATH for python[.exe]
import os, sys
ospath = os.environ['PATH'].split(os.path.pathsep)
_, extension = os.path.splitext(sys.executable)
target = 'python' + extension
matches = [os.path.join(directory, target) for directory in ospath]
matches = [match for match in matches if os.path.isfile(match)]
if not matches: raise OSError('{!r} was not found in any PATH directory'.format(target))
python = matches[0]
# Test:
import subprocess
print('I am Python {} and I am calling "{}" -V'.format(sys.version.split()[0], python))
subprocess.call([python, '-V'])
Of course, that won't do what you want if the py -3 wrapper works by actually changing the path before launching Python... I'm not familiar with that wrapper so I don't know whether that will be a problem.
I have written a python script which depends on paramiko to work. The system that will run my script has the following limitations:
No internet connectivity (so it can't download dependencies on the fly).
Volumes are mounted with 'noexec' (so I cannot run my code as a binary file generated with something like 'pyInstaller')
End-user cannot be expected to install any dependencies.
Vanilla python is installed (without paramiko)
Python version is 2.7.5
Pip is not installed either and cannot be installed on the box
I however, have access to pip on my development box (if that helps in any way).
So, what is the way to deploy my script so that I am able to provide it to the end-user with the required dependencies, i.e paramiko (and sub-dependencies of paramiko), so that the user is able to run the script out-of-the-box?
I have already tried the pyinstaller 'one folder' approach but then faced the 'noexec' issue. I have also tried to directly copy paramiko (and sub-dependencies of paramiko) to a place from where my script is able to find it with no success.
pip is usually installed with python installation. You could use it to install the dependencies on the machine as follows:
import os, sys
def selfInstallParamiko():
# assuming paramiko tar-ball/wheel is under the current working directory
currentDir = os.path.abspath(os.path.dirname(__file__))
# paramikoFileName: name of the already downloaded paramiko tar-ball,
# which you'll have to ship with your scripts
paramikoPath = os.path.join(currentDir, paramikoFileName)
print("\nInstalling {} ...".format(paramikoPath))
# To check for which pip to use (pip for python2, pip3 for python3)
pipName = "pip"
if sys.version_info[0] >= 3:
pipName = "pip3"
p = subprocess.Popen("{} install --user {}".format(pipName, paramikoPath).split())
out, err= p.communicate("")
if err or p.returncode != 0:
print("Unable to self-install {}\n".format(paramikoFileName))
sys.exit(1)
# Needed, because sometimes windows command shell does not pick up changes so good
print("\n\nPython paramiko module installed successfully.\n"
"Please start the script again from a new command shell\n\n")
sys.exit()
You can invoke it when your script starts and an ImportError occurs:
# Try to self install on import failure:
try:
import paramiko
except ImportError:
selfInstallParamiko()
Suppose I have a file called a_python2_script.py with content
print "Hello"
and call it in Python 3 with:
from subprocess import Popen
Popen(["python2", "a_python2_script.py"])
This works, but I do not want to hardcode python2 interpreter call, since the program should run on Windows and Linux.
I know about sys.executable but that would give me the path of the current (i.e. Python 3) interpreter. What is the pythonic way to get the Python 2 interpreter path from within the Python 3 script?
Note: I am calling an external Python 2 library and have no chance to convert it to Python 3.
Use os.get_exec_path() (available from 3.2) to get a list of directories that will be searched for an executable.
In [1]: import os
In [2]: paths = os.get_exec_path()
In [5]: for p in paths:
...: for prog in ["python2", "python2.exe"]:
...: attempt = os.path.join(p, prog)
...: if os.path.isfile(attempt):
...: print(attempt)
...:
/usr/local/bin/python2
This might be one possible way to figure it out. I am currently running Windows and have Python installed using Anaconda and Anaconda3. My PATH contains both of these folders.
A simple script could check each PATH location for the presence of the Python executable. You could then execute python --version at each location:
import sys, os
from subprocess import Popen
python_paths = []
for path in os.environ["PATH"].split(os.pathsep):
for executable in ["python", "python.exe", "python2"]:
if os.path.isfile(os.path.join(path, executable)):
python_paths.append(path)
for path in python_paths:
Popen([os.path.join(path, "python"), "--version"])
print(python_paths)
On my Windows machine this gives:
['d:\\Anaconda', 'd:\\Anaconda3']
Python 2.7.9 :: Anaconda 2.2.0 (32-bit)
Python 3.4.3 :: Anaconda 2.3.0 (32-bit)
Tested using Python 3.4.3. I am not currently able to check the outcome for Linux though.
I'm working on several Python projects who run on various versions of Python. I'm hoping to set up my vim environment to use ropevim, pyflakes, and pylint but I've run into some issues caused by using a single vim (compiled for a specific version of Python which doesn't match the project's Python version).
I'm hoping to build vim into each of my virtualenv directories but I've run into an issue and I can't get it to work. When I try to build vim from source, despite specifying the Python config folder in my virtualenv, the system-wide Python interpreter is always used.
Currently, I have Python 2.6.2 and Python 2.7.1 installed with several virtualenvs created from each version. I'm using Ubuntu 10.04 where the system-default Python is 2.6.5. Every time I compile vim and call :python import sys; print(sys.version) it returns Python 2.6.5.
configure --prefix=/virtualenv/project --enable-pythoninterp=yes --with-python-config-dir=/virtualenv/project/lib/python2.6/config
Results in the following in config.log:
...
configure:5151: checking --enable-pythoninterp argument
configure:5160: result: yes
configure:5165: checking for python
configure:5195: result: /usr/bin/python
...
It should be /virtualenv/project/bin/python. Is there any way to specify the Python interpreter for vim to use?
NOTE: I can confirm that /virtualenv/project/bin appears at the front of PATH environment variable.
I'd recommend building vim against the 2 interpreters, then invoking it using the shell script I provided below to point it to a particular virtualenv.
I was able to build vim against Python 2.7 using the following command (2.7 is installed under $HOME/root):
% LD_LIBRARY_PATH=$HOME/root/lib PATH=$HOME/root/bin:$PATH \
./configure --enable-pythoninterp \
--with-python-config-dir=$HOME/root/lib/python2.7/config \
--prefix=$HOME/vim27
% make install
% $HOME/bin/vim27
:python import sys; print sys.path[:2]
['/home/pat/root/lib/python27.zip', '/home/pat/root/lib/python2.7']
Your virtualenv is actually a thin wrapper around the Python interpreter it was created with -- $HOME/foobar/lib/python2.6/config is a symlink to /usr/lib/python2.6/config.
So if you created it with the system interpreter, VIM will probe for this and ultimately link against the real interpreter, using the system sys.path by default, even though configure will show the virtualenv's path:
% PATH=$HOME/foobar/bin:$PATH ./configure --enable-pythoninterp \
--with-python-config-dir=$HOME/foobar/lib/python2.6/config \
--prefix=$HOME/foobar
..
checking for python... /home/pat/foobar/bin/python
checking Python's configuration directory... (cached) /home/pat/foobar/lib/python2.6/config
..
% make install
% $HOME/foobar/bin/vim
:python import sys; print sys.path[:1]
['/usr/lib/python2.6']
The workaround: Since your system vim is most likely compiled against your system python, you don't need to rebuild vim for each virtualenv: you can just drop a shell script named vim in your virtualenv's bin directory, which extends the PYTHONPATH before calling system vim:
Contents of ~/HOME/foobar/bin/vim:
#!/bin/sh
ROOT=`cd \`dirname $0\`; cd ..; pwd`
PYTHONPATH=$ROOT/lib/python2.6/site-packages /usr/bin/vim $*
When that is invoked, the virtualenv's sys.path is inserted:
% $HOME/foobar/bin/vim
:python import sys; print sys.path[:2]
['/home/pat/foobar/lib/python2.6/site-packages', '/usr/lib/python2.6']
For what it's worth, and no one seems to have answered this here, I had some luck using a command line like the following:
vi_cv_path_python=/usr/bin/python26 ./configure --includedir=/usr/include/python2.6/ --prefix=/home/bcrowder/local --with-features=huge --enable-rubyinterp --enable-pythoninterp --disable-selinux --with-python-config-dir=/usr/lib64/python2.6/config
I would like to give a similar solution to crowder's that works quite well for me.
Imagine you have Python installed in /opt/Python-2.7.5 and that the structure of that folder is
$ tree -d -L 1 /opt/Python-2.7.5/
/opt/Python-2.7.5/
├── bin
├── include
├── lib
└── share
and you would like to build vim with that version of Python. All you need to do is
$ vi_cv_path_python=/opt/Python-2.7.5/bin/python ./configure --enable-pythoninterp --prefix=/SOME/FOLDER
Thus, just by explicitly giving vi_cv_path_python variable to configure the script will deduce everything on it's own (even the config-dir).
This was tested multiple times on vim 7.4+ and lately with vim-7-4-324.
I was having this same issue with 3 different versions of python on my system.
for me the easiest thing was to change my $PATH env variable so that the folder that has the version of python I wanted was (in my case /usr/local/bin) was found before another.
During my compiling vim80, the system python is 2.6, I have another python 2.7 under ~/local/bin, I find that, to make the compiling work:
update $PATH to place my python path ahead
add a soft link, ln -s python python2 ( the configure file would try to locate python config by probing python2 )
make distclean before re-run ./configure to make sure no cached wrong value is picked.
Is it possible to determine if the current script is running inside a virtualenv environment?
The most reliable way to check for this is to check whether sys.prefix == sys.base_prefix. If they are equal, you are not in a virtual environment; if they are unequal, you are. Inside a virtual environment, sys.prefix points to the virtual environment, and sys.base_prefix is the prefix of the system Python the virtualenv was created from.
The above always works for Python 3 stdlib venv and for recent virtualenv (since version 20). Older versions of virtualenv used sys.real_prefix instead of sys.base_prefix (and sys.real_prefix did not exist outside a virtual environment), and in Python 3.3 and earlier sys.base_prefix did not ever exist. So a fully robust check that handles all of these cases could look like this:
import sys
def get_base_prefix_compat():
"""Get base/real prefix, or sys.prefix if there is none."""
return getattr(sys, "base_prefix", None) or getattr(sys, "real_prefix", None) or sys.prefix
def in_virtualenv():
return get_base_prefix_compat() != sys.prefix
If you only care about supported Python versions and latest virtualenv, you can replace get_base_prefix_compat() with simply sys.base_prefix.
Using the VIRTUAL_ENV environment variable is not reliable. It is set by the virtualenv activate shell script, but a virtualenv can be used without activation by directly running an executable from the virtualenv's bin/ (or Scripts) directory, in which case $VIRTUAL_ENV will not be set. Or a non-virtualenv Python binary can be executed directly while a virtualenv is activated in the shell, in which case $VIRTUAL_ENV may be set in a Python process that is not actually running in that virtualenv.
Try using pip -V (notice capital V)
If you are running the virtual env. it'll show the path to the env.'s location.
This is an improvement of the accepted answer by Carl Meyer. It works with virtualenv for Python 3 and 2 and also for the venv module in Python 3:
import sys
def is_venv():
return (hasattr(sys, 'real_prefix') or
(hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix))
The check for sys.real_prefix covers virtualenv, the equality of non-empty sys.base_prefix with sys.prefix covers venv.
Consider a script that uses the function like this:
if is_venv():
print('inside virtualenv or venv')
else:
print('outside virtualenv or venv')
And the following invocation:
$ python2 test.py
outside virtualenv or venv
$ python3 test.py
outside virtualenv or venv
$ python2 -m virtualenv virtualenv2
...
$ . virtualenv2/bin/activate
(virtualenv2) $ python test.py
inside virtualenv or venv
(virtualenv2) $ deactivate
$ python3 -m virtualenv virtualenv3
...
$ . virtualenv3/bin/activate
(virtualenv3) $ python test.py
inside virtualenv or venv
(virtualenv3) $ deactivate
$ python3 -m venv venv3
$ . venv3/bin/activate
(venv3) $ python test.py
inside virtualenv or venv
(venv3) $ deactivate
Check the $VIRTUAL_ENV environment variable.
The $VIRTUAL_ENV environment variable contains the virtual environment's directory when in an active virtual environment.
>>> import os
>>> os.environ['VIRTUAL_ENV']
'/some/path/project/venv'
Once you run deactivate / leave the virtual environment, the $VIRTUAL_ENV variable will be cleared/empty. Python will raise a KeyError because the environment variable was unset.
>>> import os
>>> os.environ['VIRTUAL_ENV']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py", line 678, in __getitem__
raise KeyError(key) from None
KeyError: 'VIRTUAL_ENV'
These same environment variable checks can of course also be done outside of the Python script, in the shell.
There are multiple good answers here, and some less robust ones.
Here's an overview.
How not to do it
Do not rely on on the location of Python or the site-packages folder.
If these are set to non-standard locations, that does not mean
you're actually in a virtual environment. Users can have more than one
Python version installed, and those are not always where you expect them to be.
Avoid looking at:
sys.executable
sys.prefix
pip -V
which python
Also, do not check for the presence of venv, .venv or envs in any of these paths.
This will break for environments with a more unique location. For example,
Pipenv uses hash values as the name for its environments.
VIRTUAL_ENV environment variable
Both virtualenv and venv set the environment variable $VIRTUAL_ENV when activating an environment.
See PEP 405.
You can read out this variable in shell scripts, or use this Python code to determine if it's set.
import os
running_in_virtualenv = "VIRTUAL_ENV" in os.environ
# alternative ways to write this, also supporting the case where
# the variable is set but contains an empty string to indicate
# 'not in a virtual environment':
running_in_virtualenv = bool(os.environ.get("VIRTUAL_ENV"))
running_in_virtualenv = bool(os.getenv("VIRTUAL_ENV"))
The problem is, this only works when the environment is activated by the activate shell script.
You can start the environment's scripts without activating the environment, so if that is a concern, you have to use a different method.
sys.base_prefix
virtualenv, venv and pyvenv point sys.prefix to the Python installed inside of the virtualenv as you would expect.
At the same time, the original value of sys.prefix is also made available as sys.base_prefix.
We can use that to detect if we're in a virtualenv.
import sys
# note: Python versions before 3.3 don't have sys.base_prefix
# if you're not in virtual environment
running_in_virtualenv = sys.prefix != sys.base_prefix
Fallback: sys.real_prefix
Now watch out, virtualenv before version 20 did not set sys.base_prefix but it set sys.real_prefix instead.
So to be safe, check both as suggested in hroncok's answer:
import sys
real_prefix = getattr(sys, "real_prefix", None)
base_prefix = getattr(sys, "base_prefix", sys.prefix)
running_in_virtualenv = (base_prefix or real_prefix) != sys.prefix
Anaconda
If you're using Anaconda virtual environments, check
Victoria Stuart's answer.
According to the virtualenv pep at http://www.python.org/dev/peps/pep-0405/#specification you can just use sys.prefix instead os.environ['VIRTUAL_ENV'].
the sys.real_prefix does not exist in my virtualenv and same with sys.base_prefix.
To check whether your inside Virtualenv:
import os
if os.getenv('VIRTUAL_ENV'):
print('Using Virtualenv')
else:
print('Not using Virtualenv')
You can also get more data on your environment:
import sys
import os
print(f'Python Executable: {sys.executable}')
print(f'Python Version: {sys.version}')
print(f'Virtualenv: {os.getenv("VIRTUAL_ENV")}')
You can do which python and see if its pointing to the one in virtual env.
Updated Nov 2019 (appended).
I routinely use several Anaconda-installed virtual environments (venv). This code snippet/examples enables you to determine whether or not you are in a venv (or your system environment), and to also require a specific venv for your script.
Add to Python script (code snippet):
# ----------------------------------------------------------------------------
# Want script to run in Python 3.5 (has required installed OpenCV, imutils, ... packages):
import os
# First, see if we are in a conda venv { py27: Python 2.7 | py35: Python 3.5 | tf: TensorFlow | thee : Theano }
try:
os.environ["CONDA_DEFAULT_ENV"]
except KeyError:
print("\tPlease set the py35 { p3 | Python 3.5 } environment!\n")
exit()
# If we are in a conda venv, require the p3 venv:
if os.environ['CONDA_DEFAULT_ENV'] != "py35":
print("\tPlease set the py35 { p3 | Python 3.5 } environment!\n")
exit()
# See also:
# Python: Determine if running inside virtualenv
# http://stackoverflow.com/questions/1871549/python-determine-if-running-inside-virtualenv
# [ ... SNIP! ... ]
Example:
$ p2
[Anaconda Python 2.7 venv (source activate py27)]
(py27) $ python webcam_.py
Please set the py35 { p3 | Python 3.5 } environment!
(py27) $ p3
[Anaconda Python 3.5 venv (source activate py35)]
(py35) $ python webcam.py -n50
current env: py35
processing (live): found 2 faces and 4 eyes in this frame
threaded OpenCV implementation
num_frames: 50
webcam -- approx. FPS: 18.59
Found 2 faces and 4 eyes!
(py35) $
Update 1 -- use in bash scripts:
You can also use this approach in bash scripts (e.g., those that must run in a specific virtual environment). Example (added to bash script):
if [ $CONDA_DEFAULT_ENV ] ## << note the spaces (important in BASH)!
then
printf 'venv: operating in tf-env, proceed ...'
else
printf 'Note: must run this script in tf-env venv'
exit
fi
Update 2 [Nov 2019]
For simplicity, I like Matt's answer (https://stackoverflow.com/a/51245168/1904943).
Since my original post I've moved on from Anaconda venv (and Python itself has evolved viz-a-viz virtual environments).
Reexamining this issue, here is some updated Python code that you can insert to test that you are operating in a specific Python virtual environment (venv).
import os, re
try:
if re.search('py37', os.environ['VIRTUAL_ENV']):
pass
except KeyError:
print("\n\tPlease set the Python3 venv [alias: p3]!\n")
exit()
Here is some explanatory code.
[victoria#victoria ~]$ date; python --version
Thu 14 Nov 2019 11:27:02 AM PST
Python 3.8.0
[victoria#victoria ~]$ python
Python 3.8.0 (default, Oct 23 2019, 18:51:26)
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, re
>>> re.search('py37', os.environ['VIRTUAL_ENV'])
<re.Match object; span=(20, 24), match='py37'>
>>> try:
... if re.search('py37', os.environ['VIRTUAL_ENV']):
... print('\n\tOperating in Python3 venv, please proceed! :-)')
... except KeyError:
... print("\n\tPlease set the Python3 venv [alias: p3]!\n")
...
Please set the Python3 venv [alias: p3]!
>>> [Ctrl-d]
now exiting EditableBufferInteractiveConsole...
[victoria#victoria ~]$ p3
[Python 3.7 venv (source activate py37)]
(py37) [victoria#victoria ~]$ python --version
Python 3.8.0
(py37) [victoria#victoria ~]$ env | grep -i virtual
VIRTUAL_ENV=/home/victoria/venv/py37
(py37) [victoria#victoria ~]$ python
Python 3.8.0 (default, Oct 23 2019, 18:51:26)
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, re
>>> try:
... if re.search('py37', os.environ['VIRTUAL_ENV']):
... print('\n\tOperating in Python3 venv, please proceed! :-)')
... except KeyError:
... print("\n\tPlease set the Python3 venv [alias: p3]!\n")
...
Operating in Python3 venv, please proceed! :-)
>>>
Easiest way is to just run: which python, if you are in a virtualenv it will point to its python instead of the global one
(edited) I found that way, what do you think of it ? (it also returns the venv base path and works even for readthedocs where checking the env variable does not):
import os
import sys
from distutils.sysconfig import get_config_vars
def get_venv_basedir():
"""Returns the base directory of the virtualenv, useful to read configuration and plugins"""
exec_prefix = get_config_vars()['exec_prefix']
if hasattr(sys, 'real_prefix') is False or exec_prefix.startswith(sys.real_prefix):
raise EnvironmentError('You must be in a virtual environment')
return os.path.abspath(get_config_vars()['exec_prefix'] + '/../')
In windows OS you see something like this:
C:\Users\yourusername\virtualEnvName\Scripts>activate
(virtualEnvName) C:\Users\yourusername\virtualEnvName\Scripts>
Parentheses mean that you are actually in the virtual environment called "virtualEnvName".
There are a lot of great methods posted here already, but just adding one more:
import site
site.getsitepackages()
tells you where pip installed the packages.
If you are using Anaconda here is the solution. This command lists all the discoverable environments
conda info --envs
I typically use a bash wrapper script to start my application and was running into the problem of imports not working because of pip packages not being installed outside the virtual environment.
Using the solution provided by Matt Harasymczuk, here are the few lines of bash script to check the virtual environment has been activated before starting the application.
#!/bin/bash
# check the virtual environment is active
if [ -z "$VIRTUAL_ENV" ]
then
echo "[ERROR] Virtual environment not activated"
exit 1
fi
# start the application
python3 # amend with application name and other details
# end of file
You can look for the 'signature' of whatever venv method you're trying to support. In my case I want to support:
python -m venv venv_dir
or
virtualenv venv_dir
So I can write:
import sys
from pathlib import Path
IS_VENV_ENVIRONMENT = (Path(sys.prefix) / "pyvenv.cfg").exists()
See: https://docs.python.org/3/library/venv.html
"Running this command creates the target directory (creating any parent directories that don’t exist already) and places a pyvenv.cfg file in it"
NB: virtualenv also creates the same file but I couldn't see any docs about that, I only noticed by experiment.
It's not bullet-proof but for UNIX environments simple test like
if run("which python3").find("venv") == -1:
# something when not executed from venv
works great for me. It's simpler then testing existing of some attribute and, anyway, you should name your venv directory venv.
A potential solution is:
os.access(sys.executable, os.W_OK)
In my case I really just wanted to detect if I could install items with pip as is. While it might not be the right solution for all cases, consider simply checking if you have write permissions for the location of the Python executable.
Note: this works in all versions of Python, but also returns True if you run the system Python with sudo. Here's a potential use case:
import os, sys
can_install_pip_packages = os.access(sys.executable, os.W_OK)
if can_install_pip_packages:
import pip
pip.main(['install', 'mypackage'])