Develop Reference

Activate virtual environment and run command with single call to subprocess.Popen with shell=False possible? [duplicate]

I want to activate a virtualenv instance from a Python script.
I know it's quite easy to do, but all the examples I've seen use it to run commands within the env and then close the subprocess.
I simply want to activate the virtualenv and return to the shell, the same way that bin/activate does.
Something like this:
$me: my-script.py -d env-name
$(env-name)me:
Is this possible?
Relevant:
virtualenv › Invoking an env from a script

If you want to run a Python subprocess under the virtualenv, you can do that by running the script using the Python interpreter that lives inside virtualenv's /bin/ directory:
import subprocess
# Path to a Python interpreter that runs any Python script
# under the virtualenv /path/to/virtualenv/
python_bin = "/path/to/virtualenv/bin/python"
# Path to the script that must run under the virtualenv
script_file = "must/run/under/virtualenv/script.py"
subprocess.Popen([python_bin, script_file])
However, if you want to activate the virtualenv under the current Python interpreter instead of a subprocess, you can use the activate_this.py script:
# Doing execfile() on this file will alter the current interpreter's
# environment so you can import libraries in the virtualenv
activate_this_file = "/path/to/virtualenv/bin/activate_this.py"
execfile(activate_this_file, dict(__file__=activate_this_file))

The simplest solution to run your script under virtualenv's interpreter is to replace the default shebang line with path to your virtualenv's interpreter like so at the beginning of the script:
#!/path/to/project/venv/bin/python
Make the script executable:
chmod u+x script.py
Run the script:
./script.py
Voila!

It turns out that, yes, the problem is not simple, but the solution is.
First I had to create a shell script to wrap the "source" command. That said I used the "." instead, because I've read that it's better to use it than source for Bash scripts.
#!/bin/bash
. /path/to/env/bin/activate
Then from my Python script I can simply do this:
import os
os.system('/bin/bash --rcfile /path/to/myscript.sh')
The whole trick lies within the --rcfile argument.
When the Python interpreter exits it leaves the current shell in the activated environment.
Win!

To run another Python environment according to the official Virtualenv documentation, in the command line you can specify the full path to the executable Python binary, just that (no need to active the virtualenv before):
/path/to/virtualenv/bin/python
The same applies if you want to invoke a script from the command line with your virtualenv. You don't need to activate it before:
me$ /path/to/virtualenv/bin/python myscript.py
The same for a Windows environment (whether it is from the command line or from a script):
> \path\to\env\Scripts\python.exe myscript.py

Just a simple solution that works for me. I don't know why you need the Bash script which basically does a useless step (am I wrong ?)
import os
os.system('/bin/bash --rcfile flask/bin/activate')
Which basically does what you need:
[hellsing#silence Foundation]$ python2.7 pythonvenv.py
(flask)[hellsing#silence Foundation]$
Then instead of deactivating the virtual environment, just Ctrl + D or exit. Is that a possible solution or isn't that what you wanted?

The top answer only works for Python 2.x
For Python 3.x, use this:
activate_this_file = "/path/to/virtualenv/bin/activate_this.py"
exec(compile(open(activate_this_file, "rb").read(), activate_this_file, 'exec'), dict(__file__=activate_this_file))
Reference: What is an alternative to execfile in Python 3?

The child process environment is lost in the moment it ceases to exist, and moving the environment content from there to the parent is somewhat tricky.
You probably need to spawn a shell script (you can generate one dynamically to /tmp) which will output the virtualenv environment variables to a file, which you then read in the parent Python process and put in os.environ.
Or you simply parse the activate script in using for the line in open("bin/activate"), manually extract stuff, and put in os.environ. It is tricky, but not impossible.

For python2/3, Using below code snippet we can activate virtual env.
activate_this = "/home/<--path-->/<--virtual env name -->/bin/activate_this.py" #for ubuntu
activate_this = "D:\<-- path -->\<--virtual env name -->\Scripts\\activate_this.py" #for windows
with open(activate_this) as f:
code = compile(f.read(), activate_this, 'exec')
exec(code, dict(__file__=activate_this))

I had the same issue and there was no activate_this.py in the Scripts directory of my environment.
activate_this.py
"""By using execfile(this_file, dict(__file__=this_file)) you will
activate this virtualenv environment.
This can be used when you must use an existing Python interpreter, not
the virtualenv bin/python
"""
try:
__file__
except NameError:
raise AssertionError(
"You must run this like execfile('path/to/active_this.py', dict(__file__='path/to/activate_this.py'))")
import sys
import os
base = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
if(sys.platform=='win32'):
site_packages = os.path.join(base, 'Lib', 'site-packages')
else:
site_packages = os.path.join(base, 'lib', 'python%s' % sys.version[:3], 'site-packages')
prev_sys_path = list(sys.path)
import site
site.addsitedir(site_packages)
sys.real_prefix = sys.prefix
sys.prefix = base
# Move the added items to the front of the path:
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
Copy the file to the Scripts directory of your environment and use it like this:
def activate_virtual_environment(environment_root):
"""Configures the virtual environment starting at ``environment_root``."""
activate_script = os.path.join(
environment_root, 'Scripts', 'activate_this.py')
execfile(activate_script, {'__file__': activate_script})
activate_virtual_environment('path/to/your/venv')
Refrence: https://github.com/dcreager/virtualenv/blob/master/virtualenv_support/activate_this.py

You should create all your virtualenvs in one folder, such as virt.
Assuming your virtualenv folder name is virt, if not change it
cd
mkdir custom
Copy the below lines...
#!/usr/bin/env bash
ENV_PATH="$HOME/virt/$1/bin/activate"
bash --rcfile $ENV_PATH -i
Create a shell script file and paste the above lines...
touch custom/vhelper
nano custom/vhelper
Grant executable permission to your file:
sudo chmod +x custom/vhelper
Now export that custom folder path so that you can find it on the command-line by clicking tab...
export PATH=$PATH:"$HOME/custom"
Now you can use it from anywhere by just typing the below command...
vhelper YOUR_VIRTUAL_ENV_FOLDER_NAME
Suppose it is abc then...
vhelper abc

Python script to activate and keep open a Virtualenv

I need a python script that will activate a virtualenv, run another python program inside the virtualenv, and then close the virutalenv after the second python program closes. Here is my code:
import os
import subprocess
from subprocess import Popen
activate_dir = "C:/Users/JohnDoe/theprogram/Scripts/"
os.chdir(activate_dir)
subprocess.Popen(["activate.bat"])
cal_dir = "C:/Users/JohnDoe/theprogram/"
os.chdir(cal_dir)
os.system('python program_file.py')
However, when this code run, I get an import error from the program_file which means the virtualenv is not activated. How can I fix this?
Thanks
Edit:
This is on a Windows environment.

The issue is that you are creating a new process with subprocess.Popen(["activate.bat"]) that is using that virtual environment, you're not changing your environment. What you need to do is to either call the python script in the same process you span:
os.system("source activate;python -V")
Or you could write a shell script that starts the virtual environment and calls any python script you send to it. In bash (on linux) this would be:
#!/bin/bash
# start a virtual environment and call a python module
# usage: ./runVirenvPythonModule module.py
source activate
python $1 # this is the first cmd line argument passed in

I've found a method to detect, activate, and create (if needed) a virtual environment inside a Python script and run inside that virtual environment while remaining inside that script and without the use of shell commands issued from that script (except to display installed packages via pip list). Without the use of shell commands, the script becomes OS agnostic.
Here is some example code. You simply run it from whatever OS shell you are using (Windows, Linux, MacOS):
import os
import sys
import venv
def isvirtualenv():
return sys.prefix != sys.base_prefix
def findfile(startdir, pattern):
for root, dirs, files in os.walk(startdir):
for name in files:
if name.find(pattern) >= 0:
return root + os.sep + name
return None
venv_path = 'venv' # This is an example path
if isvirtualenv():
print('Already in virtual environment.')
else:
if findfile(os.getcwd(), 'activate') is None:
print('No virtual environment found. Creating one.')
env = venv.EnvBuilder(with_pip = True)
env.create(venv_path)
else:
print('Not in virtual environment. Virtual environment directory found.')
# This is the heart of this script that puts you inside the virtual environment.
# There is no need to undo this. When this script ends, your original path will
# be restored.
os.environ['PATH'] = os.path.dirname(findfile(os.getcwd(), 'activate')) + os.pathsep + os.environ['PATH']
sys.path.insert(1, os.path.dirname(findfile(venv_path, 'easy_install.py')))
# Run your script inside the virtual environment from here
print(os.environ['PATH'])
os.system('pip list')

Automatically load a virtualenv when running a script

I have a python script that needs dependencies from a virtualenv. I was wondering if there was some way I could add it to my path and have it auto start it's virtualenv, run and then go back to the system's python.
I've try playing around with autoenv and .env but that doesn't seem to do exactly what I'm looking for. I also thought about changing the shabang to point to the virtualenv path but that seems fragile.

There are two ways to do this:
Put the name of the virtual env python into first line of the script. Like this
#!/your/virtual/env/path/bin/python
Add virtual environment directories to the sys.path. Note that you need to import sys library. Like this
import sys
sys.path.append('/path/to/virtual/env/lib')
If you go with the second option you might need to add multiple paths to the sys.path (site etc). The best way to get it is to run your virtual env python interpreter and fish out the sys.path value. Like this:
/your/virtual/env/bin/python
Python blah blah blah
> import sys
> print sys.path
[ 'blah', 'blah' , 'blah' ]
Copy the value of sys.path into the snippet above.

I'm surprised that nobody has mentioned this yet, but this is why there is a file called activate_this.py in the virtualenv's bin directory. You can pass that to execfile() to alter the module search path for the currently running interpreter.
# doing execfile() on this file will alter the current interpreter's
# environment so you can import libraries in the virtualenv
activate_this_file = "/path/to/virtualenv/bin/activate_this.py"
execfile(activate_this_file, dict(__file__=activate_this_file))
You can put this file at the top of your script to force the script to always run in that virtualenv. Unlike the modifying hashbang, you can use relative path with by doing:
script_directory = os.path.dirname(os.path.abspath(__file__))
activate_this_file = os.path.join(script_directory, '../../relative/path/to/env/bin/activate_this.py')

From the virtualenv documentation:
If you directly run a script or the python interpreter from the
virtualenv’s bin/ directory (e.g. path/to/env/bin/pip or
/path/to/env/bin/python script.py) there’s no need for activation.
So if you just call the python executable in your virtualenv, your virtualenv will be 'active'. So you can create a script like this:
#!/bin/bash
PATH_TO_MY_VENV=/opt/django/ev_scraper/venv/bin
$PATH_TO_MY_VENV/python -c 'import sys; print(sys.version_info)'
python -c 'import sys; print(sys.version_info)'
When I run this script on my system, the two calls to python print what you see below. (Python 3.2.3 is in my virtualenv, and 2.7.3 is my system Python.)
sys.version_info(major=3, minor=2, micro=3, releaselevel='final', serial=0)
sys.version_info(major=2, minor=7, micro=3, releaselevel='final', serial=0)
So any libraries you have installed in your virtualenv will be available when you call $PATH_TO_MY_VENV/python. Calls to your regular system python will of course be unaware of whatever is in the virtualenv.

I think the best answer here is to create a simple script and install it inside your virtualenv. Then you can either directly use the script, or create a symlink, or whatever.
Here's an example:
$ mkdir my-tool
$ cd my-tool
$ mkdir scripts
$ touch setup.py
$ mkdir scripts
$ touch scripts/crunchy-frog
$ chmod +x scripts/crunchy-frog
crunchy-frog
#!/usr/bin/env python
print("Constable Parrot ate one of those!")
setup.py
from setuptools import setup
setup(name="my-cool-tool",
scripts=['scripts/crunchy-frog'],
)
Now:
$ source /path/to/my/env/bin/activate
(env) $ python setup.py develop
(env) $ deactivate
$ cd ~
$ ln -s /path/to/my/env/bin/crunchy-frog crunchy-frog
$ ./crunchy-frog
Constable Parrot ate one of those!
When you install your script (via setup.py install or setup.py develop) then it will replace the first line of the scripts with a shebang line for the env python (which you can verify with $ head /path/to/my/env/bin/crunchy-frog). So whenever you run that particular script, it will use that specific Python env.

Does this help?
import site
site.addsitedir('/path/to/virtualenv/lib/python2.7/site-packages/')

I had this problem before and I made a simple script to look for a virtualenv folder recursively just importing and calling a function:
script_autoenv.py
# -*- coding:utf-8 -*-
import os, site
def locate_env(path, env_name):
"""search for a env directory name in each directory in the path"""
if os.path.isdir(path + "/env"):
env_26_path = '%s/%s/lib/python2.6/site-packages/' % (path, env_name)
env_27_path = '%s/%s/lib/python2.7/site-packages/' % (path, env_name)
if os.path.isdir(env_26_path):
site.addsitedir(env_26_path)
print "Virtualenv 2.6 founding"
elif os.path.isdir(env_27_path):
site.addsitedir(env_27_path)
print "Virtualenv 2.7 founding"
else:
new_path, old_dir = os.path.split(path)
if old_dir:
locate_env(new_path, env_name)
else:
print "No envs found"
You just need to specify the script directory and the env name folder and the script do the rest:
test.py
# -*- coding:utf-8 -*-
import os
import script_autoenv
script_autoenv.locate_env(os.path.realpath(__file__), 'env')
import django
print django.VERSION
I hope it's works for you

The answer may be pipenv (https://pipenv.readthedocs.io/en/latest/).
It will allow you to do something like:
pipenv run python main.py
to run main.py in the python environment with the specified libraries.
You can give it a try here https://rootnroll.com/d/pipenv/
...Maybe is not exactly what you are looking for, but it may be worth taking a look before reinventing it.

How to find a module in a virtualenv without activating said virtualenv?

Suppose I have the following setup:
mkdir test && cd test
virtualenv .venv
source .venv/bin/activate
pip install django
mkdir mod1
touch mod1/__init__.py
echo "a = 1" > mod1/mod2.py
Which gives me:
test/.venv
test/mod1/__init__.py
test/mod1/mod2.py
How would I write this function:
def get_module(module_name, root_path, virtualenv_path=None)
In order for this to work:
project_root_path = "./test"
project_virtualenv_path = "./test/.venv"
get_module("mod1.mod2", project_root_path, project_virtualenv_path)
get_module("django.contrib.auth", project_root_path, project_virtualenv_path)
Assuming I don't have ./test/.venv activated.
The reason I want to do this, is because I'm working on a vim plugin which would implement gf functionality in a python file on an import statement. I'm trying to support virtualenvs as well.
EDIT:
Also, the script should not alter the current runtime, by adding or appending to sys.path. This should run inside vim, via the vim python bindings, and I don't think altering the vim python runtime would be a good idea.
get_module could either return a module object, or the path to the module, which is what I'm basically looking for.

You can add your virtualenv on python path like:
import site
site.addsitedir('/home/user/.virtualenvs/myapp1/lib/python2.7/site-packages')
and then import should work

The only practical solution I could find here is to run the virtualenv's activate_this.py script, look for what I need, then remove it's changes from sys.path.
import sys
import os
old_sys_path = list(sys.path)
virtualenv_path = "/path/to/venv"
activate_this_path = os.path.join(virtualenv_path, "bin", "activate_this.py")
execfile(activate_this_path, dict(__file__=activate_this_path))
# get my module here
# restore sys.path
sys.path = old_sys_path
If you have a better answer, please add it, and I'll change the accepted answer gladly.

How do you set your pythonpath in an already-created virtualenv?

What file do I edit, and how? I created a virtual environment.

The most elegant solution to this problem is here.
Original answer remains, but this is a messy solution:
If you want to change the PYTHONPATH used in a virtualenv, you can add the following line to your virtualenv's bin/activate file:
export PYTHONPATH="/the/path/you/want"
This way, the new PYTHONPATH will be set each time you use this virtualenv.
EDIT: (to answer #RamRachum's comment)
To have it restored to its original value on deactivate, you could add
export OLD_PYTHONPATH="$PYTHONPATH"
before the previously mentioned line, and add the following line to your bin/postdeactivate script.
export PYTHONPATH="$OLD_PYTHONPATH"

The comment by #s29 should be an answer:
One way to add a directory to the virtual environment is to install virtualenvwrapper (which is useful for many things) and then do
mkvirtualenv myenv
workon myenv
add2virtualenv . #for current directory
add2virtualenv ~/my/path
If you want to remove these path edit the file myenvhomedir/lib/python2.7/site-packages/_virtualenv_path_extensions.pth
Documentation on virtualenvwrapper can be found at http://virtualenvwrapper.readthedocs.org/en/latest/
Specific documentation on this feature can be found at
http://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html?highlight=add2virtualenv

You can create a .pth file that contains the directory to search for, and place it in the {venv-root}/lib/{python-version}/site-packages directory. E.g.:
cd $(python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")
echo /some/library/path > some-library.pth
The effect is the same as adding /some/library/path to sys.path, and remain local to the virtualenv setup.

Initialize your virtualenv
cd venv
source bin/activate
Just set or change your python path by entering command following:
export PYTHONPATH='/home/django/srmvenv/lib/python3.4'
for checking python path enter in python:
python
\>\> import sys
\>\> sys.path

I modified my activate script to source the file .virtualenvrc, if it exists in the current directory, and to save/restore PYTHONPATH on activate/deactivate.
You can find the patched activate script here.. It's a drop-in replacement for the activate script created by virtualenv 1.11.6.
Then I added something like this to my .virtualenvrc:
export PYTHONPATH="${PYTHONPATH:+$PYTHONPATH:}/some/library/path"

It's already answered here -> Is my virtual environment (python) causing my PYTHONPATH to break?
UNIX/LINUX
Add "export PYTHONPATH=/usr/local/lib/python2.0" this to ~/.bashrc file and source it by typing "source ~/.bashrc" OR ". ~/.bashrc".
WINDOWS XP
1) Go to the Control panel
2) Double click System
3) Go to the Advanced tab
4) Click on Environment Variables
In the System Variables window, check if you have a variable named PYTHONPATH. If you have one already, check that it points to the right directories. If you don't have one already, click the New button and create it.
PYTHON CODE
Alternatively, you can also do below your code:-
import sys
sys.path.append("/home/me/mypy")