Develop Reference

subprocess in python cannot create & activate conda venv

Currently I am tryting to, in a python script,
create a conda venv in a temp dir with a different python version I am using in my system
install some packages into this temp conda venv
Execute other python script using this new venv
Kill the process (which is automatic since it is under with .... as ..:)
import subprocess
from tempfile import TemporaryDirectory
with TemporaryDirectory() as tmpdir:
subprocess.call([
f"""
conda create -p {tmpdir}/temp_venv python=3.8 <<< y;
conda activate {tmpdir}/temp_venv && pip install <some_package>==XXX;
{tmpdir}/temp_venv/bin/python /path/to/python/script/test.py
"""
],
shell=True)
The point is that when I try this approach, I get the following error
**CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
You may need to close and restart your shell after running 'conda init'.**
I have already tried running conda init bash but the error persists.
I have also tried to use the Venv package for that but unfortunately it does not let me create a venv with a python version that is not installed in the system.

So, the problem is that conda expects your shell to be initialized normally (an interactive shell). But when you use subprocess, you are in a non-login, non-interactive shell. So, one hack would be to manually call the shell startup script. So for example, on my Macbook Pro:
subprocess.run([
f"""
conda create -y -p {tmpdir}/temp_venv python=3.8;
conda init
source ~/.bash_profile
conda activate {tmpdir}/temp_venv && pip install <some_package>==XXX;
{tmpdir}/temp_venv/bin/python /path/to/python/script/test.py
"""
],
shell=True)
Of course, this is going to be a bit platform dependent. For example, on Ubuntu, you are going to want to use:
source ~/.bashrc
instead.
A more portable solution would be to get subprocess.run to use an interactive shell, that would automatically call those scripts according to the convention of your OS (which conda handles setting up correctly).
So, this is definitely a hack, but it should work.
BTW, if you are using conda, you might as well use:
conda create -y -p {tmpdir}/temp_venv python=3.8 <some_package>==XXX
instead of a seperate:
pip install <some_package>==XXX;
A less hacky alternative is to use conda run, which will run a script in the conda environment. So something like:
subprocess.run([
f"""
conda create -y -p {tmpdir}/temp_venv python=3.8;
conda run -p {tmpdir}/temp_venv --no-capture-output pip install <some_package>==XXX;
conda run -p {tmpdir}/temp_venv/bin/python --no-capture-output /path/to/python/script/test.py
"""
],
shell=True)
I hesitate to use conda run because, at least a few years ago, it was considered "broken" for various subtle reasons, although, in simple cases it works. I think it is still considered an "experimental feature", so use with that caveat in mind, but it should be more portable.

Using conda activate or specifying python path in bash script?

I'm running some python scripts on some Linux clusters using SGE or SLURM. I already have my conda environment set up properly using the login node. I have been writing something like
source ~/.bashrc
module purge #Kill all active modules
conda init bash
conda deactivate
conda deactivate
conda activate my_env
python my_script.py
to activate the environment properly. (I have done a lot of work to figure this out) However, I just found some example codes like
/anaconda3/envs/my_env/bin/python my_script.py
seems to do the same thing without the need for tedious deactivation and activation. Are they actually doing the same thing? If so, which would be the better practice?

Programmatic execution with an environment is usually better done through the conda run subcommand. E.g.,
my_slurm_script.sh
#!/bin/bash -l
conda run -n my_env python my_script.py
Read the conda run --help for details.

When you activate an env, it just changes the python executable to /anaconda3/envs/my_env/bin/python instead of the system's python executable /usr/bin/python in layman terms.
A bertter approach will be to use conda's built in method conda run -n env_name script.py.

When I use virtualenv to create a virtual environment, why the project use it?

In my project, I use virtualenv created the ENV:
(ENV) bora-MBP:testDemo01 ldl$ ls
ENV db.sqlite3 manage.py templates test01 testDemo01
You see I have an active virtual environment of the project.
In one python file of my project:
import six
print(six.PY3) # print True
I checked the python version of my project using, there it shows it uses Python3.
but I cd into the ENV/lib, there only shows python2.7:
(ENV) bora-MBP:testDemo01 ldl$ cd ENV/lib/
(ENV) bora-MBP:lib ldl$ ls
python2.7
there is no python3, this is my first question.
My second question is, why my project will use the ENV virtual environment? where is the configuration I can check?
EDIT-01
In my first question:
I know I'm using python3 in my project now, I can use six.PY3 to check, or other ways, but why I create a virtual environment, there only gets python2.7 directory under the ENV/lib/?
My second question:
we know my project will use the ENV environment to run program, but why? is there any default settings for my project to use this environment(dependencies, packages and so on)? is there any configuration file for us to check(clearly point out my project testDemo01 will use the ENV as run environment)?

For your first question:
Please check python version on your virtual environment using python --version to make sure it is python 3 or not. Check whether you have multiple versions of python using whereis python in the command line.
For your second question:
To make a virtual environment management process easier install virtualenvwrapper pip install virtualenvwrapper-win.
Then you will be able to check and change your ENV directory from 'edit the system environment variables'. Change or create 'WORKON_HOME' variable with your require path.
Based on your EDIT-01:
It seems like you have multiple version of python installed in your machine and virtualenv is using python2.7. Please use virtualenv -p path_of_python3 envname to create virtualenv with python3. You can find path for all python using whereis python in the command line.
You can check your dependencies using pip freeze inside your virtualenv.

First question, python2.7/ under ENV/lib/ is because of the creation virtualenv method, if you create the ENV like this:
virtualenv -p python3 ./ENV
there should be python3.x/ under ENV/lib/.
For the second question:
is there any configuration file for us to check(clearly point out my project testDemo01 will use the ENV as run environment)?
for the source reason, you should check the ENV/bin/active bash file, when you active the virtual env, then the project use that yet:
VIRTUAL_ENV="/Users/aircraft/Desktop/TestPython/Demo/venv"
export VIRTUAL_ENV
_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
# unset PYTHONHOME if set
if ! [ -z "${PYTHONHOME+_}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME"
unset PYTHONHOME
fi
if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT-}" ] ; then
_OLD_VIRTUAL_PS1="$PS1"
if [ "x" != x ] ; then
PS1="$PS1"
else
PS1="(`basename \"$VIRTUAL_ENV\"`) $PS1"
fi
export PS1
fi
you see, when you execute the active bash, there will export the VIRTUAL_ENV, and add it into the PATH, and unset PYTHONHOME if set. This is why the project will use the virtualenv.

source bin/activate not updating prompt: Is my virtualenv really active? If not, how can this be fixed?

I am trying to run virtualenv for the first time and when I run the following code the shell prompt does not change.
$ virtualenv test
New python executable in test/bin/python
Installing setuptools............done.
$ source test/bin/activate
$
I'm assuming it did not activate my environment because there is no (test)$ prompt. The virtualenv command creates the directory correctly, so the activate file is there, it just seems to not be working?
I have tried this command as well and it gives the same result.
$ . test/bin/activate
$
Is my virtual environment running and I just cannot see the prompt(and is there something I can run to check that)?
or
Is my virtual environment simply not running and I am not activating it correctly?
I'm using Python 3.6.0(installed from Homebrew), virtualenv 15.1.0, Mac OS X 10.11.6, and zsh(oh-my-zsh)

To tell if your virtualenv is active without depending on the prompt (which shell configuration can override in persistent ways), use:
declare -p VIRTUAL_ENV
If the output is:
declare: no such variable: VIRTUAL_ENV
...then your virtualenv is genuinely not active. By contrast, if the output is:
typeset -x VIRTUAL_ENV=/Users/jess/test
...then it's active, but your prompt has been overridden.
To diagnose this (either the failure to activate, or the prompt override not sticking) in zsh, use:
PS4=':%x:%I+' # the equivalent to this in bash would be PS4=':$BASH_SOURCE:$LINENO+'
set -x
source test/bin/activate
...which will print source file, line number, and command executed for everything done both during and after the activate script is sourced; this should include the activate script updating your PS1 (or, potentially deciding for some reason not to execute); and may include a function hook changing your PS1 back to a version that doesn't have the prompt updated to show the virtualenv's state.

Thanks to the help from Charles, I discovered it was my oh-my-zsh theme(powerlevel9k) that was just not displaying the prompt. I simply needed to add in the virtualenv Python Segment in my ~/.zshrc described here.

If you are using oh-my-zsh on iterm and want to prompt virtual environment then ,
Do open ~/.zshrc Add/update virtualenv in that file like this POWERLEVEL9K_LEFT_PROMPT_ELEMENTS=(context time battery dir vcs virtualenv custom_wifi_signal) Then finaly don't lforget to source ~/.zshrc

How to leave/exit/deactivate a Python virtualenv

I'm using virtualenv and the virtualenvwrapper. I can switch between virtualenv's just fine using the workon command.
me#mymachine:~$ workon env1
(env1)me#mymachine:~$ workon env2
(env2)me#mymachine:~$ workon env1
(env1)me#mymachine:~$
How do I exit all virtual environments and work on my system environment again? Right now, the only way I have of getting back to me#mymachine:~$ is to exit the shell and start a new one. That's kind of annoying. Is there a command to work on "nothing", and if so, what is it? If such a command does not exist, how would I go about creating it?

Usually, activating a virtualenv gives you a shell function named:
$ deactivate
which puts things back to normal.
I have just looked specifically again at the code for virtualenvwrapper, and, yes, it too supports deactivate as the way to escape from all virtualenvs.
If you are trying to leave an Anaconda environment, the command depends upon your version of conda. Recent versions (like 4.6) install a conda function directly in your shell, in which case you run:
conda deactivate
Older conda versions instead implement deactivation using a stand-alone script:
source deactivate

Use:
$ deactivate
If this doesn't work, try
$ source deactivate
Anyone who knows how Bash source works will think that's odd, but some wrappers/workflows around virtualenv implement it as a complement/counterpart to source activate. Your mileage may vary.

I defined an alias, workoff, as the opposite of workon:
alias workoff='deactivate'
It is easy to remember:
[bobstein#host ~]$ workon django_project
(django_project)[bobstein#host ~]$ workoff
[bobstein#host ~]$

To activate a Python virtual environment:
$cd ~/python-venv/
$./bin/activate
To deactivate:
$deactivate

Running deactivate [name of your environment] is able to exit/deactivate from your python environment.
Example with python3.6 Windows 10 in PowerShell:
PS C:\Users\kyrlon\Desktop> py -m venv env1
PS C:\Users\kyrlon\Desktop> .\env1\Scripts\activate
(env1) PS C:\Users\kyrlon\Desktop> deactivate env1
PS C:\Users\kyrlon\Desktop> py -m venv env1
Example with python3.9 on Linux Ubuntu 20.04 LTS Desktop:
kyrlon#pc1:~$ python3 -m venv venv1
kyrlon#pc1:~$ source venv1/bin/activate
(venv1) kyrlon#pc1:~$ deactivate venv1
kyrlon#pc1:~$

I found that when within a Miniconda3 environment I had to run:
conda deactivate
Neither deactivate nor source deactivate worked for me.

You can use virtualenvwrapper in order to ease the way you work with virtualenv.
Installing virtualenvwrapper:
pip install virtualenvwrapper
If you are using a standard shell, open your ~/.bashrc or ~/.zshrc if you use Oh My Zsh. Add these two lines:
export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
To activate an existing virtualenv, use command workon:
$ workon myenv
(myenv)$
In order to deactivate your virtualenv:
(myenv)$ deactivate
Here is my tutorial, step by step on how to install virtualenv and virtualenvwrapper.

For my particular case, I go to to the working directory
CD /myworkingdirectory
Then I activate my env like this:
my-env/scripts/activate
From this same working folder (/myworkingdirectory) to deactivate, I tried this but it does nothing:
my-env/scripts/deactivate
This does work:
deactivate

Using the deactivate feature provided by the venv's activate script requires you to trust the deactivation function to be properly coded to cleanly reset all environment variables back to how they were before— taking into account not only the original activation, but also any switches, configuration, or other work you may have done in the meantime.
It's probably fine, but it does introduce a new, non-zero risk of leaving your environment modified afterwards.
However, it's not technically possible for a process to directly alter the environment variables of its parent, so we can use a separate sub-shell to be absolutely sure our venvs don't leave any residual changes behind:
To activate:
$ bash --init-file PythonVenv/bin/activate
This starts a new shell around the venv. Your original bash shell remains unmodified.
To deactivate:
$ exit OR [CTRL]+[D]
This exits the entire shell the venv is in, and drops you back to the original shell from before the activation script made any changes to the environment.
Example:
[user#computer ~]$ echo $VIRTUAL_ENV
No virtualenv!
[user#computer ~]$ bash --init-file PythonVenv/bin/activate
(PythonVenv) [user#computer ~]$ echo $VIRTUAL_ENV
/home/user/PythonVenv
(PythonVenv) [user#computer ~]$ exit
exit
[user#computer ~]$ echo $VIRTUAL_ENV
No virtualenv!

Since the deactivate function created by sourcing ~/bin/activate cannot be discovered by the usual means of looking for such a command in ~/bin, you may wish to create one that just executes the function deactivate.
The problem is that a script named deactivate containing a single command deactivate will cause an endless loop if accidentally executed while not in the venv. A common mistake.
This can be avoided by only executing deactivate if the function exists (i.e. has been created by sourcing activate).
#!/bin/bash
declare -Ff deactivate && deactivate

$ conda deactivate or $ source deactivate
would work. If it doesn't work, try deactivate [name of your environment] instead.

I use zsh-autoenv which is based off autoenv.
zsh-autoenv automatically
sources (known/whitelisted) .autoenv.zsh files, typically used in
project root directories. It handles "enter" and leave" events,
nesting, and stashing of variables (overwriting and restoring).
Here is an example:
; cd dtree
Switching to virtual environment: Development tree utiles
;dtree(feature/task24|✓); cat .autoenv.zsh
# Autoenv.
echo -n "Switching to virtual environment: "
printf "\e[38;5;93m%s\e[0m\n" "Development tree utiles"
workon dtree
# eof
dtree(feature/task24|✓); cat .autoenv_leave.zsh
deactivate
So when I leave the dtree directory, the virtual environment is automatically exited.
"Development tree utiles" is just a name… No hidden mean linking to the Illuminati in here.

I my case, I was able to activate virtual environment using env-name\scripts\activate and deactivate it using deactivate. However, after running update on my windows PC deactivate was no longer recognized as an internal or external command. What I used from that moment onward is env-name\scripts\deactivate and that solved the problem.

I had the same problem while working on an installer script. I took a look at what the bin/activate_this.py did and reversed it.
Example:
#! /usr/bin/python
# -*- coding: utf-8 -*-
import os
import sys
# Path to virtualenv
venv_path = os.path.join('/home', 'sixdays', '.virtualenvs', 'test32')
# Save old values
old_os_path = os.environ['PATH']
old_sys_path = list(sys.path)
old_sys_prefix = sys.prefix
def deactivate():
# Change back by setting values to starting values
os.environ['PATH'] = old_os_path
sys.prefix = old_sys_prefix
sys.path[:0] = old_sys_path
# Activate the virtualenvironment
activate_this = os.path.join(venv_path, 'bin/activate_this.py')
execfile(activate_this, dict(__file__=activate_this))
# Print list of pip packages for virtualenv for example purpose
import pip
print str(pip.get_installed_distributions())
# Unload pip module
del pip
# Deactivate/switch back to initial interpreter
deactivate()
# Print list of initial environment pip packages for example purpose
import pip
print str(pip.get_installed_distributions())
I am not 100% sure if it works as intended. I may have missed something completely.