I'm running some python scripts on some Linux clusters using SGE or SLURM. I already have my conda environment set up properly using the login node. I have been writing something like
source ~/.bashrc
module purge #Kill all active modules
conda init bash
conda deactivate
conda deactivate
conda activate my_env
python my_script.py
to activate the environment properly. (I have done a lot of work to figure this out) However, I just found some example codes like
/anaconda3/envs/my_env/bin/python my_script.py
seems to do the same thing without the need for tedious deactivation and activation. Are they actually doing the same thing? If so, which would be the better practice?
Programmatic execution with an environment is usually better done through the conda run subcommand. E.g.,
my_slurm_script.sh
#!/bin/bash -l
conda run -n my_env python my_script.py
Read the conda run --help for details.
When you activate an env, it just changes the python executable to /anaconda3/envs/my_env/bin/python instead of the system's python executable /usr/bin/python in layman terms.
A bertter approach will be to use conda's built in method conda run -n env_name script.py.
Related
Currently I am tryting to, in a python script,
create a conda venv in a temp dir with a different python version I am using in my system
install some packages into this temp conda venv
Execute other python script using this new venv
Kill the process (which is automatic since it is under with .... as ..:)
import subprocess
from tempfile import TemporaryDirectory
with TemporaryDirectory() as tmpdir:
subprocess.call([
f"""
conda create -p {tmpdir}/temp_venv python=3.8 <<< y;
conda activate {tmpdir}/temp_venv && pip install <some_package>==XXX;
{tmpdir}/temp_venv/bin/python /path/to/python/script/test.py
"""
],
shell=True)
The point is that when I try this approach, I get the following error
**CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
You may need to close and restart your shell after running 'conda init'.**
I have already tried running conda init bash but the error persists.
I have also tried to use the Venv package for that but unfortunately it does not let me create a venv with a python version that is not installed in the system.
So, the problem is that conda expects your shell to be initialized normally (an interactive shell). But when you use subprocess, you are in a non-login, non-interactive shell. So, one hack would be to manually call the shell startup script. So for example, on my Macbook Pro:
subprocess.run([
f"""
conda create -y -p {tmpdir}/temp_venv python=3.8;
conda init
source ~/.bash_profile
conda activate {tmpdir}/temp_venv && pip install <some_package>==XXX;
{tmpdir}/temp_venv/bin/python /path/to/python/script/test.py
"""
],
shell=True)
Of course, this is going to be a bit platform dependent. For example, on Ubuntu, you are going to want to use:
source ~/.bashrc
instead.
A more portable solution would be to get subprocess.run to use an interactive shell, that would automatically call those scripts according to the convention of your OS (which conda handles setting up correctly).
So, this is definitely a hack, but it should work.
BTW, if you are using conda, you might as well use:
conda create -y -p {tmpdir}/temp_venv python=3.8 <some_package>==XXX
instead of a seperate:
pip install <some_package>==XXX;
A less hacky alternative is to use conda run, which will run a script in the conda environment. So something like:
subprocess.run([
f"""
conda create -y -p {tmpdir}/temp_venv python=3.8;
conda run -p {tmpdir}/temp_venv --no-capture-output pip install <some_package>==XXX;
conda run -p {tmpdir}/temp_venv/bin/python --no-capture-output /path/to/python/script/test.py
"""
],
shell=True)
I hesitate to use conda run because, at least a few years ago, it was considered "broken" for various subtle reasons, although, in simple cases it works. I think it is still considered an "experimental feature", so use with that caveat in mind, but it should be more portable.
I've made a python script for multiple users and I want to be able to run it by just double clicking the script.
I also have made a conda env for this script and it seems I am unable to activate the environment within my python script (?).
I thought I could write a script which will activate my conda env and then run my python script in command prompt but I'm completely lost and can't find out how to even activate my conda env in command prompt.
Am I heading in the right direction with this - is the best way to activate a conda env and run a python script within one executable script via command prompt?
Sorry if this is a really obvious and/or stupid question, I am very new to all of this!
What I've tried so far:
I have now added conda and python to my path (thank you #Nesha25, I didn't need admin!). I then tried to run my script in the command prompt with: conda run -n Ngon_env python C:\Users\jlp\Desktop\Local_BLAST_scripts\Neisseria\ngon_script.py --live-stream But I get the following error which seems to occur after it tried to use the input function in my python script: EOFError: EOF when reading a line.
I then tried conda run -n Ngon_env --live-stream python C:\Users\jlp\Desktop\Local_BLAST_scripts\Neisseria\ngon_script.py and it doesn't seem to do anything and just gets stuck.
The conda run --no-capture-output flag worked instead, hooray!
Not a Windows user, but conda activate is for interactive shell sessions. The conda run command is for programmatic execution within an environment. So, you would have a script with a line like:
conda run -n my_env python your_script.py
or possibly
conda run -p /path/to/my_env python your_script.py
if trying to share the environment across users.
If the script requires interaction, you may need to add flags (like --live-stream and/or --no-capture-output). See conda run --help.
I'm using virtualenv and the virtualenvwrapper. I can switch between virtualenv's just fine using the workon command.
me#mymachine:~$ workon env1
(env1)me#mymachine:~$ workon env2
(env2)me#mymachine:~$ workon env1
(env1)me#mymachine:~$
How do I exit all virtual environments and work on my system environment again? Right now, the only way I have of getting back to me#mymachine:~$ is to exit the shell and start a new one. That's kind of annoying. Is there a command to work on "nothing", and if so, what is it? If such a command does not exist, how would I go about creating it?
Usually, activating a virtualenv gives you a shell function named:
$ deactivate
which puts things back to normal.
I have just looked specifically again at the code for virtualenvwrapper, and, yes, it too supports deactivate as the way to escape from all virtualenvs.
If you are trying to leave an Anaconda environment, the command depends upon your version of conda. Recent versions (like 4.6) install a conda function directly in your shell, in which case you run:
conda deactivate
Older conda versions instead implement deactivation using a stand-alone script:
source deactivate
Use:
$ deactivate
If this doesn't work, try
$ source deactivate
Anyone who knows how Bash source works will think that's odd, but some wrappers/workflows around virtualenv implement it as a complement/counterpart to source activate. Your mileage may vary.
I defined an alias, workoff, as the opposite of workon:
alias workoff='deactivate'
It is easy to remember:
[bobstein#host ~]$ workon django_project
(django_project)[bobstein#host ~]$ workoff
[bobstein#host ~]$
To activate a Python virtual environment:
$cd ~/python-venv/
$./bin/activate
To deactivate:
$deactivate
Running deactivate [name of your environment] is able to exit/deactivate from your python environment.
Example with python3.6 Windows 10 in PowerShell:
PS C:\Users\kyrlon\Desktop> py -m venv env1
PS C:\Users\kyrlon\Desktop> .\env1\Scripts\activate
(env1) PS C:\Users\kyrlon\Desktop> deactivate env1
PS C:\Users\kyrlon\Desktop> py -m venv env1
Example with python3.9 on Linux Ubuntu 20.04 LTS Desktop:
kyrlon#pc1:~$ python3 -m venv venv1
kyrlon#pc1:~$ source venv1/bin/activate
(venv1) kyrlon#pc1:~$ deactivate venv1
kyrlon#pc1:~$
I found that when within a Miniconda3 environment I had to run:
conda deactivate
Neither deactivate nor source deactivate worked for me.
You can use virtualenvwrapper in order to ease the way you work with virtualenv.
Installing virtualenvwrapper:
pip install virtualenvwrapper
If you are using a standard shell, open your ~/.bashrc or ~/.zshrc if you use Oh My Zsh. Add these two lines:
export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
To activate an existing virtualenv, use command workon:
$ workon myenv
(myenv)$
In order to deactivate your virtualenv:
(myenv)$ deactivate
Here is my tutorial, step by step on how to install virtualenv and virtualenvwrapper.
For my particular case, I go to to the working directory
CD /myworkingdirectory
Then I activate my env like this:
my-env/scripts/activate
From this same working folder (/myworkingdirectory) to deactivate, I tried this but it does nothing:
my-env/scripts/deactivate
This does work:
deactivate
Using the deactivate feature provided by the venv's activate script requires you to trust the deactivation function to be properly coded to cleanly reset all environment variables back to how they were before— taking into account not only the original activation, but also any switches, configuration, or other work you may have done in the meantime.
It's probably fine, but it does introduce a new, non-zero risk of leaving your environment modified afterwards.
However, it's not technically possible for a process to directly alter the environment variables of its parent, so we can use a separate sub-shell to be absolutely sure our venvs don't leave any residual changes behind:
To activate:
$ bash --init-file PythonVenv/bin/activate
This starts a new shell around the venv. Your original bash shell remains unmodified.
To deactivate:
$ exit OR [CTRL]+[D]
This exits the entire shell the venv is in, and drops you back to the original shell from before the activation script made any changes to the environment.
Example:
[user#computer ~]$ echo $VIRTUAL_ENV
No virtualenv!
[user#computer ~]$ bash --init-file PythonVenv/bin/activate
(PythonVenv) [user#computer ~]$ echo $VIRTUAL_ENV
/home/user/PythonVenv
(PythonVenv) [user#computer ~]$ exit
exit
[user#computer ~]$ echo $VIRTUAL_ENV
No virtualenv!
Since the deactivate function created by sourcing ~/bin/activate cannot be discovered by the usual means of looking for such a command in ~/bin, you may wish to create one that just executes the function deactivate.
The problem is that a script named deactivate containing a single command deactivate will cause an endless loop if accidentally executed while not in the venv. A common mistake.
This can be avoided by only executing deactivate if the function exists (i.e. has been created by sourcing activate).
#!/bin/bash
declare -Ff deactivate && deactivate
$ conda deactivate or $ source deactivate
would work. If it doesn't work, try deactivate [name of your environment] instead.
I use zsh-autoenv which is based off autoenv.
zsh-autoenv automatically
sources (known/whitelisted) .autoenv.zsh files, typically used in
project root directories. It handles "enter" and leave" events,
nesting, and stashing of variables (overwriting and restoring).
Here is an example:
; cd dtree
Switching to virtual environment: Development tree utiles
;dtree(feature/task24|✓); cat .autoenv.zsh
# Autoenv.
echo -n "Switching to virtual environment: "
printf "\e[38;5;93m%s\e[0m\n" "Development tree utiles"
workon dtree
# eof
dtree(feature/task24|✓); cat .autoenv_leave.zsh
deactivate
So when I leave the dtree directory, the virtual environment is automatically exited.
"Development tree utiles" is just a name… No hidden mean linking to the Illuminati in here.
I my case, I was able to activate virtual environment using env-name\scripts\activate and deactivate it using deactivate. However, after running update on my windows PC deactivate was no longer recognized as an internal or external command. What I used from that moment onward is env-name\scripts\deactivate and that solved the problem.
I had the same problem while working on an installer script. I took a look at what the bin/activate_this.py did and reversed it.
Example:
#! /usr/bin/python
# -*- coding: utf-8 -*-
import os
import sys
# Path to virtualenv
venv_path = os.path.join('/home', 'sixdays', '.virtualenvs', 'test32')
# Save old values
old_os_path = os.environ['PATH']
old_sys_path = list(sys.path)
old_sys_prefix = sys.prefix
def deactivate():
# Change back by setting values to starting values
os.environ['PATH'] = old_os_path
sys.prefix = old_sys_prefix
sys.path[:0] = old_sys_path
# Activate the virtualenvironment
activate_this = os.path.join(venv_path, 'bin/activate_this.py')
execfile(activate_this, dict(__file__=activate_this))
# Print list of pip packages for virtualenv for example purpose
import pip
print str(pip.get_installed_distributions())
# Unload pip module
del pip
# Deactivate/switch back to initial interpreter
deactivate()
# Print list of initial environment pip packages for example purpose
import pip
print str(pip.get_installed_distributions())
I am not 100% sure if it works as intended. I may have missed something completely.
my anaconda (4.5.4) works fine as long as I just use it via a linux terminal (bash shell). However, running conda commands in a bash script does not work at all.
The script test.sh containes these lines:
#!/bin/bash
conda --version
conda activate env
Now, running bash test.sh results in the error
test.sh: line 2: conda: command not found
test.sh: line 3: conda: command not found
As recommended for anaconda version > 4.4 my .bashrc does not contain
export PATH="/opt/anaconda/bin:$PATH",
but
. /opt/anaconda/etc/profile.d/conda.sh
Thank you.
I solved the problem thanks to #darthbith 's comment.
Since conda is a bash function and bash functions can not be propagated to independent shells (e.g. opened by executing a bash script), one has to add the line
source /opt/anaconda/etc/profile.d/conda.sh
to the bash script before calling conda commands. Otherwise bash will not know about conda.
If #randomwalker's method doesn't work for you, which it won't any time your script is run in a more basic shell such as sh, then you have two options.
Add this to your script: eval $(conda shell.bash hook)
Call your script with: bash -i <scriptname> so that it runs in your interactive environment.
Let's say you try to access user name with "miky" # "server" address.First when you login to your user ; learn conda path with "which conda" then probably you will get a path such as "/home/miky/anaconda3/bin/conda"
then put your conda commands as follow (in my example i use conda to install a mysql plugin forexample.): shh miky#server -t "/home/miky/anaconda3/bin/conda install -y -c anaconda mysql-connector-python" thats all.
do sudo ln -s /home/<user>/miniconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh and try again. This should activate conda for all users permenantly
source
I'm using virtualenv and the virtualenvwrapper. I can switch between virtualenv's just fine using the workon command.
me#mymachine:~$ workon env1
(env1)me#mymachine:~$ workon env2
(env2)me#mymachine:~$ workon env1
(env1)me#mymachine:~$
How do I exit all virtual environments and work on my system environment again? Right now, the only way I have of getting back to me#mymachine:~$ is to exit the shell and start a new one. That's kind of annoying. Is there a command to work on "nothing", and if so, what is it? If such a command does not exist, how would I go about creating it?
Usually, activating a virtualenv gives you a shell function named:
$ deactivate
which puts things back to normal.
I have just looked specifically again at the code for virtualenvwrapper, and, yes, it too supports deactivate as the way to escape from all virtualenvs.
If you are trying to leave an Anaconda environment, the command depends upon your version of conda. Recent versions (like 4.6) install a conda function directly in your shell, in which case you run:
conda deactivate
Older conda versions instead implement deactivation using a stand-alone script:
source deactivate
Use:
$ deactivate
If this doesn't work, try
$ source deactivate
Anyone who knows how Bash source works will think that's odd, but some wrappers/workflows around virtualenv implement it as a complement/counterpart to source activate. Your mileage may vary.
I defined an alias, workoff, as the opposite of workon:
alias workoff='deactivate'
It is easy to remember:
[bobstein#host ~]$ workon django_project
(django_project)[bobstein#host ~]$ workoff
[bobstein#host ~]$
To activate a Python virtual environment:
$cd ~/python-venv/
$./bin/activate
To deactivate:
$deactivate
Running deactivate [name of your environment] is able to exit/deactivate from your python environment.
Example with python3.6 Windows 10 in PowerShell:
PS C:\Users\kyrlon\Desktop> py -m venv env1
PS C:\Users\kyrlon\Desktop> .\env1\Scripts\activate
(env1) PS C:\Users\kyrlon\Desktop> deactivate env1
PS C:\Users\kyrlon\Desktop> py -m venv env1
Example with python3.9 on Linux Ubuntu 20.04 LTS Desktop:
kyrlon#pc1:~$ python3 -m venv venv1
kyrlon#pc1:~$ source venv1/bin/activate
(venv1) kyrlon#pc1:~$ deactivate venv1
kyrlon#pc1:~$
I found that when within a Miniconda3 environment I had to run:
conda deactivate
Neither deactivate nor source deactivate worked for me.
You can use virtualenvwrapper in order to ease the way you work with virtualenv.
Installing virtualenvwrapper:
pip install virtualenvwrapper
If you are using a standard shell, open your ~/.bashrc or ~/.zshrc if you use Oh My Zsh. Add these two lines:
export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
To activate an existing virtualenv, use command workon:
$ workon myenv
(myenv)$
In order to deactivate your virtualenv:
(myenv)$ deactivate
Here is my tutorial, step by step on how to install virtualenv and virtualenvwrapper.
For my particular case, I go to to the working directory
CD /myworkingdirectory
Then I activate my env like this:
my-env/scripts/activate
From this same working folder (/myworkingdirectory) to deactivate, I tried this but it does nothing:
my-env/scripts/deactivate
This does work:
deactivate
Using the deactivate feature provided by the venv's activate script requires you to trust the deactivation function to be properly coded to cleanly reset all environment variables back to how they were before— taking into account not only the original activation, but also any switches, configuration, or other work you may have done in the meantime.
It's probably fine, but it does introduce a new, non-zero risk of leaving your environment modified afterwards.
However, it's not technically possible for a process to directly alter the environment variables of its parent, so we can use a separate sub-shell to be absolutely sure our venvs don't leave any residual changes behind:
To activate:
$ bash --init-file PythonVenv/bin/activate
This starts a new shell around the venv. Your original bash shell remains unmodified.
To deactivate:
$ exit OR [CTRL]+[D]
This exits the entire shell the venv is in, and drops you back to the original shell from before the activation script made any changes to the environment.
Example:
[user#computer ~]$ echo $VIRTUAL_ENV
No virtualenv!
[user#computer ~]$ bash --init-file PythonVenv/bin/activate
(PythonVenv) [user#computer ~]$ echo $VIRTUAL_ENV
/home/user/PythonVenv
(PythonVenv) [user#computer ~]$ exit
exit
[user#computer ~]$ echo $VIRTUAL_ENV
No virtualenv!
Since the deactivate function created by sourcing ~/bin/activate cannot be discovered by the usual means of looking for such a command in ~/bin, you may wish to create one that just executes the function deactivate.
The problem is that a script named deactivate containing a single command deactivate will cause an endless loop if accidentally executed while not in the venv. A common mistake.
This can be avoided by only executing deactivate if the function exists (i.e. has been created by sourcing activate).
#!/bin/bash
declare -Ff deactivate && deactivate
$ conda deactivate or $ source deactivate
would work. If it doesn't work, try deactivate [name of your environment] instead.
I use zsh-autoenv which is based off autoenv.
zsh-autoenv automatically
sources (known/whitelisted) .autoenv.zsh files, typically used in
project root directories. It handles "enter" and leave" events,
nesting, and stashing of variables (overwriting and restoring).
Here is an example:
; cd dtree
Switching to virtual environment: Development tree utiles
;dtree(feature/task24|✓); cat .autoenv.zsh
# Autoenv.
echo -n "Switching to virtual environment: "
printf "\e[38;5;93m%s\e[0m\n" "Development tree utiles"
workon dtree
# eof
dtree(feature/task24|✓); cat .autoenv_leave.zsh
deactivate
So when I leave the dtree directory, the virtual environment is automatically exited.
"Development tree utiles" is just a name… No hidden mean linking to the Illuminati in here.
I my case, I was able to activate virtual environment using env-name\scripts\activate and deactivate it using deactivate. However, after running update on my windows PC deactivate was no longer recognized as an internal or external command. What I used from that moment onward is env-name\scripts\deactivate and that solved the problem.
I had the same problem while working on an installer script. I took a look at what the bin/activate_this.py did and reversed it.
Example:
#! /usr/bin/python
# -*- coding: utf-8 -*-
import os
import sys
# Path to virtualenv
venv_path = os.path.join('/home', 'sixdays', '.virtualenvs', 'test32')
# Save old values
old_os_path = os.environ['PATH']
old_sys_path = list(sys.path)
old_sys_prefix = sys.prefix
def deactivate():
# Change back by setting values to starting values
os.environ['PATH'] = old_os_path
sys.prefix = old_sys_prefix
sys.path[:0] = old_sys_path
# Activate the virtualenvironment
activate_this = os.path.join(venv_path, 'bin/activate_this.py')
execfile(activate_this, dict(__file__=activate_this))
# Print list of pip packages for virtualenv for example purpose
import pip
print str(pip.get_installed_distributions())
# Unload pip module
del pip
# Deactivate/switch back to initial interpreter
deactivate()
# Print list of initial environment pip packages for example purpose
import pip
print str(pip.get_installed_distributions())
I am not 100% sure if it works as intended. I may have missed something completely.