Develop Reference

subprocess in python cannot create & activate conda venv

Currently I am tryting to, in a python script,
create a conda venv in a temp dir with a different python version I am using in my system
install some packages into this temp conda venv
Execute other python script using this new venv
Kill the process (which is automatic since it is under with .... as ..:)
import subprocess
from tempfile import TemporaryDirectory
with TemporaryDirectory() as tmpdir:
subprocess.call([
f"""
conda create -p {tmpdir}/temp_venv python=3.8 <<< y;
conda activate {tmpdir}/temp_venv && pip install <some_package>==XXX;
{tmpdir}/temp_venv/bin/python /path/to/python/script/test.py
"""
],
shell=True)
The point is that when I try this approach, I get the following error
**CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
You may need to close and restart your shell after running 'conda init'.**
I have already tried running conda init bash but the error persists.
I have also tried to use the Venv package for that but unfortunately it does not let me create a venv with a python version that is not installed in the system.

So, the problem is that conda expects your shell to be initialized normally (an interactive shell). But when you use subprocess, you are in a non-login, non-interactive shell. So, one hack would be to manually call the shell startup script. So for example, on my Macbook Pro:
subprocess.run([
f"""
conda create -y -p {tmpdir}/temp_venv python=3.8;
conda init
source ~/.bash_profile
conda activate {tmpdir}/temp_venv && pip install <some_package>==XXX;
{tmpdir}/temp_venv/bin/python /path/to/python/script/test.py
"""
],
shell=True)
Of course, this is going to be a bit platform dependent. For example, on Ubuntu, you are going to want to use:
source ~/.bashrc
instead.
A more portable solution would be to get subprocess.run to use an interactive shell, that would automatically call those scripts according to the convention of your OS (which conda handles setting up correctly).
So, this is definitely a hack, but it should work.
BTW, if you are using conda, you might as well use:
conda create -y -p {tmpdir}/temp_venv python=3.8 <some_package>==XXX
instead of a seperate:
pip install <some_package>==XXX;
A less hacky alternative is to use conda run, which will run a script in the conda environment. So something like:
subprocess.run([
f"""
conda create -y -p {tmpdir}/temp_venv python=3.8;
conda run -p {tmpdir}/temp_venv --no-capture-output pip install <some_package>==XXX;
conda run -p {tmpdir}/temp_venv/bin/python --no-capture-output /path/to/python/script/test.py
"""
],
shell=True)
I hesitate to use conda run because, at least a few years ago, it was considered "broken" for various subtle reasons, although, in simple cases it works. I think it is still considered an "experimental feature", so use with that caveat in mind, but it should be more portable.

Using conda activate or specifying python path in bash script?

I'm running some python scripts on some Linux clusters using SGE or SLURM. I already have my conda environment set up properly using the login node. I have been writing something like
source ~/.bashrc
module purge #Kill all active modules
conda init bash
conda deactivate
conda deactivate
conda activate my_env
python my_script.py
to activate the environment properly. (I have done a lot of work to figure this out) However, I just found some example codes like
/anaconda3/envs/my_env/bin/python my_script.py
seems to do the same thing without the need for tedious deactivation and activation. Are they actually doing the same thing? If so, which would be the better practice?

Programmatic execution with an environment is usually better done through the conda run subcommand. E.g.,
my_slurm_script.sh
#!/bin/bash -l
conda run -n my_env python my_script.py
Read the conda run --help for details.

When you activate an env, it just changes the python executable to /anaconda3/envs/my_env/bin/python instead of the system's python executable /usr/bin/python in layman terms.
A bertter approach will be to use conda's built in method conda run -n env_name script.py.

Best way to activate my conda environment for a python script on a windows PC

I've made a python script for multiple users and I want to be able to run it by just double clicking the script.
I also have made a conda env for this script and it seems I am unable to activate the environment within my python script (?).
I thought I could write a script which will activate my conda env and then run my python script in command prompt but I'm completely lost and can't find out how to even activate my conda env in command prompt.
Am I heading in the right direction with this - is the best way to activate a conda env and run a python script within one executable script via command prompt?
Sorry if this is a really obvious and/or stupid question, I am very new to all of this!
What I've tried so far:
I have now added conda and python to my path (thank you #Nesha25, I didn't need admin!). I then tried to run my script in the command prompt with: conda run -n Ngon_env python C:\Users\jlp\Desktop\Local_BLAST_scripts\Neisseria\ngon_script.py --live-stream But I get the following error which seems to occur after it tried to use the input function in my python script: EOFError: EOF when reading a line.
I then tried conda run -n Ngon_env --live-stream python C:\Users\jlp\Desktop\Local_BLAST_scripts\Neisseria\ngon_script.py and it doesn't seem to do anything and just gets stuck.
The conda run --no-capture-output flag worked instead, hooray!

Not a Windows user, but conda activate is for interactive shell sessions. The conda run command is for programmatic execution within an environment. So, you would have a script with a line like:
conda run -n my_env python your_script.py
or possibly
conda run -p /path/to/my_env python your_script.py
if trying to share the environment across users.
If the script requires interaction, you may need to add flags (like --live-stream and/or --no-capture-output). See conda run --help.

Why do I got "conda: command not found" when building a docker, while in base image the "conda" command works fine?

I am trying to build a docker from a base image, the Dockerfile is like this:
FROM my/base/image:latest
RUN conda install package-name
RUN rm -rf /tmp/*
CMD ["/bin/bash"]
and I got "conda: command not found". But if I directly run the base image as a container, the conda command works fine.
Also, if I add RUN source ~/.bashrc in the Dockerfile, when building process run into this line, it will enter another terminal and will not response to any command. What‘s happening in this situation? I would be very grateful if anyone could give me any advice and suggestion. Thanks in advance.

~/.bashrc is sourced by interactive shells, it's the reason why it works when you use it from within the container. This is not the case for a Dockerfile during the build process.
In the conda docker image, the conda executable is simply added to the PATH.
ENV PATH /opt/conda/bin:$PATH
It's a simple solution. You can also call /opt/conda/bin/conda install ... (to adapt according to your conda installation) if you do not want to alter the PATH.
Alternative
If you want to have ~/.bashrc sourced you can use the SHELL instruction to override the default shell used for commands with a bash interactive shell (thanks to the i flag).
SHELL ["/bin/bash", "-i", "-c"]
RUN conda install ...
However it seems to be a less conventional approach.

Install Anaconda on Ubuntu (or Linux) via command line

I would like to install Anaconda on a remote server.
The server is running Ubuntu 12.04.
I only have access to this server via SSH.
How can I install Anaconda via the command line?

Something along the lines of:
wget https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh
to get the installer for 64 bit linux followed by:
bash Anaconda3-2020.07-Linux-x86_64.sh
You can get the latest release from here

Please take a look at the Anaconda repo archive page and select an appropriate version that you'd like to install.
After that, just do:
# replace this `Anaconda3-version.num-Linux-x86_64.sh` with your choice
~$ wget -c https://repo.continuum.io/archive/Anaconda3-vers.num-Linux-x86_64.sh
~$ bash Anaconda3-version.num-Linux-x86_64.sh
Concrete Example:
As of this writing, Anaconda3-2020.07 is the latest version. So,
~$ wget -c https://repo.anaconda.com/archive/Anaconda3-2020.07-Linux-x86_64.sh
~$ bash Anaconda3-2020.07-Linux-x86_64.sh
P.S. Based on comments, this should also work in CentOS systems.

You can do as Prashant said or you can use bash scripts to automate the installation. Just simply copy and paste depending on the version of Python you want
If you are trying to it entirely in command line you use a bash script
python 2 anaconda install bash script:
# Go to home directory
cd ~
# You can change what anaconda version you want at
# https://repo.continuum.io/archive/
wget https://repo.continuum.io/archive/Anaconda2-4.2.0-Linux-x86_64.sh
bash Anaconda2-4.2.0-Linux-x86_64.sh -b -p ~/anaconda
rm Anaconda2-4.2.0-Linux-x86_64.sh
echo 'export PATH="~/anaconda/bin:$PATH"' >> ~/.bashrc
# Refresh basically
source .bashrc
conda update conda
python 3 anaconda install bash script
# Go to home directory
cd ~
# You can change what anaconda version you want at
# https://repo.continuum.io/archive/
wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh
bash Anaconda3-4.2.0-Linux-x86_64.sh -b -p ~/anaconda
rm Anaconda3-4.2.0-Linux-x86_64.sh
echo 'export PATH="~/anaconda/bin:$PATH"' >> ~/.bashrc
# Refresh basically
source .bashrc
conda update conda
Source: https://medium.com/#GalarnykMichael/install-python-on-ubuntu-anaconda-65623042cb5a

Download Anaconda for linux, place in your ubuntu system through WinScp, then
$ sudo bash Anaconda2-4.3.0-Linux-x86_64.sh
after this logout of your ssh session and then login, you will get base environment.

1 - Go to Anaconda Repository, find the installation for your OS and copy the address
2 - wget {paste}. Ex: https://repo.continuum.io/archive/Anaconda3-5.2.0-Linux-x86_64.sh
3 - Execute with: bash. Ex: bash Anaconda3-5.2.0-Linux-x86_64.sh
Run!

$ sudo bash Anaconda2-4.3.0-Linux-x86_64.sh
Video tutorial::
https://youtu.be/JP60kTsVJ8E

Just download the anaconda installer and execute it as it is a shell script. Follow the steps :
In the terminal type "wget https://repo.continuum.io/archive/Anaconda-2.3.0-Linux-x86_64.sh"
The file will be downloaded in current directory. Now execute the downloaded file by "bash ./Anaconda-2.3.0-Linux-x86_64.sh"
Restart the terminal. This is very important for python version provided by anaconda to be set to default for that user.
Note- Try using environment for using different version of python. Changing the default python version for root might result in non functioning of some functionalities like yum.