Recently, my anaconda environment got broken due to certain bad conda package upgrade. Due to this back experience, I would like to back up my anaconda environment for future easy recovery.
What I did to back up was to zip up the entire folder at C:\ProgramData\Anaconda3. Is this the correct way?
I am using Windows 10, anaconda python v3.6 64-bit.
It doesn't really make sense to zip up a conda environment for back up purposes since there are other ways to do this which may be more appropriate and use the in-built functions designed to do just this.
You can create a .txt version of the conda environment that details each module and version within, and can then be used to re-create the EXACT environment in the future.
# Create list of the environment
conda list --explicit environment_backup.txt
# Use the newly created text file to recreate the environment
conda create --name my_env_name --file environment_backup.txt
See docs for more information on managing conda environments.
N.B. As an additional point, conda environment directories can be fairly large (often >1GB) whereas the txt file created here is ~25KB, offering a clear advantage when archiving something for safe-keeping.
There are numerous ways you can achieve that as the way the anaconda website has shared. However, if you have limited or no internet accessibility, using a tool named "conda-pack" is recommended (unfortunately with the same OS for now).
Follow steps below:
conda install -c conda-forge conda-pack
And then on your source machine:
# Pack environment my_env into my_env.tar.gz
$ conda pack -n my_env
# Pack environment my_env into out_name.tar.gz
$ conda pack -n my_env -o out_name.tar.gz
# Pack environment located at an explicit path into my_env.tar.gz
$ conda pack -p /explicit/path/to/my_env
Lastly, on your target machine:
# Unpack environment into directory `my_env`
$ mkdir -p my_env
$ tar -xzf my_env.tar.gz -C my_env
# Use python without activating or fixing the prefixes. Most python
# libraries will work fine, but things that require prefix cleanups
# will fail.
$ ./my_env/bin/python
# Activate the environment. This adds `my_env/bin` to your path
$ source my_env/bin/activate
# Run python from in the environment
(my_env) $ python
# Cleanup prefixes from in the active environment.
# Note that this command can also be run without activating the environment
# as long as some version of python is already installed on the machine.
(my_env) $ conda-unpack
# At this point the environment is exactly as if you installed it here
# using conda directly. All scripts should work fine.
(my_env) $ ipython --version
# Deactivate the environment to remove it from your path
(my_env) $ source my_env/bin/deactivate
Related
I have the following in a CMakeLists.txt file. I'm trying to use cmake to check if a conda environment named myenv is installed on the system and activate that environment. If the environment does not exist, then create the environment and activate it. This assumes that conda is already installed via Anaconda (or Miniconda).
# Create and activate a Python environment.
cmake_minimum_required(VERSION 3.18)
# Define the project
project(MyExample)
# Specify the C++ standard
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED True)
# Make sure Python is installed
find_package(Python REQUIRED)
# Activate conda environment, assume Anaconda or Miniconda is already installed
if(EXISTS /opt/miniconda3/envs/myenv)
execute_process(COMMAND conda activate myenv)
else()
execute_process(COMMAND conda create --yes --quiet --name myenv python)
execute_process(COMMAND conda activate myenv)
endif()
When I run the above cmake file, I get the error:
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
However, conda is installed on my system and I can activate the environment manually in the terminal. Why does the environment not get activated via cmake?
"Why does the environment not get activated via cmake?"
The first issue is in the error message, which indicates that the shell that CMake evaluates the execute_process command under doesn't know what conda activate means.
This could be because conda init <shell> has never been run for the particular shell that CMake uses (bash, perhaps?). This answer has some details on what conda init does. It could also be because the Conda initialization code that defines conda activate only gets loaded in interactive shells - which I wouldn't expect CMake to be using. There might be hacky ways to force execution in an interactive shell (bash -l -c 'conda activate foo'), but that doesn't matter because...
Even if the above were working, the procedures here don't make sense: a Conda environment's activation status is scoped to the shell process. I would expect that the (sub)shell dies with the completion of the execute_process. So, even if the activation worked, it wouldn't persist any further in the CMake script.
Discussion
Generally, this CMake script does not seem like a good approach. Tight-coupling the compilation of code to the existence of a particularly-named Conda environment at the user level seems to go against the spirit of CMake, which aims to automate the discovery of software dependencies so that users don't need commonly hardcoded locations. Perhaps it might be worth reassessing what is trying to be accomplished and the strategy to get there.
For example, on Conda Forge, lots of packages compile with CMake, but there CMake is executed in the context of an already activated environment and itself knows nothing about Conda. This makes it so the CMake code is completely agnostic to how its dependencies are provided, which I regard as cleaner engineering.
It often happens that when I create a conda environment, I forgot about activating it, despite the clear conda message. And I end up installing my packages in the conda base environment. (yeah... I'm a bit of a dreamer)
My questions are:
Are there use cases where you create a conda environment but don't activate it right after?
How to create a conda environment and activate it in a single line? (in a Linux prompt shell, and non-interactive)
As a concrete example in answer to your first question, I have used conda as a way to wrap a disposable build environment in some Makefile targets, i.e., I create the environment, and then subsequent commands or targets may make use of the environment via conda run.
Adapting a snippet from one Makefile, you could create a function in a bash startup file:
conda_create_and_run() {
ENV_NAME=$1
CONDA_PY_VER=$2
. ${CONDA_ENV_FILE}
conda config --append envs_dirs ${CONDA_DIR}
conda create -p ${CONDA_DIR}/${ENV_NAME} python=${CONDA_PY_VER} -y
conda activate ${ENV_NAME}
}
Here CONDA_PY_VER is the non-default python version you want the environment to possibly be created with, and CONDA_DIR and CONDA_ENV_FILE are, respectively, the location where conda keeps its environments and the conda environment file you need to source (or have part of your shell init) in order to have the conda commands available.
You would then use it as:
conda_create_and_run myenv 3.8
to create an environment for python3.8 named myenv.
I read that the prefix line in the environment.yaml file is not used by conda env create. Two of the posts on SO pointing to this fact are:
export conda environment without prefix variable which shows local path to executable
Anaconda export Environment file
I have the reverse problem of most of these posts
I want to specify inside the file the actual prefix, so that different users setup their environments in their home directory in a shared machine.
However, as previously mentioned the command for creating environments is completely ignoring the prefix line.
I managed to setup an environment to a specific path using a prefix like this:
conda env create --prefix=<prefix> --file=environment.yaml
but I am trying to figure a way to define the prefix so the user will not have to type it themselves but it will be automatically configured to be their home directory.
I work around the lacking of proper solution by using Makefile target
# project_root/Makefile
# from my experience mamba installs and updates faster than conda
# conda install mamba -n base -c conda-forge
env-create:
mamba env create -p ./envs -f environment.yml
env-update:
mamba env update -p ./envs -f environment.yml
usage:
$ make env-create
# or
$ make env-update
I tried to create a new environment for anaconda installation that I want to tweak apart from the original install. I found this not working:
$ conda env create --name pandas018numpy111 pandas018test1
Using Anaconda Cloud api site https://api.anaconda.org
Error: Invalid name, try the format: user/package
pandas018test1 does not exist or can't be accessed
environment.yml file not found
There is no requirements.txt
What is wrong here?
What I wanted to accomplish was to create the copy of the original environment, add some modules for testing around, then toss away the test environment (pandas018numpy111).
Ah, I found the answer.
What I need is the conda create command, actually,
$ conda create -n pandas018numpy111 --clone root
Then I have the throw-away environment to modify, etc and later just toss. To switch then I will just use:
$ source activate pandas018numpy111
and to exit from this environment,
$ source deactivate
I have installed Anaconda 2.2.0 for Windows and created a virtual environment via:
> conda create -n my-env anaconda
The environment is sucessfully created and I see it in my list of envinronments (and indeed the directory is there in Anaconda\envs..)
> conda info -e
# conda environments:
#
my-env D:\Anaconda\envs\my-env
root * D:\Anaconda
However, when running the activate.bat script to switch envinronment, although it appears to be successful the switch isn't actually made:
> activate.bat my-env
Activating environment "astropy-dev"...
> conda list -e
# conda environments:
#
my-env D:\Anaconda\envs\my-env
root * D:\Anaconda
With the * indicating the active environment.
I have seen some issues with conda activate on Windows but haven't found this sepecific issue.
For further info: I am looking to copy the whole Anaconda package distribution and then install a dev version over one package.
If you are using Powershell, activate currently does not support it. You will need to modify your PATH manually, or else use the cmd shell.
Are you calling activate from within a batch script? Then it should be call activate my-env.
You don't need the .bat. It's just activate my-env.
You command
activate astropy-dev
must be run from the D:\Anaconda directory. Then it should work.
To check, type:
conda info -e