Develop Reference

Does it make sense to use Conda + Poetry?

Does it make sense to use Conda + Poetry for a Machine Learning project? Allow me to share my (novice) understanding and please correct or enlighten me:
As far as I understand, Conda and Poetry have different purposes but are largely redundant:
Conda is primarily a environment manager (in fact not necessarily Python), but it can also manage packages and dependencies.
Poetry is primarily a Python package manager (say, an upgrade of pip), but it can also create and manage Python environments (say, an upgrade of Pyenv).
My idea is to use both and compartmentalize their roles: let Conda be the environment manager and Poetry the package manager. My reasoning is that (it sounds like) Conda is best for managing environments and can be used for compiling and installing non-python packages, especially CUDA drivers (for GPU capability), while Poetry is more powerful than Conda as a Python package manager.
I've managed to make this work fairly easily by using Poetry within a Conda environment. The trick is to not use Poetry to manage the Python environment: I'm not using commands like poetry shell or poetry run, only poetry init, poetry install etc (after activating the Conda environment).
For full disclosure, my environment.yml file (for Conda) looks like this:
name: N
channels:
- defaults
- conda-forge
dependencies:
- python=3.9
- cudatoolkit
- cudnn
and my poetry.toml file looks like that:
[tool.poetry]
name = "N"
authors = ["B"]
[tool.poetry.dependencies]
python = "3.9"
torch = "^1.10.1"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
To be honest, one of the reasons I proceeded this way is that I was struggling to install CUDA (for GPU support) without Conda.
Does this project design look reasonable to you?

I have experience with a Conda + Poetry setup, and it's been working fine. The great majority of my dependencies are specified in pyproject.toml, but when there's something that's unavailable in PyPI, or installing it with Conda is easier, I add it to environment.yml. Moreover, Conda is used as a virtual environment manager, which works well with Poetry: there is no need to use poetry run or poetry shell, it is enough to activate the right Conda environment.
Tips for creating a reproducible environment
Add Poetry, possibly with a version number (if needed), as a dependency in environment.yml, so that you get Poetry installed when you run conda create, along with Python and other non-PyPI dependencies.
Add conda-lock, which gives you lock files for Conda dependencies, just like you have poetry.lock for Poetry dependencies.
Consider using mamba which is generally compatible with conda, but is better at resolving conflicts, and is also much faster. An additional benefit is that all users of your setup will use the same package resolver, independent from the locally-installed version of Conda.
By default, use Poetry for adding Python dependencies. Install packages via Conda if there's a reason to do so (e.g. in order to get a CUDA-enabled version). In such a case, it is best to specify the package's exact version in environment.yml, and after it's installed, to add an entry with the same version specification to Poetry's pyproject.toml (without ^ or ~ before the version number). This will let Poetry know that the package is there and should not be upgraded.
If you use a different channels that provide the same packages, it might be not obvious which channel a particular package will be downloaded from. One solution is to specify the channel for the package using the :: notation (see the pytorch entry below), and another solution is to enable strict channel priority. Unfortunately, in Conda 4.x there is no way to enable this option through environment.yml.
Note that Python adds user site-packages to sys.path, which may cause lack of reproducibility if the user has installed Python packages outside Conda environments. One possible solution is to make sure that the PYTHONNOUSERSITE environment variable is set to True (or to any other non-empty value).
Example
environment.yml:
name: my_project_env
channels:
- pytorch
- conda-forge
# We want to have a reproducible setup, so we don't want default channels,
# which may be different for different users. All required channels should
# be listed explicitly here.
- nodefaults
dependencies:
- python=3.10.* # or don't specify the version and use the latest stable Python
- mamba
- pip # pip must be mentioned explicitly, or conda-lock will fail
- poetry=1.* # or 1.1.*, or no version at all -- as you want
- tensorflow=2.8.0
- pytorch::pytorch=1.11.0
- pytorch::torchaudio=0.11.0
- pytorch::torchvision=0.12.0
# Non-standard section listing target platforms for conda-lock:
platforms:
- linux-64
virtual-packages.yml (may be used e.g. when we want conda-lock to generate CUDA-enabled lock files even on platforms without CUDA):
subdirs:
linux-64:
packages:
__cuda: 11.5
First-time setup
You can avoid playing with the bootstrap env and simplify the example below if you have conda-lock, mamba and poetry already installed outside your target environment.
# Create a bootstrap env
conda create -p /tmp/bootstrap -c conda-forge mamba conda-lock poetry='1.*'
conda activate /tmp/bootstrap
# Create Conda lock file(s) from environment.yml
conda-lock -k explicit --conda mamba
# Set up Poetry
poetry init --python=~3.10 # version spec should match the one from environment.yml
# Fix package versions installed by Conda to prevent upgrades
poetry add --lock tensorflow=2.8.0 torch=1.11.0 torchaudio=0.11.0 torchvision=0.12.0
# Add conda-lock (and other packages, as needed) to pyproject.toml and poetry.lock
poetry add --lock conda-lock
# Remove the bootstrap env
conda deactivate
rm -rf /tmp/bootstrap
# Add Conda spec and lock files
git add environment.yml virtual-packages.yml conda-linux-64.lock
# Add Poetry spec and lock files
git add pyproject.toml poetry.lock
git commit
Usage
The above setup may seem complex, but it can be used in a fairly simple way.
Creating the environment
conda create --name my_project_env --file conda-linux-64.lock
conda activate my_project_env
poetry install
Activating the environment
conda activate my_project_env
Updating the environment
# Re-generate Conda lock file(s) based on environment.yml
conda-lock -k explicit --conda mamba
# Update Conda packages based on re-generated lock file
mamba update --file conda-linux-64.lock
# Update Poetry packages and re-generate poetry.lock
poetry update

To anyone using #michau's answer but having issues including poetry in the environment.yml. Currently, poetry versions 1.2 or greater aren't supported by conda-forge. You can still include poetry v1.2 in the .yml with the below as an alternative:
dependencies:
- python=3.9.*
- mamba
- pip
- pip:
- "poetry>=1.2"

How to export and import a conda environment without errors

I exported a conda environment in this way:
conda env export > environment.yml´
Then commited and pulled the environment.yml file to the git repo.
From another computer I cloned the repo and then tried to create the conda environment:
conda env create -f environment.yml
First I got a warning:
Warning: you have pip-installed dependencies in your environment file,
but you do not list pip itself as one of your conda dependencies.
Conda may not use the correct pip to install your packages, and they
may end up in the wrong place. Please add an explicit pip dependency.
I'm adding one for you, but still nagging you
I don't know why conda export does not include pip in the environment definition.
Then I got errors like wrong/unavailable versions of packages:
es-core-news-sm==3.0.0 version not found
I just removed the version part and only left the name of the package and got it work with:
conda env update --prefix ./env --file environment.yml --prune
Here additional details:
I would like to know how can I avoid this behavior?

es-core-news-sm==3.0 does not exist on pypi, where only 3.1 and 2.3.1 are available, hence your error message.
This is of course something very specific to the environment that you have and the packages that you have installed. In your specific case, just removing the version can be a fix, but no guarantee that this will work in all cases.
As for the cause, I can only guess, but what I expect happened in your case is:
You installed es-core-news-sm==3.0 to your environment
The developers of that package created a newer version and decided to delete the old version
Exporting the environment does correctly state that it contains es-core-news-sm==3.0
Creating an environment from the .yaml from step 3 fails, because the packe is not available any longer (see 2.)
An alternative (depending on your usecase) coul;d be to use conda-pack, which can create a packed version of your environment that you can then unpack. This only works though if the OS on the source and target machine are the same

pip-installed package inside conda virtual environment is not isolated

I am developing a simple python package (on macOS 10.14) and I have problems in setting the instructions how to install it. Right now the package is not available anywhere yet, but think of it as a dummy "hello world" package with a dummy "hello world" function inside a dummy "hello world" module. Of course it has a proper setup.py script that would allow users to install and uninstall the package with pip.
When I install and test it myself everything works fine, the problem is not related to the package itself.
The issue is that I cannot make conda virtual environments and pip work together... Next to my setup.py script there is a environment.yaml file that specifies the dependancies required for my package. Based on this file I create a virtual environment with:
conda env create --prefix ENV -f environment.yaml
I have conda 4.7.12 with Python 3.7.3 inside so the virtual environment has it's own pip. So I activate the virtual environment and explicitly call the pip inside to install my package in the virtual environment:
/Users/my_name/Desktop/dev_dir/ENV/bin/pip install . --user
The installation is successful and the package can be imported. However when I deactivate the virtual environment with conda deactivate and run python interpreter from the conda base environment (version 3.6.9) I can still load my package! For some reason it is available outside of that particular virtual environment...
Later, when I run the 'inner' pip from conda base shell:
/Users/my_name/Desktop/dev_dir/ENV/bin/pip uninstall pkg
The removal seems to go through as well. I get a message:
Uninstalling pkg-0.0.0.9000:
Would remove:
/Users/my_name/.local/lib/python3.7/site-packages/pkg-0.0.0.9000.dist-info/*
/Users/my_name/.local/lib/python3.7/site-packages/pkg/*
Proceed (y/n)? y
Successfully uninstalled pkg-0.0.0.9000
suggesting that the package was indeed installed in a directory .local, outside conda virtual environments.
And the best for the last: even after this uninstallation when I run python interpreters (regardless of which environment from) and I try to import pkg it still works! when I then type pkg in the interpreter I get the path to my development directory:
>>> import pkg
>>> pkg
<module 'pkg' from '/Users/my_name/Desktop/dev_dir/pkg/__init__.py'>
Could someone please help me disentangle this mess? I would like to have my package installed inside the virtual environment, nicely isolated. And also - it should be gone after uninstallation, right?
PS. PYTHONPATH variable is never set anywhere at any stage, I have been checking that...

when I then type pkg in the interpreter I get the path to my development directory
This can only happen if:
You modified your PYTHONPATH to include /Users/my_name/Desktop/dev_dir which you didn't do
You are running the interpreter while you are in the folder /Users/my_name/Desktop/dev_dir, seems likely as you called it your development folder.
Check the output of print(sys.path), which lists all directories that are searched when doing import (standard locations + PYTHONPATH) and also print(os.getcwd()) as the current working directory is also searched
You tried installing your package to your activated conda environment using
/Users/my_name/Desktop/dev_dir/ENV/bin/pip install . --user
Looking at [the docs](https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-user] however:
--user
Install to the Python user install directory for your platform. Typically ~/.local/
So the --user option is messing with your intention to install into the currently active environment. But pip actually does that by default when run inside a virtual environment. So simply do:
conda activate <your envname>
pip install .

#FlyingTeller already correctly identified the issue. I just wanted to point out that you could further streamline your process by adding the installation for your package into your YAML definition. For example,
name: my_env
channels:
- defaults
dependencies:
- python=3.7.3
- pip
- pip:
- -e /Users/my_name/Desktop/dev_dir/pkg
This is also further in line with the best practices (see "Using Pip in a Conda Environment").

Just wanted to hopefully clear some up by telling you this keeps happen to many and if you forget the rule that is NO root install with conda, all rules for your files might change and suddenly it keeps asking for sudo AND fails. Conda = NO SUDO! Hope you got it fixed!

You have to add the pip package to your environment (see https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html), otherwise the packages will be installed by the global pip installation such that those packages can be accessed by all environments.
Therefore, create an environment using
conda create --name exampleenv pip
instead of
conda create --name exampleenv

How to install my own python module (package) via conda and watch its changes

I have a file mysql.py, which I use in almost all of my projects. Since I do not want to copy and paste the same file into each of these projects I wrote a module - possibly a package in the future.
Question
How do I add a local module to my conda environment, and automatically update or watch it when I change something in mysql.py? How to do the same for a package?
I would like to avoid to set up a local channel/repository and just reference to the folder with mysql.py.

If you install the conda build package (and you have a package, not just a script), you can install in "editable" mode:
conda develop .
(running from the directory with your script). This is very similar to the "editable" mode from pip
pip install -e .
Either approach lets you uninstall packages with either
conda develop -u .
or
pip uninstall .
If you just have a script (not a package), you can edit or set the PYTHONPATH environment variable to include the directory with the script.

While the previous answers are doing what I need, I just want to show what I will be using instead. Since it was my plan to learn about conda packages anyway...
0. Good sources
Michael Sarahan - Making packages and packaging "just work" | YouTube
GitHub - audreyr/cookiecutter: A command-line utility that creates projects from cookiecutters (project templates)
and use one of these templates:
GitHub - audreyr/cookiecutter-pypackage: Cookiecutter template for a Python package.
GitHub - conda/cookiecutter-conda-python: A cookiecutter template for conda packages using Python (This is what I used)
1. Create a python package template for conda using cookiecutter
conda install -c conda-forge cookiecutter
Now change to the directory where you want to initialize your package, then do:
cookiecutter https://github.com/conda/cookiecutter-conda-python.git
This will ask for some basic information about the package that you want to create. Then change into your repo
cd myrepo
2. Build your package
make sure conda-build is installed, if not run
conda install conda-build
Make sure to set the CONDA_BLD_PATH as mentioned in anaconda - using a different conda-build root directory - Stack Overflow. This will be the directory where you can find your packages, then run:
conda build conda.recipe
to build your package and clean up after you with
conda build purge
3. Set up your own local channel (no uploading to anaconda.org)
Read
Creating custom channels — Conda documentation
python - add local channel to .condarc on windows - Stack Overflow
for help.
Index each platform. Maybe someone can confirm that this step is not needed, as my builds already contain the repodata.json. Otherwise:
conda index D:\CODE\condamychannel\win-64
Test if the package can be found with
conda search -c file:///D:\CODE\condamychannel --override-channels mypackage
or add the channel to the config directly (per environment)
conda config --add channels file:///D:\CODE\condamychannel
4. Install (and update) the package
activate myenv
and
conda install mypackage
Once I change mypackage, I give it a new version number in meta.yaml and setup.py and build the package with conda build conda.recipe.
Updating is simply
conda update mypackage
See if your package works:
python
>>> import cli from mypackage
>>> cli.cli()
CLI template
This may not be the optimal way, but I could not find a tutorial that contains all the steps I outlined above.

I had a wrapper function that I had to call multiple times on different scripts.So, I copied that file wrappers.py to the Anaconda site-packages folder. On my computer this was at: C:\ProgramData\Anaconda3\Lib\site-packages. Then, whenever I needed wrappers.py, I'd just import it in my scripts, like this:
import wrappers
If you want to make sure that the import was successful, you could either select Anaconda as your dev environment in your IDE and then invoke the Intellisense after import: from wrappers import (intellisense suggestions). Or you could also use IDLE:
>>> import wrappers
>>>(if you get this prompt, this means the import was successful).
Conda integration is only possible if you create custom channels. This is because conda searches for packages in the following locations(based on your OS) and to tell conda to search for your module at certain location(s), you must install conda-build to create a custom channel:
- https://repo.continuum.io/pkgs/main/win-64
- https://repo.continuum.io/pkgs/main/noarch
- https://repo.continuum.io/pkgs/free/win-64
- https://repo.continuum.io/pkgs/free/noarch
- https://repo.continuum.io/pkgs/r/win-64
- https://repo.continuum.io/pkgs/r/noarch
- https://repo.continuum.io/pkgs/pro/win-64
- https://repo.continuum.io/pkgs/pro/noarch
- https://repo.continuum.io/pkgs/msys2/win-64
- https://repo.continuum.io/pkgs/msys2/noarch
If you want to avoid creating a repo/channel, then a simple import as above should do the job. In addition, if you make any changes to your module and save it, you will always have the latest import in your scripts.

How to make sure which program is run behind a terminal command

If I run
ipython notebook
in terminal
it reports
Could not start notebook. Please install ipython-notebook
But I am sure the notebook is indeed install by
conda install ipython-notebook
because
conda install ipython-notebook
gives me
Fetching package metadata: ..
Solving package specifications: .
# All requested packages already installed.
# packages in environment at /home/a/anaconda:
#
ipython-notebook 2.3.1 py27_0
so I guess the command line bunded to a wrong location.
So how can I figure out which binary or directory the command line pointed to?

I am not terribly familar with conda, but looking at the description tells me it is some soft of package management tool. One of its strengths, like the virtualenv package, is that you can have completely different environments (installations) of python packages. This allows you to have a separate set of packages for different requirements, for example.
One drawback is that the different environments need to be activated so that the packages contained therein can be used.
For conda and your particular case, it seems that:
cd ~
source activate anaconda
Will activate the environment stored in $HOME/anaconda/.
Not that conda tells you where the environment is stored:
Fetching package metadata: ..
Solving package specifications: .
# All requested packages already installed.
# packages in environment at /home/a/anaconda:
#
ipython-notebook 2.3.1 py27_0