There is syntax used in requirements.txt that allows dependencies from a github project (see here).
I want to work with conda and keep requirements file compatible with pip (for others), so I don't want to use yml environment files.
Is there a way to create a file that both pip and conda could install from?
Is there a way to create a file that both pip and conda could install from?
No, not generally. This is because Conda additionally manages non-Python dependencies (e.g., dynamic libraries compiled from other languages), which are things that typically are not provided by PyPI. So, even though Conda can output a Pip-like requirements.txt (i.e., with conda list --export), some of the packages are specific to Conda.
As for installing GitHub projects into Conda envs, anything one can do with Pip can be accomplished in a YAML file, e.g., working from the referenced question:
YAML
name: my_env
channels:
- defaults
dependencies:
- pip
- pip:
- -e git://github.com/mozilla/elasticutils.git#egg=elasticutils
would install a GitHub-hosted package using Pip. See the Advanced Pip Example in the Conda repository, for additional examples.
Using a Pip Requirements File in Conda
One can also use a Pip requirements.txt indirectly in Conda by creating a YAML wrapper for it. For example,
requirements.txt
-e git://github.com/mozilla/elasticutils.git#egg=elasticutils
my_env.yaml
name: my_env
channels:
- defaults
dependencies:
- pip
- pip:
- -r requirements.txt
which you can use either to create a new env:
conda env create -f my_env.yaml
or to update an existing env:
conda env update -f my_env.yaml
and it will use Pip to install from GitHub.
for this add -e in front of the link and add the link normally.
-e https://github.com/something.git
like this you have to add .
the requirements file can be used for both pip and conda
in pip
pip install -r requirements.txt
in conda
conda install --yes --file requirements.txt
Related
I'm trying to follow the best practice of installing fully pinned dependencies (for repeatable builds and better Docker caching, see this pythonspeed.com article).
My project needs to use both conda and pip (conda for complex ML packages, pip for stuff not available on conda). The conda-lock and pip-compile tools are able to generate all transitive dependencies at pinned versions. However, these tools are independent: when I run pip-compile, it's not aware of the dependencies that conda-lock wants to install, and vice versa.
This results in different package versions, causing wasted space in the Docker image and potentially causing breakage/incompatibility, as the pip install step installs different versions of some transitive dependencies.
Does anyone have a better solution for creating pinned Python dependency lists when using both conda and pip?
(Edit: here's a github ticket on conda-lock to support pip dependencies: https://github.com/conda-incubator/conda-lock/issues/4)
Instead of using a tool that solves the depedencies, you could just install all the dependencies and then use conda env export to generate a pinned/versioned environment.yaml.
Main downside: this is heavier weight, as it actually has to install all the dependencies. On the upside, you end up with just a single environment "spec" environment file as input, and a single environment "lock" file as output.
Specify direct dependencies in environment-spec.yaml
Specify both conda and pip dependencies together. Example:
name: base
channels:
- conda-forge
- defaults
# etc.
dependencies:
- matplotlib
- pandas
- pip # needed to have a pip section below
- scikit-learn
- pip:
- pyplot_themes # only available on PyPI
Install dependencies and export pinned versions (including transitive dependencies)
This could be done directly on your local machine, but here's how to isolate this process in Docker:
# syntax=docker/dockerfile:1
# Note: using miniconda instead of micromamba because micromamba lacks the
# `conda env export` command.
FROM continuumio/miniconda3:4.9.2
COPY environment-spec.yml /environment-spec.yml
# mounts are for conda caching and pip caching
RUN --mount=type=cache,target=/opt/conda/pkgs --mount=type=cache,target=/root/.cache \
conda env create -n regen_env --file /environment-spec.yml
# Export dependencies.
RUN conda env export -n regen_env > /environment-lock-raw.yml
CMD ["cat", "/environment-lock.yml"]
Then you can create a pinned environment file like so (assuming the above dockerfile was named regen_environment.Dockerfile):
docker build -t regen_env -f regen_enviroment.Dockerfile .
docker run --rm regen_env > environment-lock.yaml
This outputs the pinned enviroment file to environment-lock.yaml, which you can then install with conda install -f environment-lock.yaml.
(Here's a gist with some more references and details: https://gist.github.com/jli/b2d2d62ad44b7fcb5101502c08dca1ae)
I'm trying to set a conda environment using a requirements.txt file that a coworker shared with me. My coworker uses Python in a Mac without Anaconda, and I'm using it in a Windows machine with Anaconda. The file requirements.txt was generated with the command pip freeze and looks like this:
absl-py==0.7.1
affine==2.3.0
agate==1.6.0
agate-dbf==0.2.0
agate-excel==0.2.1
agate-sql==0.5.2
...
After checking the answer of this question, I tried the following in the Anaconda terminal:
conda create --name my-env-name --file requirements.txt
Which fails with the following error message:
PackagesNotFoundError: The following packages are not available from current channels:
- appscript==1.0.1
- style==1.1.0
- senticnet==1.3
- scikits.optimization==0.3
...
My understanding is that this happens because those packages are not available in the Anaconda package installation channels, and that they should be installed instead via pip with my conda environment activated, using pip install -r requirements.txt
The problem is that this list of packages is very long, and I would like to avoid having to manually check and separating which packages are included in Anaconda channels and which should be installed via pip. Then, is there a way to tell Anaconda to create an environment by automatically recognizing the packages included in its channels, installing them, and then installing the rest using pip?
Using requirements.txt with conda
There's no problem at all using a requirements.txt file when creating a conda environment.
In fact, you can also set additional channels at creation time:
conda create --name my-env-name --file requirements.txt --channel <NAME_OF_CHANNEL>
for example, in the case of the first package you mention, you can install it from anaconda channel. So you could run:
conda create --name my-env-name --file requirements.txt --channel default --channel anaconda
Why using default channel first? Well, just to give preference to the default one (the priority of channels is specified by the order they are listed: higher priority from left to right).
When at least some of the packages are not available using conda
Well, when no conda channel can provide any of your required packages, there are several alternatives:
Install through conda those packages available in any of its channels.
Install through pip the rest.
Create a conda environment.yml file:
conda env export > environment.yml
When you need to recreate this environment, then you can do:
conda env create --name my-env-name --file environment.yml
and it will install the packages using conda, will install pip, and then will install those packages only available with the latter one.
This approach has good and bad properties:
one of the good properties is that it separates those packages installed through conda from those installed using pip.
one of the bad properties is that it's only useful for conda, but not for pip alone.
I want to set up an environment.myl file for a project's conda environment. I have a local package that I would normally use pip install -e . so I can work on the code locally. Is there a way to use pip to install this package with the env file?
I tried this based on something I found using install options with github links, but doesn't work.
name: foo
channels:
- defaults
dependencies:
- python=3.7
- pip
- pip:
- /Users/me/projects/package/ --install-option="-e"
As far as I can tell from reading the code, conda-env will copy the entries in the pip dictionary and place them into a temporary pip requirements file. Hence, you should follow the Requirements File Format, namely,
name: foo
channels:
- defaults
dependencies:
- python=3.7
- pip
- pip:
- -e /Users/me/projects/package
I did a quick test on a local package and I was able to verify that the package installed and shows up in pip list -e.
There is also an advanced-pip/ example in the repo that illustrates some additional options.
If I have a directory with setup.py, in pip, I can pip install . in the directory to install the package.
What if I am using conda?
conda install . makes conda to find a package named dot.
conda packages are a different structure than standard python packaging. As a result, the official, recommended and best-practice approach is to use conda to install pip within an activated conda environment, and use that to install standard packages:
conda install pip
NOTE: You want to use conda packages whenever they're available, as they have more features within a conda environment than non-conda packages.
conda install pip will install pip within the currently activated conda environment, and will ensure that it is integrated with conda so that, for example, conda list, will include any packages installed with pip.
NOTE: Commands like conda update will ignore pip installed packages, as it only checks conda channels for available updates, so they still need to be updated using pip. See this Question/Answer discussion:
Does conda update packages from pypi installed using pip install?
NOTE: See #kalefranz comment below regarding conda 4.6 experimental handling of packages.
If you're interested in creating your own conda package(s), take a look at this question/1st answer for a great run-down:
How to install my own python module (package) via conda and watch its changes
If you simply wish to install non-conda packages, using pip is the correct, and expected, way to go.
You can use pip install from within conda environment.
Just activate your environment using:
$ conda activate myenvironment
and use pip install . to install your package in environment's directory.
EDIT: As pointed by Chris Larson in another answert, you should install pip inside the environment using
$ conda install pip
in order to register packages correctly.
If I have a whl file, I can use pip install xxx.whl to install it.
From the documentation, conda install from a local file is also available, but the file should be a tarball file, i.e. .tar.bz2 files.
conda install /package-path/package-filename.tar.bz2 works. And if I have multiple tarballs, I can tar them to get a .tar file, then conda install /packages-path/packages-filename.tar installs the packages in it.
Can I install/upgrade packages from GitHub using conda?
For example, with pip I can do:
pip install git+git://github.com/scrappy/scrappy#master
to install scrappy directly from the master branch in GitHub. Can I do something equivalent with conda?
If this is not possible, would it make any sense to install pip with conda and manage such local installations with pip?
The answers are outdated. You simply have to conda install pip and git. Then you can use pip normally:
Activate your conda environment source activate myenv
conda install git pip
pip install git+git://github.com/scrappy/scrappy#master
There's better support for this now through conda-env. You can, for example, now do:
name: sample_env
channels:
dependencies:
- requests
- bokeh>=0.10.0
- pip:
- "--editable=git+https://github.com/pythonforfacebook/facebook-sdk.git#8c0d34291aaafec00e02eaa71cc2a242790a0fcc#egg=facebook_sdk-master"
It's still calling pip under the covers, but you can now unify your conda and pip package specifications in a single environment.yml file.
If you wanted to update your root environment with this file, you would need to save this to a file (for example, environment.yml), then run the command: conda env update -f environment.yml.
It's more likely that you would want to create a new environment:
conda env create -f environment.yml (changed as supposed in the comments)
conda doesn't support this directly because it installs from binaries, whereas git install would be from source. conda build does support recipes that are built from git. On the other hand, if all you want to do is keep up-to-date with the latest and greatest of a package, using pip inside of Anaconda is just fine, or alternately, use setup.py develop against a git clone.
I found a reference to this in condas issues. The following should now work.
name: sample_env
channels:
dependencies:
- requests
- bokeh>=0.10.0
- pip:
- git+https://github.com/pythonforfacebook/facebook-sdk.git