In some documents and resources, it has been noted that installing packages using pip after installing conda could cause some problems like crashing conda and so on. could anybody tell me when and in which cases is that happening? I tried pip for some packages due to the smallness of conda's repository and I found no problem using both together.
I'm using pip 21.1.2 and conda 4.10.1.
The point is that pip is less strict wrt. to dependency checks, so it's clearly better to prioritize conda installs over pip:
"Pip and conda differ in how dependency relationships within an environment are fulfilled. When installing packages, pip installs dependencies in a recursive, serial loop. No effort is made to ensure that the dependencies of all packages are fulfilled simultaneously. This can lead to environments that are broken in subtle ways, if packages installed earlier in the order have incompatible dependency versions relative to packages installed later in the order. In contrast, conda uses a satisfiability (SAT) solver to verify that all requirements of all packages installed in an environment are met. This check can take extra time but helps prevent the creation of broken environments." (https://www.anaconda.com/blog/understanding-conda-and-pip)
However, some packages are only available via PyPI, and you have to resort to pip. In that case stick to the best practices as recommended in the conda docs.
(https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#using-pip-in-an-environment)
So far I never experienced any inconsistencies with conda and pip installs myself, and when you are a serious programmer and have your test suites properly set up, you'll anyhow get aware of issues well in advance and before they can cause any trouble.
I've been asked to look at some dev ops stuff regarding python and I'm a bit stuck. The network I'm working is not internet connected so I've been setting up Nexus repositories to bring in dependencies for docker, java and pypi that the other developers can access and pull down locally. However, they have started using conda more and more and we are on a fixed version on our dev network to match a delivery network.
I'm trying to use nexus' conda repos although every time I try and install something it tries to update everything else, including the python and conda versions which are:
conda version : 4.8.3
conda-build version : 3.18.11
python version : 3.8.3.final.0
I've edited my .condarc file to read:
channels:
- http://master:8041/repository/anaconda-proxy/main/
- http://master:8041/repository/conda-forge/
remote_read_timeout_secs: 1200.0
auto_update_conda: false
channel_priority: false
However every time i try to install something to cache the dependencies I get an huge list of updates. For example:
conda install cudatoolkit
<snip>
The following packages will be downloaded:
package | build
---------------------------|-----------------
alabaster-0.7.12 | py_0 16 KB http://master:8041/repository/anaconda-proxy/main
anaconda-client-1.7.2 | py38_0 172 KB http://master:8041/repository/anaconda-proxy/main
anaconda-project-0.8.4 | py_0 210 KB http://master:8041/repository/anaconda-proxy/main
argh-0.26.2 | py38_0 36 KB http://master:8041/repository/anaconda-proxy/main
.....
Any advice would be great. I've added the auto_update_conda and channel_priority flags but to no avail. Thanks in advance.
Additonal info:
I'm a Java developer and I only use a bit of python, so I'm not massively familar with the anaconda setup so apologies if this is simpler than I'm making it.
How Conda Solves
Conda always first attempts to solve the install directive without changing existing packages (i.e., it runs first with a --freeze-installed flag) and will only proceed to a full solve (what you are seeing) if it can't find any version of your requested package that already has all its dependencies satisfied in the environment. That is, this result implies that what you are asking for is not possible. Or at least not via the CLI if you want a valid environment.1
At the core of the issue is that even if there is only a single dependency that needs updating, there is no intermediate mode to indicate that you want to minimize the total number of changes (which I think would actually be a nice enhancement). Conda only has two solving modes:
Change nothing else (--freeze-installed).
All dependencies are allowed to update (--update-deps).
The exception to this are the aggressive_update_packages and the auto_update_conda, which it will always attempt to update whenever the environment is mutated. But it seems you've already realized those can be disabled through configuration settings.2
Manual Dependency Updating
This doesn't mean what you are hoping to accomplish is impossible, but that there isn't a clean way to automate it via the CLI. Instead, you might need to manually track down the dependencies that need updating (e.g., conda search cudatoolkit --info), update them first (conda install with specific versions), and then try installing your package again. I would strongly recommend first settling on the exact version of cudatoolkit you plan to install, otherwise conda search cudatoolkit --info will be too much info.
Package Pinning
For packages that you really do want absolutely fixed there is package pinning. You could do this for conda, python, and other core packages.
Base Environment
I find it a bit odd that the base environment (the one that has the conda package) is being mutated at all. Instead, I would expect software engineers to always use non-base environments for development and production. It is easy to create new environments, one can define them with version controlled YAML files, use them modularly by creating them on a per project or per task-type basis, and they can be mutated without worrying about affecting the Conda infrastructure. However, I'm not entirely clear on your setup, so this comment may not apply.
[1] If one doesn't care about validity (probably not a good idea for production) then there is always the --no-deps flag.
[2] The default aggressive_update_packages packages are ones that frequently become vulnerable to exploits (e.g., openssl), so carefully consider the implications of leaving them outdated.
I have recently started using the Anaconda Python distribution as it offers a lot of Data Analysis libraries out of the box. And using conda to create environments and install packages is also a breeze. But I have faced some serious issues when I want to update Python itself or any other module, I am informed beforehand that a LOT of my existing libraries will be removed.
For example, this is what I get when I use conda update [package_name]
$ conda update pandas
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done
## Package Plan ##
environment location: C:\Users\User\Anaconda3
added / updated specs:
- matplotlib
The following packages will be REMOVED:
[Almost half of my existing packages]
The following packages will be UPDATED:
[Some packages including my desired library, in this case, pandas]
I have searched the web on how to update packages and Python using conda and almost everywhere I saw that conda update [package name] was suggested. But why doesn't it work for me? I mean it will work but at the expense of tons of important libraries that I need.
So I have tried using the Anaconda Navigator to update the desired libraries (like matplotlib and pandas) hoping that the removal of existing libraries might be a command line issue on my computer. But I had seriously messed up my base (root) environment by updating pandas using Navigator. I didn't get any warnings that a lot of my modules will be removed so I thought I was doing fine. But after the update was done and I wrote some matplotlib code, I wasn't able to run it. I got errors that resembled something that indicated matplotlib was a "non-conda module". So I had to do conda install --revision n to go back to a state where I had my modules.
Right now, the only way for me to update any package or Python is to do this:
conda install pandas=[package_version_that_is_higher_than_mine]
But there's got to be a reason why I am facing this issue. Any help is absolutely appreciated.
EDIT: It turns out that the issue is mainly when I am trying to update using the base environment. When I use my other conda environments, the conda update [package_name] or conda update --all works fine.
Anaconda (as distinct from Conda) is designed to be used as a fixed set of package builds that have been vetted for compatibility (see "What's in a Name? Clarifying the Anaconda Metapackage). When you try to introduce new packages or package upgrades into that context, Conda can be rather unpredictable as to how it will solve that. I think it helps to keep in mind that commands like conda (install|upgrade|remove) mean requesting a distinct environment as a whole, and do not represent low-level commands to change a single package.
Conda does offer some options to get this more low-level behavior. One thing to try is the --freeze-installed flag, which would do what you're asking for. Recent versions of Conda do this by default in the first round of solves, and if it doesn't work then it attempts a full solve. There is also the more dangerous and brute force --no-dep flag, which won't do a solve at all and just install the package. The documentation for this literally says,
"This WILL lead to broken environments and inconsistent behavior. Use at your own risk."
Typically, if you want to use newer packages, it is better to create a new env (conda create -n my_env [pkg1 pkg2 ...]) because the fact is that you no longer want the Anaconda distribution, but instead a custom one with newer versions. My personal view is that most non-beginners should be using Miniconda and relegate their base env to only having conda, while being very liberal about creating envs for projects that have different package requirements. If you ever need a true Anaconda distribution, there's always the anaconda package for that.
I have a reasonable understanding of the difference between conda install & pip install; How pip installs python only packages & conda can install non-python binaries. However, there is some overlap between these two. Which leads me to ask:
What's the rule of thumb for whether to use conda or pip when both offer a package?
For example, TensorFlow is available on both repositories but from the tensorflow docs:
within Anaconda, we recommend installing TensorFlow with the
pip install command, not with the conda install command.
But, there are many other packages that overlap, like numpy, scipy etc.
However, this Stackoverflow answer suggests that conda install should be the default & pip should only be used if a package is unavailable from conda. Is this true even for TensorFlow or other python-only packages?
The Tensorflow maintainers actually publish the wheels of TensorFlow on PyPI that's why it's the recommended official way. The conda packages are created by the Anaconda staff and/or the community. That doesn't mean the conda packages are bad, it just means that the TensorFlow maintainers don't participate there (officially). Basically they are just saying: "If you have trouble installing it with pip the TensorFlow devs will try to help you. But we don't officially support the conda packages so if something goes wrong with the conda package you need to ask the conda-package maintainers. You've been warned."
In the more general case:
For Python-only packages you should always use conda install. There might be exceptions, for example if there is no conda-package at all or the conda package is out-of-date (and nobody is releasing a new version of that package) and you really need that package/version.
However it's different for packages that require compilation (e.g. C-Extensions, etc.). It's different because with pip you can install a package either:
as pre-compiled wheel
as package, compiled on your computer
While conda just provides the
compiled conda package
With compiled packages you have to be careful with binary compatibility. That means that a package is compiled against specific binary interface of another library - which could depend on the version of the libraries or the compilation flags, etc.
With conda you have to take the package as-is, which means that you have to assume that the packages are binary-compatible. If they aren't it won't work (segfault or linking errors or whatever).
If you use pip and can choose which wheel (if any) to install or compile it against the available libraries on your computer. That means it's less likely that you get a binary-incompatibility. That is (or was) a big problem if you install conda packages from different conda-channels. Because they might simply be binary-incompatible (e.g. conda-forge and the anaconda-channel have or had a few problems there).
However it should probably be decided on a case-by-case basis. I had no problems with my tensorflow conda environment where I installed all packages from the conda-forge channel, including tensorflow. However I have heard that several people had trouble with tensorflow in mixed conda-forge and anaconda channel environments. For example NumPy from the main channel and TensorFlow from the conda-forge channel might just be binary-incompatible.
My rule of thumb is:
If it's a Python-only package just install it (it's unlikely to make trouble). Use the conda package when possible but it won't make (much) trouble if you use pip. If you install it using pip it's not managed by conda so it's possible it won't be recognized as available dependency and you have to update it yourself, but that's about all the difference.
If it's a compiled package (like a C extension or a wrapper around a C library or such like) it becomes a bit more complicated. If you want to be "careful" or you have reason to expect problems:
Always create a new environment if you need to test compiled packages from different channels and/or conda and pip. It's easy to discard a messed up conda environment but it's a lot more annoying to fix an environment that you depend on.
If possible install all compiled packages using conda install from one and only one channel (if possible the main anaconda channel).
If not possible try to mix main anaconda channel compiled packages with conda packages from a different channel.
If that doesn't work try to mix conda compiled packages and pip compiled-packages (pre-compiled wheels or self-compiled installers).
You asked about why you cannot install packages from PyPI with conda. I don't know the exact reasons but pip mostly provides the package and you have to install it yourself. With conda you get an already compiled and installed package that is just "copied" without installation. That requires that the package is installed on different operating systems (Mac, Windows, Linux) and on platforms (32-bit, 64-bit), against different Python versions (2.7, 3.5, 3.6) and possibly against different NumPy versions. That means conda has to provide several packages instead of just one. That takes resources (space for the final installed packages and time for the installation) which probably aren't available or feasible. Aside from that there is probably no converter for a pypi package to a conda recipe aside from all the specifics you have to know about a package (compilation, installation) to make it work. That's just my guess though.
I recently discovered Conda after I was having trouble installing SciPy, specifically on a Heroku app that I am developing.
With Conda you create environments, very similar to what virtualenv does. My questions are:
If I use Conda will it replace the need for virtualenv? If not, how do I use the two together? Do I install virtualenv in Conda, or Conda in virtualenv?
Do I still need to use pip? If so, will I still be able to install packages with pip in an isolated environment?
Conda replaces virtualenv. In my opinion it is better. It is not limited to Python but can be used for other languages too. In my experience it provides a much smoother experience, especially for scientific packages. The first time I got MayaVi properly installed on Mac was with conda.
You can still use pip. In fact, conda installs pip in each new environment. It knows about pip-installed packages.
For example:
conda list
lists all installed packages in your current environment.
Conda-installed packages show up like this:
sphinx_rtd_theme 0.1.7 py35_0 defaults
and the ones installed via pip have the <pip> marker:
wxpython-common 3.0.0.0 <pip>
Short answer is, you only need conda.
Conda effectively combines the functionality of pip and virtualenv in a single package, so you do not need virtualenv if you are using conda.
You would be surprised how many packages conda supports. If it is not enough, you can use pip under conda.
Here is a link to the conda page comparing conda, pip and virtualenv:
https://docs.conda.io/projects/conda/en/latest/commands.html#conda-vs-pip-vs-virtualenv-commands.
I use both and (as of Jan, 2020) they have some superficial differences that lend themselves to different usages for me. By default Conda prefers to manage a list of environments for you in a central location, whereas virtualenv makes a folder in the current directory. The former (centralized) makes sense if you are e.g. doing machine learning and just have a couple of broad environments that you use across many projects and want to jump into them from anywhere. The latter (per project folder) makes sense if you are doing little one-off projects that have completely different sets of lib requirements that really belong more to the project itself.
The empty environment that Conda creates is about 122MB whereas the virtualenv's is about 12MB, so that's another reason you may prefer not to scatter Conda environments around everywhere.
Finally, another superficial indication that Conda prefers its centralized envs is that (again, by default) if you do create a Conda env in your own project folder and activate it the name prefix that appears in your shell is the (way too long) absolute path to the folder. You can fix that by giving it a name, but virtualenv does the right thing by default.
I expect this info to become stale rapidly as the two package managers vie for dominance, but these are the trade-offs as of today :)
EDIT: I reviewed the situation again in 04/2021 and it is unchanged. It's still awkward to make a local directory install with conda.
Virtual Environments and pip
I will add that creating and removing conda environments is simple with Anaconda.
> conda create --name <envname> python=<version> <optional dependencies>
> conda remove --name <envname> --all
In an activated environment, install packages via conda or pip:
(envname)> conda install <package>
(envname)> pip install <package>
These environments are strongly tied to conda's pip-like package management, so it is simple to create environments and install both Python and non-Python packages.
Jupyter
In addition, installing ipykernel in an environment adds a new listing in the Kernels dropdown menu of Jupyter notebooks, extending reproducible environments to notebooks. As of Anaconda 4.1, nbextensions were added, adding extensions to notebooks more easily.
Reliability
In my experience, conda is faster and more reliable at installing large libraries such as numpy and pandas. Moreover, if you wish to transfer your preserved state of an environment, you can do so by sharing or cloning an env.
Comparisons
A non-exhaustive, quick look at features from each tool:
Feature
virtualenv
conda
Global
n
y
Local
y
n
PyPI
y
y
Channels
n
y
Lock File
n
n
Multi-Python
n
y
Description
virtualenv creates project-specific, local environments usually in a .venv/ folder per project. In contrast, conda's environments are global and saved in one place.
PyPI works with both tools through pip, but conda can add additional channels, which can sometimes install faster.
Sadly neither has an official lock file, so reproducing environments has not been solid with either tool. However, both have a mechanism to create a file of pinned packages.
Python is needed to install and run virtualenv, but conda already ships with Python. virtualenv creates environments using the same Python version it was installed with. conda allows you to create environments with nearly any Python version.
See Also
virtualenvwrapper: global virtualenv
pyenv: manage python versions
mamba: "faster" conda
In my experience, conda fits well in a data science application and serves as a good general env tool. However in software development, dropping in local, ephemeral, lightweight environments with virtualenv might be convenient.
Installing Conda will enable you to create and remove python environments as you wish, therefore providing you with same functionality as virtualenv would.
In case of both distributions you would be able to create an isolated filesystem tree, where you can install and remove python packages (probably, with pip) as you wish. Which might come in handy if you want to have different versions of same library for different use cases or you just want to try some distribution and remove it afterwards conserving your disk space.
Differences:
License agreement. While virtualenv comes under most liberal MIT license, Conda uses 3 clause BSD license.
Conda provides you with their own package control system. This package control system often provides precompiled versions (for most popular systems) of popular non-python software, which can easy ones way getting some machine learning packages working. Namely you don't have to compile optimized C/C++ code for you system. While it is a great relief for most of us, it might affect performance of such libraries.
Unlike virtualenv, Conda duplicating some system libraries at least on Linux system. This libraries can get out of sync leading to inconsistent behavior of your programs.
Verdict:
Conda is great and should be your default choice while starting your way with machine learning. It will save you some time messing with gcc and numerous packages. Yet, Conda does not replace virtualenv. It introduces some additional complexity which might not always be desired. It comes under different license. You might want to avoid using conda on a distributed environments or on HPC hardware.
Another new option and my current preferred method of getting an environment up and running is Pipenv
It is currently the officially recommended Python packaging tool from Python.org
Conda has a better API no doubt. But, I would like to touch upon the negatives of using conda since conda has had its share of glory in the rest of the answers:
Solving environment Issue - One big thorn in the rear end of conda environments. As a remedy, you get advised to not use conda-forge channel. But, since it is the most prevalent channel and some packages (not just trivial ones, even really important ones like pyspark) are exclusively available on conda-forge you get cornered pretty fast.
Packing the environment is an issue
There are other known issues as well. virtualenv is an uphill journey but, rarely a wall on the road. conda on the other hand, IMO, has these occasional hard walls where you just have to take a deep breath and use virtualenv
1.No, if you're using conda, you don't need to use any other tool for managing virtual environments (such as venv, virtualenv, pipenv etc).
Maybe there's some edge case which conda doesn't cover but virtualenv (being more heavyweight) does, but I haven't encountered any so far.
2.Yes, not only can you still use pip, but you will probably have to. The conda package repository contains less than pip's does, so conda install will sometimes not be able to find the package you're looking for, more so if it's not a data-science package.
And, if I remember correctly, conda's repository isn't updated as fast/often as pip's, so if you want to use the latest version of a package, pip might once again be your only option.
Note: if the pip command isn't available within a conda virtual environment, you will have to install it first, by hitting:
conda install pip
Yes, conda is a lot easier to install than virtualenv, and pretty much replaces the latter.
I work in corporate, behind several firewall with machine on which I have no admin acces
In my limited experience with python (2 years) i have come across few libraries (JayDeBeApi,sasl) which when installing via pip threw C++ dependency errors
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools
these installed fine with conda, hence since those days i started working with conda env.
however it isnt easy to stop conda from installing dependency inside c.programfiles where i dont have write access.