I've had to install a package with pip in a conda environment to get it to work for my application (link).
The package works fine. However, every time I modify the virtual environment in any way, conda tries to install the "missing" package - which would effectively result in downgrading it.
Question: is there a way to mark the pip package as 'manually installed' in the conda venv (e.g. in the same way apt-mark would handle it)? The intention is to get miniconda to leave it alone while still handling the remaining dependencies for the desired additional package.
The pip installed package indeed shows up when typing conda list, with Channel "pypi".
Can give any additional information if needed.
Thanks in advance for any help.
Related
I am using Spyder and trying to add /usr/local/lib/python3.7/site-packages to the PYTHONPATH Manager. However, I receive an error informing me "This directory cannot be added to PATH. If you want to set a different Python interpreter, please go to Preferences > Main Interpreter".
However, I have already changed my interpreter to point to /usr/bin/python3
At the moment, I am using the rather annoying work around of putting the following at the top of all my code.
import sys
sys.path.append("/usr/local/lib/python3.7/site-packages")
Typing the following gives me the below. Is there a way which I can even ensure after running pip3 install XXX in the terminal, that the packages are downloaded somewhere such as the below?
for p in sys.path: print(p)
/Users/user
/usr/local/lib/python3.7
/Users/user/opt/anaconda3/lib/python37.zip
/Users/user/opt/anaconda3/lib/python3.7
/Users/user/opt/anaconda3/lib/python3.7/lib-dynload
/Users/user/opt/anaconda3/lib/python3.7/site-packages
/Users/user/opt/anaconda3/lib/python3.7/site-packages/aeosa
/Users/user/opt/anaconda3/lib/python3.7/site-packages/IPython/extensions
/Users/user/.ipython
Alternatively, and preferably, advice on how to add the above site-packages directory to my PATH? I feel I am missing something obvious.
(Spyder maintainer here) We forbid adding site-packages directories through our PYTHONPATH manager because it allows people to mix two different Python versions (which is what you're trying to do by adding your system site-packages to your Anaconda's Python).
And we do that because it usually generates odd errors and segfaults for binary packages such as Numpy, Pandas and Matplotlib, given that binary packages for one Python version are incompatible with packages for another one.
Finally, even though you found a workaround for that (by using sys.path), we strongly suggest you to stop doing that because it'll give you nothing by headaches in the future.
Doing what you are asking isn't the recommended path forward but you can solve the underlying problem in either of the following ways (A or B).
To "ensure pip installs packages to another location which Spyder can see" as the asker guessed in a comment on the accepted answer which got no answer (Method B below) is usually not a good idea. Keeping a clean environment for Spyder will ensure that you can determine requirements (including package version) for each of your projects reliably. Therefore, do the reverse of what you guessed: Ensure Spyder uses the Python interpreter in the environment where pip installed your project's required packages.
A. Change the Python interpreter
Go to Tools, Preferences, and set Python interpreter to the python executable that was used to install the package (If using a virtual environment, it would be your_other_env/bin/python).
Close and reopen Spyder (Spyder says to restart the IPython console, but it may not work in this case and show the error where Spyder cannot restart a kernel it didn't start).
Open Spyder again and run any py file. You will get an error that says to install the spyder-kernels package (for some reason pip 22.0.4 will only install spyder_kernels: This issue is at "spyder-kernels should be spyder_kernels" :edit: but the issue is invalid, so upgrade pip first such as via pip install --upgrade pip in your virtual environment). Take note of the version in the error, since that is the version you need.
If you are using conda or are on Windows the instructions will differ, so see Common Illnesses in the Spyder documentation instead of continuing this step.
source your_other_env/bin/activate
pip install --upgrade pip setuptools
pip install spyder-kernels=...
deactivate
but change ... to the version shown in your Spyder error from step 3. If you installed Spyder with conda as recommended, use the commands from the URL above instead.
I don't recommend Method B, as I've explained. However, it may be useful if you are manually installing Spyder plugins or test suites that apply to all projects but aren't in the requirements.txt or setup.py requirements for your project(s) (and therefore don't affect determining requirements for your users).
B. To "ensure pip installs packages to another location which Spyder can see" you would run "spyder_env/bin/python -m pip install ..." to install the package, where spyder_env is the virtualenv where Spyder is installed (but if Spyder is installed in the system using an installer or linux distro package, you may need to use your system's python such as via python3 -m pip install --user ... where ... is the package name. Always use --user instead of sudo or root to avoid mismatched files caused by mashing together the distro-packaged modules and your manually installed modules).
I have just tried to update my anaconda environment to the latest version and I am now receiving errors. I opened the conda environment as an admin, and the commands issued were:
conda update conda
conda update anaconda
First command finished fine. Second command produced error:
pythonw.exe - Entry Point Not Found
The procedure entry point ?PyWinObject_FromULARGE_INTEGER##YAPEAU_object##AEAT_ULARGE_INTEGER###Z could not be located in the dynamic link library c:\ProgramData\Anaconda3\pythoncom37.dll
I have found a reference to this sort of error that requires me to copy a file libssl-1-1-x64.dll from Anaconda3/Library/bin with the one from Anaconda3/DLLs.
How to Fix Entry Point Not Found while installing libraries in conda environment
However, I do not have that file, in the source location. Is there any commands I can issue to download this file again, or somewhere online I can safely download that one file from?
Got the same error, when updating conda.
However, the file pythoncom37.dll was located in C:\Windows\System32.
Turns out the file was a left-over from a previous update of Python 3.7.5 to Python 3.8,
i.e. not related to the installation of conda itself. My guess is that conda registered with Python 3.7 and then failed to use the dll from an incompatible installation.
Solution: Removed pythoncom37.dll and pywintypes37 from C:\Windows\System32.
I had the same problem while updating tensorflow and other packages using anaconda python3 with sublime text3.
To solve this, I've deleted all the pythoncom37.dll in directory shown from the error window.
Replacing the file from other directory did not work.
Also reinstalling conda, upgrading conda, reinstalling sublimetext3 or tensorflow did not help as well.
Given that there seem to be a lot of answers and some work for for different people with different setups, python versions and circumstances, a quick summary of things to try.
Go to [envpath]\Scripts and run py pywin32_postinstall.py -install to update the pywin32 dependencies
Copy both files found in [anacondaPath]\Lib\site-packages\pywin32_system32 to C:\Windows\System32
Install pywin32 with conda instead of pip with conda install pywin32
Force pywin32 to a particular version (e.g. 224 for Python 3.7) pip install --upgrade pywin32==224
Add \Lib\site-packages\pywin32_system32 to your path environment variables
Uninstall pypiwin32 and install pywin32. pip uninstall and pip install pywin32
Download the latest Visual C++ version and restart the computer (https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0)
Downgrade to e.g. Python 3.6 if possible for your purposes
If any of those worked, commenting which one in your case may be helpful to understand what works when:)
List item
Sorry all - the clue was in the error message. The entry on how to fix entry point led me in the right direction. but it was the pythoncom37.dll file I needed to copy.
That's what you get for blindly following instructions.
Many thanks.
When I had this error, it did not show a path for the entry point.
I tried reinstalling anaconda and it didn't resolve the issue.
I found the path by doing pip install win32, which stated the path to the library that was was in use. It turned out it was connecting to a corrupt roaming profile version, so renaming the roaming profile folder (to _OLD) resolved the issue.
Had the same problem as on the picture above, solved it using these steps.
removed the file pythoncom37.dll from the environment in question
removed the file C:\tools\Anaconda3\Library\bin\pythoncom37.dll
run conda install --force-reinstall nb_conda_kernels ipykernel
repeat per environment.
Be aware that this will also upgrade all environment packages in the active environment.
I had the same problem. But my virtual environments all worked okay, so I had a workaround:
Create a new virtual environment called 'env_base' with all standard anaconda packages
conda create -n env_base anaconda python=3.7
Activate it
conda activate env_base
Create the kernel
conda install -c anaconda ipykernel
python -m ipykernel install --user --name=env_base
Then use this virtual environment as the base jupyter notebook. You can replace your launch shortcut with the link for this one and it is as good as having the actual Jupyter notebook working with base packages.
It doesn't fix the problem, but it sidesteps it effectively.
I currently have scikit-learn 0.19 installed. I'd like to test my code using the latest development version as there seems to be a fix for Incremental PCA.
How do I go about installing this new version if I've previously installed scikit-learn using anaconda?
Also, how would I revert back to the stable release in the event that 0.20 does not solve my problem?
I am in need of some hand holding here, as I've read the docs on the website and not sure I completely understand the process (especially being able to revert back to the stable version if needed).
The whole point of the Anaconda Python distribution (apart from the convenience of having a bunch of useful packages included) is that you get the conda environment manager, which exists to meet exactly this sort of requirement.
What you want to do is to create a new conda environment by launching the Anaconda prompt and typing
conda create -n myenv scikit-learn other-package other-package2 etc
where myenv is the name you want to give the new environment and other-package other-package2 etc are the names of any other packages you will want to use (import) in your code. conda will figure out any dependencies of these packages and show you a list of what is going to be installed before it proceeds.
If you want to specify that a package should be a particular version, add that to the package name e.g. other-package=1.1.0, otherwise conda will install the latest versions of each package that are mutually compatible. You can also specify a particular version of Python by including it in the package list, e.g. python=3.4. You can check what versions of a package are available with conda search package-name (where package-name is the name of the package you want, obviously).
To run your code in the newly created environment, first activate the environment at the Anaconda prompt. If you use the Spyder IDE, launch it after activating the correct environment, or use the start menu shortcut specific to that environment if you have one. Other IDEs may have their own method of selecting a specific environment to work in.
To revert to the version(s) you were using before, activate the environment containing those versions - if you've never created a new environment before, that'll be root.
Just in case someone comes here looking for a solution without conda:
The website recommends that you download the latest code via
git clone git://github.com/scikit-learn/scikit-learn.git
and then include it in pip via (after changing to the directory)
pip install --editable .
You can also add the --user flag to have pip install to a local directory. Then, uninstalling should be as easy as pip uninstall sklearn.
I tried the conda search --outdated, there are lots of outdated packages, for example the scipy is 0.17.1 but the latest is 0.18.0. However, when I do the conda update --all. It will not update any packages.
update 1
conda update --all --alt-hint
Fetching package metadata .......
Solving package specifications: ..........
# All requested packages already installed.
# packages in environment at /home/user/opt/anaconda2:
#
update 2
I can update those packages separately. I can do conda update scipy. But why I cannot update all of them in one go?
TL;DR: dependency conflicts: Updating one requires (by it's requirements) to downgrade another
You are right:
conda update --all
is actually the way to go1. Conda always tries to upgrade the packages to the newest version in the series (say Python 2.x or 3.x).
Dependency conflicts
But it is possible that there are dependency conflicts (which prevent a further upgrade). Conda usually warns very explicitly if they occur.
e.g. X requires Y <5.0, so Y will never be >= 5.0
That's why you 'cannot' upgrade them all.
Resolving
Update 1: since a while, mamba has proven to be an extremely powerful drop-in replacement for conda in terms of dependency resolution and (IMH experience) finds solutions to problems where conda fails. A way to invoke it without installing mamba is via the --solver=libmamba flag (requires conda-libmamba-solver), as pointed out by matteo in the comments.
To add: maybe it could work but a newer version of X working with Y > 5.0 is not available in conda. It is possible to install with pip, since more packages are available in pip. But be aware that pip also installs packages if dependency conflicts exist and that it usually breaks your conda environment in the sense that you cannot reliably install with conda anymore. If you do that, do it as a last resort and after all packages have been installed with conda. It's rather a hack.
A safe way you can try is to add conda-forge as a channel when upgrading (add -c conda-forge as a flag) or any other channel you find that contains your package if you really need this new version. This way conda does also search in this places for available packages.
Considering your update: You can upgrade them each separately, but doing so will not only include an upgrade but also a downgrade of another package as well. Say, to add to the example above:
X > 2.0 requires Y < 5.0, X < 2.0 requires Y > 5.0
So upgrading Y > 5.0 implies downgrading X to < 2.0 and vice versa.
(this is a pedagogical example, of course, but it's the same in reality, usually just with more complicated dependencies and sub-dependencies)
So you still cannot upgrade them all by doing the upgrades separately; the dependencies are just not satisfiable so earlier or later, an upgrade will downgrade an already upgraded package again. Or break the compatibility of the packages (which you usually don't want!), which is only possible by explicitly invoking an ignore-dependencies and force-command. But that is only to hack your way around issues, definitely not the normal-user case!
1 If you actually want to update the packages of your installation, which you usually don't. The command run in the base environment will update the packages in this, but usually you should work with virtual environments (conda create -n myenv and then conda activate myenv). Executing conda update --all inside such an environment will update the packages inside this environment. However, since the base environment is also an environment, the answer applies to both cases in the same way.
To answer more precisely to the question:
conda (which is conda for miniconda as for Anaconda) updates all but ONLY within a specific version of a package -> major and minor. That's the paradigm.
In the documentation you will find "NOTE: Conda updates to the highest version in its series, so Python 2.7 updates to the highest available in the 2.x series and 3.6 updates to the highest available in the 3.x series."
doc
If Wang does not gives a reproducible example, one can only assist.
e.g. is it really the virtual environment he wants to update or could Wang get what he/she wants with
conda update -n ENVIRONMENT --all
*PLEASE read the docs before executing "update --all"!
This does not lead to an update of all packages by nature. Because conda tries to resolve the relationship of dependencies between all packages in your environment, this can lead to DOWNGRADED packages without warnings.
If you only want to update almost all, you can create a pin file
echo "conda ==4.0.0" >> ~/miniconda3/envs/py35/conda-meta/pinned
echo "numpy 1.7.*" >> ~/miniconda3/envs/py35/conda-meta/pinned
before running the update. conda issues not pinned
If later on you want to ignore the file in your env for an update, you can do:
conda update --all --no-pin
You should not do update --all. If you need it nevertheless you are saver to test this in a cloned environment.
First step should always be to backup your current specification:
conda list -n py35 --explicit
(but even so there is not always a link to the source available - like for jupyterlab extensions)
Next you can clone and update:
conda create -n py356 --clone py35
conda activate py356
conda config --set pip_interop_enabled True # for conda>=4.6
conda update --all
conda config
update:
Currently I would use mamba (or micromamba) as conda pkg-manager replacement
update:
Because the idea of conda is nice but it is not working out very well for complex environments I personally prefer the combination of nix-shell (or lorri) and poetry [as superior pip/conda .-)] (intro poetry2nix).
Alternatively you can use nix and mach-nix (where you only need you requirements file. It resolves and builds environments best.
On Linux / macOS you could use nix like
nix-env -iA nixpkgs.python37
to enter an environment that has e.g. in this case Python3.7 (for sure you can change the version)
or as a very good Python (advanced) environment you can use mach-nix (with nix) like
mach-nix env ./env -r requirements.txt
(which even supports conda [but currently in beta])
or via api like
nix-shell -p nixFlakes --run "nix run github:davhau/mach-nix#with.ipython.pandas.seaborn.bokeh.scikit-learn "
Finally if you really need to work with packages that are not compatible due to its dependencies, it is possible with technologies like NixOS/nix-pkgs.
Imagine the dependency graph of packages, when the number of packages grows large, the chance of encountering a conflict when upgrading/adding packages is much higher. To avoid this, simply create a new environment in Anaconda.
Be frugal, install only what you need. For me, I installed the following packages in my new environment:
pandas
scikit-learn
matplotlib
notebook
keras
And I have 84 packages in total.
I agree with Mayou36.
For example, I was doing the mistake to install new packages in the base environment using conda for some packages and pip for some other packages.
Why this is bad?
1.None of this is going to help with updating packages that have been > installed >from PyPI via pip, or any packages installed using python
setup.py install. conda list will give you some hints about the
pip-based Python packages you have in an environment, but it won't do
anything special to update them.
And I had all my projects in the same one environment! And I used update all -which is bad and did not update all-.
So, the best thing to do is to create a new environment for each project. Why?
2. A Conda environment is a directory that contains a specific collection of Conda packages that you have installed. For example, you
may be working on a research project that requires NumPy 1.18 and its
dependencies, while another environment associated with an finished
project has NumPy 1.12 (perhaps because version 1.12 was the most
current version of NumPy at the time the project finished). If you
change one environment, your other environments are not affected. You
can easily activate or deactivate environments, which is how you
switch between them.
So, to wrap it up:
Create a new environment for each project
Be aware for the differences in conda and pip
3.Only include the packages that you will actually need and update them properly only if necessary.
if working in MS windows, you can use Anaconda navigator. click on the environment, in the drop-down box, it's "installed" by default. You can select "updatable" and start from there
To update all possible packages I used conda update --update-all
It works!
I solved this problem with conda and pip.
Firstly, I run:
conda uninstall qt and conda uninstall matplotlib and conda uninstall PyQt5
After that, I opened the cmd and run this code that
pip uninstall qt , pip uninstall matplotlib , pip uninstall PyQt5
Lastly, You should install matplotlib in pip by this code that pip install matplotlib
I am working on a project where they are using Ansible to run several conda installs. I need to install two additional packages from github that have dependencies that are already covered by the existing conda installs with the second package having a dependency on the first.
Using the Ansible code below, I can get the first package to install without reinstalling the dependencies.
- name: install mypackage
shell: /home/myname/envs/myproject/bin/pip install --install-option="--prefix=/home/myname/envs/myproject" --egg https://github.com/myname/mypackage/archive/my_branch.zip
This gets me 95% of the way there, however, when I try to install the second package, it doesn't recognize the first package as having been installed and fails.
I am new to this and I have been throwing things up against the wall but I'm not able to install the first package in such a way where:
It recognizes the existing conda installs
The second package identifies the first one
From what I can understand from your task you are using a venv to install the packages, that's good. I don't understand why, though, you are using the shell module to handle the install.. This not good.
You can handle all this with ansible' pip module :
- name: "Install mypackage"
pip:
virtualenv: /home/{{ lookup('env','USER') }}/envs/myproject/
name: "{{ item }}"
with_items:
- "https://github.com/myname/mypackage1/archive/my_branch.zip"
- "https://github.com/myname/mypackage2/archive/my_branch.zip"
This should install correctly the packages in the order you require, without the hassle of having to work your way through shell output.
Note that you can mix normal python packages with eggs etc..
As an alternative to virtualenv you can use executable.
Have a look at the docs
I believe the question is how to use ansible to pip install packages within a conda environment. Noting that it is perfectly possible to use pip install within a conda environment, which is particularly useful in cases where the desired package does not exist on the conda repositories and cannot be installed with conda install.
The goal is thus to use the environment created by conda, and not a virtualenv (for which, btw, ansible's pip module provides specific parameters).
I have managed to do so by using ansible's pip module and pointing the pip executable to the one installed within the desired conda environment.
See code below, notice usage of the executable variable:
- name: Install pip packages WITHIN a designated conda environment
pip:
name: some_package_name
executable: "/home/[username]/[anaconda3]/envs/[conda_env_name]/bin/pip"
# ^-- Of course you will need to ensure the correct path.
This will pip install the packages inside the designated conda environment.