In my Ubuntu 14.04 computer, I have installed two kinds of Python, one is called native python, which comes along with the Ubuntu operating system, and the other is the conda version, which is installed after I installed the conda package.
If I launch python command, the default python refers to the conda version.
Using conda can bring a lot of advantages for package management. But before I installed conda, I have already installed some Python modules with the native Python by using pip install command. These modules, however, are not reachable by conda Python. So, here is my question: how can I set conda so that it can use the packages managed by native python?
When I ask this question, I cannot help asking another questions:
Is it a good practice to mix packages managed by conda and native python? Any practice I can follow?
How can I switch Python to native Python?
It is not a good practice to mix packages managed by conda and native python. You can still, however, install Python modules using pip into Anaconda. I would recommend strictly using Anaconda (and use conda virtual environments as well as the conda package manager), and not using native python any longer.
Your best bet is to use strictly Anaconda moving forward. I would reinstall the packages into a conda virtual environment.
conda create --name NAME_HERE
or
conda create --name NAME_HERE --clone root if you want to include all packages that come with Anaconda by default.
Then switch to your new environment with source activate NAME_HERE (Linux, macOS) or activate NAME_HERE (Windows). Then you can install packages with both the conda package manager and pip.
See the conda docs on managing conda virtual environments for details.
Kind user #MikhailKnyazev has pointed out that this is how you would use the packages managed by native python. It is still not recommended.
Although it is certainly not a good practice, it is useful to know
that you can add system-side packages inside virtual environment by
symlinking them like this: ln -s /usr/lib/<PYTHON_VER>/dist-packages/<PACKAGE> <virtualenv_path>/lib/<PYTHON_VER>/site-packages/
Related
I've been coding with R for quite a while but I want to start learning and using python more for its machine learning applications. However, I'm quite confused as to how to properly install packages and set up the whole working environment. Unlike R where I suppose most people just use RStudio and directly install packages with install.packages(), there seems to be a variety of ways this can be done in python, including pip install conda install and there is also the issue of doing it in the command prompt or one of the IDEs. I've downloaded python 3.8.5 and anaconda3 and some of my most burning questions right now are:
When to use which command for installing packages? (and also should I always do it in the command prompt aka cmd on windows instead of inside jupyter notebook)
How to navigate the cmd syntax/coding (for example the python documentation for installing packages has this piece of code: py -m pip install "SomeProject" but I am completely unfamiliar with this syntax and how to use it - so in the long run do I also have to learn what goes on in the command prompt or does most of the operations occur in the IDE and I mostly don't have to touch the cmd?)
How to set up a working directory of sorts (like setwd() in R) such that my .ipynb files can be saved to other directories or even better if I can just directly start my IDE from another file destination?
I've tried looking at some online resources but they mostly deal with coding basics and the python language instead of these technical aspects of the set up, so I would greatly appreciate some advice on how to navigate and set up the python working environment in general. Thanks a lot!
Python uses a different way of installing packages. Python has a thing named venv which stands for Virtual Environment. You install all of your packages in venv. Usually for each new project you make a new venv.
By using Anaconda on windows you install everything within the anaconda environment that you have specified.
python -m pip install "modulename" is a command that will install modulename to your default venv. You will be able to use this module when no other venv is specified. Here is the docs page. And here is a tutorial on how to use venv
By default python uses the same directory you have your code in. e.g. C:/Users/me/home/mypythonfile.py will run in C:/Users/me/home/ and will be able to access files in this directory. However you can use ../ to navigate directories or you can specify an absolute path to file you want to open e.g. with open("C:/system32/somesystemfile.sys") as file
Going over the technical differences of conda and pip:
So Conda is a packaging tool and installer that aims to do more than what pip does; handle library dependencies outside of the Python packages as well as the Python packages themselves. Both have many similar functionalities as well, you can install packages or create virtual environments with both.
It is generally advisable to generally have both conda and pip installed since there are some packages which might not be available with conda but with pip and vice versa.
The commands to install in both the ways is easy enough, but one thing to keep in mind is that
conda stores packages in the anaconda/pkgs directory
pip stores it in directory under /usr/local/bin/ for a Unix-based system, or \Program Files\ for Windows
You can use both pip or conda inside the jupyter notebook, it will work just fine, but it may be possible that you get multiple versions of the same package.
Most of the times, you will use cmd only to install a module used in your code, or to create environments, py -m pip install "SomeProject" here basically means that the module "SomeProject" will be downloaded in base env.
You could think of conda as python with a variety of additional functionalities, such as certain pre-installed packages and tools, such as spyder and jupyter. Hence, you must be precise when you say:
I've downloaded python 3.8.5 and anaconda3
Does it mean you installed python in your computer and then also anaconda?
In general, or at least in my opinion, using anaconda has advantages for development, but typically you'll just use a simple python installation in production (if that applies to you).
Anaconda has it's own package registry/repository . When you call conda install <package>, it will search for the package there and install it if available. You would better search it first, for instance matplotlib.
pip is a package manager for the Python Package Index. pip also ships with anaconda. Hence, in an anaconda environment you may install packages from either sources (either using pip install or conda install). For instance, pandas from PyPI and pandas from conda. There is no guarantee that packages exist in both sources. You must either search it first or simply try it.
In your first steps, I would suggest you to stick to only one dev env (either simple python or anaconda, recommend the second). Because that simplifies the question: "which python and which pip is executed in the cmd line?". That said, those commands should work as expected in any terminal, it be a simple cmd or an embedded one like in PyCharm or VS Code.
You could inspect that by running (on windows and linux at least):
which python, which pip.
Honestly, this is a question/answer that falls outside the scope of SO and for more info you would better check official websites, such as for anaconda or search for python vs anaconda blogs.
I need python3.6 for tensorflow installation, so I downloaded python3.6.12.tar. And I found that I should pip install tarfile. However, in this case it is an older version of python. FYI, In my computer(laptop) I installed python3.9.
My question is: can I pip install python.tar inside a virtualenv?
This is not how virtual environments work. I suggest you to do a little bit more research on virtual environments in Python.
Virtual Environments and Packages
Basically you need to install the necessary python version onto your machine. Then go ahead and use that specific python (which is version 3.6 in your case), to create a virtual environment with the command
~ /usr/bin/<path-to-python3.6> -m venv venv
This command will create a folder called venv. Now you need to source the activation script inside this folder to activate your environment.
Handy note: if you are dealing with different versions of python, a more robust way of handling such situations is via using a tool called pyenv.
My Ubuntu system is on python 2.7.15
conda install -c anaconda flask
Anaconda always installs python 3.5 with Flask and other packages. How can I not install python 3.7 and leave python 2.7.15 as is when installing anaconda packages?
The Python you install with anaconda does not interfere at all with your system Python. You can use Anaconda to have multiple Pythons (in multiple conda environments) besides the system Python. You just have to make sure which one is invoked when you run scripts and make sure it's the one you intended.
To answer the "literal" question you asked, you can specify the Python version when installing something:
conda install -c anaconda flask python=2
This will keep your Python at version 2 or report a mismatch if the package you want to install isn't available on anaconda for Python 2. The number of packages dropping Python 2 support is increasing because Python 2 is near it's "end of life", so don't expect to get latest or even near-latest releases of the packages when keeping at Python 2.
Personally I would recommend to create a different environment instead of trying to install to much into the base environment:
conda create -n mypython2environment python=2 flask
And by activating that environment you should be able to use the packages you installed in that environment:
activate mypython2environment
Several IDEs have built-in support for conda environments, so these may be helpful (especially in making sure you use the correct environment and thus the correct Python).
We have an application which does some of its work in Python in a python virtual environment setup using virtualenv.
We've hit a problem where the version of a system library does not match the version installed in the virtual environment. That is we have NetCDF4 installed into the virtual environment and and previously had libnetcdf.so.7 installed through yum. The python package appears to be dependent on having libnetcdf.so.7 available.
Due to a system update libnetcdf.so.7 no longer exists and has been replaced by libnetcdf.so.11.
So the question is this: Does setting up the virtual environment detect the system library version or is there some other mechanism? Also do we need to re-build the environment to fix this or is there another option?
When you use virtualenv to create a virtual environment you have the option of whether or not to include the standard site packages as part of the environment. Since this is now default behaviour (though it can be asserted by using --no-site-packages in the command line) it's possible that you are using an older version of virtualenv that doesn't insist on this.
In that case you should be able to re-create the environment fairly easily. First of all capture the currently-installed packages in the existing environment with the commmand
pip freeze > /tmp/requirements.txt
Then delete the virtual environment, and re-create it with the following commands:
virtualenv --no-site-packages envname
source envname/bin/activate
pip install -r /tmp/requirements.txt
However none of this addresses the tricky issue of not having the required support libraries installed. You might try creating a symbolic link to the new library from the old library's position - it may be thatNetCDF4 can work with multiple versions of libnetCDF and is simply badly configured to use a specific version. If not then solving thsi issue might turn out to be long and painful.
I am a ruby programmer trying to learn python. I am pretty familiar with pyenv since it is like a copy and paste from rbenv. Pyenv helps allow to have more than one version of python in a system and also to isolate the python without touching sensitive parts of system.
I suppose every python installation comes with pip package. What I still don't understand is, there are many good python libs out there that suggest to use this virtualenv and anaconda. I can even find a virtualenv plugin for pyenv.
Now I am getting confused with the purpose of these two pyenv and virtualenv.
worse inside pyenv there is a virtualenv plugin.
My questions are:
what is the difference between pyenv and virtualenv?
Is there any difference in using pip command inside both pyenv and virtualenv?
what does this pyenv virutalenv do?
Your explanation with example will be highly appreciated.
Edit: It's worth mentioning pip here as well, as conda and pip have similarities and differences that are relevant to this topic.
pip: the Python Package Manager.
You might think of pip as the python equivalent of the ruby gem command
pip is not included with python by default.
You may install Python using homebrew, which will install pip automatically: brew install python
The final version of OSX did not include pip by default. To add pip to your mac system's version of python, you can sudo easy_install pip
You can find and publish python packages using PyPI: The Python Package Index
The requirements.txt file is comparable to the ruby gemfile
To create a requirements text file, pip freeze > requirements.txt
Note, at this point, we have python installed on our system, and we have created a requirements.txt file that outlines all of the python packages that have been installed on your system.
pyenv: Python Version Manager
From the docs: pyenv lets you easily switch between multiple versions of Python. It's simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well. This project was forked from rbenv and ruby-build, and modified for Python.
Many folks hesitate to use python3.
If you need to use different versions of python, pyenv lets you manage this easily.
virtualenv: Python Environment Manager.
From the docs: The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.
To create a virtualenv, simply invoke virtualenv ENV, where ENV is is a directory to place the new virtual environment.
To initialize the virtualenv, you need to source ENV/bin/activate. To stop using, simply call deactivate.
Once you activate the virtualenv, you might install all of a workspace's package requirements by running pip install -r against the project's requirements.txt file.
Anaconda: Package Manager + Environment Manager + Additional Scientific Libraries.
**Anaconda is a commercial distribution of Python with the most popular python libraries, you are not permitted to use Anaconda in an organisation with more than 200 employees.
From the docs: Anaconda 4.2.0 includes an easy installation of Python (2.7.12, 3.4.5, and/or 3.5.2) and updates of over 100 pre-built and tested scientific and analytic Python packages that include NumPy, Pandas, SciPy, Matplotlib, and IPython, with over 620 more packages available via a simple conda install <packagename>
As a web developer, I haven't used Anaconda. It's ~3GB including all the packages.
There is a slimmed down miniconda version, which seems like it could be a more simple option than using pip + virtualenv, although I don't have experience using it personally.
While conda allows you to install packages, these packages are separate than PyPI packages, so you may still need to use pip additionally depending on the types of packages you need to install.
See also:
conda vs pip vs virtualenv (section in documentation from anaconda)
the difference between pip and conda (stackoverflow)
the relationship between virtualenv and pyenv (stackoverflow)
Simple analogy:
pyenv ~ rbenv
pip ~ bundler
virtual env ~ gemset in rvm. This can be managed by bundler directly without gemset.
Since I use python3 I prefer the python3 built-in virtual environment named venv. venv is simple and easy to use. I would recommend you to read its official docs. The doc is short and concise.
In ruby, we don't really need a virtual environment because the bundler takes care of it. Both virtual env and bundler are great, however, they have different solutions to solve the same problem.
Simple explanation: https://docs.conda.io/projects/conda/en/latest/commands.html#conda-vs-pip-vs-virtualenv-commands
If you have used pip and virtualenv in the past, you can use conda to perform all of the same operations.
Pip is a package manager
virtualenv is an environment manager
Conda is both