Develop Reference

How to completely reset everything Python to default in Ubuntu 18.04?

Edit:
So now which pip, which python and which python3 have started working correctly somehow. I was also able to install libraries using pip. The only remaining problem is that of the apt_pkg. Running apt-get install python-apt gives:
Reading package lists... Done
Building dependency tree
Reading state information... Done
python-apt is already the newest version (1.6.3ubuntu1).
Running any python application still gives error: ImportError: No module named apt_pkg
Running echo $PATH returns:
/home/hamza/bin:/home/hamza/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
Running find /usr -name python -o -name python3 returns:
/usr/local/lib/python3.5/dist-packages/notebook/static/components/codemirror/mode/python
/usr/local/lib/python2.7/dist-packages/notebook/static/components/codemirror/mode/python
/usr/lib/python3
/usr/lib/gimp/2.0/python
/usr/lib/libreoffice/share/Scripts/python
/usr/lib/virtualbox/sdk/bindings/xpcom/python
/usr/lib/wx/python
/usr/bin/python
/usr/bin/python3
/usr/share/code/resources/app/extensions/python
/usr/share/bash-completion/helpers/python
/usr/share/bash-completion/completions/python
/usr/share/bash-completion/completions/python3
/usr/share/python
/usr/share/javascript/codemirror/mode/python
/usr/share/python3
/usr/share/lintian/overrides/python
/usr/share/lintian/overrides/python3
/usr/share/gcc-8/python
/usr/share/gdb/python
/usr/share/librevenge/python
/usr/share/doc/python
/usr/share/doc/python3
/usr/share/doc/python-statsmodels/examples/python
/usr/share/playonlinux/python
Original:
Python on my system is a wreck. I have got multiple versions installed. None work properly. Apparently some applications using python are not working either (eg: Unity Tweaks). My python libraries will not install because pip doesn't work, although it says it is already installed
I was trying to install some libraries for python on my system using pip. Turned out pip was not working. I tried to reinstall pip only to learn that pip was already installed. I went through a process of understanding the difference between python and python3. I have tried enough things and have basically lost track of the things I have done to get the libraries installed. Long story short, pip does not work no matter what, my libraries will not install and consequently my code doesn't run.
Here is some of the things I have already done:
which pip
This return no output
apt-get install python-pip
Says I have pip already installed
apt-get install python-apt
To resolve ImportError: No module named apt_pkg. It says:
Reading package lists... Done
Building dependency tree
Reading state information... Done
python-apt is already the newest version (1.6.3ubuntu1).
But the error persists
And countless other stuff I do not remember.
What I want now is simple. I wish to somehow get out of this hassle by resetting all of my python settings and installation to default, and then install the pandas and numpy libraries for my system. Any and all help would be appreciated.
Result for python -m pip -V:
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
Result for python3 -m pip -V:
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.5)

One of the following commands worked for me:
pip3 install -U pip
python -m pip install -U pip
python3 -m pip install -U pip
Make sure the version of your python3 is 3.6, which is the default version of ubuntu18, then upgrade pip.
export py3_path=`which python3`
rm $py3_path
ln -s `which python3.6` $py3_path
python3 -m pip install -U pip
Or use pyenv to manage multi versions https://github.com/pyenv/pyenv

pip 10 and apt: how to avoid "Cannot uninstall X" errors for distutils packages

I am dealing with a legacy Dockerfile. Here is a very simplified version of what I am dealing with:
FROM ubuntu:14.04
RUN apt-get -y update && apt-get -y install \
python-pip \
python-numpy # ...and many other packages
RUN pip install -U pip
RUN pip install -r /tmp/requirements1.txt # includes e.g., numpy==1.13.0
RUN pip install -r /tmp/requirements2.txt
RUN pip install -r /tmp/requirements3.txt
First, several packages are installed using apt, and then several packages are installed using pip. pip version 10 has been released, and part of the release is this new restriction:
Removed support for uninstalling projects which have been installed using distutils. distutils installed projects do not include metadata indicating what files belong to that install and thus it is impossible to actually uninstall them rather than just remove the metadata saying they've been installed while leaving all of the actual files behind.
This leads to the following problem in my setup. For example, first apt installs python-numpy. Later pip tries to install a newer version of numpy from e.g., /tmp/requirements1.txt, and tries to uninstall the older version, but because of the new restriction, it cannot remove this version:
Installing collected packages: numpy
Found existing installation: numpy 1.8.2
Cannot uninstall 'numpy'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
Now I know at this point there are several solutions.
I could not install python-numpy through apt. However, this causes issues because python-numpy installs a few different packages as requirements, and I do not know if another part of the system relies on these packages. And in reality, there are several apt packages installed through the Dockerfile, and each one I remove seems to reveal another Cannot uninstall X error, and removes a number of other packages along with it, that our app may or may not rely on.
I could also use the --ignore-installed option when I try to pip install things that have already been installed through apt, but then again I have the same problem of every --ignore-installed argument revealing yet another thing that needs to be ignored.
I could pin pip at an older version that does not have this restriction, but I don't want to be stuck using an outdated version of pip forever.
I have been going around in circles trying to come up with a good solution that involves minimal changes to this legacy Dockerfile, and allows the app we deploy with that file to continue to function as it has been. Any suggestions as to how I can safely get around this problem of pip 10 not being able to install newer versions of distutils packages? Thank you!
UPDATE:
I did not realize that --ignore-installed could be used without a package as an argument to ignore all installed packages. I am considering whether or not this might be a good option for me, and have asked about it here.

This is the solution I ended up going with, and our apps have been running in production without any issues for close to a month with this fix in place:
All I had to do was to add
--ignore-installed
to the pip install lines in my dockerfile that were raising errors. Using the same dockerfile example from my original question, the fixed dockerfile would look something like:
FROM ubuntu:14.04
RUN apt-get -y update && apt-get -y install \
python-pip \
python-numpy # ...and many other packages
RUN pip install -U pip
RUN pip install -r /tmp/requirements1.txt --ignore-installed # don't try to uninstall existing packages, e.g., numpy
RUN pip install -r /tmp/requirements2.txt
RUN pip install -r /tmp/requirements3.txt
The documentation I could find for --ignore-installed was unclear in my opinion (pip install --help simply says "Ignore the installed packages (reinstalling instead)."), and I asked about the potential dangers of this flag here, but have yet to get satisfying answer. However, if there are any negative side effects, our production environment has yet to see the effects of them, and I think the risk is low/none (at least that has been our experience). I was able to confirm that in our case, when this flag was used, the existing installation was not uninstalled, but that the newer installation was always used.
Update:
I wanted to highlight this answer by #ivan_pozdeev. He provides some information that this answer does not include, and he also outlines some potential side-effects of my solution.

This is what worked for me--
pip install --ignore-installed <Your package name>
or
sudo pip install --ignore-installed <Your package name>
or (inside jupyter notebook)
import sys
!{sys.executable} -m pip install --ignore-installed <Your package name>

For windows
write
conda update --all
pip install --upgrade <Your package name>
OR
conda update --all
pip install <Your package name>
OR
pip install wrapt --upgrade --ignore-installed
pip install <Your package name>
from ERROR: Cannot uninstall 'wrapt'. during upgrade

You can just remove numpy manually but keep the other dependencies installed by apt. Then use pip as before to install the latest version of numpy.
#Manually remove just numpy installed by distutils
RUN rm /usr/lib/python2.7/dist-packages/numpy-1.8.2.egg-info
RUN rm -r /usr/lib/python2.7/dist-packages/numpy
RUN pip install -U pip
RUN pip install -r /tmp/requirements1.txt
The location of numpy should be the same. But if you want to confirm the location you can run the container without running the requirements.txt files and issue the following commands in the python console inside the container.
>>> import numpy
>>> print numpy.__file__
/usr/lib/python2.7/dist-packages/numpy/__init__.pyc

Python 3: ImportError "No Module named Setuptools"

I'm having troubles with installing packages in Python 3.
I have always installed packages with setup.py install. But now, when I try to install the ansicolors package I get:
importerror "No Module named Setuptools"
I have no idea what to do because I didn't have setuptools installed in the past. Still, I was able to install many packages with setup.py install without setuptools. Why should I get setuptools now?
I can't even install setuptools because I have Python 3.3 and setuptools doesn't support Python 3.
Why doesn't my install command work anymore?

Your setup.py file needs setuptools. Some Python packages used to use distutils for distribution, but most now use setuptools, a more complete package. Here is a question about the differences between them.
To install setuptools on Debian:
sudo apt-get install python3-setuptools
For an older version of Python (Python 2.x):
sudo apt-get install python-setuptools

EDIT: Official setuptools dox page:
If you have Python 2 >=2.7.9 or Python 3 >=3.4 installed from
python.org, you will already have pip and setuptools, but will need to
upgrade to the latest version:
On Linux or OS X:
pip install -U pip setuptools
On Windows:
python -m pip install -U pip setuptools
Therefore the rest of this post related to Distribute is obsolete (e.g. some links don't work).
EDIT 2022-02-04
From Python 3.10 Distutils is deprecated and will be removed in Python 3.12 - use setuptools:
The entire distutils package is deprecated, to be removed in Python 3.12. Its functionality
for specifying package builds has already been completely replaced by
third-party packages setuptools and packaging ...
Distribute (deprecated)
Distribute - is a setuptools fork which "offers Python 3 support". Installation instructions for distribute(setuptools) + pip:
curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py
easy_install pip
Similar issue here.
UPDATE: Distribute seems to be obsolete, i.e. merged into Setuptools: Distribute is a deprecated fork of the Setuptools project. Since the Setuptools 0.7 release, Setuptools and Distribute have merged and Distribute is no longer being maintained. All ongoing effort should reference the Setuptools project and the Setuptools documentation.
You may try with instructions found on setuptools pypi page (I haven't tested this, sorry :( ):
wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py -O - | python
easy_install pip

Make sure you are running the latest version of pip
I tried to install Ansible and it failed with
ModuleNotFoundError: No module named 'setuptools_rust'
python3-setuptools was already in place, so upgrading pip solved it.
pip3 install -U pip

I was doing this inside a virtualenv on Oracle Linux 6.4 using Python 2.6, so the apt-based solutions weren't an option for me, nor were the Python 2.7 ideas. My fix was to upgrade my version of setuptools that had been installed by virtualenv:
pip install --upgrade setuptools
After that, I was able to install packages into the virtualenv.

The solution which worked for me was to upgrade my setuptools:
python3 -m pip install --upgrade pip setuptools wheel

For others with the same issue due to a different reason: This can also happen when there's a pyproject.toml in the same directory as the setup.py, even when setuptools is available.
Removing pyproject.toml fixed the issue for me.

pip uninstall setuptools
and then:
pip install setuptools
This works for me and fixes my issue.

When there's a pyproject.toml in the same directory as the setup.py, it can be the cause of the issue. I renamed that file, but it didn't solve the issue, so I restablished the original file name, and did the following change.
Under the [build-system] section, I added "setuptools" to the requires= list, and it worked.

First step #1
You have to install setuptools
On Linux:
pip install -U pip setuptools
On Mac OS:
pip install -U pip setuptools
On Windows:
python -m pip install -U pip setuptools
Second step #2
Make sure you have made it accessible (make it available in environmental variables)
On Linux
export PATH="INSTALLATIONDIRECTORY:$PATH"
On Mac OS
Sorry, I don't know.
On Windows
Open the Start Search, type in “env”, and choose “Edit the system environment variables”
Click the “Environment Variable” button.
Set the environment variables as needed. The New button adds an additional variable.
Dismiss all of the dialogs by choosing “OK”. Your changes are saved!

The distribute package provides a Python 3-compatible version of setuptools: http://pypi.python.org/pypi/distribute
Also, use pip to install the modules. It automatically finds dependencies and installs them for you.
It works just fine for me with your package:
[~] pip --version
pip 1.2.1 from /usr/lib/python3.3/site-packages (python 3.3)
[~] sudo pip install ansicolors
Downloading/unpacking ansicolors
Downloading ansicolors-1.0.2.tar.gz
Running setup.py egg_info for package ansicolors
Installing collected packages: ansicolors
Running setup.py install for ansicolors
Successfully installed ansicolors
Cleaning up...
[~]

this is how my problem was solved => pip3 install setuptools-rust
If you want to check your list => pip3 list
i faced this problem while trying to install elastalert2
System informations
CentOS Linux release 7.9.2009 (Core)
Python 3.6.8
pip 21.3.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)

I ran into this problem when my pip requirements.txt file contained an editable library that was built using poetry and contained a pyproject.toml file. Following the documentation for setuptools, my solution was to add setuptools to the build-system requirements in the pyproject.toml file as follows:
[build-system]
requires = ["poetry-core>=1.0.0", "setuptools"]
build-backend = "poetry.core.masonry.api"

If pip isn't installed, like for example if it's coming from the Deadsnakes PPA, or a Docker environment, the best way to fix this error is by bootstrapping it by running
python -m ensurepip

Windows 7:
I have given a complete solution here for Python Selenium WebDriver:
Setup easy install (Windows - simplified)
download ez.setup.py (https://bootstrap.pypa.io/ez_setup.py) from 'https://pypi.python.org/pypi/setuptools'
move ez.setup.py to C:\Python27\
open cmd prompt
cd C:\Python27\
C:\Python27\python.exe ez.setup.py install

I ran sudo python setup.py build_ext -i and it failed with No module named setuptools.
I solved it with this command:
<i>sudo apt-get install python-setuptools</i>

A few years ago I inherited a Python (2.7.1) project running under Django-1.2.3 and now was asked to enhance it with QR possibilities. I got the same problem and did not find pip or apt-get either. So I solved it in a totally different, but easy way.
I /bin/vi-ed the setup.py and changed the line
"from setuptools import setup"
into:
"from distutils.core import setup"

The PyPA recommended tool for installing and managing Python packages is pip. pip is included with Python 3.4 (PEP 453), but for older versions here's how to install it (on Windows):
Download https://bootstrap.pypa.io/get-pip.py
>c:\Python33\python.exe get-pip.py
Downloading/unpacking pip
Downloading/unpacking setuptools
Installing collected packages: pip, setuptools
Successfully installed pip setuptools
Cleaning up...
>c:\Python33\Scripts\pip.exe install pymysql
Downloading/unpacking pymysql
Installing collected packages: pymysql
Successfully installed pymysql
Cleaning up...

On macOS, if you have homebrew installed, simply run:
brew install rust

If you still find this issue, try this:
python3 -m pip install scrapy --upgrade --force --user

While trying to install socketIO I had the same issue. At my system (Windows 11) setupTools were there twice. At C:\Program Files\Python310\Lib\site-packages\ and at C:\Users\user\AppData\Roaming\Python\Python310\site-packages\
Had to uninstall one of them.

Working on Debian, sometime it can be because of older pip version. Therefore, update pip>
python -m pip install -U pip

I had such a problem to install pyunicorn. I wanted to install it on Google Colab. I tried installing pyunicorn directly from Github and it worked for me. For example in my case it was like this:
pip install git+https://github.com/pik-copan/pyunicorn.git#egg=pyunicorn

Python GDAL package missing header file when installing via pip

I'm trying to install gdal from pip pip install gdal inside a virtual environment (Ubuntu). It fails because it cannot find cpl_port.h
extensions/gdal_wrap.cpp:2853:22: fatal error: cpl_port.h: No such file or directory
compilation terminated
However GDAL is installed correctly and the header file is located at /usr/include/gdal/cpl_port.h. Is there some environment variable for GDAL that needs to be set in order for pip to find the header files?

As suggested in the other thread, exporting some shell variables before running pip worked flawlessly. A path for *_INCLUDE_PATH can be found with gdal-config --cflags.
# GDAL library must have been installed
sudo apt-get install libgdal-dev
# Set up pip and/or virtualenv stuff
...
# Now install Python binding for GDAL
export CPLUS_INCLUDE_PATH=/usr/include/gdal
export C_INCLUDE_PATH=/usr/include/gdal
pip install GDAL

Tomyun's answer worked for me, with the proviso that you have to ensure that the version of GDAL-dev installed via apt-get matches the version being installed by pip.
For Ubuntu 14.04, the commands are:
# GDAL library must have been installed
sudo apt-get install libgdal-dev
# Set up pip and/or virtualenv stuff
...
# Now install Python binding for GDAL
export CPLUS_INCLUDE_PATH=/usr/include/gdal
export C_INCLUDE_PATH=/usr/include/gdal
pip3 install GDAL=1.10.0

Taken from this comment, it solved my issue directly
pip3 install GDAL==$(gdal-config --version)
Explicit version number should match your existing libgdal version. –

Using PIP :
pip install --no-install GDAL
Then cd into ENV/build/GDAL
python setup.py build_ext --include-dirs=/usr/include/gdal
pip install --no-download GDAL
(Source: http://ubuntuforums.org/showthread.php?t=1769445)
Using Buildout :
[gdal-bindings]
recipe = zc.recipe.egg:custom
egg = GDAL==1.9.1
include-dirs = /usr/include/gdal
library-dirs = /usr/lib

I was also getting this error when test installing in a virtual environment a package of mine that depends on GDAL. In this case the solution is to change the dependecy from GDAL to pygdal in the install_requires parameter in setup.py. Like so:
install_requires=['pygdal'],

This is what worked for me:
I had to get the latest hearder versions for installing gdal 2.2.4 through pip:
sudo apt-add-repository ppa:ubuntugis/ubuntugis-unstable
sudo apt update
sudo apt install libgdal-dev
Before that, I was getting extensions/gdal_wrap.cpp:3172:27: fatal error: cpl_vsi_error.h: No such file or directory, even when including the correct "include" path to pip.
The the pip installation (in a virtualenv):
pip install --global-option=build_ext --global-option="-I/usr/include/gdal" gdal

try to do:
brew install gdal
after that try again.

Installing SciPy with pip

It is possible to install NumPy with pip using pip install numpy.
Is there a similar possibility with SciPy? (Doing pip install scipy does not work.)
Update
The package SciPy is now available to be installed with pip!

Prerequisite:
sudo apt-get install build-essential gfortran libatlas-base-dev python-pip python-dev
sudo pip install --upgrade pip
Actual packages:
sudo pip install numpy
sudo pip install scipy
Optional packages:
sudo pip install matplotlib OR sudo apt-get install python-matplotlib
sudo pip install -U scikit-learn
sudo pip install pandas
src

An attempt to easy_install indicates a problem with their listing in the Python Package Index, which pip searches.
easy_install scipy
Searching for scipy
Reading http://pypi.python.org/simple/scipy/
Reading http://www.scipy.org
Reading http://sourceforge.net/project/showfiles.php?group_id=27747&package_id=19531
Reading http://new.scipy.org/Wiki/Download
All is not lost, however; pip can install from Subversion (SVN), Git, Mercurial, and Bazaar repositories. SciPy uses SVN:
pip install svn+http://svn.scipy.org/svn/scipy/trunk/#egg=scipy
Update (12-2012):
pip install git+https://github.com/scipy/scipy.git
Since NumPy is a dependency, it should be installed as well.

In Ubuntu 10.04 (Lucid), I could successfully pip install scipy (within a virtualenv) after installing some of its dependencies, in particular:
$ sudo apt-get install libamd2.2.0 libblas3gf libc6 libgcc1 libgfortran3 liblapack3gf libumfpack5.4.0 libstdc++6 build-essential gfortran libatlas-sse2-dev python-all-dev

To install scipy on windows follow these instructions:-
Step-1 : Press this link http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy to download a scipy .whl file (e.g. scipy-0.17.0-cp34-none-win_amd64.whl).
Step-2: Go to the directory where that download file is there from the command prompt (cd folder-name ).
Step-3: Run this command:
pip install scipy-0.17.0-cp27-none-win_amd64.whl

I tried all the above and nothing worked for me. This solved all my problems:
pip install -U numpy
pip install -U scipy
Note that the -U option to pip install requests that the package be upgraded. Without it, if the package is already installed pip will inform you of this and exit without doing anything.

If I first install BLAS, LAPACK and GCC Fortran as system packages (I'm using Arch Linux), I can get SciPy installed with:
pip install scipy

On Fedora, this works:
sudo yum install -y python-pip
sudo yum install -y lapack lapack-devel blas blas-devel
sudo yum install -y blas-static lapack-static
sudo pip install numpy
sudo pip install scipy
If you get any public key errors while downloading, add --nogpgcheck as parameter to yum, for example:
yum --nogpgcheck install blas-devel
On Fedora 23 onwards, use dnf instead of yum.

For the Arch Linux users:
pip install --user scipy prerequisites the following Arch packages to be installed:
gcc-fortran
blas
lapack

Addon for Ubuntu (Ubuntu 10.04 LTS (Lucid Lynx)):
The repository moved, but a
pip install -e git+http://github.com/scipy/scipy/#egg=scipy
failed for me... With the following steps, it finally worked out (as root in a virtual environment, where python3 is a link to Python 3.2.2):
install the Ubuntu dependencies (see elaichi), clone NumPy and SciPy:
git clone git://github.com/scipy/scipy.git scipy
git clone git://github.com/numpy/numpy.git numpy
Build NumPy (within the numpy folder):
python3 setup.py build --fcompiler=gnu95
Install SciPy (within the scipy folder):
python3 setup.py install

In my case, it wasn't working until I also installed the following package : libatlas-base-dev, gfortran
sudo apt-get install libatlas-base-dev gfortran
Then run pip install scipy

install python-3.4.4
scipy-0.15.1-win32-superpack-python3.4
apply the following commend doc
py -m pip install --upgrade pip
py -m pip install numpy
py -m pip install matplotlib
py -m pip install scipy
py -m pip install scikit-learn

The answer is yes, there is.
First you can easily install numpy use commands:
pip install numpy
Then you should install mkl, which is required by Scipy, and you can download it here
After download the file_name.whl you install it
C:\Users\****\Desktop\a> pip install mkl_service-1.1.2-cp35-cp35m-win32.whl
Processing c:\users\****\desktop\a\mkl_service-1.1.2-cp35-cp35m-win32.whl
Installing collected packages: mkl-service
Successfully installed mkl-service-1.1.2
Then at the same website you can download scipy-0.18.1-cp35-cp35m-win32.whl
Note:You should download the file_name.whl according to you python version, if you python version is 32bit python3.5 you should download this one, and the "win32" is about your python version, not your operating system version.
Then install file_name.whl like this:
C:\Users\****\Desktop\a>pip install scipy-0.18.1-cp35-cp35m-win32.whl
Processing c:\users\****\desktop\a\scipy-0.18.1-cp35-cp35m-win32.whl
Installing collected packages: scipy
Successfully installed scipy-0.18.1
Then there is only one more thing to do: comment out a specfic line or there will be error messages when you imput command "import scipy".
So comment out this line
from numpy._distributor_init import NUMPY_MKL # requires numpy+mkl
in this file: your_own_path\lib\site-packages\scipy__init__.py
Then you can use SciPy :)
Here tells you more about the last step.
Here is a similar anwser to a similar question.

Besides all of these answers,
If you install python of 32bit on your 64bit machine, you have to download scipy of 32-bit irrespective of your machine.
http://www.lfd.uci.edu/~gohlke/pythonlibs/
In the above URL you can download the packages and command is: pip install

For gentoo, it's in the main repository:
emerge --ask scipy

You can also use this in windows with python 3.6 python -m pip install scipy