Develop Reference

pip installed module but python gives Import error [duplicate]

During the past years, I have installed many Python libraries with various Python versions. To make them ready to work immediately, I installed them blindly without control. Currently they're causing problems when I tried to install pynest which invokes numpy, scipy and matplotlib. After struggling, I am going to clean and reinstall Python and the libraries.
After investigation, I found Python 2.5/2.6/2.7/3.2 on my system, and each of them has some copies or other things at: (my OS == Mac OS X 10.7.5 Lion)
/Library/Frameworks/
/opt/local/Library/Frameworks/
/opt/local/bin/
/Applications/
/usr/local/bin/
/usr/bin/
/System/Library/Frameworks/
I know I'm crazy to have these. Now I have removed all these except the things in /System/Libarary/Frameworks (I never remove any thing from /System/Library/). After the clean work, which python now gives /usr/bin/python which links to /System/Library/Frameworks.
Now, is it a clear environment for me to reinstall python? How to double check that there's no other versions existing? How should I reinstall them to guarantee that they and their libraries won't be everywhere and have many copies again?
I want to install a clean Python 2.7 onto a proper location, and make my system know exactly where it is and never install any libraries somewhere else. Please give me some advice that how to manage it like in a professional way.
For your information, here is my current $PATH, I think it should be modified:
/opt/local/bin:/opt/local/sbin:/opt/nest/lib/python2.7/site-packages:/usr/local/lib/python2.7/site-packages:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/texbin:/Library/Frameworks/Python.framework/Versions/3.2/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/texbin:/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/django/bin:/usr/X11/bin:/opt/local/bin:/opt/local/sbin:/usr/local/lib/python2.7/site-packages:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Library/Frameworks/Python.framework/Versions/3.2/bin
Please let me know If you need more information. Thank you!
UPDATE:
I'm rethinking profoudly why it becomes so crazy. I believe it's because I installed things via:
easy_install / macports / homebrew / fink / pip sometimes;
.dmg sometimes;
.pkg sometimes;
compile source code sometimes;
and they made things at different locations. I wonder what's the mechanism behind these ways? How do they choose target location? How to prevent them from messing things up?

Why did it get messed up?
There're a couples of different way to install Python, as the update of OP says, and they locate files in different locations. For example, macports puts things into /opt/local/, while homebrew puts things into /usr/local/. Also, Mac OS X brings a few python versions with itself. So, if you install python many times via different ways, you will get many python versions existing independently on your system.
What problem does it cause?
I don't know exactly. I guess the problem is that if you have many versions of python, then which one to use and where to find packages will be determined by the path order in your system PATH and the PYTHONPATH respectively. So you may lose control of where to install python modules. Consider that if you run sudo python setup.py install to install a module (it finds python by the root's PATH) and then try to import the module by python -c "import it" (this time it finds python by your PATH), maybe something will go wrong. This is my guess, I didn't validate it. But in my own case, something did go wrong.
How to avoid this?
I think the principle would be that be aware of that different ways and tools install things independently to different locations, so use them mindfully.
Unless you intend to, don't install the same thing twice via different
ways. (If you intend to do it for python, you might want to check out virtualenv)
Keep an eye on the path order in your PATH and consider if it's
correct.
When installing modules, be clear which python (or pip) is
running and where the module is installed.
So, how did I solve my own case?
Since it had been messing up already and seemed to be very hard to cure, so finally I solved this question by a full OS re-installation, and started to follow the DOs-and-DONTs above. For the installation of the scientific environment with python (numpy/scipy/matplotlib, which had shown problems to make me ask this question), I found this tutorial was extremely helpful. So, problem solved finally.

Here is what was confusing me and how I solved it.
$ which python
/usr/bin/python
$ which python3
/usr/local/bin/python3
$ ls /usr/local/bin/python
ls: /usr/local/bin/python: No such file or directory
So notice I didn't have a HomeBrew installation of python2.7, but did have the python3 installation. The version under /usr/bin/python is using the system default. You can tell based on the module search path:
$ /usr/bin/python
Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
`enter code here`Type "help", "copyright", "credits" or "license" for
more information.
>>> import sys
>>> sys.path
['', '/Library/Python/2.7/...
Notice the '/Library/Python'... that's Mac OS's version of python. But I want to stay strictly on a user installed version (i.e. HomeBrew).
So here's what I did to fix this:
$ brew install python
...
Warning: python 2.7.13 is already installed, it's just not linked.
You can use `brew link python` to link this version.
$ brew link --overwrite python
$ which python
/usr/local/bin/python
$ python
Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/local/Cellar/python/2.7.13...
Its no longer /Library/.. but /usr/local.
Now its finding all of my pip installed modules! Problem solved!
UPDATE:
After updating brew to version 1.5.4, it seems the symbolic links were removed. And now you have to add this to your path:
export PATH="/usr/local/opt/python/libexec/bin:$PATH"
Read the Caveats section in 'brew info python':
==> Caveats
This formula installs a python2 executable to /usr/local/bin.
If you wish to have this formula's python executable in your PATH then add
the following to ~/.bash_profile:
export PATH="/usr/local/opt/python/libexec/bin:$PATH"
Pip and setuptools have been installed. To update them
pip2 install --upgrade pip setuptools
You can install Python packages with
pip2 install <package>
They will install into the site-package directory
/usr/local/lib/python2.7/site-packages
See: https://docs.brew.sh/Homebrew-and-Python.html

In order to install a python distributions into specific folder, you can use the --prefix scheme during python installation. Using the prefix scheme, you can for example install Python 2.7 into the folder /opt/py27. Now, in order to use the new installed Python distribution you have to: cleanup you PATH and LD_LIBRARY_PATH:
Remove all 'old' Python paths and
configure (according to my example) the environment variables like this:
PATH: Add /opt/py27/bin
LD_LIBRARY_PATH: Add /opt/py27/lib
That's it.
(In case you need multiple environments of Python installed at the same time, I'd suggest to have a look at virtualenv)

tl;dr
brew install python
Symptoms
I had similar issues with python programs not finding dependencies.
My python3 version was a broken symlink.
My pip was pointing to a python 3.8
And my pip3 was pointing to 3.9
python -V was outputting some python 2.7 version
python3 -V was outputting some python3.8 version
Solution
I ran brew install python and it fixed all my problems.

Install a newer Python when there are installed packages linked against the old one

(Please bear with me for the long description, it has been quite a troubleshooting journey.)
My ultimate goal is to get python package graph_tool working on my system and also on ipynb if possible. I have already brew install graph-tool, as indicated here, but that's still insufficient.
So I follow conda instructions here, and try to fulfill all requirements including having cgal and python3.6. I had anaconda with python2.7 and python3.5 originally, but since python3.6 is necessary, I download anaconda3 with python3.6. (conda update python still kept giving me 3.5)
I delete /Users/mymacbook/anaconda, so that /Users/mymacbook/anaconda3 would the default search directory. And I have a mini-success!
$ python3
Python 3.6.2 |Anaconda, Inc.| (default, Sep 21 2017, 18:29:43)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Unfortunately, as I continue to conda install the other packages required (boost, cgal, etc), my python3 is reverted back to 3.5 :(
$ python3
Python 3.5.4 |Anaconda custom (x86_64)| (default, Oct 5 2017, 02:58:14)
$ conda install python=3.6
Fetching package metadata ...........
Solving package specifications: .
UnsatisfiableError: The following specifications were found to be in conflict:
- cgal -> python 3.5*
- python 3.6*
Use "conda info <package>" to see the dependencies for each package.
$ conda info cgal
Fetching package metadata ...........
ResolvePackageNotFound:
- cgal
Quite a pickle isn't it... A recommendation was to uninstall cgal due to ResolvePackageNotFound, but like I mentioned, I need both cgal and python3.6.
Thanks in advance for the help! (and please include applicable command lines with your suggestions - I'm a beginner)

Looking for Packages when you do a conda create .. or conda install ... , or even a conda search, it will only present you results consistent with your OS and in some cases, base python version.
What's really available You can go directly to Anaconda.org and search for you package. This will surface builds on channels other than the "official" anaconda distribution packages. When looking here, however, you must manually look through the files available as some may not be built for your os.
view the results of conda info to see your current channels. It looks like the channel "conda-forge" may have what you are craving.
conda create -n tamtams_project ipython cgal --channel conda-forge
I prefer knowing where packages are coming from when I stray from the default channels. you can always see that info by adding to your .condarc file with the command:
conda config --set show_channel_urls True
The above create would then show that nearly all of the packages would be conda-forge packages since the command line channel becomes the preferred channel.
So, a minor preference might be to use:
conda create -n tamtams_project ipython cgal -c defaults -c conda-forge
Then you see most packages being default packages and two coming from conda-forge.

Your installed copy of cgal is compiled for Python 3.5. If you look at https://anaconda.org/conda-forge/cgal/files , there's is a separate package for each Python minor version.
Managing Python — Conda documentation recommends either
installing a new Python to a new environment if you don't want to replace an existing installation, or
update'ing it (together with other packages if needed) if you do want to replace the current one
Of course, you can instead uninstall all version-specific packages, install the new Python, then reinstall the packages - but that's more work.

Conda showing two versions of requests library

I'm new to conda and seeing something strange but I don't actually know if it's a problem or not.
I'm currently in the root environment. At some point I was trying to install pip in another environment, but accidentally just ran pip install requests. This seems to have installed it in my root environment:
$ conda list | grep requests
requests 2.12.4 py36_0
requests 2.13.0 <pip>
And it looks like the pip version is what's getting picked up when I run python:
$ python
Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 12:22:00)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests; requests.__version__
'2.13.0'
My guess is that having two versions of the same package lying around is going to cause headaches later. Then my assumption was that I'd be better off keeping the non-pip version, so I tried this:
$ pip uninstall requests
[asks for confirmation, say yes]
$ conda list
[traceback, which I can post if helpful. Summary is:]
ModuleNotFoundError: No module named 'requests'
Then pip install requests brings me back to square 1 (having both versions of requests).
Finally, I want to know how to prevent this from happening again. According to the docs, if I want to use pip to install a package in a conda environment, I should:
Activate the conda environment where you want to install the package
run pip install whatever
It should show up in conda list for the current environment.
However, this isn't working for me - the installed package shows up under conda list --name root rather than in the current environment.
So, handful of questions:
Is it a problem to have two copies of requests in my conda root?
If this is a problem, how do I fix it?
How do I use pip within a conda environment?

Is it a problem to have two copies of requests in my conda root?
Probably.
If this is a problem, how do I fix it?
In my testing, conda remove followed by pip uninstall does the trick. (After which you can just re-install requests using only conda this time.) But if something goes wrong, remove .../lib/python3.6/site-packages/requests-2.13.0.dist-info. That seemed to work for me.
FWIW, I was only able to reproduce the double-install by installing with pip first, then installing again with conda.
How do I use pip within a conda environment?
Your summary in the OP is correct. Just activate the conda environment and use pip as you normally would. My rule of thumb is to install packages with conda if they are available, and resort to pip otherwise.

Import theano gives the AttributeError: module 'theano' has no attribute 'gof'

I have python 3.
I installed "Theano" bleeding edge and "Keras" using
pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git
and also
pip install --upgrade git+git://github.com/Theano/Theano.git
and
pip install git+git://github.com/fchollet/keras.git
But when I try to import Theano, I receive the following error:
AttributeError: module 'theano' has no attribute 'gof'
I looked for a solution online and reached nothing...
This is the piece of code I receive an error on (the last line produces error):
import sys
import numpy as np
import pandas as pd
from sklearn import preprocessing
from keras.models import Sequential
Since I don't have enough experience with python I'm completely lost and can't figure out what to do...
Any help would be appreciated.

The problem arises from a broken installation of theano and has nothing to do with keras itself.
This error seems to be due to conflicts in the installed version of theano, as also suggested in this answer to a related question.
An easy way that should solve the problem without having to fiddle with the installed version and all that is to use conda as package manager and let it do the dirty work.
If you choose to do this be aware that you should manage all of your python modules with it (even though with the latest versions you can install packages with the pip shipped with anaconda itself).
See the official documentation for how to install Anaconda.
Once anaconda is set up you can install theano using simply conda install theano.
With conda is also often convenient to install the packages needed for some particular application, like Keras in your case, in an environment isolated from the rest of your python installation, for easier maintenance.
Read the relevant docs to see how this would work.

I used conda to install theano and still got the same error. After much trial and error and StackOverflow searches, what worked for me was to first run:
conda install m2w64-toolchain
followed by:
conda install theano
Alternatively you can chain the modules together when you create an environment, for example:
conda create -n myenv python=3.5 m2w64-toolchain theano
Also important to follow #gtnbz2nyt's advice and restart your Python instance.

The problem seems to be with your g++ compiler. Try uninstalling it and running your script again. It'll spit a warning implying a degradation in performance, but it'll work nonetheless.
'Python 3.6.3 |Anaconda custom (32-bit)|
(default, Oct 15 2017, 07:29:16)
[MSC v.1900 32 bit (Intel)]
Type "copyright", "credits" or "license" for more information.
IPython 6.1.0 -- An enhanced Interactive Python.
import theano
WARNING (theano.tensor.blas): Using NumPy C-API based implementation
for BLAS functions.
'

For macOS Catalina:
conda create -n pymc3 python=3.8
conda activate pymc3
pip install pymc3

Anaconda Runtime Error: Python is not installed as a framework?

I've installed Anaconda with the pkg installer:
Python 2.7.10 |Continuum Analytics, Inc.| (default, May 28 2015, 17:04:42)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
but when I attempt to use anything from matplotlib, i.e.:
from matplotlib import pyplot as plt
I get
RuntimeError: Python is not installed as a framework.
The Mac OS X backend will not be able to function correctly if Python is not installed
as a framework. See the Python documentation for more information on installing Python
as a framework on Mac OS X. Please either reinstall Python as a framework,
or try one of the other backends.
I'm really not sure what this means, or how to go about fixing it.

Posting since I just had this issue and this was a quick fix:
If you used pip to install:
Create ~/.matplotlib/matplotlibrc
Add "backend: TkAgg" (without the quotations) to the file.

I was having the same problem with anaconda 2 & matplotlib 1.5.3.
Running a simple conda install matplotlib to reinstall matplotlib did the trick for me.

If the problem is only matplotlib, is worth try to change the backend:
import matplotlib
matplotlib.use('TkAgg')
import matplotlib.pyplot as plt
plt.plot([1, 2, 3], [4, 5, 6])
plt.show()
If it works you can change the backend permanently from the matplotlibrc file.

I was having the same problem. Installing an older version of matplotlib did the trick for me. Try this command in your terminal while in your virtual environment:
pip install matplotlib==1.4.3

Run the file using pythonw instead of python.
This happens because python is not installed as a framework.
Therefore use pythonw myScript.py instead of python myScript.py
I am sure this will fix it.
I had a similar error.
RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are using (Ana)Conda please install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib on OSX' in the Matplotlib FAQ for more information.

From the matplotlib documentation;
$ conda install python.app
You need a framwork build of Python for matplotlib, but
The default python provided in (Ana)conda is not a framework build. However, a framework build can easily be installed, both in the main environment and in conda envs: install python.app (conda install python.app) and use pythonw rather than python
NB I had to add the conda-forge channel as python.app isn't included in the default miniconda channels
$ conda config --add channels conda-forge

If you experience this error, don't forget to check your bash_profile.
You can do this in terminal by:
cd
then
nano .bash_profile
check the contents. Macports and Homebrew add their own headings for things they've done here. You can remove the declarations they make to $PATH. Just leave the one Anaconda has made. I had a If you would like, you can:
cp .bash_profile ./bash_profile_backup_yyyy_mm_dd
and have a backup of the file, with filename indexing to the date you changed it. That is, provided you actually put in the date in instead of just the formatting characters I'm suggesting.
source ~/.bash_profile
will refresh your system's reference to the bash_profile and you should be good to go in importing and using matplotlib

if using inside a virtualenv, I recommend following the instructions here:
http://matplotlib.org/faq/virtualenv_faq.html

A reinstall of matplotlib should fix the issue for you as it did for me with
conda install matplotlib

Quickfix: Run your file using pythonw, instead of python.
e.g pythonw testFile.py.