Develop Reference

Pip not installing Python modules to proper directory

So I downloaded a module, but when I tried to use it in a program, Python threw the error:
"ImportError: No module named markovify"
When I go to try to change my .bash_profile, it shows this:
"# Setting PATH for Python 3.5
The orginal version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.5/bin:${PATH}"
export PATH"
I've been using 2.7.10 all this time, and Terminal defaults to it. (When I type in Python, that's the version it says I'm using)
I'm trying to get pip to install modules for python 2.7.10 to a different folder.
What's weird is this: I seem to have installed pyparsing with pip and it seems to work. I tried installing markovify and it throws this error message.
What am I missing?

When I go to try to change my .bash_profile, it prints this
That suggests you're using some tool to change your profile. Did you also use that tool to install markovify? You need to be clear about your environment when you have environment questions! :-)
If you installed markovify from the command line, I'd suggest opening a new terminal and running these commands, just to make sure you're working from where you think you are:
$ python --version
Python 2.7.10
$ command -v python
/usr/bin/python
The most likely problem, it seems to me, is that you got a bit tangled up and wound up doing something other than what you intended. From a fresh start, provided you're starting with the environment you want, pip should, you know, Just Work.
I'm trying to get pip to install modules for python 2.7.10 to a different folder.
That's possible to do, but are you sure that's what you want? Usually if you're using the Python interpreter you intend to, and haven't putzed with PYTHON_PATH & friends, when you run pip, it will install to its default location, which sure enough is where Python will look for it when you say to import it.
As an experiment, you might check if markovify was somehow installed for Python 3.5. Try
$ /Library/Frameworks/Python.framework/Versions/3.5/bin/python -c 'import markovify'
(You might also want to check that the Python interpreter is in fact in that folder and has that name.)

When you're working on more than one Python version, you should work on Virtual Environments:
Virtual Env on Hitchhiker's Guide

xlrd, xlwt with sagemath

I want to use xlrd, xlwt, xlutils with sagemath. I have installed them in my system's Python(2.7.3) but sage environment is not recognizing them. Getting following error when I try to import with sage
ImportError: No module named xlrd`
But without sage its working fine. Any idea how can I make them work inside sage?? I am using Ubuntu 12(quantal),32 bits and Sage Version 5.8 with Python 2.7.5(sage's Python).
Faced same problem with Tkinter too but solved it using sudo apt-get install tk8.5-dev got solution form http://www.sagemath.org/doc/faq/faq-usage.html#how-to-get-sage-s-python-to-recognize-my-system-s-tcl-tk-install
But what for other libraries?
I tried to install them manually inside /usr/lib/sagemath/local/lib/python2.7/site-packages/ folder by putting tar files then extract them and then sudo python setup.py install
but still not working.

Rather than installing them with the system Python, install them with Sage's Python: you can either do sage --sh: this starts a subshell with various environment variables set appropriately for use with Sage, in particular $PATH will have $SAGE_ROOT/local/bin first. Then install the packages with python setup.py install etc. Or you can just run Sage's Python directly with sage --python setup.py install ....

Sage does not use the python installed in your system. It uses its own copy of python so while working with sage programs you would have to run the programs from terminal using this command
$ sage --python <Python_File>
Similarly sage does not uses the modules installed in the system's python. The Modules need to be installed under sage libraries. You have downloaded and extracted the folder right but used a wrong command. Now go again in the xlrd extracted folder and use this command.
$ sage --python setup.py install
Now Everything should work.
Cheers

Python ImportError while module is installed [Ubuntu]

I'd like to make a switch from Windows to Linux (Ubuntu) writing my python programs but I just can't get things to work. Here's the problem: I can see that there are quite the number of modules pre-installed (like numpy, pandas, matplotlib, etc.) in Ubuntu. They sit nicely in the /host/Python27/Lib/site-packages directory. But when I write a test python script and try to execute it, it gives me an ImportError whenever I try to import a module (for instance import numpy as np gives me ImportError: No module named numpy). When I type which python in the commandline I get the /usr/bin/python path. I think I might need to change things related to the python path, but I don't know how to do that.

You can use the following command in your terminal to see what folders are in your PYTHONPATH.
python -c "import sys, pprint; pprint.pprint(sys.path)"
I'm guessing /host/Python27/Lib/site-packages wont be in there (it doesn't sound like a normal python path. How did you install these packages?).
If you want to add folders to your PYTHONPATH then use the following:
export PYTHONPATH=$PYTHONPATH:/host/Python27/Lib/site-packages
Personally here are some recommendations for developing with Python:
Use virtualenv. It is a very powerful tool that creates sandboxed python environments so you can install modules and keep them separate from the main interpreter.
Use pip - When you've created a virtualenv, and activated it you can use pip install to install packages for you. e.g. pip install numpy will install numpy into your virtual environment and will be accessible from only this virtualenv. This means you can also install different versions for testing etc. Very powerful. I would recommend using pip to install your python packages over using ubuntu apt-get install as you are more likely to get the newer versions of modules (apt-get relies on someone packaging the latest versions of your python libraries and may not be available for as many libraries as pip).
When writing python scripts that you will make executable (chmod +x my_python_script.py) make sure you put #!/usr/bin/env python at the top as this will pick up the python interpreter in your virtual environment. If you don't (and put #!/usr/bin/python) then running ./my_python_script.py will always use the system python interpreter.

/host/Python27/Lib/site-packages is not a default python directory on linux installations as far as I am aware.
The normal python installation (and python packages) should be found under /usr/lib or /usr/lib64 depending on your processor architecture.
If you want to check where python is searching in addition to these directories you can use a terminal with the following command:
echo $PYTHONPATH
If the /host/Python27/Lib/site-packages path is not listed, attempt to use the following command and try it again:
export PYTHONPATH=$PYTHONPATH:host/Python27/Lib/site-packages
If this should work and you do not want to write this in a terminal every time you want to use these packages, simply put it into a file called .bashrc in your home folder (normally /home/<username>).

When installing other python libraries, specify the pip version you want to install it to, if it's python2 you use, then enter this syntax:
pip2 install <package>
For python3
pip3 install <package>

"Proper way" to manage multiple versions of Python on archlinux

So I have read this - https://wiki.archlinux.org/index.php/Python
And it is clear from this wiki that I can install Python 2.7.2 via
pacman -S python2
Is it reasonable for me to create a symlink to python2
ln -s python2 /usr/bin/python
if I don't forsee myself switching to python 3.0 any time soon? Or is there a better way of managing multiple python versions like what I usually use on a debian system (update-alternatives --config python) or on a mac os x system (python select)?
CLARIFICATION:
What I am trying to find out is - what is the "best practice" of managing various python versions on an archlinux system?
I am new to archlinux but familiar with ubuntu, debian and mac os x

I would argue you shouldn't create any symlinks like this at all. Especially if you are going to distribute some of your python code, you should not assume a user has python2 or python3 at /usr/bin/python.
If your script requires python2, just use:
#!/usr/bin/env python2
If your script requires python3, use:
#!/usr/bin/env python3
This way your scripts will work fine even through updates to Python. It will also be much more clear what version your script actually needs.

Most unices already have a /usr/bin/python. Overwriting that one is a bad idea, as this is the Python version used by all packages in the system, and changing that one may break them. When installing the Python 2.7 package the executable should be installed as /usr/bin/python2.7 (if not I would claim Archlinux is broken) and it's better to use that when you want to run Python 2.7.
Archlinux is a bit special, since it will use /usr/bin/python for Python 3, despite the default executable name for Python 3 being /usr/bin/python3. This is confusing and can be seen as a bug, but it does mean you can't use that symlink for Python 2, as any other Archlinux script that uses Python 3 will almost certainly break if you do.
So where on other Unices, symlinking /usr/bin/python to Python 2.7 is a bad idea, on Archlinux it is a terrible idea. Instead just install all version you need and call them with /usr/bin/pythonX.X.

There is a nice project on github what helps you with it's called pyenv
What helps you to manage multiple python instances

As others have said, the short answer is "don't do this, it will most likely break things on your system", however, if you mostly use Python 2 you can still set your personal default in your shell (and still have the option of switching to Python 3 at any time). To do this, first become root and install python2-virtualenv:
# pacman -S python2-virtualenv
Then create a virtual environment that uses Python 2 (this will automatically install Python, setuptools, wheel, and pip in the environment):
$ virtualenv -p /usr/bin/python2 --system-site-packages ~/env # (Or wherever you want your environment to live)
If you only want to use locally installed packages (eg. packages you install with pip and not the ones installed by pacman) remove the --system-site-packages option when creating your environment.
Now in your ~/.bash_profile or ~/.profile (or whatever your preferred shells configuration file is), set something like the following:\
source ~/env/bin/activate
This will activate the virtual environment, making your default version Python 2.
This could still break anything that is launched in a shell, but it's not likely that anything will be unless you're explicitly running it from a shell, at which point you can turn the virtual environment off by running:
deactivate
or simply manually run Python 3.

I just stumbled over this post, no necro-bumping intended but I was wondering nobody mentioned virtualenvs. I'm using ArchLinux as well and I use the python packages virtualenv and virtualenvwrapper to create multiple python environments. You can reference to the python 2 or python3 binaries in /usr/bin/ to determine the python version used in the virtual environment.
The benefit is that packages installed in a virtual environment don't mess with the python the system is using and there are many ways to automate project handling.

I know this may be a very old answer, but it took me two days to solve the problem so I'm going to share.
The proper way to manage python versions in your system to work on different projects without them driving you crazy is to use pyenv and its plugins pyenv-virtualenv and pyenv-virtualenvwrapper as described by Henrique Bastos into this blog post. Notice that this way to work is kinda platform independent, since pyenv is a python package and it can be run quite similarly on Windows, Linux and Mac OSx.
The problems start with Arch Linux. The OS doesn't provide a pacman version of pyenv, so you have to install it cloning it from github as described in the installation section of the release. The installation process is the same both for pyenv-virtualenv and pyenv-virtualenvwrapper. Notice that the shell initialization configuration may be different, in my case it didn't work for ~/.bash_profile, but worked for ~/.bashrc .
Running pyenv is not straightforward if your installation is very fresh like the one I'm setting up in these days, since pip requires openSSL and even if you install it via pacman, pyenv doesn't see it. So, if you want to install an older version of Python (namely 3.4.3), you will find the shell complaining about you haven't installed the openSSL plugin, even if you have it. To be honest, I didn't have the right packages the first time I tried to install it; you have to download the following packages
sudo pacman -S openssl
sudo pacman -S openssl-1.0
sudo pacman -S python-pyopenssl
sudo pacman -S python2-pyopenssl
The way I solved the problem is to add the flags as described in the pyenv installation FAQs: that solution eventually led me to install the python version I wanted:
LDFLAGS="-L/usr/lib/openssl-1.0" \
CFLAGS="-I/usr/include/openssl-1.0" \
pyenv install -v 3.4.3
To avoid to go on the FAQs page everytime you want to renew the python installation environment, you can add an alias in ~/.bashrc or whatever is you shell as follows:
echo alias pyenv='LDFLAGS="-L/usr/lib/openssl-1.0" \
CFLAGS="-I/usr/include/openssl-1.0" \
pyenv' >> ~/.bashrc
In this way you can install python properly with a clean pyenv syntax, and manage it via its plugins in the same way (since the syntax is pyenv [COMMAND] [OTHERSTUFF]).

No, there is no better way to do this. The python symlink is part of the Python 3 package.
I guess changing this link won't break anything for now but it might be possible that some packages will depend on it in the future.

Problem with virtualenv in Mac OS X

I've installed virtualenv via pip and get this error after creating a new environment:
selenium:~ auser$ virtualenv new
New python executable in new/bin/python
ERROR: The executable new/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/System/Library/Frameworks/Python.framework/ Versions/2.6' (should be '/Users/user/new')
ERROR: virtualenv is not compatible with this system or executable
In my environment:
PYTHONPATH=/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages
PATH=/System/Library/Frameworks/Python.framework/Versions/2.6/bin:/Library/Frameworks/Python.framework/Versions/2.6/bin:/Library/Frameworks/Python.framework/Versions/2.6/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
How can I repair this?
Thanks.

Just in case there's someone still seeking for the answer.
I ran into this same problem just today and realized since I already have Anaconda installed, I should not have used pip install virtualenv to install virtual environment as this would give me the error message when trying to initiate it later. Instead, I tried conda install virtualenv then entered virtualenv env_mysite and problem solved.

Like #RyanWilcox mentioned, you might be inadvertently pointing virtualenv to the wrong Python installation. Virtualenv comes with a -p flag to let you specify which interpreter to use.
In my case,
virtualenv test_env
threw the same error as yours, while
virtualenv -p python test_env
worked perfectly.
If you call virtualenv -h, the documentation for the -p flag will tell you which python it thinks it should be using; if it looks wonky, try passing -p python. For reference, I'm on virtualenv 1.11.6.

In case anyone in the future runs into this problem - this is caused by your default Python distribution being conda. Conda has it's own virtual env set up process but if you have the conda distribution of python and still wish to use virtualenv here's how:
Find the other python distribution on your machine: ls -ls /usr/bin/python*
Take note of the availble python version that is not conda and run the code below (note for python 3 and above you have to upgrade virtualenv first): virtualenv -p python2.7(or your python version) flaskapp

I've run across this problem myself. I wrote down the instructions in a README, which I have pasted below....
I have found there are two things that work:
Make sure you're running the latest virtualenv (1.5.1, of this writting)
If you're using a non system Python as your standard Python (which python to check) Forcefully use the System supplied one.
Instead of virtualenv thing use /usr/bin/python2.6 PATH/TO/VIRTUALENV thing (or whatever which
python returned to you - this is what it did for me when I ran into this issue)

I had the same problem and as I see it now, it was caused by a messy Python installation. I have OS X installed for over a year since I bought a new laptop and I have already installed and reinstalled Python for several times using different sources (official binaries, homebrew, official binaries + hand-made adjustments as described here). Don't ask me why I did that, I'm just a miserable newbie believing everything will fix itself after being re-installed.
So, I had a number of different Pythons installed here and there as well as many hardlinks pointing at them inconsistently. Eventually I got sick of all of them and reinstalled OS X carefully cleaned the system from all the Pythons I found using find utility. Also, I have unlinked all the links pointing to whatever Python from everywhere. Then I've installed a fresh Python using homebrew, installed virtualenv and everything works as a charm now.
So, my recipe is:
sudo find / -iname "python*" > python.log
Then analyze this file, remove and unlink everything related to the version of Python you need, reinstall it (I did it with homebrew, maybe official installation will also work) and enjoy. Make sure you unlink everything python-related from /usr/bin and /usr/local/bin as well as remove all the instances of Frameworks/Python.framework/Versions/<Your.Version> in /Library and /System/Library.
It may be a dirty hack, but it worked for me. I prefer not to keep any system-wide Python libraries except pip and virtualenv and create virtual environments for all of my projects, so I do not care about removing the important libraries. If you don't want to remove everything, still try to understand whether your Pythons are, what links point to them and from where. Then think what may cause the problem and fix it.

I ran into a variation of this "not functioning" error.
I was trying to create an environment in a folder that included the path ".../Programming/Developing..." which is actually "/Users/eric/Documents/Programming:Developing/"
and got this error:
ImportError: No module named site
ERROR: The executable env/bin/python2.7 is not functioning
ERROR: It thinks sys.prefix is u'/Users/eric/Documents/Programming:Developing/heroku' (should be u'/Users/eric/Documents/Programming:Developing/heroku/env')
ERROR: virtualenv is not compatible with this system or executable
I tried the same in a different folder and it worked fine, no errors and env/bin has what I expect (activate, etc.).

I got the same problem and I found that it happens when you do not specify the python executable name properly. So for python 2x, for example:
virtualenv --system-site-packages -p python mysite
But for python 3.6 you need to specify the executable name like python3.6
virtualenv --system-site-packages -p python3.6 mysite

On on OSX 10.6.8 leopard, after having "upgraded" to Lion, then downgrading again (ouch - AVOID!), I went through the Wolf Paulus method a few months ago, completely ignorant of python. Deleted python 2.7 altogether and "replaced" it with 3.something. My FTP program stopped working (Fetch) and who knows what else relies on Python 2.7. So at that point I downloaded the latest version of 2.7 from python.org and it's installer got me up and running - until i tried to use virtualenv.
What seems to have worked for me this time was totally deleting Python 2.7 with this code:
sudo rm -R /System/Library/Frameworks/Python.framework/Versions/2.7
removing all the links with this code:
sudo rm /usr/bin/pydoc
sudo rm /usr/bin/python
sudo rm /usr/bin/pythonw
sudo rm /usr/bin/python-config
I had tried to install python with homebrew, but apparently it will not work unless all of XTools is installed, which I have been avoiding, since the version of XTools compatible with 10.6 is ancient and 4GB and mostly all I need is GCC, the compiler, which you can get here.
So I just installed with the latest download from python.org.
Then had to reinstall easy_install, pip, virtualenv.
Definitely wondering when it will be time for a new laptop, but there's a lot to be said for buying fewer pieces of hardware (slave labor, unethical mining, etc).

The above solutions failed for me, but the following worked:
python3 -m venv --without-pip <ENVIRONMENT_NAME>
. <ENVIRONMENT_NAME>/bin/activate
curl https://bootstrap.pypa.io/get-pip.py | python
deactivate
It's hacky, but yes, the core problem really did just seem to be pip.

I did the following steps to get virtualenv working :
Update virtualenv as follows :
==> sudo pip install --upgrade virtualenv
Initialize python3 virtualenv :
==> virtualenv -p python3 venv

I had this same issue, and I can confirm that the problem was with an outdated virtualenv.py file.
It was not necessary to do a whole install --upgrade.
Replacing the virtualenv.py file with the most recent version sufficed.

I also had this problem, and I tried the following method which worked for me:
conda install virtualenv
virtualenv --system-site-packages /anaconda/envs/tensorflow (here envs keeps all the virtual environments made by user)
source /anaconda/envs/tensorflow/bin/activate
Hope it's helpful.

I had this same issue when trying to install py2.7 on a newer system. The root issue was that virtualenv was part of py3.7 and thus was not compatible:
$ virtualenv -p python2.7 env
Running virtualenv with interpreter /usr/local/bin/python2.7
New python executable in /Users/blah/env/bin/python
ERROR: The executable /Users/blah/env/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/Library/Frameworks/Python.framework/Versions/2.7' (should be u'/Users/blah/env')
ERROR: virtualenv is not compatible with this system or executable
$ which virtualenv
/Library/Frameworks/Python.framework/Versions/3.7/bin/virtualenv
# install proper version of virtualenv
$ pip2.7 install virtualenv
$ /Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenv -p python2.7 env
$ . ./env/bin/activate
(env) $

Open terminal and type /Library/Frameworks/Python.framework/Versions/
then type ls /Library/Frameworks/Python.framework/Versions/2.7/bin/
if you are using Python2(or any other else).
Edit ~/.bash_profile and add the following line:
export PATH=$PATH:/Library/Frameworks/Python.framework/Versions/2.7/bin/
cat ~/.bash_profile
In my case the content of ~/.bash_profile is as follows:
export PATH=$PATH:/Library/Frameworks/Python.framework/Versions/2.7/bin/
Now the virtualenv command should work.

If you continue to have trouble with virtualenv, you might try pythonbrew, instead. It's an alternate solution to the same problem. It works more like Ruby's rvm: It builds and creates an entire instance of Python, under $HOME/.pythonbrew, and then sets up some bash functions that allow you to switch easily between versions. Where virtualenv shadows the system version of Python, using symbolic links as part of its solution, pythonbrew builds entirely self-contained installations of Python.
I used virtualenv for years. It's a decent solution, but I've switched to pythonbrew lately. Having completely self-contained Python instances means that installing a new one takes awhile (since pythonbrew actually compiles Python from scratch), but the self-contained nature of each installation appeals to me. And disk is cheap.