I've been working with Python for a little while now but have come to absolutely detest installing new modules. It always seems to take me a full work day to install one additional module. This last happened with mpl_toolkits and now it is happening with gdal.
The main issue seems to be that easy_install/pip/get-app aren't saving a file in the correct location and so Python (I'm using Spyder) can't find it. How do I install a module in a location so that Python can find it?
I have been reading guides, articles, manuals and Stack Overflow articles all day now and I feel this needs a new question as I can't find an answer.
It is installing fine, it's just Python (and I) can't find it
It's not in the /lib/python2.7/site-packages folder, which seems to be the Python default, although half the modules that do work aren't in there when I list it
I installed using:
sudo apt-get install gdal-bin
I also tried with:
pip install gdal
but this fails with the error:
Command "python setup.py egg_info" failed with the error code 1 in /tmp/pip-build-NWJT2f/gdal/
I looked for the files using
dpkg -L gdal
and then added the file path this said into Spyder's preferences option for PYTHONPATH, but it still couldn't find it and so I'm guessing this is wrong.
I have read the official documentation of PYTHONPATH, but it's very short and doesn't really explain it at all.
I recommend trying anaconda or miniconda, which manage environments and install packages - it 'just works'.
https://www.continuum.io/downloads
Related
I'm installing a bunch of python modules on my system that are specific to this code I am going to be working with. Specifically, I used pip install pyda to get the pyda module. To make sure I had gotten all the modules, I went through and ran some of the code snippets, and came across the following error:
ModuleNotFoundError: No module named 'pyda.utilities'
I tried using pip install pyda.utilities, but that honestly doesn't make sense, it should have just come with the pyda module. According to this website https://pypi.org/project/pyda/ it seems like it should come with the package. I tried determining if I just installed it in the wrong python version, but I'm having a difficult time forcing it to use the specific python version that I installed the module in (specifically, I tried to create an alias for /usr/bin/python3.7 or something like this as I have seen on other websites, but it just fusses at me that this is simply a directory, incredibly unhelpful because I can't find the corresponding executable, so I'm a bit confused here).
This is a very long question likely with a very simple answer, any thoughts or help on what the issue might be would be appreciated.
Edit: I have determined that it's a package problem, not a python problem. The command 'pip install pyda' is not actually installing everything, oddly enough, which is why it cannot find the pyda.utilities module. Unfortunately, I think this means I will have to install the package manually. I will keep this question posted because of the useful answer on virtual environments, so thanks everyone.
The answer is indeed straightforward. As #Chris indicated in the comments, start using virtual environments.
It's not as complicated as it sounds and there's plenty of tutorials on getting started with virtualenv for Python, like https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/26/python-virtual-env/
The basic steps:
check you're using the version of Python you want in your virtual environment
if you don't, change directories to where it lives
ensure you have pip and it works
check if you have virtualenv and if you don't pip install virtualenv
create a virtual environment virtualenv /your/env/folder/here
activate the virtual environment with /your/env/folder/here/Scripts/activate
After that, just install the packages you need with pip and they will end up in your virtual environment, with no interference from other Python versions or packages.
Check your python version, if it does not work restart your computer and try run setup.py install on the python command line
I am relatively new to Python so please pardon my ignorance. I want to know answer to following questions
How does pip know the location to install packages that it installs? After a built of trial and error
I suspect that it maybe hardcoded at time of installation.
Are executables like pip.exe what they call frozen binaries? In essence, does it mean that pip.exe will run without python. Again after a bit of trial and error i suspect that it requires a python installation to execute.
P.S: I know about sys.prefix,sys.executable and sys.exec_prefix. If there is anything else on which the questions i asked on depends, pls link me to same.
PIP is a package manager for Python packages, or modules if you like.
pip when used with virtualenv will generally install packages in the path /lib//site-packages.
For example, I created a test virtualenv named test, and the django folder is in test/lib/python3.7/site-packages/django.
At the time of installation, you must have set up environment variables, and that is how pip recognizes directories.
pip.exe which is placed under path\Scripts needs a python installation and can't run without one. It is hardwired against a specific python interpreter, and can't install packages for another one. If you have 7 different python versions installed on your system, you will also have 7 different versions of pip.
Since it is bound so tightly, pip was at some point even included with the python standard library (see pep-0453 for details).
This also answers the other part of your question of how pip figures out the right location - there is only one location it can install to, the side-packages of the python interpreter it is bundled against.
I have installed Python using Homebrew. I would like for this installation of python to recognize only one site-pakages directory -- /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
However, when I start the python shell and give the command print [f for f in sys.path if f.endswith('packages')] I get the following output:
['/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages', '/Library/Python/2.7/site-packages']
Why is /Library/Python/2.7/site-packages sitting at the end there? Shouldn't the Homebrew installation of Python have just one main site-packages directory under /usr/local/ ???
Edit: I am using virtualenv -- but this is occurring after a fresh install of python using homebrew before I have even had a chance to install virtualenv. Also, I'm interested in why it is happening, not on what I should do to get around it.
Created a stack overflow account just for this, since I was so annoyed by it. I actually had a bug inadvertently caused by that extra "/Library/Python/2.7/site-packages" in the sys.path
Researching it led me to this:
http://bugs.python.org/issue4865
tl;dr: From what I can tell, they throw that path in there to help people that installed packages for the apple-shipped python, but then later would try to use those packages in other versions of python (like the ones used by homebrew, macports, etc).
So I was really eager to try generating a django model diagram with django-extensions by doing the following:
manage.py graph_models -a -g -o my_project_visualized.png
Then I realized I need pygraphviz to make it work. I'm running Windows 7 64bit, so I installed Activestate and Graphviz as suggested by the documentation as a requirement. Graphviz 2.28 was installed quite successfully and then Activestate was installed with some tweaks msiexec /package activestate_install.exe /qr because it always stuck at "finish checking disk spaces required".
Later, it turns out activestate wasn't really helping in installing pygraphviz. So after a lot of trial and errors, I managed to install pygraphviz by following this guide http://blog.ropardo.ro/2011/11/28/installing-pygraphviz-on-windows/. Just as I tried to run the graph_models command again, python once again tells me that pygraphviz module is not installed... so I pip freeze to see if it's really not installed. Now comes the good part, pip freeze returns a traceback of ImportError: DLL load failed: %1 is not a valid Win32 application. But that only happens in virtualenvs, when get out of virturalenvs and pip freeze, it works as expected.
By googling around, I realized I might have installed some 64bit version package while having 32bit python 2.7.3 on my machine, and that's indeed what I did. Just that I don't remember which ones I installed were 64bit. It's late at night, and I'm tired and frustrated, so I started uninstalling things that I thought might be causing the problem. I uninstalled mingw32 and Activestate, and the next thing I know, I can't even get into python from command line. pip is not recognized either; every python package is not recognized. Checking back at the Python27 directory, half of the folders are gone in C:\Python27\Lib. There used to be a lot of files and folders which it's impossible for me to recall what they are, and now there's only 4 directories:
bsddb, importlib, site-packages, and test
and the site-packages folder is a lot "cleaner", too! I'm sure there used to be a lot of package files and scripts (some .pth files and .py files). Now there's only two files easy-install.pth and PIL.pth and some folders for some packages I installed. However, the most critical issue is I can't even use python in cmd now.
I thought maybe uninstalling activestate messed up some of the environmental variables I set, then I went into the system settings and added in things like C:\Python27\Scripts;C:\Python27\Lib\site-packages to PATH. I went back into cmd, and typed "python" and "pip", the response I get for python is The system cannot find the path specified., and for pip it's still 'pip' is not recognized as an internal or external command, operable program or batch file.
This is kind of the only time that I regret for not setting up a system restore point..
Could anyone please tell me what's going on here, what am I doing wrong, and how am I supposed to get everything working again? Thanks a lot!
Boy, you indeed messed up your installation. "ActiveState" is a python distribution for Windows, so you installed a new python distribution over your old one. When you uninstalled, it obligingly removed your python installation, including the default library.
You'll have to reinstall your Python installation, I'm afraid. Start from scratch or backup.
You have my sympathies, if that's any consolation.
I'm working with PyInstaller under Python 2.6, which is only partially supported due to the mess MS have created with their manifest nonense which now affects Python since it is now MSVC8 compiled.
The problem is that the manifest embedding support relies on the pywin32 extensions in order to build which is a pain because without including the host's site-packages folder when I create the virtualenv (kinda defeats the point in a build environment) I cannot find a way to install the required extensions so they are accessible to PyInstaller.
Has anyone found a solution to this issue?
I found http://old.nabble.com/Windows:-virtualenv-and-pywin32--td27658201.html (now a dead link) which offered the following solution:
Browse http://sourceforge.net/projects/pywin32/files/ for the URL of the exe you want
Activate your virtualenv
Run easy_install http://PATH.TO/EXE/DOWNLOAD
This works with modern versions of setuptools (circa February 2014, reported by tovmeod in the comments).
If you are using an old version of setuptools (or distribute it merged back into setuptools), you may get this error message:
error: c:\users\blah\appdata\local\temp\easy_install-ibkzv7\pywin32-214.win32-py2.6.exe is not a valid distutils Windows .exe
In which case:
Download the exe yourself
Activate your virtualenv
Run easy_install DOWNLOADED_FILE.exe
I rather hopefully tried "pip install" rather than "easy_install", but this didn't work, and likely never will (citation needed).
Finally, I found but haven't tested a solution at http://www.mail-archive.com/python-list#python.org/msg272040.html which is:
Solved this by copying the pywin32.pth file into my virtualenv site-packages
and editing the file to point to the path.
If the other options don't work for you, maybe this will?
For Python 2.7 or 3.x use pypiwin32.
pip install pypiwin32
OK, well since I had to find a way forward I improvised. I've internally created a git repository with a hacked-together version of pywin32 that will install within a virtualenv using the standard setup.py script. It took a lot of fiddling to make it work right but I managed to get it to load and the dependent code now works as I need it to. If people feel this would be of benefit to the community please post a comment: if I get enough I'll try and put something up on my github account.
This may have been improved since previous answer, since I've successfully installed pywin32 on sandbox on several machines without any specific "hacks" :
$ virtualenv sandbox
$ sandbox\scripts\activate
(sandbox) $ git clone https://github.com/Travis-Sun/pywin32.git
(sandbox) $ cd pywin32
(sandbox) $ python setup.py install
Tested with following environment :
windows 7
git
python 2.7.10 with virtualenv
VS2008. It may also work (but I've not tested yet) with
http://www.microsoft.com/en-us/download/details.aspx?id=44266
Edit: Scratch this for now, appears to be some problems with the installation still...
I got rather tired of the whole situation, and just created a set of converted wheels ("wheel convert <.exe>"). I'll try and keep them maintained for the most recent build, but do shout if there are any issues.
https://tr00st.co.uk/python/wheel/pywin32/
Installation can be done easily using pip and pointing to the package matching your version and architecture. For example, for Python 3.5/amd64:
pip install https://tr00st.co.uk/python/wheel/pywin32/pywin32-219-cp35-none-win_amd64.whl
Caveat: The --upgrade process currently fails, as the uninstall procedure is unable to clean up after itself (Access Denied when cleaning up win32api.pyd) - this is only when removing the temporary directory, which can be manually deleted. Easiest way around this is to uninstall and reinstall instead of upgrading, then manually delete the temporary folder.