I am relatively new to Python so please pardon my ignorance. I want to know answer to following questions
How does pip know the location to install packages that it installs? After a built of trial and error
I suspect that it maybe hardcoded at time of installation.
Are executables like pip.exe what they call frozen binaries? In essence, does it mean that pip.exe will run without python. Again after a bit of trial and error i suspect that it requires a python installation to execute.
P.S: I know about sys.prefix,sys.executable and sys.exec_prefix. If there is anything else on which the questions i asked on depends, pls link me to same.
PIP is a package manager for Python packages, or modules if you like.
pip when used with virtualenv will generally install packages in the path /lib//site-packages.
For example, I created a test virtualenv named test, and the django folder is in test/lib/python3.7/site-packages/django.
At the time of installation, you must have set up environment variables, and that is how pip recognizes directories.
pip.exe which is placed under path\Scripts needs a python installation and can't run without one. It is hardwired against a specific python interpreter, and can't install packages for another one. If you have 7 different python versions installed on your system, you will also have 7 different versions of pip.
Since it is bound so tightly, pip was at some point even included with the python standard library (see pep-0453 for details).
This also answers the other part of your question of how pip figures out the right location - there is only one location it can install to, the side-packages of the python interpreter it is bundled against.
Related
Currently, I feel like my Python ecosystem is out of whack. Several years ago, I had shared my computer with someone else, and now I am discovering that my machine has many versions of Python scattered about. From what I could find, starting from Macintosh HD:
anaconda
Applications/Python 3.6
Contains IDLE, Python Launcher, etc...
Library/Python/2.7/site-packages
This contains things like pip and wheel (I am unsure what this is)
Library/Frameworks/Python.framework/Versions
This contains two folders, 3.5 and 3.6.
System/Library/Frameworks/Python.framework/Versions
Inside here there are many alias folders, all which point to 2.7
Is all of this supposed to be normal? I am trying to run Python from the terminal, yet I have been getting messages such as ImportError: No module named site. pip has also not been working.
Is there a way for me to reset the Python on my machine to just that which comes with macOS? I feel like starting over from a blank slate would be helpful, since I want to get things setup in some type of comprehendible way (e.g. a way in which I've set things up so I know what is on my machine instead of many random things put on it by another).
This is very normal. you have a preinstalled python2.7 which comes with macOS and another one "Anaconda" which has installed manually. You first need to check what is your default python path (version) on your macOS, I mean if you are using anaconda or the default pre-installed python2.x. To remind you can check like below:
python --version
output (for me): Python 3.7.6
then if you want to change it to another version/or use another version under conda you can check these two answeres of mine.
How to add anaconda to PATH?
and here:
How to set the default python3 to python3.7?
It is normal that when you install a package for your default python version you don't expect it to be installed on the other one too. Normally it is better to install python2.x as an environment of conda and switch between two environments with "conda activate py2" and "conda deactivate" to go back to your default version. for each of them, you need to be in the environment and then use pip.
if pip is not working, it may need to be installed. On macOS, as you remember, you can use
brew update
brew install pip
or "easy-install" instead of brew (or whatever you use for installation).
I'm installing a bunch of python modules on my system that are specific to this code I am going to be working with. Specifically, I used pip install pyda to get the pyda module. To make sure I had gotten all the modules, I went through and ran some of the code snippets, and came across the following error:
ModuleNotFoundError: No module named 'pyda.utilities'
I tried using pip install pyda.utilities, but that honestly doesn't make sense, it should have just come with the pyda module. According to this website https://pypi.org/project/pyda/ it seems like it should come with the package. I tried determining if I just installed it in the wrong python version, but I'm having a difficult time forcing it to use the specific python version that I installed the module in (specifically, I tried to create an alias for /usr/bin/python3.7 or something like this as I have seen on other websites, but it just fusses at me that this is simply a directory, incredibly unhelpful because I can't find the corresponding executable, so I'm a bit confused here).
This is a very long question likely with a very simple answer, any thoughts or help on what the issue might be would be appreciated.
Edit: I have determined that it's a package problem, not a python problem. The command 'pip install pyda' is not actually installing everything, oddly enough, which is why it cannot find the pyda.utilities module. Unfortunately, I think this means I will have to install the package manually. I will keep this question posted because of the useful answer on virtual environments, so thanks everyone.
The answer is indeed straightforward. As #Chris indicated in the comments, start using virtual environments.
It's not as complicated as it sounds and there's plenty of tutorials on getting started with virtualenv for Python, like https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/26/python-virtual-env/
The basic steps:
check you're using the version of Python you want in your virtual environment
if you don't, change directories to where it lives
ensure you have pip and it works
check if you have virtualenv and if you don't pip install virtualenv
create a virtual environment virtualenv /your/env/folder/here
activate the virtual environment with /your/env/folder/here/Scripts/activate
After that, just install the packages you need with pip and they will end up in your virtual environment, with no interference from other Python versions or packages.
Check your python version, if it does not work restart your computer and try run setup.py install on the python command line
I have a variety of different Python versions installed on my Windows system- a 2.7 version, a 3.5 version, and a 3.6 version (there are a bunch of different packages that only work with one version, or are too buggy in the 3.6 version, etc.).
Long story short, I'm trying to keep my all my pips and python.exes in order. I've added my C:/Python35 and C:/Python36 and their Scripts folders to my path, but I also want to make sure that I am using the right pip from my command line (for example, I don't want to pip install pyinstaller to the 3.6 version, since Python 3.6 doesn't play well with pyinstaller as of right now.
I see that inside my Python3x/Scripts/ folder, there are three different pips available: pip, pip3.5, and pip3.
So whenever I want to install a module for 3.5, I plan to issue the following command pip3.5 install package_name. Whenever I want to install something for 3.6, I'd use pip or pip3. Seems like a decent enough plan to me.
However, can anyone confirm if the three pips are all the same executable? If so, I'd like to delete pip and pip3 so that I don't accidentally confuse it with my Python 3.6 pip- is this acceptable practice or am I missing something? This SO post provides some insights but doesn't explain why there's multiple pips in the same folder. Also, why are three separate pips provided? Is it simply for convenience from the command line?
Within the same python installation all the different pip files you find should be the same executable, there is the multiple versions simply to help keep everything in order if there are multiple installs of python on a single computer.
I personally only have the main version of python I use for development set to my PATH variable on my windows laptop and then if I need to do anything to a different python I instead link directly to the necessary file with something like C:\Python36\Scripts\pip3 install natsort but that is simply personal preference and my way of organizing.
If you do have them all on path you can then simply call out pip3.6 install <package name> or whatever python version you are using
The difference between them is that each one install the package in its own folder, for example if i type pip install Django, it will be placed for python 2 version, but is a little bit complex when you have multiple version of python3 like you showed, the solution: Don't delete the files and makevirtualenv when you're working, that avoid problems.
This prevents dependency issues with different versions of Python. You also check out virtualenvwrapper which is a convenient way to manage your virtual environments
If you want to manage the version with virtualenv
virtualenv python2_project -p usr/bin/python2.7
virtualenv p35_project -p usr/bin/python3.5
virtualenv p3x_project -p usr/bin/python3.x
I have installed Python using Homebrew. I would like for this installation of python to recognize only one site-pakages directory -- /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
However, when I start the python shell and give the command print [f for f in sys.path if f.endswith('packages')] I get the following output:
['/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages', '/Library/Python/2.7/site-packages']
Why is /Library/Python/2.7/site-packages sitting at the end there? Shouldn't the Homebrew installation of Python have just one main site-packages directory under /usr/local/ ???
Edit: I am using virtualenv -- but this is occurring after a fresh install of python using homebrew before I have even had a chance to install virtualenv. Also, I'm interested in why it is happening, not on what I should do to get around it.
Created a stack overflow account just for this, since I was so annoyed by it. I actually had a bug inadvertently caused by that extra "/Library/Python/2.7/site-packages" in the sys.path
Researching it led me to this:
http://bugs.python.org/issue4865
tl;dr: From what I can tell, they throw that path in there to help people that installed packages for the apple-shipped python, but then later would try to use those packages in other versions of python (like the ones used by homebrew, macports, etc).
I'm working with PyInstaller under Python 2.6, which is only partially supported due to the mess MS have created with their manifest nonense which now affects Python since it is now MSVC8 compiled.
The problem is that the manifest embedding support relies on the pywin32 extensions in order to build which is a pain because without including the host's site-packages folder when I create the virtualenv (kinda defeats the point in a build environment) I cannot find a way to install the required extensions so they are accessible to PyInstaller.
Has anyone found a solution to this issue?
I found http://old.nabble.com/Windows:-virtualenv-and-pywin32--td27658201.html (now a dead link) which offered the following solution:
Browse http://sourceforge.net/projects/pywin32/files/ for the URL of the exe you want
Activate your virtualenv
Run easy_install http://PATH.TO/EXE/DOWNLOAD
This works with modern versions of setuptools (circa February 2014, reported by tovmeod in the comments).
If you are using an old version of setuptools (or distribute it merged back into setuptools), you may get this error message:
error: c:\users\blah\appdata\local\temp\easy_install-ibkzv7\pywin32-214.win32-py2.6.exe is not a valid distutils Windows .exe
In which case:
Download the exe yourself
Activate your virtualenv
Run easy_install DOWNLOADED_FILE.exe
I rather hopefully tried "pip install" rather than "easy_install", but this didn't work, and likely never will (citation needed).
Finally, I found but haven't tested a solution at http://www.mail-archive.com/python-list#python.org/msg272040.html which is:
Solved this by copying the pywin32.pth file into my virtualenv site-packages
and editing the file to point to the path.
If the other options don't work for you, maybe this will?
For Python 2.7 or 3.x use pypiwin32.
pip install pypiwin32
OK, well since I had to find a way forward I improvised. I've internally created a git repository with a hacked-together version of pywin32 that will install within a virtualenv using the standard setup.py script. It took a lot of fiddling to make it work right but I managed to get it to load and the dependent code now works as I need it to. If people feel this would be of benefit to the community please post a comment: if I get enough I'll try and put something up on my github account.
This may have been improved since previous answer, since I've successfully installed pywin32 on sandbox on several machines without any specific "hacks" :
$ virtualenv sandbox
$ sandbox\scripts\activate
(sandbox) $ git clone https://github.com/Travis-Sun/pywin32.git
(sandbox) $ cd pywin32
(sandbox) $ python setup.py install
Tested with following environment :
windows 7
git
python 2.7.10 with virtualenv
VS2008. It may also work (but I've not tested yet) with
http://www.microsoft.com/en-us/download/details.aspx?id=44266
Edit: Scratch this for now, appears to be some problems with the installation still...
I got rather tired of the whole situation, and just created a set of converted wheels ("wheel convert <.exe>"). I'll try and keep them maintained for the most recent build, but do shout if there are any issues.
https://tr00st.co.uk/python/wheel/pywin32/
Installation can be done easily using pip and pointing to the package matching your version and architecture. For example, for Python 3.5/amd64:
pip install https://tr00st.co.uk/python/wheel/pywin32/pywin32-219-cp35-none-win_amd64.whl
Caveat: The --upgrade process currently fails, as the uninstall procedure is unable to clean up after itself (Access Denied when cleaning up win32api.pyd) - this is only when removing the temporary directory, which can be manually deleted. Easiest way around this is to uninstall and reinstall instead of upgrading, then manually delete the temporary folder.