I am using pyenv to manage my python installations on both my Windows desktop (using pyenv-win) and on my MacBook.
I currently have one global installation of version 3.10.4 which is my "main" python version where I play around, work on simple projects and have all my pip packages installed. I'd like to update my python version to 3.10.5, but maintain all my packages and I am sure how to do that.
Before using pyenv, I, simply, installed python from python.org. When I wanted to update, I'd simply download the installer of the new version and run it, which would override the python installation, but not touch any of the packages, so it was that simple.
I am not sure how it's done with pyenv though. I couldn't find any update command, so it seems I am supposed to install the new version, set it as the new global version, and remove the old one. However, does it mean that I have to reinstall all my pip packages? I can export a list of them to install them again, but it feels like there should be a simple update command that would update the version and maintain all the packages.
Am I missing something?
For this you need to understand how pyenv actually works, So when you install pyenv and install python, it sets your environment variables to your python location. And when you change python version in pyenv, it changes those python version to it. Since with every python environment, it has its own pip(which is also set by pyenv). You will be having different version of it. The best way I think is that you can do
pip freeze > requirements.txt
Change your python version, And then install all listed dependencies from requirements.txt
pip install -r requirements.txt
This will be fast and easy method. To update and other stuff related to pyenv is here https://realpython.com/intro-to-pyenv/, These are more good then pyenv official doc.
Edit: My question is not regarding how to immediately get pip to work. I fixed this by relinking as you can see in the question.
The question is: What is best/pythonic practice on a fresh *NIX system?
One would probably want to have the most recent version of both Python2 and especially Python3 with pip and setuptools and so on. A safeguarded virtual environment seems to be the way, but anyway one has to start somewhere.
So what is the recommendation?
From the comments it seems that one should use the get-pip.py script to install and then use pip to install virtualenv. I guess this has to be done both for Python2 and Python3 if one wants to have both.
This is what happened some days ago:
Ubuntu 18.04 and Python 2.7
I am a heavy R user, but use more and more Python. Virtually every time I set up any kind of customized python environment on Linux or OSX, something breaks and has to be fiddled with.... links, binaries, paths, dependencies. Each and every time. How many thousands of people are sitting on a Ubuntu/Debian box right now and do apt install python pip; pip install --upgrade pip and -duh- it breaks? Really?
Specifically right now: I want to install django on a webserver and started with apt install python-pip.
Then I did the recommended pip install --upgrade pip which installed pip-18.0 but gave the message Not uninstalling pip at /usr/lib/python2.7/dist-packages
After that pip --version threw an error
Traceback (most recent call last): File "/usr/bin/pip", line 9, in
from pip import main
It turned out that the old pip still resides in /usr/bin/pip while the new one is in /usr/local/bin/pip.
I fixed it brute force with:
rm /usr/bin/pip
ln -s /usr/local/bin/pip /usr/bin/pip
Did I do the right thing or is this the way to doom down the road?
Is there a pythonic or elegant way to handle such issues or to prevent them in the first place?
With best regards and thanks for any kind of suggestions.
My conclusion after investigating the hints in the comments is:
Python conflicts with itself.
Do not try to avoid this - cope with it.
Use virtual environments to handle the unavoidable multiple Python versions consistently.
pyenv seems to work best for my purposes. Use the following to install first pyenv (1), update it (2), install a Pythons version (for this example 3.5.6) with pyenv (3), set your user-global python to the installed version (4), upgrade the Python package mangement tools (5) and install anything inside the virtual Python enviroment (6).
curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
pyenv update
CONFIGURE_OPTS=--enable-shared pyenv install 3.5.6
pyenv global 3.5.6
pip install --upgrade pip setuptools wheel
pip install ipython jupyter snowflake-connector-python seaborn
CONFIGURE_OPTS=--enable-shared in (3) is needed for some python packages which depend on shared libraries. seaborn will not install without it.
Note that the pyenvinstalled versions includes the package manager pip. Now every call python or pip everything references to the correct version.
When referencing python in scripts in MacOS bash and other *NIXes use #!/usr/bin/env python.
It is possible to install pyenv install 2.7.15 the latest python 2.7 and change to it whenever necessary. pyenv shell 2.7.15 switches to Python2.7 only for a particular shell session when needed to run an adhoc script.
Since I use this flow I have no headaches anymore.
The same setup works superfine so far both on a local MacOS and on a remote Ububtu based jupyter notebook server.
Disclaimer: I am not a hardcore Python fullstack developer, but use Python whenever necessary in Data Science pipelines. So for application development in Python this might not be the best solution, but for consistent management of a Data Science environment it seems to work well.
Hope this is helpful.
I am managing several modules on an HPC, and want to install some requirements for a tool using pip.
I won't use virtualenv because they don't work well with our module system. I want to install module-local versions of packages and will set PYTHONPATH correctly when the module is loaded, and this has worked just fine when the packages I am installing are not also installed in the default python environment.
What I do not want to do is uninstall the default python's versions of packages while I am installing module-local versions.
For example, one package requires numpy==1.6, and the default version installed with the python I am using is 1.8.0. When I
pip install --install-option="--prefix=$RE_PYTHON" numpy==1.6
where RE_PYTHON points to the top of the module-local site-packages directory, numpy==1.6 installs fine, then pip goes ahead and starts uninstalling 1.8.0 from the tree of the python I am using (why it wants to uninstall a newer version is beyond me but I want to avoid this even when I am doing a local install of e.g. numpy==1.10.1).
How can I prevent pip from doing that? It is really annoying and I have not been able to find a solution that doesn't involve virtualenv.
You have to explicitly tell pip to ignore the current installed package by specifying the -I option (or --ignore-installed). So you should use:
PYTHONUSERBASE=$RE_PYTHON pip install -I --user numpy==1.6
This is mentioned in this answer by Ian Bicking.
I have both Python 2.7 and 3.3 installed on my system. I'm trying to hopefully write everything for 3.3. I've run into a snag. I need to install Pillow on 3.3 so I can use Images. How do I get it to install on 3.3. If I try
pip install Pillow
It comes up and says 'Requirement already satisfied(use --upgrade to upgrade): Pillow in /usr/lib/python2.7/dist-packages'
How do I get Pillow to install on 3.3, since that is where I need it?
I ran through the rest of the install(setuptools and prerequisites). Not sure where they installed. I did use the python3-setuptools for the initial install attempt. I don't know if that helped to control the flow of the prerequisites to 3.3 or not.
This is on a Linux system.
Your 2.7 and 3.3 have their own separate site-packages locations.
And, just as they have their own separate executables (usually python and python2.7 for 2.7, and python3 and python3.3 for 3.3), when you install pip for each one, they'll each have their own pip scripts (usually pip, pip2, and pip2.7 vs. pip3 and pip3.3).
So, just do this:
pip3.3 install Pillow
As a side note, if you use virtual environments, either via the third-party virtualenv package or the stdlib venv package (3.3+ only, and really not worth using until 3.4), this problem goes away: when you're inside a virtual environment, it's either a 2.7 environment or a 3.3 environment, and it's as if nothing else exists.
The basic design is explained in PEP 394. How pip fits into that design is explained in… as far as I know, docs that haven't been written yet, but will hopefully be part of Python 3.4.0 and/or pip 1.5.something.
Older versions of pip (I believe before 1.5.0) would use pip and pip-2.7 vs. pip and pip-3.3, which obviously leads to a bit of confusion.
And some distros have their own python-pip packages that do things differently. And then there's Arch, where python actually means 3.2. And so on.
But the basic idea is that when you have Python X.Y and V.W side by side, there will be some way to differentiate explicitly.
I'm starting to learn python and loving it. I work on a Mac mainly as well as Linux. I'm finding that on Linux (Ubuntu 9.04 mostly) when I install a python module using apt-get it works fine. I can import it with no trouble.
On the Mac, I'm used to using Macports to install all the Unixy stuff. However, I'm finding that most of the python modules I install with it are not being seen by python. I've spent some time playing around with PATH settings and using python_select . Nothing has really worked and at this point I'm not really understanding, instead I'm just poking around.
I get the impression that Macports isn't universally loved for managing python modules. I'd like to start fresh using a more "accepted" (if that's the right word) approach.
So, I was wondering, what is the method that Mac python developers use to manage their modules?
Bonus questions:
Do you use Apple's python, or some other version?
Do you compile everything from source or is there a package manger that works well (Fink?).
The most popular way to manage python packages (if you're not using your system package manager) is to use setuptools and easy_install. It is probably already installed on your system. Use it like this:
easy_install django
easy_install uses the Python Package Index which is an amazing resource for python developers. Have a look around to see what packages are available.
A better option is pip, which is gaining traction, as it attempts to fix a lot of the problems associated with easy_install. Pip uses the same package repository as easy_install, it just works better. Really the only time use need to use easy_install is for this command:
easy_install pip
After that, use:
pip install django
At some point you will probably want to learn a bit about virtualenv. If you do a lot of python development on projects with conflicting package requirements, virtualenv is a godsend. It will allow you to have completely different versions of various packages, and switch between them easily depending your needs.
Regarding which python to use, sticking with Apple's python will give you the least headaches, but If you need a newer version (Leopard is 2.5.1 I believe), I would go with the macports python 2.6.
Your question is already three years old and there are some details not covered in other answers:
Most people I know use HomeBrew or MacPorts, I prefer MacPorts because of its clean cut of what is a default Mac OS X environment and my development setup. Just move out your /opt folder and test your packages with a normal user Python environment
MacPorts is only portable within Mac, but with easy_install or pip you will learn how to setup your environment in any platform (Win/Mac/Linux/Bsd...). Furthermore it will always be more up to date and with more packages
I personally let MacPorts handle my Python modules to keep everything updated. Like any other high level package manager (ie: apt-get) it is much better for the heavy lifting of modules with lots of binary dependencies. There is no way I would build my Qt bindings (PySide) with easy_install or pip. Qt is huge and takes a lot to compile. As soon as you want a Python package that needs a library used by non Python programs, try to avoid easy_install or pip
At some point you will find that there are some packages missing within MacPorts. I do not believe that MacPorts will ever give you the whole CheeseShop. For example, recently I needed the Elixir module, but MacPorts only offers py25-elixir and py26-elixir, no py27 version. In cases like these you have:
pip-2.7 install --user elixir
( make sure you always type pip-(version) )
That will build an extra Python library in your home dir. Yes, Python will work with more than one library location: one controlled by MacPorts and a user local one for everything missing within MacPorts.
Now notice that I favor pip over easy_install. There is a good reason you should avoid setuptools and easy_install. Here is a good explanation and I try to keep away from them. One very useful feature of pip is giving you a list of all the modules (along their versions) that you installed with MacPorts, easy_install and pip itself:
pip-2.7 freeze
If you already started using easy_install, don't worry, pip can recognize everything done already by easy_install and even upgrade the packages installed with it.
If you are a developer keep an eye on virtualenv for controlling different setups and combinations of module versions. Other answers mention it already, what is not mentioned so far is the Tox module, a tool for testing that your package installs correctly with different Python versions.
Although I usually do not have version conflicts, I like to have virtualenv to set up a clean environment and get a clear view of my packages dependencies. That way I never forget any dependencies in my setup.py
If you go for MacPorts be aware that multiple versions of the same package are not selected anymore like the old Debian style with an extra python_select package (it is still there for compatibility). Now you have the select command to choose which Python version will be used (you can even select the Apple installed ones):
$ port select python
Available versions for python:
none
python25-apple
python26-apple
python27 (active)
python27-apple
python32
$ port select python python32
Add tox on top of it and your programs should be really portable
Please see Python OS X development environment. The best way is to use MacPorts. Download and install MacPorts, then install Python via MacPorts by typing the following commands in the Terminal:
sudo port install python26 python_select
sudo port select --set python python26
OR
sudo port install python30 python_select
sudo port select --set python python30
Use the first set of commands to install Python 2.6 and the second set to install Python 3.0. Then use:
sudo port install py26-packagename
OR
sudo port install py30-packagename
In the above commands, replace packagename with the name of the package, for example:
sudo port install py26-setuptools
These commands will automatically install the package (and its dependencies) for the given Python version.
For a full list of available packages for Python, type:
port list | grep py26-
OR
port list | grep py30-
Which command you use depends on which version of Python you chose to install.
I use MacPorts to install Python and any third-party modules tracked by MacPorts into /opt/local, and I install any manually installed modules (those not in the MacPorts repository) into /usr/local, and this has never caused any problems. I think you may be confused as to the use of certain MacPorts scripts and environment variables.
MacPorts python_select is used to select the "current" version of Python, but it has nothing to do with modules. This allows you to, e.g., install both Python 2.5 and Python 2.6 using MacPorts, and switch between installs.
The $PATH environment variables does not affect what Python modules are loaded. $PYTHONPATH is what you are looking for. $PYTHONPATH should point to directories containing Python modules you want to load. In my case, my $PYTHONPATH variable contains /usr/local/lib/python26/site-packages. If you use MacPorts' Python, it sets up the other proper directories for you, so you only need to add additional paths to $PYTHONPATH. But again, $PATH isn't used at all when Python searches for modules you have installed.
$PATH is used to find executables, so if you install MacPorts' Python, make sure /opt/local/bin is in your $PATH.
There's nothing wrong with using a MacPorts Python installation. If you are installing python modules from MacPorts but then not seeing them, that likely means you are not invoking the MacPorts python you installed to. In a terminal shell, you can use absolute paths to invoke the various Pythons that may be installed. For example:
$ /usr/bin/python2.5 # Apple-supplied 2.5 (Leopard)
$ /opt/local/bin/python2.5 # MacPorts 2.5
$ /opt/local/bin/python2.6 # MacPorts 2.6
$ /usr/local/bin/python2.6 # python.org (MacPython) 2.6
$ /usr/local/bin/python3.1 # python.org (MacPython) 3.1
To get the right python by default requires ensuring your shell $PATH is set properly to ensure that the right executable is found first. Another solution is to define shell aliases to the various pythons.
A python.org (MacPython) installation is fine, too, as others have suggested. easy_install can help but, again, because each Python instance may have its own easy_install command, make sure you are invoking the right easy_install.
If you use Python from MacPorts, it has it's own easy_install located at: /opt/local/bin/easy_install-2.6 (for py26, that is). It's not the same one as simply calling easy_install directly, even if you used python_select to change your default python command.
Have you looked into easy_install at all? It won't synchronize your macports or anything like that, but it will automatically download the latest package and all necessary dependencies, i.e.
easy_install nose
for the nose unit testing package, or
easy_install trac
for the trac bug tracker.
There's a bit more information on their EasyInstall page too.
For MacPython installations, I found an effective solution to fixing the problem with setuptools (easy_install) in this blog post:
http://droidism.com/getting-running-with-django-and-macpython-26-on-leopard
One handy tip includes finding out which version of python is active in the terminal:
which python
When you install modules with MacPorts, it does not go into Apple's version of Python. Instead those modules are installed onto the MacPorts version of Python selected.
You can change which version of Python is used by default using a mac port called python_select. instructions here.
Also, there's easy_install. Which will use python to install python modules.
You may already have pip3 pre-installed, so just try it!
Regarding which python version to use, Mac OS usually ships an old version of python. It's a good idea to upgrade to a newer version. You can download a .dmg from http://www.python.org/download/ . If you do that, remember to update the path. You can find the exact commands here http://farmdev.com/thoughts/66/python-3-0-on-mac-os-x-alongside-2-6-2-5-etc-/
I use easy_install with Apple's Python, and it works like a charm.
Directly install one of the fink packages (Django 1.6 as of 2013-Nov)
fink install django-py27
fink install django-py33
Or create yourself a virtualenv:
fink install virtualenv-py27
virtualenv django-env
source django-env/bin/activate
pip install django
deactivate # when you are done
Or use fink django plus any other pip installed packages in a virtualenv
fink install django-py27
fink install virtualenv-py27
virtualenv django-env --system-site-packages
source django-env/bin/activate
# django already installed
pip install django-analytical # or anything else you might want
deactivate # back to your normally scheduled programming