"Proper way" to manage multiple versions of Python on archlinux

"Proper way" to manage multiple versions of Python on archlinux - python

So I have read this - https://wiki.archlinux.org/index.php/Python
And it is clear from this wiki that I can install Python 2.7.2 via
pacman -S python2
Is it reasonable for me to create a symlink to python2
ln -s python2 /usr/bin/python
if I don't forsee myself switching to python 3.0 any time soon? Or is there a better way of managing multiple python versions like what I usually use on a debian system (update-alternatives --config python) or on a mac os x system (python select)?
CLARIFICATION:
What I am trying to find out is - what is the "best practice" of managing various python versions on an archlinux system?
I am new to archlinux but familiar with ubuntu, debian and mac os x

I would argue you shouldn't create any symlinks like this at all. Especially if you are going to distribute some of your python code, you should not assume a user has python2 or python3 at /usr/bin/python.
If your script requires python2, just use:
#!/usr/bin/env python2
If your script requires python3, use:
#!/usr/bin/env python3
This way your scripts will work fine even through updates to Python. It will also be much more clear what version your script actually needs.

Most unices already have a /usr/bin/python. Overwriting that one is a bad idea, as this is the Python version used by all packages in the system, and changing that one may break them. When installing the Python 2.7 package the executable should be installed as /usr/bin/python2.7 (if not I would claim Archlinux is broken) and it's better to use that when you want to run Python 2.7.
Archlinux is a bit special, since it will use /usr/bin/python for Python 3, despite the default executable name for Python 3 being /usr/bin/python3. This is confusing and can be seen as a bug, but it does mean you can't use that symlink for Python 2, as any other Archlinux script that uses Python 3 will almost certainly break if you do.
So where on other Unices, symlinking /usr/bin/python to Python 2.7 is a bad idea, on Archlinux it is a terrible idea. Instead just install all version you need and call them with /usr/bin/pythonX.X.

There is a nice project on github what helps you with it's called pyenv
What helps you to manage multiple python instances

As others have said, the short answer is "don't do this, it will most likely break things on your system", however, if you mostly use Python 2 you can still set your personal default in your shell (and still have the option of switching to Python 3 at any time). To do this, first become root and install python2-virtualenv:
# pacman -S python2-virtualenv
Then create a virtual environment that uses Python 2 (this will automatically install Python, setuptools, wheel, and pip in the environment):
$ virtualenv -p /usr/bin/python2 --system-site-packages ~/env # (Or wherever you want your environment to live)
If you only want to use locally installed packages (eg. packages you install with pip and not the ones installed by pacman) remove the --system-site-packages option when creating your environment.
Now in your ~/.bash_profile or ~/.profile (or whatever your preferred shells configuration file is), set something like the following:\
source ~/env/bin/activate
This will activate the virtual environment, making your default version Python 2.
This could still break anything that is launched in a shell, but it's not likely that anything will be unless you're explicitly running it from a shell, at which point you can turn the virtual environment off by running:
deactivate
or simply manually run Python 3.

I just stumbled over this post, no necro-bumping intended but I was wondering nobody mentioned virtualenvs. I'm using ArchLinux as well and I use the python packages virtualenv and virtualenvwrapper to create multiple python environments. You can reference to the python 2 or python3 binaries in /usr/bin/ to determine the python version used in the virtual environment.
The benefit is that packages installed in a virtual environment don't mess with the python the system is using and there are many ways to automate project handling.

I know this may be a very old answer, but it took me two days to solve the problem so I'm going to share.
The proper way to manage python versions in your system to work on different projects without them driving you crazy is to use pyenv and its plugins pyenv-virtualenv and pyenv-virtualenvwrapper as described by Henrique Bastos into this blog post. Notice that this way to work is kinda platform independent, since pyenv is a python package and it can be run quite similarly on Windows, Linux and Mac OSx.
The problems start with Arch Linux. The OS doesn't provide a pacman version of pyenv, so you have to install it cloning it from github as described in the installation section of the release. The installation process is the same both for pyenv-virtualenv and pyenv-virtualenvwrapper. Notice that the shell initialization configuration may be different, in my case it didn't work for ~/.bash_profile, but worked for ~/.bashrc .
Running pyenv is not straightforward if your installation is very fresh like the one I'm setting up in these days, since pip requires openSSL and even if you install it via pacman, pyenv doesn't see it. So, if you want to install an older version of Python (namely 3.4.3), you will find the shell complaining about you haven't installed the openSSL plugin, even if you have it. To be honest, I didn't have the right packages the first time I tried to install it; you have to download the following packages
sudo pacman -S openssl
sudo pacman -S openssl-1.0
sudo pacman -S python-pyopenssl
sudo pacman -S python2-pyopenssl
The way I solved the problem is to add the flags as described in the pyenv installation FAQs: that solution eventually led me to install the python version I wanted:
LDFLAGS="-L/usr/lib/openssl-1.0" \
CFLAGS="-I/usr/include/openssl-1.0" \
pyenv install -v 3.4.3
To avoid to go on the FAQs page everytime you want to renew the python installation environment, you can add an alias in ~/.bashrc or whatever is you shell as follows:
echo alias pyenv='LDFLAGS="-L/usr/lib/openssl-1.0" \
CFLAGS="-I/usr/include/openssl-1.0" \
pyenv' >> ~/.bashrc
In this way you can install python properly with a clean pyenv syntax, and manage it via its plugins in the same way (since the syntax is pyenv [COMMAND] [OTHERSTUFF]).

No, there is no better way to do this. The python symlink is part of the Python 3 package.
I guess changing this link won't break anything for now but it might be possible that some packages will depend on it in the future.

Related

How can I ensure that my Python version is updated after installing version 3.11 using Mac?

I've just installed Python version 3.11 (I also moved versions 3.8 and 3.9 to the trash from my Applications folder).
I can see it in the following:
$ myname#name-MBP miniconda3 % ls /usr/local/bin/py*
/usr/local/bin/pydoc3 /usr/local/bin/python3-intel64
/usr/local/bin/pydoc3.11 /usr/local/bin/python3.11
/usr/local/bin/python3 /usr/local/bin/python3.11-config
/usr/local/bin/python3-config /usr/local/bin/python3.11-intel64
(Any additional recommendations on whether I need to clean things up would be much appreciated.)
Checking python3 --version still displays Python 3.8.13.
First attempt to solve
Initially I tried installing it from the command line using homebrew and specifying the version:
brew install python#3.11
I also tried using conda, but neither of these worked.
Second attempt to solve
My initial thought was to check my PATH (I think this is how Python decides which version to use, but please correct me if I'm wrong).
This still only contained version 8 paths:
/Users/myname/miniconda3/lib/python38.zip
/Users/myname/miniconda3/lib/python3.8
/Users/myname/miniconda3/lib/python3.8/lib-dynload
/Users/myname/.local/lib/python3.8/site-packages
/Users/myname/miniconda3/lib/python3.8/site-packages
So I added the Python3.11 path to it using
PYTHONPATH="/usr/local/bin/pydoc3.11/:$PYTHONPATH"
export PYTHONPATH
Now it includes the v3.11 path when I print out sys.path:
/usr/local/bin/python3.11
/usr/local/bin/pydoc3.11
/Users/myname/miniconda3
/Users/myname/miniconda3/lib/python38.zip
/Users/myname/miniconda3/lib/python3.8
/Users/myname/miniconda3/lib/python3.8/lib-dynload
/Users/myname/.local/lib/python3.8/site-packages
/Users/myname/miniconda3/lib/python3.8/site-packages
But the python3 --version output is still unchanged.
Questions
I'm nervous to keep playing around with the contents of my path and entering random command line executions to try to solve this because I really have no clue what I'm doing.
What's happening here?
How can I get the output of python3 --version to be 3.11?

So first thing to understand is setting the variable PYTHONPATH will not affect which version of python is executed by the shell. The shell (bash/zsh) only knows to scan the paths in the PATH env var to figure out all the executables.
Now there are two ways to solve this.
1. Using the python#3.11 from homebrew.
There are several downsides to using this method. Currently, the default python3 by brew is 3.10.x. Whenever you will install any cask or formula that depends on python#3, it'll invariably install the python#3 formula aka 3.10.x. Installing python3 will make brew symlink 3.10.x into /opt/homebrew/bin.
Python 3.11.x can be used by installing python#3.11 and invoking python3.11. This should drop you into the Python 3.11 interpreter. Append all python executable names with the version like pip will be pip3.11
Trying to force 3.11 over 3.10 links will be complicated and cause instability. It'll only cause frustration during development.
2. Using VirtualEnvs
Your best bet for the most stable and no-headache approach to python is to create virtual envs using either venv or pyenv. use pyenv-virtualenv for max ease of use.
One limitation of venv is that it'll create a virtualenv of the same version that invoked it. Aka if you do brew install python#3 && python3 -m venv <folder>, it'll create virtualenv of python3. For 3.11 you'll have to brew install python#3.11 && python3.11 -m venv <folder>. Pyenv on the other hand can install any version of python.
Go through https://www.dataquest.io/blog/a-complete-guide-to-python-virtual-environments/ and https://github.com/pyenv/pyenv-virtualenv to understand and learn more.

pyenv - how to get more info re: "system", including version

I'm on MacOS, and I have latest python installed via brew. A few months ago, I started using pyenv to be able to switch between latest python and a project that was fixed at python v3.8. I think I got it all set up but I'm confused. pyenv refuses to show me information about system:
$ pyenv versions
* system (set by /Users/<user>/.pyenv/version)
3.8.6
$ pyenv version system
system (set by /Users/<user>/.pyenv/version)
$ cat ~/.pyenv/version
system
How do I get pyenv to show me the version and/or location of system?? Obviously, I can get info about system python when it's the one in use, but why doesn't pyenv show anything about it? Showing info about the current config seems like basic functionality for a config management tool.
By comparison, when I run apt list --installed, it shows me what's installed, whether installed by me or bundled with the OS. It doesn't just show a placeholder for things installed by the system.
So I'm frustrated that pyenv is doing this.
Edit: Wow, nvm is the same way. How? Why? Why do these tools have a built-in disregard for the system config?

Pyenv is a shim tool. It's used to intercept calls to python and pip and transparently give the correct binaries for these commands. When you call python -m some_module, pyenv will find the correct python binary that makes sense for that call, either because it's locally set by a .python-version file or set by the current shell using pyenv shell or pyenv activate commands.
Pyenv tries to not mess up with current python installations, because the environment system may use these tools, and should keep using whatever version they are already using. Thus, it simply packs the currently existing binaries as "system" and does not touch it, so that the existing system prior to installing pyenv keeps working as it was before.
As a best practice, I recommend you to create a new environment and set it with pyenv global to something different of system (e.g. pyenv install 3.9.10 && pyenv virtualenv 3.9.10 myglobal && pyenv global myglobal) -- so that your default application python is different from the operating system python, and thus you can't mess with the system; just leave system as is. If you want to check what system is, you can activate it and call it normally, pyenv activate system ; python --version will show the version, but it's not something you should be doing in the first place, so pyenv doesn't support it.

How to uninstall python 3.7 from Ubuntu 19.10?

I'm using Ubuntu 19.10, in which there is python of version 3.7. After the release of python 3.8, which I have installed, now I want to uninstall python 3.7 so that whenever I would call python3 in my terminal, it would always call python3.8?

You do not need to uninstall old version for this.
You need to update your update-alternatives , then you will be able to set your default python version.
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.4 1
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2
Then run :
sudo update-alternatives --config python
Set python3.6 as default.
Or use the following command to set python3.6 as default:
sudo update-alternatives --set python /usr/bin/python3.6
This should do it.

I suggest you do not touch your default OS python instalation. Other parts of the system may depend on it and there is no way to know if an upgrade can break something, even if it should not.
I also suggest that you learn (if you haven't already, but I suppose you did not because of the question) about using a python virtual environment, like virtualenv. This allows you to setup specific python environments for each project you write. This means each environment can have its own python version, and, besides the standard python lib for that version, any other third-party python lib you would like to install with pip for that project. This isolates projects from each other. You won't break one because of an upgrade of another.
That said, if you want to keep cutting edge with Python versions, make an install from source, and then install it in the system with the altinstall parameter (see the README.rst of the Python distribution. This way all the installed versions are available with different names (the same for pip) and then you create each of your virtual environments with the wanted version. There is a parameter for virtualenv to apply a specific (older) version if you want.
Quoting the README on the "Installing multiple versions" section:
On Unix and Mac systems if you intend to install multiple versions of Python
using the same installation prefix (--prefix argument to the configure
script) you must take care that your primary python executable is not
overwritten by the installation of a different version. All files and
directories installed using make altinstall contain the major and minor
version and can thus live side-by-side. make install also creates
${prefix}/bin/python3 which refers to ${prefix}/bin/pythonX.Y. If you
intend to install multiple versions using the same prefix you must decide which
version (if any) is your "primary" version. Install that version using make
install. Install all other versions using make altinstall.
For example, if you want to install Python 2.7, 3.6, and 3.8 with 3.8 being the
primary version, you would execute make install in your 3.8 build directory
and make altinstall in the others.
Finally, the other answers are ok to do exactly what you asked, if you still want to.

so that whenever I would call python3 in my terminal, it would always call python3.8
You can simply create an alias for Python 3.8 in your .bashrc file in your /home path.
Open ~/.bashrc with your preferred editor, and go to the last line.
Append this line:
alias python='python3.8'
And whenever you call python in your terminal, it opens Python 3.8.
Note: This doesn't work as expected when creating virtual environments with specific version. You might want to keep that in mind.

Installing pip3 with a virtualenv

I am trying to clean up a few things now that I have Mavericks. I used Homebrew to install python 2.7 and 3.3. I also have their respective pips. Now I want to have a sane virtualenv setup.
I see from virtualenv with python2 and python3 via Homebrew that I can specify which version of python a newly-created virtualenv will use. What is the best (i.e., easiest to use/remember) way to make sure that a virtualenv using Python 3.3 uses pip3?
ETA: I've not had to use 3.3 before, so this is my first experience trying to maintain different versions. Sorry if this has an obvious answer.

Credit goes to scythargon for this one (left in a comment to Kristof's answer in this thread) and, since it did exactly what I needed, I figured it deserved its own answer.
When creating a virtualenv with virtualenvwrapper use the -p flag to explicitly associate it with a specific version of Python. In the examples below replace {virtualenv_name} with your intended virtual environment name:
Python 3.3:
$ mkvirtualenv {virtualenv_name} -ppython3.3
Python 3.4:
$ mkvirtualenv {virtualenv_name} -ppython3.4
Python 2.7:
$ mkvirtualenv {virtualenv_name} -ppython2.7

Have a look at this article: Python Development Environment on Mac OS X Mavericks 10.9. It describes quite clearly how to set up both versions of Python on the same machine using Homebrew and contains some helpful pointers.
In the article, the author appends -py3 to a Python3 virtualenv, as #moliware already suggested.

Problem with virtualenv in Mac OS X

I've installed virtualenv via pip and get this error after creating a new environment:
selenium:~ auser$ virtualenv new
New python executable in new/bin/python
ERROR: The executable new/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/System/Library/Frameworks/Python.framework/ Versions/2.6' (should be '/Users/user/new')
ERROR: virtualenv is not compatible with this system or executable
In my environment:
PYTHONPATH=/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages
PATH=/System/Library/Frameworks/Python.framework/Versions/2.6/bin:/Library/Frameworks/Python.framework/Versions/2.6/bin:/Library/Frameworks/Python.framework/Versions/2.6/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
How can I repair this?
Thanks.

Just in case there's someone still seeking for the answer.
I ran into this same problem just today and realized since I already have Anaconda installed, I should not have used pip install virtualenv to install virtual environment as this would give me the error message when trying to initiate it later. Instead, I tried conda install virtualenv then entered virtualenv env_mysite and problem solved.

Like #RyanWilcox mentioned, you might be inadvertently pointing virtualenv to the wrong Python installation. Virtualenv comes with a -p flag to let you specify which interpreter to use.
In my case,
virtualenv test_env
threw the same error as yours, while
virtualenv -p python test_env
worked perfectly.
If you call virtualenv -h, the documentation for the -p flag will tell you which python it thinks it should be using; if it looks wonky, try passing -p python. For reference, I'm on virtualenv 1.11.6.

In case anyone in the future runs into this problem - this is caused by your default Python distribution being conda. Conda has it's own virtual env set up process but if you have the conda distribution of python and still wish to use virtualenv here's how:
Find the other python distribution on your machine: ls -ls /usr/bin/python*
Take note of the availble python version that is not conda and run the code below (note for python 3 and above you have to upgrade virtualenv first): virtualenv -p python2.7(or your python version) flaskapp

I've run across this problem myself. I wrote down the instructions in a README, which I have pasted below....
I have found there are two things that work:
Make sure you're running the latest virtualenv (1.5.1, of this writting)
If you're using a non system Python as your standard Python (which python to check) Forcefully use the System supplied one.
Instead of virtualenv thing use /usr/bin/python2.6 PATH/TO/VIRTUALENV thing (or whatever which
python returned to you - this is what it did for me when I ran into this issue)

I had the same problem and as I see it now, it was caused by a messy Python installation. I have OS X installed for over a year since I bought a new laptop and I have already installed and reinstalled Python for several times using different sources (official binaries, homebrew, official binaries + hand-made adjustments as described here). Don't ask me why I did that, I'm just a miserable newbie believing everything will fix itself after being re-installed.
So, I had a number of different Pythons installed here and there as well as many hardlinks pointing at them inconsistently. Eventually I got sick of all of them and reinstalled OS X carefully cleaned the system from all the Pythons I found using find utility. Also, I have unlinked all the links pointing to whatever Python from everywhere. Then I've installed a fresh Python using homebrew, installed virtualenv and everything works as a charm now.
So, my recipe is:
sudo find / -iname "python*" > python.log
Then analyze this file, remove and unlink everything related to the version of Python you need, reinstall it (I did it with homebrew, maybe official installation will also work) and enjoy. Make sure you unlink everything python-related from /usr/bin and /usr/local/bin as well as remove all the instances of Frameworks/Python.framework/Versions/<Your.Version> in /Library and /System/Library.
It may be a dirty hack, but it worked for me. I prefer not to keep any system-wide Python libraries except pip and virtualenv and create virtual environments for all of my projects, so I do not care about removing the important libraries. If you don't want to remove everything, still try to understand whether your Pythons are, what links point to them and from where. Then think what may cause the problem and fix it.

I ran into a variation of this "not functioning" error.
I was trying to create an environment in a folder that included the path ".../Programming/Developing..." which is actually "/Users/eric/Documents/Programming:Developing/"
and got this error:
ImportError: No module named site
ERROR: The executable env/bin/python2.7 is not functioning
ERROR: It thinks sys.prefix is u'/Users/eric/Documents/Programming:Developing/heroku' (should be u'/Users/eric/Documents/Programming:Developing/heroku/env')
ERROR: virtualenv is not compatible with this system or executable
I tried the same in a different folder and it worked fine, no errors and env/bin has what I expect (activate, etc.).

I got the same problem and I found that it happens when you do not specify the python executable name properly. So for python 2x, for example:
virtualenv --system-site-packages -p python mysite
But for python 3.6 you need to specify the executable name like python3.6
virtualenv --system-site-packages -p python3.6 mysite

On on OSX 10.6.8 leopard, after having "upgraded" to Lion, then downgrading again (ouch - AVOID!), I went through the Wolf Paulus method a few months ago, completely ignorant of python. Deleted python 2.7 altogether and "replaced" it with 3.something. My FTP program stopped working (Fetch) and who knows what else relies on Python 2.7. So at that point I downloaded the latest version of 2.7 from python.org and it's installer got me up and running - until i tried to use virtualenv.
What seems to have worked for me this time was totally deleting Python 2.7 with this code:
sudo rm -R /System/Library/Frameworks/Python.framework/Versions/2.7
removing all the links with this code:
sudo rm /usr/bin/pydoc
sudo rm /usr/bin/python
sudo rm /usr/bin/pythonw
sudo rm /usr/bin/python-config
I had tried to install python with homebrew, but apparently it will not work unless all of XTools is installed, which I have been avoiding, since the version of XTools compatible with 10.6 is ancient and 4GB and mostly all I need is GCC, the compiler, which you can get here.
So I just installed with the latest download from python.org.
Then had to reinstall easy_install, pip, virtualenv.
Definitely wondering when it will be time for a new laptop, but there's a lot to be said for buying fewer pieces of hardware (slave labor, unethical mining, etc).

The above solutions failed for me, but the following worked:
python3 -m venv --without-pip <ENVIRONMENT_NAME>
. <ENVIRONMENT_NAME>/bin/activate
curl https://bootstrap.pypa.io/get-pip.py | python
deactivate
It's hacky, but yes, the core problem really did just seem to be pip.

I did the following steps to get virtualenv working :
Update virtualenv as follows :
==> sudo pip install --upgrade virtualenv
Initialize python3 virtualenv :
==> virtualenv -p python3 venv

I had this same issue, and I can confirm that the problem was with an outdated virtualenv.py file.
It was not necessary to do a whole install --upgrade.
Replacing the virtualenv.py file with the most recent version sufficed.

I also had this problem, and I tried the following method which worked for me:
conda install virtualenv
virtualenv --system-site-packages /anaconda/envs/tensorflow (here envs keeps all the virtual environments made by user)
source /anaconda/envs/tensorflow/bin/activate
Hope it's helpful.

I had this same issue when trying to install py2.7 on a newer system. The root issue was that virtualenv was part of py3.7 and thus was not compatible:
$ virtualenv -p python2.7 env
Running virtualenv with interpreter /usr/local/bin/python2.7
New python executable in /Users/blah/env/bin/python
ERROR: The executable /Users/blah/env/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/Library/Frameworks/Python.framework/Versions/2.7' (should be u'/Users/blah/env')
ERROR: virtualenv is not compatible with this system or executable
$ which virtualenv
/Library/Frameworks/Python.framework/Versions/3.7/bin/virtualenv
# install proper version of virtualenv
$ pip2.7 install virtualenv
$ /Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenv -p python2.7 env
$ . ./env/bin/activate
(env) $

Open terminal and type /Library/Frameworks/Python.framework/Versions/
then type ls /Library/Frameworks/Python.framework/Versions/2.7/bin/
if you are using Python2(or any other else).
Edit ~/.bash_profile and add the following line:
export PATH=$PATH:/Library/Frameworks/Python.framework/Versions/2.7/bin/
cat ~/.bash_profile
In my case the content of ~/.bash_profile is as follows:
export PATH=$PATH:/Library/Frameworks/Python.framework/Versions/2.7/bin/
Now the virtualenv command should work.

If you continue to have trouble with virtualenv, you might try pythonbrew, instead. It's an alternate solution to the same problem. It works more like Ruby's rvm: It builds and creates an entire instance of Python, under $HOME/.pythonbrew, and then sets up some bash functions that allow you to switch easily between versions. Where virtualenv shadows the system version of Python, using symbolic links as part of its solution, pythonbrew builds entirely self-contained installations of Python.
I used virtualenv for years. It's a decent solution, but I've switched to pythonbrew lately. Having completely self-contained Python instances means that installing a new one takes awhile (since pythonbrew actually compiles Python from scratch), but the self-contained nature of each installation appeals to me. And disk is cheap.

Develop Reference

Python is a programming language that lets you work quickly and integrate systems more effectively.