Develop Reference

How to install multiple versions of Python in Windows?

up until recently I have only worked with one version of Python and used virtual environments every now and then. Now, I am working with some libraries that require older version of Python. So, I am very confused. Could anyone please clear up some of my confusion?
How do I install multiple Python versions?
I initially had Python version 3.8.x but upgraded to 3.10.x last month. There is currently only that one version on my PC now.
I wanted to install one of the Python 3.8.x version and went to https://www.python.org/downloads/. It lists a lot of versions and subversions like 3.6, 3.7, 3.8 etc. etc. with 3.8.1, 3.8.2 till 3.8.13. Which one should I pick?
I actually went ahead with 3.8.12 and downloaded the Tarball on the page: https://www.python.org/downloads/release/python-3812/
I extracted the tarball (23.6MB) and it created a folder with a setup.py file.
Is Python 3.8.12 now installed? Clicking on the setup.py file simply flashes the terminal for a second.
I have a few more questions. Hopefully, they won't get me downvoted. I am just confused and couldn't find proper answers for them.
Why does Python have such heavy dependency on the exact versions of libraries and packages etc?
For example, this question
How can I run Mozilla TTS/Coqui TTS training with CUDA on a Windows system?. This seems very beginner unfriendly. Slightly mismatched package
version can prevent any program from running.
Do virtual environments copy all the files from the main Python installation to create a virtual environment and then install specific packages inside it? Isn't that a lot of wasted resources in duplication because almost all projects require there own virtual environment.

Your questions depend a bit on "all the other software". For example, as #leiyang indicated, the answer will be different if you use conda vs just pip on vanilla CPython (the standard Windows Python).
I'm also going to assume you're actually on Windows, because on Linux I would recommend looking at pyenv. There is a pyenv-win, which may be worth looking into, but I don't use it myself because it doesn't play as nice if you also want (mini)conda environments.
1. (a) How do I install multiple Python versions?
Simply download the various installers and install them in sensible locations. E.g. "C:\Program Files\Python39" for Python 3.9, or some other location where you're allowed to install software.
Don't have Python add itself to the PATH though, since that'll only find the last version to do so and can really confuse things.
Also, you probably want to use virtual environments consistently, as this ties a specific project very clearly to a specific Python version, avoiding future confusion or problems.
1. (b) "3.8.1, 3.8.2 till 3.8.13" which should I pick?
Always pick the latest 3.x.y, so if there's a 3.8.13 for Windows, but no 3.8.14, pick that. Check if the version is actually available for your operating system, sometimes there are later versions for one OS, but not for another.
The reason is that between a verion like 3.6 and 3.7, there may be major changes that change how Python works. Generally, there will be backwards compatibility, but some changes may break how some of your packages work. However, when going up a minor version, there won't be any such breaking changes, just fixes and additions that don't get in the way of what was already there. A change from 2.x to 3.x only happens if the language itself goes through a major change, and rarely happens (and perhaps never will again, depending on who you ask).
An exception to the "no minor version change problems" is of course if you run some script that very specifically relies on something that was broken in 3.8.6, but no fixed in 3.8.7+ (as an example). However, that's very bad coding, to rely on what's broken and not fixing it later, so only go along with that if you have no other recourse. Otherwise, just the latest minor version of any version you're after.
Also: make sure you pick the correct architecture. If there's no specific requirement, just pick 64-bit, but if your script needs to interact with other installed software at the binary level, it may require you to install 32-bit Python (and 32-bit packages as well). If you have no such requirement, 64-bit allows more memory access and has some other benefits on modern computers.
2. Why does Python have such heavy dependency on the exact versions of libraries and packages etc?
It's not just Python, this is true for many languages. It's just more visible to the end user for Python, because you run it as an interpreted language. It's only compiled at the very last moment, on the computer it's running on.
This has the advantage that the code can run on a variety of computers and operating systems, but the downside that you need the right environment where you're running it. For people who code in languages like C++, they have to deal with this problem when they're coding, but target a much smaller number of environments (although there's still runtimes to contend with, and DirectX versions, etc.). Other languages just roll everything up into the program that's being distributed, while a Python script by itself can be tiny. It's a design choice.
There are a lot of tools to help you automate the process though and well-written packages will make the process quite painless. If you feel Python is very shakey when it comes to this, that's probable to blame on the packages or scripts you're using, not really the language. The only fault of the language is that it makes it very easy for developers to make such a mess for you and make your life hard with getting specific requirements.
Look for alternatives, but if you can't avoid using a specific script or package, once you figure out how to install or use it, document it or better yet, automate it so you don't have to think about it again.
3. Do virtual environments copy all the files from the main Python installation to create a virtual environment and then install specific packages inside it? Isn't that a lot of wasted resources in duplication because almost all projects require there own virtual environment.
Not all of them, but quite a few of them. However, you still need the original installation to be present on the system. Also, you can't pick up a virtual environment and put it somewhere else, not even on the same PC without some careful changes (often better to just recreate it).
You're right that this is a bit wasteful - but this is a difficult choice.
Either Python would be even more complicated, having to manage many different version of packages in a single environment (Java developers will be able to tell you war stories about this, with their dependency management - or wax lyrically about it, once they get it themselves).
Or you get what we have: a bit wasteful, but in the end diskspace is a lot cheaper than your time. And unlike your time, diskspace is almost infinitely expandable.
You can share virtual environments between very similar projects though, but especially if you get your code from someone else, it's best to not have to worry and just give up a few dozen MB for the project. On the upside: you can just delete a virtual environment directory and that pretty much gets rid of the whole things. Some applications like PyCharm may remember that it was once there, but other than that, that's the virtual environment gone.

Just install them. You can have any number of Python installations side by side. Unless you need to have 2 different minor versions, for example 3.10.1 and 3.10.2, there is no need to do anything special. (And if you do need that then you don't need any advice.) Just set up separate shortcuts for each one.
Remember you have to install any 3rd-party libraries you need in each version. To do this, navigate to the Scripts folder in the version you want to do the install in, and run pip from that folder.
Python's 3rd-party libraries are open-source and come from projects that have release schedules that don't necessarily coincide with Python's. So they will not always have a version available that coincides with the latest version of Python.
Often you can get around this by downloading unofficial binaries from Christoph Gohlke's site. Google Python Gohlke.

Install Python using the windows executable installers from python.org. If the version is 3.x.y, use the highest y that has a windows executable installer. Unless your machine is very old, use the 64-bit versions. Do not have them add python to your PATH environment variable, but in only one of the installs have it install the python launcher py. That will help you in using multiple versions. See e.g. here.
Python itself does not. But some modules/libraries do. Especially those that are not purely written in Python but contain extensions written in C(++). The reason for this is that compiling programs on ms-windows can be a real PITA. Unlike UNIX-like operating systems with Linux, ms-windows doesn't come with development tools as standard. Nor does it have decent package management. Since the official Python installers are built with microsoft tools, you need to use those with C(++) extensions as well. Before 2015, you even had to use exactly the same version of the compiler that Python was built with. That is still a good idea, but no longer strictly necessary. So it is a signigicant amount of work for developers to release binary packages for each supported Python version. It is much easier for them to say "requires Python 3.x".

How to change python's "full name" (something like `cpython-34m-x86_64-linux-gnu`)?

I installed Python 3.5 from source and broke a number of modules on Python 3.4, which unfortunately, was an essential part of Ubuntu. I've been trying to fix the system, now I'm almost there, with (hopefully) the last problem: My Python 3.4 only recognize C modules with name *.cpython-34m.so, while all packages from Ubuntu repository are named *.cpython-34m-x86_64-linux-gnu.so. It seems that the cpython-34m stuff is the full name of Python, so I need to change it in accord with Ubuntu's expectation. How can I achieve this?

What you are trying makes no sense. The name cannot be changed, for a good reason. The reason the names are different is to prevent incompatible versions from mixing up each other. You can compile a different version with different options and then the name will be different, too.

The essence of the package management is to satisfy all dependencies through the repositories. Theoretically there should be no need to install anything from the sources.
If you need a different Python version for some specifical reasons, virtual environments would be the thing to look for:
http://docs.python-guide.org/en/latest/dev/virtualenvs/

I reinstalled Python 3.4 via Ubuntu package system, and suddenly everything worked fine. I still have no clue how Ubuntu customize its own Python, since Python's configure command has no related option at all. Anyway, it works, so why bother :P. Finally, thank you for helping me with this problem.

Python Code Completion

After using C# for long time I finally decided to switch to Python.
The question I am facing for the moment has to do about auto-complete.
I guess I am spoiled by C# and especially from resharper and I was expecting something similar to exist for Python.
My editor of choice is emacs and after doing some research I found autocomplete.pl, yasnippet and rope although it is not clear to me if and how they can be installed in a cygwin based system which is what I use since all the related documentation appears to be linux specific...
The version of emacs I currently use is 23.2.1 which bundles the python mode that although useful is far behind from whatever research has to offer.
My question to python users has to do about how common is autocomplete vs manual typing (using M-/ where possible) ?
I am thinking about just memorizing python build-in functions like len, append, extend etc. and revert close to a pre-autocomplete editing mode. How different such an approach is from what other pythonistas are doing?

I found this post
My Emacs Python environment
to be the most useful and comprehensive list of instructions and references on how to setup a decent Python development environment in Emacs regardless of OS platform. It is still a bit of work to setup but at least it covers the popular packages and components generally recommended for Python in Emacs that provide auto-completion functionality.
I loosely used this post as a guide to do the setup on my Windows machine with Emacs 23.2.1 and Python 2.6.5. Although, I also have Cygwin installed in some cases instead of running the *nix shell commands mentioned in the post, I just download the packages via a web browser, unzip them with 7zip, and copy them to my Emacs' plugin directory.
Also, to install Pymacs, Rope, and Ropemacs, I used Python's EasyInstall package manager. To use it, I downloaded and installed the setuptools package using the Windows install version. Once installed, at the command line, cd to their respective download locations and run the command
easy_install .
instead of the shell commands shown in the post.
Generally, I saved any *.el files in my ~\.emacs.d\plugins (e.g. in %USERPROFILE%\Application Data\.emacs.d\) and then updated my .emacs file to reference them as documented in the post.
Despite all this, on occasion, I've used DreamPie since it does have overall better auto-completion out of the box than my Emacs setup.

I'm spoiled by Intellisense too. The PyDev extensions for Eclipse offer a pretty good auto-complete substitute.

I find that PyDev + Eclipse can meet most of my needs. There is also PyCharm from the Intellij team. PyCharm has the added advantage of smooth integration with git.

I've been using PyScripter, an IDE for Windows, for a while now, and have found it very good. It has autocompletion among many other features. It's written in Delphi -- not that there's anything wrong with that -- it just bothers me a bit, though...

Take a look at Spyderlib, support most of the features including code completion

IMO, by far the easiest way to take advantage of the python tools available for emacs is to take advantage of the defaults that are all set up at:
https://github.com/gabrielelanaro/emacs-for-python
I actually took the time to get pymacs and ropemacs and python-mode all working independently before finding that little gem, and now I rely on it entirely for all my python based customizations. If you are new, I would definitely start there.

How do I set up a Python development environment on Linux?

I'm a .NET developer who knows very little about Python, but want to give it a test drive for a small project I'm working on.
What tools and packages should I install on my machine? I'm looking for a common, somewhat comprehensive, development environment.
I'll likely run Ubuntu 9.10, but I'm flexible. If Windows is a better option, that's fine too.
Edit: To clarify, I'm not looking for the bare minimum to get a Python program to run. I wouldn't expect a newbie .NET dev to use notepad and a compiler. I'd recommend Visual Studio, NUnit, SQL Server, etc.

Your system already has Python on it. Use the text editor or IDE of your choice; I like vim.
I can't tell you what third-party modules you need without knowing what kind of development you will be doing. Use apt as much as you can to get the libraries.
To speak to your edit:
This isn't minimalistic, like handing a .NET newbie notepad and a compiler: a decent text editor and the stdlib are all you really need to start out. You will likely need third-party libraries to develop whatever kind of applications you are writing, but I cannot think of any third-party modules all Python programmers will really need or want.
Unlke the .NET/Windows programming world, there is no one set of dev tools that stands above all others. Different people use different editors a whole lot. In Python, a module namespace is fully within a single file and project organization is based on the filesystem, so people do not lean on their IDEs as hard. Different projects use different version control software, which has been booming with new faces recently. Most of these are better than TFS and all are 1000 times better than SourceSafe.
When I want an interactive session, I use the vanilla Python interpreter. Various more fancy interpreters exist: bpython, ipython, IDLE. bpython is the least fancy of these and is supposed to be good about not doing weird stuff. ipython and IDLE can lead to strange bugs where code that works in them doens't work in normal Python and vice-versa; I've seen this first hand with IDLE.
For some of the tools you asked about and some others
In .NET you would use NUnit. In Python, use the stdlib unittest module. There are various third-party extensions and test runners, but unittest should suit you okay.
If you really want to look into something beyond this, get unittest2, a backport of the 2.7 version of unittest. It has incorporated all the best things from the third-party tools and is really neat.
In .NET you would use SQL Server. In Python, you may use PostgreSQL, MySQL, sqlite, or some other database. Python specifies a unified API for databases and porting from one to another typically goes pretty smoothly. sqlite is in the stdlib.
There are various Object Relational Models to make using databases more abstracted. SQLAlchemy is the most notable of these.
If you are doing network programming, get Twisted.
If you are doing numerical math, get numpy and scipy.
If you are doing web development, choose a framework. There are about 200000: Pylons, zope, Django, CherryPy, werkzeug...I won't bother starting an argument by recommending one. Most of these will happily work with various servers with a quick setting.
If you want to do GUI development, there are quite a few Python bindings. The stdlib ships with Tk bindings I would not bother with. There are wx bindings (wxpython), GTK+ bindings (pygtk), and two sets of Qt bindings. If you want to do native Windows GUI development, get IronPython and do it in .NET. There are win32 bindings, but they'll make you want to pull your hair out trying to use them directly.

In order to reduce the chance of effecting/hosing the system install of python, I typically install virtualenv on the ubuntu python install. I then create a virtualenv in my home directory so that subsequent packages I install via pip or easy_install do not effect the system installation. And I add the bin from that virtualenv to my path via .bashrc
$ sudo apt-get install python-virtualenv
$ virtualenv --no-site-packages ~/local
$ PATH=~/local/bin:$PATH #<----- add this to .bashrc to make it permanent
$ easy_install virtualenv #<--- so that project environments are based off your local environment rather than the system, probably not necessary
Install your favorite editor, I like emacs + rope, but editors are a personal preference and there are plenty of choices.
When I start a new project/idea I create a new virtual environment for that project, so that I don't effect dependencies anywhere else. Since I would hate for some of my projects to break due to an upgrade of a library both that project and the new one depends on.
~/projects $ virtualenv --no-site-packages my_new_project.env
~/projects/my_new_project.env $ source bin/activate
(my_new_project.env)~/projects/my_new_project.env $ easy_install paste ipython #whatever else I think I need
(my_new_project.env)~/projects/my_new_project.env $ emacs ./ & # start hacking
When creating a new package...in order to have something that will be easy_installable/pippable use paster create
(my_new_project.env)~/projects/my_new_project.env$ paster create new_package
(my_new_project.env)~/projects/my_new_project.env/new_package$ python setup.py develop new_package
That's the common stuff as far as I can think of it. Everything else would be editor/version control tool specific

Since I'm accustomed to Eclipse, I find Eclipse + PyDev convenient for Python. For quick computations, Idle is great.
I've used Python on Windows and on Ubuntu, and Linux is much cleaner.

If you launch a terminal and type python you'll get an interpreter, where you can start trying stuff.
Just in case you haven't seen it, check out the book Dive Into Python, is free on-line.
http://www.diveintopython.org/
Follow the examples in the book using the interpreter.
For storing your work you could use any editor; Vim or EMACS could be the most powerful, but also the most difficult to learn at first. If you want a more "traditional" IDE, you could try WingIDE.
http://www.wingware.com/
After you start to get more comfortable with python you should try an enhanced interpreter; try ipython.
http://ipython.scipy.org/moin/
When you start to develop a more serious project you'll need to get additional modules. Here you have two options; 1) Use your distribution tools to install additional modules; or 2) Download the modules you need directly from their sites and install them manually. You'll be responsible to upgrade them of course.
You'll have to decide for yourself which way to go. Personally I prefer to download and install additional modules manually.

Python (duh), setuptools or pip, virtualenv, and an editor. I suggest geany, but that's just me. And of course, any other Python modules you'll need.

Getting to Python from .NET world
Jumping into the Linux world from a .NET / WIndows background can be a bit disconcerting (but I do encourage you to keep trying Linux)
But I would suggest to anyone coming from Windows, to stick with Windows for a little while. goto www.Activestate.com and download their Python package - it includes the full win32com extentions by Mark Hammond and it also includes a complete, fast IDE "pythonwin"
I have done real professional development with just this setup alone on a windows box - one 14MB .msi and off you go !
Now to use Python on the DLR (Dynamic common language runtime) you need to download IronPython. THis is a seperate interpreter, that was also originally written by Mark Hammond at Microsoft and is at ironpython.org.
With this you can run code like (from wikipedia) ::
import clr
clr.AddReference("System.Windows.Forms")
from System.Windows.Forms import MessageBox
MessageBox.Show("Hello World")
Now you can access any .NET code from python.

If you're just starting out with Python, I'd actually argue against bringing in the complexity of virtualenv (which I think can be pretty overwhelming), at least until you've got a firm grasp of Python basics (especially regarding library/dependency management).
If you're using Ubuntu and the Gnome desktop environment, gedit is the default (gui) text editor, and has great support for Python built in. So my recommendation is to start with the pre-installed Python and gedit (which is pretty extensible on its own).

You don't need much. Python comes with "Batteries Included."
Visual Studio == IDLE. You already have it. If you want more IDE-like environment, install Komodo Edit.
NUnit == unittest. You already have it in the standard library.
SQL Server == sqlite. You already have it in the standard library.
Stop wasting time getting everything ready. It's already there in the basic Python installation.
Get to work.
Linux, BTW, is primarily a development environment. It was designed and built by developers for developers. Windows is an end-user environment which has to be supplemented for development.
Linux was originally focused on developers. All the tools you need are either already there or are part of simple yum or RPM installs.

You would probably like to give NetBeans Python IDE a shot. You can choose to use either Windows/Linux.

Database: sqlite (inbuilt). You might want SQLAlchemy though.
GUI: tcl is inbuilt, but wxPython or pyQt are recommended.
IDE: I use idle (inbuilt) on windows, TextMate on Mac, but you might like PyDev. I've also heard good things about ulipad.
Numerics: numpy.
Fast inline code: lots of options. I like boost weave (part of scipy), but you could look into ctypes (to use dlls), Cython, etc.
Web server: too many options. Django (plus Apache) is the biggest.
Unit testing: inbuilt.
Pyparsing, just because.
BeautifulSoup (or another good HTML parser).
hg, git, or some other nice VC.
Trac, or another bug system.
Oh, and StackOverflow if you have any questions.

Pycharm Community is worth to try.

What version of Python (2.4, 2.5, 2.6, 3.0) do you standardize on for production development efforts (and why)?

In our group we primarily do search engine architecture and content integration work and most of that code base is in Python. All our build tools and Python module dependencies are in source control so they can be checked out and the environment loaded for use regardless of os/platform, kinda similar to the approach virtualenv uses.
For years we've maintained a code base compatible with Python 2.3 because one of the commercial products we use depends on Python 2.3. Over the years this has caused more and more issues as newer tools and libraries need newer versions of Python since 2.3 came out in ~2004.
We've recently decoupled our build environment from dependencies on the commercial product's environment and can use any version of Python (or Java) we want. Its been about a month or so since we standardized on Python 2.6 as the newest version of Python that is backwards compatible with previous versions.
Python 3.0 is not an option (for now) since we'd have to migrate too much of our code base to make our build and integration tools to work correctly again.
We like many of the new features of Python 2.6, especially the improved modules and things like class decorators, but many modules we depend on cause the Python 2.6 interpreter to spout various depreciation warnings. Another tool we're interested in for managing EC2 cloud cluster nodes, Supervisor doesn't even work correctly with Python 2.6.
Now I am wondering if we should standardize on Python 2.5 for now instead of using Python 2.6 in development of production environment tools. Most of the tools we want/need seem to work correctly with Python 2.5. We're trying to sort this out now before there are many dependencies on Python 2.6 features or modules.
Many Thanks!
-Michael

I wouldn't abandon 2.6 just because of deprecation warnings; those will disappear over time. (You can use the -W ignore option to the Python interpreter to prevent them from being printed out, at least) But if modules you need to use actually don't work with Python 2.6, that would be a legitimate reason to stay with 2.5. Python 2.5 is in wide use now and probably will be for a long time to come (consider how long 2.3 has lasted!), so even if you go with 2.5, you won't be forced to upgrade for a while.
I use Python 2.5 for all my development work, but only because it's the version that happens to be available in Gentoo (Linux)'s package repository. When the Gentoo maintainers declare Python 2.6 "stable"*, I'll switch to that. Of course, this reasoning wouldn't necessarily apply to you.
* Python 2.6 actually is stable, the reason it's not declared as such in Gentoo is that Gentoo relies on other programs which themselves depend on Python and are not yet upgraded to work with 2.6. Again, this reasoning probably doesn't apply to you.

My company is standardized in 2.5. Like you we can't make the switch to 3.0 for a million reasons, but I very much wish we could move up to 2.6.
Doing coding day to day I'll be looking through the documentation and I'll find exactly the module or function that I want, but then it'll have the little annotation: New in Version 2.6
I would say go with the newest version, and if you have depreciation warnings pop up (there will probably be very few) then just go in a find a better way to do it. Overall your code will be better with 2.6.

To me the most important to stick with python 2.5+ is because it officially supports ctypes, which changed many plugin systems.
Although you can find ctypes to work with 2.3/2.4, they are not officially bundled.
So my suggestion would be 2.5.

We're sticking with 2.5.2 for now. Our tech stack centers on Django (but we have a dozen other bits and bobs.) So we stay close to what they do.
We had to go back to docutils to 0.4 so it would work with epydoc 3.0.1. So far, this hasn't been a big issue, but it may -- at some point -- cause us to rethink our use of epydoc.
The 2.6 upgrade is part of our development plan. We have budget, but not fixed schedule right now.
The 3.0 upgrade, similarly, is something I remind folks of. We have to budget for it. We won't do it this year unless Django leaps to 3.0. We might do it next year.

I think the best solution is to give up the hope on total uniformity, although having a common environment is something to strive for. You will always will be confronted with version problems, for example when upgrading to the next best interpreter version.
So instead of dealing with it on a per issue base you could solve this problem by taking a good look on your release management.
Instead of releasing source, go for platform depending binaries (besides the source distribution).
So what you do is that you define a number of supported CPU's, for example:
x86-32, x86-64, sparc
Then which operating systems:
Linux, Windows, Solaris, FreeBSD
For each OS you support a number of their major versions.
Next step is that you provide binaries for all of them.
Yes indeed this will require quite some infrastructure investment and setting up of automatic building from your repositories (you do have them?).
The advantages is that your users only have 'one' thing to install and you can easily switch versions or even mixing versions. Actually you can even use different programming languages with this approach without affecting your release management too much.