I would like to upgrade my Python version from 2.6.6 to the newest version of 3.6.5 on a server. However, there is a program that has been running for multiple days and will be running for a week or more.
Will my program continue to run until it finishes if I upgrade my Python version?
P.S.: I ran my Python program with the screen command so I wouldn't have to stop it to be able to upgrade my Python version.
Yes, your Python 2 script should continue to run while you install/upgrade Python 3, since Python 2 and 3 are different programs.
However, the installation will be be platform specific and there might be common dependencies, so I would test this on a different machine (e.g. local virtual machine) first.
Several aspects are to be considered here:
In general, a Python program that is running does not access the installed binaries of the interpreter anymore. Also all the modules already loaded will not be read again from disk during the execution time of this program. (In this aspect it differs from e. g. a Bash script which reads its script source while executing.) So you probably are safe in even removing the complete Python installation; of course that's not what you plan on and I wouldn't recommend it for other reasons, but even that should not influence your running Python program. If, however, your running Python program tries to import a module for the first time after the removal or upgrade of your Python installation, this might fail due to incompatibilities between the versions. Most programs do their imports only at the beginning of their runtime, though.
Python 2 and 3 are different languages. Scripts for the one need to be at least adjusted to run with the other. Adjustments aren't too hard to do, though. Most distributions have both installed side-by-side. Most of the time, Python2 is still standard (and accessed by typing python). python2 and python3 are then to make it explicit. This means that you probably should not remove Python 2 from your system, even if you plan on installing Python 3. There probably are scripts relying on the installation of Python 2.
There's a lot going on here that you've not explained/described, so it's hard to give you a definitive answer (such as the operating system you're using). This is also not exactly a programming question, so you might find that the question gets voted to close. That said...
It possible to run multiple versions of Python side by side. Because 2.x and 3.x are very different, it might be much better to have both versions installed. See this answer for some details on doing that:
Official multiple Python versions on the same machine?
It's likely that your code will need some modifications to run on Python 3, so you're going to have to redeploy a new version of your code at some point anyway.
Related
So one of our servers has v2.6 installed. I’ve checked and most of the scripts that runs in this server are unix scripts. There are few python scripts but I’m not sure if it’s still being used. They are not in cron. I don’t know their users as well.
Now I want to install another version which is v3.10. In short there will be two versions in the server — v2.6 and 3.10.
My question is — is there a chance that those scripts running in v2.6 will encounter any issues once we install v3.10?
If these scripts explicitly point to your python 2.x interpreter (with a shebang), then no, you won't have any issues.
However, if your question is : 'will my scripts written for python 2.x run without issue within python 3.10', then the answer is it depends.
Some python2 will run perfectly fine with python3.
Note that even if you install python3.10 on let's say Ubuntu, then python will still refer to your python2 installation by default.
up until recently I have only worked with one version of Python and used virtual environments every now and then. Now, I am working with some libraries that require older version of Python. So, I am very confused. Could anyone please clear up some of my confusion?
How do I install multiple Python versions?
I initially had Python version 3.8.x but upgraded to 3.10.x last month. There is currently only that one version on my PC now.
I wanted to install one of the Python 3.8.x version and went to https://www.python.org/downloads/. It lists a lot of versions and subversions like 3.6, 3.7, 3.8 etc. etc. with 3.8.1, 3.8.2 till 3.8.13. Which one should I pick?
I actually went ahead with 3.8.12 and downloaded the Tarball on the page: https://www.python.org/downloads/release/python-3812/
I extracted the tarball (23.6MB) and it created a folder with a setup.py file.
Is Python 3.8.12 now installed? Clicking on the setup.py file simply flashes the terminal for a second.
I have a few more questions. Hopefully, they won't get me downvoted. I am just confused and couldn't find proper answers for them.
Why does Python have such heavy dependency on the exact versions of libraries and packages etc?
For example, this question
How can I run Mozilla TTS/Coqui TTS training with CUDA on a Windows system?. This seems very beginner unfriendly. Slightly mismatched package
version can prevent any program from running.
Do virtual environments copy all the files from the main Python installation to create a virtual environment and then install specific packages inside it? Isn't that a lot of wasted resources in duplication because almost all projects require there own virtual environment.
Your questions depend a bit on "all the other software". For example, as #leiyang indicated, the answer will be different if you use conda vs just pip on vanilla CPython (the standard Windows Python).
I'm also going to assume you're actually on Windows, because on Linux I would recommend looking at pyenv. There is a pyenv-win, which may be worth looking into, but I don't use it myself because it doesn't play as nice if you also want (mini)conda environments.
1. (a) How do I install multiple Python versions?
Simply download the various installers and install them in sensible locations. E.g. "C:\Program Files\Python39" for Python 3.9, or some other location where you're allowed to install software.
Don't have Python add itself to the PATH though, since that'll only find the last version to do so and can really confuse things.
Also, you probably want to use virtual environments consistently, as this ties a specific project very clearly to a specific Python version, avoiding future confusion or problems.
1. (b) "3.8.1, 3.8.2 till 3.8.13" which should I pick?
Always pick the latest 3.x.y, so if there's a 3.8.13 for Windows, but no 3.8.14, pick that. Check if the version is actually available for your operating system, sometimes there are later versions for one OS, but not for another.
The reason is that between a verion like 3.6 and 3.7, there may be major changes that change how Python works. Generally, there will be backwards compatibility, but some changes may break how some of your packages work. However, when going up a minor version, there won't be any such breaking changes, just fixes and additions that don't get in the way of what was already there. A change from 2.x to 3.x only happens if the language itself goes through a major change, and rarely happens (and perhaps never will again, depending on who you ask).
An exception to the "no minor version change problems" is of course if you run some script that very specifically relies on something that was broken in 3.8.6, but no fixed in 3.8.7+ (as an example). However, that's very bad coding, to rely on what's broken and not fixing it later, so only go along with that if you have no other recourse. Otherwise, just the latest minor version of any version you're after.
Also: make sure you pick the correct architecture. If there's no specific requirement, just pick 64-bit, but if your script needs to interact with other installed software at the binary level, it may require you to install 32-bit Python (and 32-bit packages as well). If you have no such requirement, 64-bit allows more memory access and has some other benefits on modern computers.
2. Why does Python have such heavy dependency on the exact versions of libraries and packages etc?
It's not just Python, this is true for many languages. It's just more visible to the end user for Python, because you run it as an interpreted language. It's only compiled at the very last moment, on the computer it's running on.
This has the advantage that the code can run on a variety of computers and operating systems, but the downside that you need the right environment where you're running it. For people who code in languages like C++, they have to deal with this problem when they're coding, but target a much smaller number of environments (although there's still runtimes to contend with, and DirectX versions, etc.). Other languages just roll everything up into the program that's being distributed, while a Python script by itself can be tiny. It's a design choice.
There are a lot of tools to help you automate the process though and well-written packages will make the process quite painless. If you feel Python is very shakey when it comes to this, that's probable to blame on the packages or scripts you're using, not really the language. The only fault of the language is that it makes it very easy for developers to make such a mess for you and make your life hard with getting specific requirements.
Look for alternatives, but if you can't avoid using a specific script or package, once you figure out how to install or use it, document it or better yet, automate it so you don't have to think about it again.
3. Do virtual environments copy all the files from the main Python installation to create a virtual environment and then install specific packages inside it? Isn't that a lot of wasted resources in duplication because almost all projects require there own virtual environment.
Not all of them, but quite a few of them. However, you still need the original installation to be present on the system. Also, you can't pick up a virtual environment and put it somewhere else, not even on the same PC without some careful changes (often better to just recreate it).
You're right that this is a bit wasteful - but this is a difficult choice.
Either Python would be even more complicated, having to manage many different version of packages in a single environment (Java developers will be able to tell you war stories about this, with their dependency management - or wax lyrically about it, once they get it themselves).
Or you get what we have: a bit wasteful, but in the end diskspace is a lot cheaper than your time. And unlike your time, diskspace is almost infinitely expandable.
You can share virtual environments between very similar projects though, but especially if you get your code from someone else, it's best to not have to worry and just give up a few dozen MB for the project. On the upside: you can just delete a virtual environment directory and that pretty much gets rid of the whole things. Some applications like PyCharm may remember that it was once there, but other than that, that's the virtual environment gone.
Just install them. You can have any number of Python installations side by side. Unless you need to have 2 different minor versions, for example 3.10.1 and 3.10.2, there is no need to do anything special. (And if you do need that then you don't need any advice.) Just set up separate shortcuts for each one.
Remember you have to install any 3rd-party libraries you need in each version. To do this, navigate to the Scripts folder in the version you want to do the install in, and run pip from that folder.
Python's 3rd-party libraries are open-source and come from projects that have release schedules that don't necessarily coincide with Python's. So they will not always have a version available that coincides with the latest version of Python.
Often you can get around this by downloading unofficial binaries from Christoph Gohlke's site. Google Python Gohlke.
Install Python using the windows executable installers from python.org. If the version is 3.x.y, use the highest y that has a windows executable installer. Unless your machine is very old, use the 64-bit versions. Do not have them add python to your PATH environment variable, but in only one of the installs have it install the python launcher py. That will help you in using multiple versions. See e.g. here.
Python itself does not. But some modules/libraries do. Especially those that are not purely written in Python but contain extensions written in C(++). The reason for this is that compiling programs on ms-windows can be a real PITA. Unlike UNIX-like operating systems with Linux, ms-windows doesn't come with development tools as standard. Nor does it have decent package management. Since the official Python installers are built with microsoft tools, you need to use those with C(++) extensions as well. Before 2015, you even had to use exactly the same version of the compiler that Python was built with. That is still a good idea, but no longer strictly necessary. So it is a signigicant amount of work for developers to release binary packages for each supported Python version. It is much easier for them to say "requires Python 3.x".
I want to develop and test my project on the up-to-date version of Python 2.7 (say 2.7.18), but I want my project to be still fully usable on earlier versions of 2.7 (say 2.7.7). Setting up many variants of 2.7 locally or/and on CI for testing can be redundant.
So there are the following questions about compatibility of 2.7.X.
Can there be any changes in syntax which make code not working?
Can there be any changes in available standard imports, for example, can some imports from __future__ be unavailable in earlier versions?
Since I have to distribute compiled Python files (.pyc, compiled via py_compile module), I'm also wondering if there can be any changes in Python bytecode which block code execution in earlier versions.
I guess if all the answers are "no", I can develop and test my project only on a single 2.7 version without worries.
I've tried to search it but there is no success. Please share your experience and/or links.
UPD 1: I should have clearly said from the beginning that it's not my desire to use 2.7, it's a requirement from the environment.
At least Python 2.7.9 introduced massive changes to the 'ssl' module, so trying to use code using SSL for 2.7.18 on Python older than 2.7.9 will fail. So a clear "yes" to number 2.
In general compatbility for most projects works the other way round, use the oldest version you need to support and work upwards from old to new, not downwards from newer to older. I do not know of any software project that makes the guarantees in the other direction.
Note that Python 2.7 dropped out of support with 2.7.18, so unless you use a compatible version like PyPy (https://www.pypy.org/) your freshly developed project will run on outdated Python versions from the start.
If you want to provide a shrink wrapped product, maybe have a look at the usual solution for this like pyinstaller (https://www.pyinstaller.org/) or freeze (https://wiki.python.org/moin/Freeze)
The #3 may work, if you study the list of bytecode opcodes which do not change that much over time (https://github.com/python/cpython/commits/2.7/Include/opcode.h) but no idea if the on-disk format changed.
I just dealt with the worse bug in my entire 3 years of computer programming! It turns out that because I wanted to work with the natural language toolkit I had to install python 3.5 even though I'm using python 3.6. So I downloaded 3.5 and now my terminal is using python 3.5 by default and I can't get it back to 3.6. Because I was using python 3.5 which does not automatically order dictionaries it was throwing my program off because it relies on ordered dictionaries. It took me 4 hours to figure that out.
You want to use virtualenv and/or virtualenvwrapper. This is a utility that allows you to use multiple different environments, with different Python versions, different pip packages installed, etc.
To find the 3.5 version, run which python in your terminal to find the path to the python executable; then look at your PATH environment, and see where the location of that Python is on your PATH. Then you need to find out where that path is getting added; this will depend on your OS/Shell.
Tough times for sure, sorry to hear that.
I use pyenv to manage the different python versions on my system. This allows you to create virtual environments using whichever version you want.
EDIT to address comments.
I totally understand that setting up virtualenv or something like pyenv is not simple. However, it is unfortunately the easiest way to deal with (and avoid) situations like this. There are two essential concepts that are important here:
1) Isolation - Virtualenv takes care of this. When you install dependencies in a virtual environment, they will not affect your other environments or system python installation.
2) Multiple Python Versions - In your case, you needed to use a module that did not support 3.6. Instead of creating a virtual environment using python 3.5, you accidentally messed up your system installation of 3.6. Recovering from these types of misconfigurations can be difficult, and it is often easier to simply prevent it in the first place.
Again, I completely understand that this might be complicated, I remember thinking the same thing, but it is less complicated than troubleshooting the misconfigurations that can occur without this tooling.
I'm looking for a way to let multiple Python programs coexist on the same Windows machine.
Here's the problem: suppose program A needs Python 2.5, B needs 2.6, C needs 3, and each of them needs its own version of Qt, Wx or whatever other modules or whatever.
Trying to install all these dependencies on the same machine will break things, e.g. you can install different versions of Python side-by-side but only one of them can have the .py file association, so if you give that to Python 2.5 then B and C won't work, etc.
The ideal state of affairs would be if program A could live in C:\A along with its own Python interpreter, Qt/Wx/MySQL driver/whatever and never touch anything outside that directory, ditto for B and C.
Is there any way to accomplish this, other than going the full virtual box route?
edit: I tried the batch file solution, but it doesn't work. That is, it works on simple test scripts but e.g. OpenRPG fails at some point in its loading process if its required version of Python doesn't own the file association.
VirtualEnv.
virtualenv is a tool to create
isolated Python environments.
The basic problem being addressed is
one of dependencies and versions, and
indirectly permissions. Imagine you
have an application that needs version
1 of LibFoo, but another application
requires version 2. How can you use
both these applications? If you
install everything into
/usr/lib/python2.4/site-packages (or
whatever your platform's standard
location is), it's easy to end up in a
situation where you unintentionally
upgrade an application that shouldn't
be upgraded.
See previous answer here.
The other tool you should look at is pip which is great for installing particular versions of a library into a virtual environment. If you need to run v 1.0 of a library in python v 2.x for one application and 1.1 of the same library in python v 2.x, for example, you will need virtualenv plus a means of installing a particular version in that environment. Virtualenv + pip is your best choice.
Use batch files to run scripts, write in notepad for example:
c:\python26\python.exe C:\Script_B\B.py
and save it as runB.bat (or anything .bat). It will run with interpreter in c:\python26\python.exe file specified after a whitespace.
One solution would be to craft a batch file that invokes the correct interpreter for a given application. THis way, you can install additional interpreters in separate folders.
Probably not perfect but it works.
Have you considered compiling them to EXEs? Once you do that, all you have to do is call the EXE, for which the machine does not require python to be installed. All the required modules etc are packaged with the distribution when you compile.
write a python script that mimics the way unix shells handle scirpts -- look at the first line and see if it matches #!(name-of-shell). Then have your python script exec that interpreter and feed it the rest of its arguments.
Then, associate .py with your script.
It looks like the best solution is a batch file that sets the file association before running the appropriate version of Python, as mentioned in the comments to one of the answers here: how to run both python 2.6 and 3.0 on the same windows XP box?