I'm looking at some software that is wanting to bring in Python 3.6 for use in an environment where 3.5 is the standard. Reading up on Python's documentation I can't find anything about whether:
3.5 is representative of a semantic version number
3.6 would represent a forwards compatible upgrade (ie: code written for a 3.5 runtime is guaranteed to work in a 3.6 runtime)
The fact that this page about porting to 3.7 exists makes me think strongly no but I can't see official docs on what the version numbers mean (if anything, ala Linux kernel versioning)
In the more general sense - is there a PEP around compatibility standards within the 3.X release stream?
The short answer is "No", the long answer is "They strive for something close to it".
As a rule, micro versions match semantic versioning rules; they're not supposed to break anything or add features, just fix bugs. This isn't always the case (e.g. 3.5.1 broke vars() on a namedtuple, because it caused a bug that was worse than the break when it came up), but it's very rare for code (especially Python level stuff, as opposed to C extensions) to break across a micro boundary.
Minor versions mostly "add features", but they will also make backwards incompatible changes with prior warning. For example, async and await became keywords in Python 3.7, which meant code using them as variable names broke, but with warnings enabled, you would have seen a DeprecationWarning in 3.6. Many syntax changes are initially introduced as optional imports from the special __future__ module, with documented timelines for becoming the default behavior.
None of the changes made in minor releases are broad changes; I doubt any individual deprecation or syntax change has affected even 1% of existing source code, but it does happen. If you've got a hundred third party dependencies, and you're jumping a minor version or two, there is a non-trivial chance that one of them will be broken by the change (example: pika prior to 0.12 used async as a variable name, and broke on Python 3.7; they released new versions that fixed the bug, but of course, moving from 0.11 and lower to 0.12 and higher changed their own API in ways that might break your code).
Major versions are roughly as you'd expect; backwards incompatible changes are expected/allowed (though they're generally not made frivolously; the bigger the change, the bigger the benefit).
Point is, it's close to semantic versioning, but in the interests of not having major releases every few years, while also not letting the language stagnate due to strict compatibility constraints, minor releases are allowed to break small amounts of existing code as long as there is warning (typically in the form of actual warnings from code using deprecated behavior, notes on the What's New documentation, and sometimes __future__ support to ease the migration path).
This is all officially documented (with slightly less detail) in their Development Cycle documentation:
To clarify terminology, Python uses a major.minor.micro nomenclature for production-ready releases. So for Python 3.1.2 final, that is a major version of 3, a minor version of 1, and a micro version of 2.
new major versions are exceptional; they only come when strongly incompatible changes are deemed necessary, and are planned very long in advance;
new minor versions are feature releases; they get released annually, from the current in-development branch;
new micro versions are bugfix releases; they get released roughly every 2 months; they are prepared in maintenance branches.
Here's the document on updating to 3.6.
If you had, for example, open(apath, 'U+') in your code in 3.5, it would fail in 3.6. So, clearly, Python 3.6 is not entirely backwards compatible to every usage in 3.5.
Realistically, you will need to test, although I feel fairly comfortable telling the average stackoverflow reader from almost every area that they should feel comfortable doing this upgrade.
As for Semantic Versioning, specifically, Python does not follow it, but it isn't entirely agnostic to the meaning of major, minor and bugfix releases. Python guidelines for its developers can be found here.
To clarify terminology, Python uses a major.minor.micro nomenclature
for production-ready releases. So for Python 3.1.2 final, that is a
major version of 3, a minor version of 1, and a micro version of 2.
new major versions are exceptional; they only come when strongly incompatible changes are deemed necessary, and are planned very long
in advance;
new minor versions are feature releases; they get released annually, from the current in-development branch;
new micro versions are bugfix releases; they get released roughly every 2 months; they are prepared in maintenance branches.
Also read PEP440, which is for modules, not about releasing new versions of python itself, but still relevant for the philosophy of the ecosystem.
Related
I've read in few places that generally, Python doesn't provide backward compatibility, which means that any newer version of Python may break code that worked fine for earlier versions. If so, what is my way as a developer to know what versions of Python can execute my code successfully? Is there any set of rules/guarantees regarding this? Or should I just tell my users: Just run this with Python 3.8 (for example) - no more no less...?
99% of the time, if it works on Python 3.x, it'll work on 3.y where y >= x. Enabling warnings when running your code on the older version should pop DeprecationWarnings when you use a feature that's deprecated (and therefore likely to change/be removed in later Python versions). Aside from that, you can read the What's New docs for each version between the known good version and the later versions, in particular the Deprecated and Removed sections of each.
Beyond that, the only solution is good unit and component tests (you are using those, right? 😉) that you rerun on newer releases to verify stuff still works & behavior doesn't change.
According to PEP387, section "Making Incompatible Changes", before incompatible changes are made, a deprecation warning should appear in at least two minor Python versions of the same major version, or one minor version in an older major version. After that, it's a free game, in principle. This made me cringe with regards to safety. Who knows if people run airplanes on Python and if they don't always read the python-dev list. So if you have something that passes 100% coverage unit tests without deprecation warnings, your code should be safe for the next two minor releases.
You can avoid this issue and many others by containerizing your deployments.
tox is great for running unit tests against multiple Python versions. That’s useful for at least 2 major cases:
You want to ensure compatibility for a certain set of Python versions, say 3.7+, and to be told if you make any breaking changes.
You don’t really know what versions your code supports, but want to establish a baseline of supported versions for future work.
I don’t use it for internal projects where I can control over the environment where my code will be running. It’s lovely for people publishing apps or libraries to PyPI, though.
This is purely a curiosity-only question.
I have noticed a difference between that when Macvim has syntax highlighting on, for python files, it makes a distinction between else and else:. The former it will recognoze as a keyword and highlight but not the latter. It does the same with other keywords like except and finally. This behavior is not present in the Windows version. I have noticed this with both versions 7.3 & 7.4.
Why is that? I thought they had the same code-base?
They are built from the same code base but that code base is constantly evolving with patches merged in every couple of days non-stop since years. Whatever build you have is guaranteed to be outdated quite quickly.
For that reason, it's rather pointless to compare Vim between platforms or major versions. What counts is what we call the "patch-level": what patches were applied to the source when your Vim was built.
Additionally, Vim can be compiled with or without this or that feature which kind of complicates things.
Your 7.3 might be very different from my 7.3 at work and both will be very different from your 7.4 or mine, here at home.
When you compare Vim builds, you have to take the whole output of :version and, most importantly, the patch-level. My Vim 7.4 has patches 1 through 26 applied, as indicated by the 2nd line of :version:
Rustines incluses : 1-26
The discrepancy you are witnessing is probably due to some changes to the Python syntax files that intervened between builds.
At present(May 2013), there are three release versions, all released on may 15
python 3.3.2
python 3.2.5
python 2.7.5
I can understand the need for 2.x and 3.x branches but why are there seperate 3.3.x and 3.2.x versions?
In this link is says The current production versions are 2.7.5 and 3.3.2..
And if you look here it says:
Python 3.2.5 was released on May 15th, 2013. This release fixes a few regressions found in Python 3.2.4, and is planned to be the final 3.2 series bug-fix release.
So you should use 2.7.5 or 3.3.2, but if you need (I don't know why) 3.2.* you have a bug-fixed version.
As wim points out, 3.2.5 is not a current production version, but I assume you're wondering why there were three versions released on 15 May 2013? That is why is the 3.2.x branch still being maintained?
Remember that each 3.n step introduces new features while 3.n.x releases are fixes to existing versions. 3.2.5 is thus a set of bugfixes to 3.2.4 while the 3.3.x branch includes new features not present in 3.2.4. Because new features are, inherently, more likely to introduce new bugs, the maintenance of the older branch allows you a higher stability choice if, for example, you're just putting together a new public release of your webserver and don't want to risk new bugs being introduced by the current branch.
This is a question of python's versioning strategy. Quote from python's wikipedia article:
CPython's public releases come in three types, distinguished by which
part of the version number is incremented:
Backwards-incompatible versions, where code is expected to break
and must be manually ported. The first part of the version number is
incremented. These releases happen infrequently—for example, version
3.0 was released 8 years after 2.0.
Major or "feature" releases, which are largely compatible but introduce new features. The second
part of the version number is incremented. These releases are
scheduled to occur roughly every 18 months, and each major version is
supported by bugfixes for several years after its release.
Bugfix releases, which introduce no new features but fix bugs. The
third and final part of the version number is incremented. These
releases are made whenever a sufficient number of bugs have been fixed
upstream since the last release, or roughly every 3 months. Security
vulnerabilities are also patched in bugfix releases.
So, 3.3 compared to 3.2 introduced new major features, that's why it's in a separate "branch".
Also see:
Python 3.2 Release Schedule
Python 3.3 Release Schedule
Python 3.4 Release Schedule
You should read bit about version numbers. The last digit means, simplified, no new features only bug fixes. So folks who use Python 3.2 can installed a newer revision without changing anything in the behavior of Python.
In our group we primarily do search engine architecture and content integration work and most of that code base is in Python. All our build tools and Python module dependencies are in source control so they can be checked out and the environment loaded for use regardless of os/platform, kinda similar to the approach virtualenv uses.
For years we've maintained a code base compatible with Python 2.3 because one of the commercial products we use depends on Python 2.3. Over the years this has caused more and more issues as newer tools and libraries need newer versions of Python since 2.3 came out in ~2004.
We've recently decoupled our build environment from dependencies on the commercial product's environment and can use any version of Python (or Java) we want. Its been about a month or so since we standardized on Python 2.6 as the newest version of Python that is backwards compatible with previous versions.
Python 3.0 is not an option (for now) since we'd have to migrate too much of our code base to make our build and integration tools to work correctly again.
We like many of the new features of Python 2.6, especially the improved modules and things like class decorators, but many modules we depend on cause the Python 2.6 interpreter to spout various depreciation warnings. Another tool we're interested in for managing EC2 cloud cluster nodes, Supervisor doesn't even work correctly with Python 2.6.
Now I am wondering if we should standardize on Python 2.5 for now instead of using Python 2.6 in development of production environment tools. Most of the tools we want/need seem to work correctly with Python 2.5. We're trying to sort this out now before there are many dependencies on Python 2.6 features or modules.
Many Thanks!
-Michael
I wouldn't abandon 2.6 just because of deprecation warnings; those will disappear over time. (You can use the -W ignore option to the Python interpreter to prevent them from being printed out, at least) But if modules you need to use actually don't work with Python 2.6, that would be a legitimate reason to stay with 2.5. Python 2.5 is in wide use now and probably will be for a long time to come (consider how long 2.3 has lasted!), so even if you go with 2.5, you won't be forced to upgrade for a while.
I use Python 2.5 for all my development work, but only because it's the version that happens to be available in Gentoo (Linux)'s package repository. When the Gentoo maintainers declare Python 2.6 "stable"*, I'll switch to that. Of course, this reasoning wouldn't necessarily apply to you.
* Python 2.6 actually is stable, the reason it's not declared as such in Gentoo is that Gentoo relies on other programs which themselves depend on Python and are not yet upgraded to work with 2.6. Again, this reasoning probably doesn't apply to you.
My company is standardized in 2.5. Like you we can't make the switch to 3.0 for a million reasons, but I very much wish we could move up to 2.6.
Doing coding day to day I'll be looking through the documentation and I'll find exactly the module or function that I want, but then it'll have the little annotation: New in Version 2.6
I would say go with the newest version, and if you have depreciation warnings pop up (there will probably be very few) then just go in a find a better way to do it. Overall your code will be better with 2.6.
To me the most important to stick with python 2.5+ is because it officially supports ctypes, which changed many plugin systems.
Although you can find ctypes to work with 2.3/2.4, they are not officially bundled.
So my suggestion would be 2.5.
We're sticking with 2.5.2 for now. Our tech stack centers on Django (but we have a dozen other bits and bobs.) So we stay close to what they do.
We had to go back to docutils to 0.4 so it would work with epydoc 3.0.1. So far, this hasn't been a big issue, but it may -- at some point -- cause us to rethink our use of epydoc.
The 2.6 upgrade is part of our development plan. We have budget, but not fixed schedule right now.
The 3.0 upgrade, similarly, is something I remind folks of. We have to budget for it. We won't do it this year unless Django leaps to 3.0. We might do it next year.
I think the best solution is to give up the hope on total uniformity, although having a common environment is something to strive for. You will always will be confronted with version problems, for example when upgrading to the next best interpreter version.
So instead of dealing with it on a per issue base you could solve this problem by taking a good look on your release management.
Instead of releasing source, go for platform depending binaries (besides the source distribution).
So what you do is that you define a number of supported CPU's, for example:
x86-32, x86-64, sparc
Then which operating systems:
Linux, Windows, Solaris, FreeBSD
For each OS you support a number of their major versions.
Next step is that you provide binaries for all of them.
Yes indeed this will require quite some infrastructure investment and setting up of automatic building from your repositories (you do have them?).
The advantages is that your users only have 'one' thing to install and you can easily switch versions or even mixing versions. Actually you can even use different programming languages with this approach without affecting your release management too much.
Suppose I've developed a general-purpose end user utility written in Python. Previously, I had just one version available which was suitable for Python later than version 2.3 or so. It was sufficient to say, "download Python if you need to, then run this script". There was just one version of the script in source control (I'm using Git) to keep track of.
With Python 3, this is no longer necessarily true. For the foreseeable future, I will need to simultaneously develop two different versions, one suitable for Python 2.x and one suitable for Python 3.x. From a development perspective, I can think of a few options:
Maintain two different scripts in the same branch, making improvements to both simultaneously.
Maintain two separate branches, and merge common changes back and forth as development proceeds.
Maintain just one version of the script, plus check in a patch file that converts the script from one version to the other. When enough changes have been made that the patch no longer applies cleanly, resolve the conflicts and create a new patch.
I am currently leaning toward option 3, as the first two would involve a lot of error-prone tedium. But option 3 seems messy and my source control system is supposed to be managing patches for me.
For distribution packaging, there are more options to choose from:
Offer two different download packages, one suitable for Python 2 and one suitable for Python 3 (the user will have to know to download the correct one for whatever version of Python they have).
Offer one download package, with two different scripts inside (and then the user has to know to run the correct one).
One download package with two version-specific scripts, and a small stub loader that can run in both Python versions, that runs the correct script for the Python version installed.
Again I am currently leaning toward option 3 here, although I haven't tried to develop such a stub loader yet.
Any other ideas?
Edit: my original answer was based on the state of 2009, with Python 2.6 and 3.0 as the current versions. Now, with Python 2.7 and 3.3, there are other options. In particular, it is now quite feasible to use a single code base for Python 2 and Python 3.
See Porting Python 2 Code to Python 3
Original answer:
The official recommendation says:
For porting existing Python 2.5 or 2.6
source code to Python 3.0, the best
strategy is the following:
(Prerequisite:) Start with excellent test coverage.
Port to Python 2.6. This should be no more work than the average port
from Python 2.x to Python 2.(x+1).
Make sure all your tests pass.
(Still using 2.6:) Turn on the -3 command line switch. This enables
warnings about features that will be
removed (or change) in 3.0. Run your
test suite again, and fix code that
you get warnings about until there are
no warnings left, and all your tests
still pass.
Run the 2to3 source-to-source translator over your source code tree.
(See 2to3 - Automated Python 2 to 3
code translation for more on this
tool.) Run the result of the
translation under Python 3.0. Manually
fix up any remaining issues, fixing
problems until all tests pass again.
It is not recommended to try to write
source code that runs unchanged under
both Python 2.6 and 3.0; you’d have to
use a very contorted coding style,
e.g. avoiding print statements,
metaclasses, and much more. If you are
maintaining a library that needs to
support both Python 2.6 and Python
3.0, the best approach is to modify step 3 above by editing the 2.6
version of the source code and running
the 2to3 translator again, rather than
editing the 3.0 version of the source
code.
Ideally, you would end up with a single version, that is 2.6 compatible and can be translated to 3.0 using 2to3. In practice, you might not be able to achieve this goal completely. So you might need some manual modifications to get it to work under 3.0.
I would maintain these modifications in a branch, like your option 2. However, rather than maintaining the final 3.0-compatible version in this branch, I would consider to apply the manual modifications before the 2to3 translations, and put this modified 2.6 code into your branch. The advantage of this method would be that the difference between this branch and the 2.6 trunk would be rather small, and would only consist of manual changes, not the changes made by 2to3. This way, the separate branches should be easier to maintain and merge, and you should be able to benefit from future improvements in 2to3.
Alternatively, take a bit of a "wait and see" approach. Proceed with your porting only so far as you can go with a single 2.6 version plus 2to3 translation, and postpone the remaining manual modification until you really need a 3.0 version. Maybe by this time, you don't need any manual tweaks anymore...
For developement, option 3 is too cumbersome. Maintaining two branches is the easiest way although the way to do that will vary between VCSes. Many DVCS will be happier with separate repos (with a common ancestry to help merging) and centralized VCS will probably easier to work with with two branches. Option 1 is possible but you may miss something to merge and a bit more error-prone IMO.
For distribution, I'd use option 3 as well if possible. All 3 options are valid anyway and I have seen variations on these models from times to times.
I don't think I'd take this path at all. It's painful whichever way you look at it. Really, unless there's strong commercial interest in keeping both versions simultaneously, this is more headache than gain.
I think it makes more sense to just keep developing for 2.x for now, at least for a few months, up to a year. At some point in time it will be just time to declare on a final, stable version for 2.x and develop the next ones for 3.x+
For example, I won't switch to 3.x until some of the major frameworks go that way: PyQt, matplotlib, numpy, and some others. And I don't really mind if at some point they stop 2.x support and just start developing for 3.x, because I'll know that in a short time I'll be able to switch to 3.x too.
I would start by migrating to 2.6, which is very close to python 3.0. You might even want to wait for 2.7, which will be even closer to python 3.0.
And then, once you have migrated to 2.6 (or 2.7), I suggest you simply keep just one version of the script, with things like "if PY3K:... else:..." in the rare places where it will be mandatory. Of course it's not the kind of code we developers like to write, but then you don't have to worry about managing multiple scripts or branches or patches or distributions, which will be a nightmare.
Whatever you choose, make sure you have thorough tests with 100% code coverage.
Good luck!
Whichever option for development is chosen, most potential issues could be alleviated with thorough unit testing to ensure that the two versions produce matching output. That said, option 2 seems most natural to me: applying changes from one source tree to another source tree is a task (most) version control systems were designed for--why not take advantages of the tools they provide to ease this.
For development, it is difficult to say without 'knowing your audience'. Power Python users would probably appreciate not having to download two copies of your software yet for a more general user-base it should probably 'just work'.