Develop Reference

Python 2.7.X to 2.7.Y compatibility

I want to develop and test my project on the up-to-date version of Python 2.7 (say 2.7.18), but I want my project to be still fully usable on earlier versions of 2.7 (say 2.7.7). Setting up many variants of 2.7 locally or/and on CI for testing can be redundant.
So there are the following questions about compatibility of 2.7.X.
Can there be any changes in syntax which make code not working?
Can there be any changes in available standard imports, for example, can some imports from __future__ be unavailable in earlier versions?
Since I have to distribute compiled Python files (.pyc, compiled via py_compile module), I'm also wondering if there can be any changes in Python bytecode which block code execution in earlier versions.
I guess if all the answers are "no", I can develop and test my project only on a single 2.7 version without worries.
I've tried to search it but there is no success. Please share your experience and/or links.
UPD 1: I should have clearly said from the beginning that it's not my desire to use 2.7, it's a requirement from the environment.

At least Python 2.7.9 introduced massive changes to the 'ssl' module, so trying to use code using SSL for 2.7.18 on Python older than 2.7.9 will fail. So a clear "yes" to number 2.
In general compatbility for most projects works the other way round, use the oldest version you need to support and work upwards from old to new, not downwards from newer to older. I do not know of any software project that makes the guarantees in the other direction.
Note that Python 2.7 dropped out of support with 2.7.18, so unless you use a compatible version like PyPy (https://www.pypy.org/) your freshly developed project will run on outdated Python versions from the start.
If you want to provide a shrink wrapped product, maybe have a look at the usual solution for this like pyinstaller (https://www.pyinstaller.org/) or freeze (https://wiki.python.org/moin/Freeze)
The #3 may work, if you study the list of bytecode opcodes which do not change that much over time (https://github.com/python/cpython/commits/2.7/Include/opcode.h) but no idea if the on-disk format changed.

For Python2 to Python3 code conversion, Which version of Python & Django best suited?

Currently I am working in big firm where we need to convert python2 old big Django project into python3 version so I have done lots of research related but still not able to find any perfect answer related to which version of Python and Django best suited for conversion.
Currently I am using Python : 2.7.16 & Django : 1.9.13 in my old version.
Anyone can suggest me best suited version of Python & Django for above old version for python2 to python3 conversion.

I thought I'd add a bit to the strategy advocated by Wim's answer - get the appropriate version of Django working on both 2.7 and 3.x first - and outline some tactics that worked for me.
Python 2.7 is your escape pod, until you pull the trigger on 3.x
your tests should run on both
don't use any 3.x specific features, like f-strings
first Python 3.x, then only later Django 2.x which doesn't run on 2.7
start early, don't over analyze, but avoid the big bang approach
file by file at first.
start with the lowest level code, like utility libraries, that you have test suites for.
if possible, try to gradually merge your changes to the 2.7 production branches and keep your 3.x porting code up to date with prod changes.
Which minor version of Django to start with?
My criteria here is that Django migrations can be fairly involved (and actually require more thinking than 2=>3 work). So I would move to the latest and greatest 1.11 that way you're already providing some value to your 2.7 users. There's probably a good number of pre-2.x compatibility shims on 1.11 and you'll be getting its 2.x deprecation warnings.
Which minor version of Python 3.x to start with?
Best to consider all angles, such as the availability of your 3rd party libs, support from your CI/devops suite and availability on your chosen server OS images. You could always install 3.8 and try a pip install of your requirements.txt by itself, for example.
Leverage git (or whatever scm you use) and virtualenv.
separate requirement.txt files, but...
if you have a file-based, git repo, you can point each venv at the same codeline with a pip install -e <your directory>. that means that, in 2 different terminals you can run 2.7 and 3.x against the same unittest(s).
you could even run 2.7 and 3.x Django servers side-by-side on different ports and point say Firefox and Chrome at them.
commit often (on the porting branch at least) and learn about git bisect.
make use of 2to3
Yes, it will break 2.7 code and Django if you let it. So...
run it in preview mode or against a single file. see what it breaks but also see what it did right.
throttle it to only certain conversions that don't break 2.7 or Django. print x=> print (x) and except(Exception) as e are 2 no-brainers.
This is what my throttled command looked like:
2to3 $tgt -w -f except -f raise -f next -f funcattrs -f print
run it file-by-file until you are really confident.
use sed or awk rather than your editor for bulk conversions.
The advantage is that, as you become more aware of your apps' specifics concerns, you can build a suite of changes that can be run on either 1 file or many files and do most of the work without breaking 2.7 or Django. Apply this after your suitably-throttled 2to3 pass. That leaves you with residual cleanups in your editor and getting your tests to pass.
(optional) start running black on 2.7 code.
black which is a code formatter, uses Python 3 ASTs to run its analysis. It doesn't try to run the code, but it will flag syntax errors that prevent it from getting to the AST stage. You will have to work some pip install global magic to get there though and you have to buy into black's usefulness.
Other people have done it - learn from them.
Listening to #155 Practical steps for moving to Python 3 should give you some ideas of the work. Look at the show links for it. They love to talk up the Instagram(?) move which involved a gradual adjustment of running 2.7 code to 3.x syntax on a common codebase, and on the same git branch, until pull-the-trigger day.
See also The Conservative Python 3 Porting Guide
and Instagram Makes a Smooth Move to Python 3 - The New Stack
Conclusion
Your time to Django 1.11 EOL (April 2020) is rather short, so if you have 2+ dev resources to throw at it, I'd consider doing the following in parallel:
DEV#1: start out on a Django 1.11 bump (the theory being that Django 1.11 is probably best positioned as a jump off point to Django 2.x), using 2.7.
DEV#2: get started on Python 3.6/3.7 of your non-Django utility code. Since the code is 2.7 compatible at this point, merge it into #1 as you go.
See how both tasks proceed, assess what the Django related project risk is and what the Python 3 pain looks like. You're already missing the Python 2.7 EOL, but an obsolete web framework is probably more dangerous than legacy Python 2.7, at least for a few months. So I wouldn't wait too long to start migrating off Django 1.9 and your work doing so won't be wasted. As you see the progress, you'll start seeing the project risks better.
Your initial 2to3 progress will be slow, but the tooling and guidance is good enough that you'll quickly pick up speed so don't overthink it before starting to gather experience. The Django side depends on your exposure to breaking changes in the framework which is why I think it's best to start early.
P.S. (controversial/personal opinion) I didn't use six or other canned 2-to-3 bridge libraries much.
It's not because I don't trust it - it's brilliant for 3rd party libs - but rather that I didn't want to add a complex permanent dependency (and I was too lazy to read its doc). I'd been writing 2.7 code in 3.x compatible syntax for a long time so I didn't really feel the need to use them. Your mileage may vary and don't set out on this path if it seems like a lot of work.
Instead, I created a py223.py (57 LOC incl. comments) with this type of content, most of which is concerned with workarounds for deprecations and name changes in the standard library.
try:
basestring_ = basestring
except (NameError,) as e:
basestring_ = str
try:
cmp_ = cmp
except (NameError,) as e:
# from http://portingguide.readthedocs.io/en/latest/comparisons.html
def cmp_(x, y):
"""
Replacement for built-in function cmp that was removed in Python 3
"""
return (x > y) - (x < y)
Then import from that py223 to work around those specific concerns. Later on I will just ditch the import and move those weird isinstance(x, basestr_) to isinstance(x, str) but I know in advance there is little to worry about.

My suggestion is to first upgrade to Django==1.11.26, which is the most recent version of Django that is supporting both Python 2 and Python 3. Stay on your current version of Python 2.7 for now.
Read carefully the release notes for 1.10.x and 1.11.x, checking for deprecations and fixing anything that stopped working from your 1.9.x code. Things WILL break. Django moves fast. For a large Django project, there may be many code changes required, and if you're using a lot of 3rd-party plugins or libraries you may have to juggle their versions around. Some of your 3rd-party dependencies will probably have been abandoned entirely, so you have to find replacements or remove the features.
To find the release notes for each version upgrade, just google "What's new in Django ". The hits will meticulously document all the deprecations and changes:
https://docs.djangoproject.com/en/2.2/releases/1.10/
https://docs.djangoproject.com/en/2.2/releases/1.11/
Once the webapp appears to be working fine on Django 1.11, with all tests passing (you do have a test suite, right?) then you can do the Python 3 conversion, whilst keeping the Django version the same. Django 1.11 supports up to Python 3.7, so that would be a good version to target. Expect unicode all over the place, since the implicit conversions between bytes and text is gone now and many Python 2 webapps relied upon that.
Once the project appears to be working fine on Django 1.11 and Python 3.7, then you can think about upgrading to Django 3.0, following the same process as before - reading the release notes, making the necessary changes, running the test suite, and checking out the webapp in a dev server manually.

I would upgrade to py3 first. You'll need to look at setup.py in the Django repo on the stable/1.9.x branch (https://github.com/django/django/blob/stable/1.9.x/setup.py) to figure out that the py3 versions supported are 3.4 (dead) and 3.5.
Once you're on py3.5 and Django 1.9 you can upgrade one at a time until you get to the version you want to end at. E.g. Django 1.11 supports py3.5 and py3.7, so
py27/dj19 -> py35/dj19 -> py35/dj1.11 -> py37/dj1.11 ... -> py37/dj2.2
dj2.2 is the first version supporting py3.8, but I would probably stop at py37/dj2.2 if you're working in a normally conservative environment.
If you have other packages you'll need to find version combinations that will work together on each step. Having a plan is key, and upgrading only one component at a time will usually end up saving you time.
The future library (https://python-future.org/) will help you with many icky situations while you need code to run on both py27 and 3.x. six is great too. I would avoid rolling your own compatibility layer (why reinvent the wheel?)
If at all possible, try to get your unit test coverage up to 75-85% before starting, and definitely set up automatic testing on both "from" and "to" versions for each upgrade step. Make sure you read and fix all warnings from Django before upgrading to the next version -- Django cares very little about backward compatibility, so I would normally suggest hitting every minor version on the upgrade path (or at least make sure you read the "backwards incompatibilities" and deprecation lists for each minor version).
Good luck (we're upgrading a 300+Kloc code base from py27/dj1.7 right now, so I feel your pain ;-)

I have same kind of issue with my project and I have tried python 3.7.5 with Django version 2.2.7.
You should not go with python latest version 3.8 or Django latest version 3.0 because you there may have been chances that for any kind of bug you may not able to get proper solution for latest versions.

You should try to shoot for the current versions. Python 3.8 and Django 3.0.The Six library will help with some convention changes. Either way you are going to have to do some refactoring so you might as well make it current.

Web page building with python 3.3?

I am going through a tutorial on building a website with django. It suggests using mod_python but I have heard to stay away from that and use wsgi instead. Problem is I am running python 3.3 (and apache 2.4.2 for that matter, everything seems to be compatible with apache 2.2). Is there any way to get all of this working on 3.3? Or is my best bet to go back to python 2.7? Thanks.
Edit: I am on Windows, so that seems to be another roadblock.

You could use nginx + uwsgi to depoly your django site instead of Apache+mod_wsgi. Here's a tutorial.
As many tutorials is about how to configure the environment in Unix-like environment, you could use cgywin to simulate a Unix-like environment on Windows.
The version of Python you use is not much critical when you develop a site using Django except that you have to use some libraries that don't support Python-3.x.

It will be hard, between python2 and python3 there is a lot of incompatibility, and somehow the developers of the most python frameworks somehow won't understand, why they should port their software to the newer version of the language.
The simplest way if you use python 2.
The best way were to start an independent, python 3 fork of your most loved python framework.
EDIT: newer django supports python3, thus it should work.

Is it possible, and/or advisable to develop Django web applications on OS X (10.6.4 and 10.5.8) using Python 2.6.5 64-bit? Why?

I'm trying to decide on which architecture to choose for developing Django 1.0.x through Django 1.2.1. I've managed to get MySQL, MySQLdb, PIL, and Python 2.65 installed on Snow Leopard using x86 64-bit builds, but I'm curious as to whether or not there is a definitive answer to this question at the moment, and if so, why?
Thank you!
Michaux

Of course it's possible. Advisable? You didn't mention httpd and mod_wsgi, or some other WSGI container. Get one installed and it should be fine.

It certainly is possible: I do it every day.
Some tips:
use virtualenv to sandbox your python packages between projects.
use mod_passenger (via Passenger.prefpane) to make VirtualHosts easier to deal with.
You may need to fiddle a bit harder with things if you run stuff under mod_python, as I recall having to work hard to get a version compiled that worked with the version of apache that is installed by default, and the python I was using.

What version of Python (2.4, 2.5, 2.6, 3.0) do you standardize on for production development efforts (and why)?

In our group we primarily do search engine architecture and content integration work and most of that code base is in Python. All our build tools and Python module dependencies are in source control so they can be checked out and the environment loaded for use regardless of os/platform, kinda similar to the approach virtualenv uses.
For years we've maintained a code base compatible with Python 2.3 because one of the commercial products we use depends on Python 2.3. Over the years this has caused more and more issues as newer tools and libraries need newer versions of Python since 2.3 came out in ~2004.
We've recently decoupled our build environment from dependencies on the commercial product's environment and can use any version of Python (or Java) we want. Its been about a month or so since we standardized on Python 2.6 as the newest version of Python that is backwards compatible with previous versions.
Python 3.0 is not an option (for now) since we'd have to migrate too much of our code base to make our build and integration tools to work correctly again.
We like many of the new features of Python 2.6, especially the improved modules and things like class decorators, but many modules we depend on cause the Python 2.6 interpreter to spout various depreciation warnings. Another tool we're interested in for managing EC2 cloud cluster nodes, Supervisor doesn't even work correctly with Python 2.6.
Now I am wondering if we should standardize on Python 2.5 for now instead of using Python 2.6 in development of production environment tools. Most of the tools we want/need seem to work correctly with Python 2.5. We're trying to sort this out now before there are many dependencies on Python 2.6 features or modules.
Many Thanks!
-Michael

I wouldn't abandon 2.6 just because of deprecation warnings; those will disappear over time. (You can use the -W ignore option to the Python interpreter to prevent them from being printed out, at least) But if modules you need to use actually don't work with Python 2.6, that would be a legitimate reason to stay with 2.5. Python 2.5 is in wide use now and probably will be for a long time to come (consider how long 2.3 has lasted!), so even if you go with 2.5, you won't be forced to upgrade for a while.
I use Python 2.5 for all my development work, but only because it's the version that happens to be available in Gentoo (Linux)'s package repository. When the Gentoo maintainers declare Python 2.6 "stable"*, I'll switch to that. Of course, this reasoning wouldn't necessarily apply to you.
* Python 2.6 actually is stable, the reason it's not declared as such in Gentoo is that Gentoo relies on other programs which themselves depend on Python and are not yet upgraded to work with 2.6. Again, this reasoning probably doesn't apply to you.

My company is standardized in 2.5. Like you we can't make the switch to 3.0 for a million reasons, but I very much wish we could move up to 2.6.
Doing coding day to day I'll be looking through the documentation and I'll find exactly the module or function that I want, but then it'll have the little annotation: New in Version 2.6
I would say go with the newest version, and if you have depreciation warnings pop up (there will probably be very few) then just go in a find a better way to do it. Overall your code will be better with 2.6.

To me the most important to stick with python 2.5+ is because it officially supports ctypes, which changed many plugin systems.
Although you can find ctypes to work with 2.3/2.4, they are not officially bundled.
So my suggestion would be 2.5.

We're sticking with 2.5.2 for now. Our tech stack centers on Django (but we have a dozen other bits and bobs.) So we stay close to what they do.
We had to go back to docutils to 0.4 so it would work with epydoc 3.0.1. So far, this hasn't been a big issue, but it may -- at some point -- cause us to rethink our use of epydoc.
The 2.6 upgrade is part of our development plan. We have budget, but not fixed schedule right now.
The 3.0 upgrade, similarly, is something I remind folks of. We have to budget for it. We won't do it this year unless Django leaps to 3.0. We might do it next year.

I think the best solution is to give up the hope on total uniformity, although having a common environment is something to strive for. You will always will be confronted with version problems, for example when upgrading to the next best interpreter version.
So instead of dealing with it on a per issue base you could solve this problem by taking a good look on your release management.
Instead of releasing source, go for platform depending binaries (besides the source distribution).
So what you do is that you define a number of supported CPU's, for example:
x86-32, x86-64, sparc
Then which operating systems:
Linux, Windows, Solaris, FreeBSD
For each OS you support a number of their major versions.
Next step is that you provide binaries for all of them.
Yes indeed this will require quite some infrastructure investment and setting up of automatic building from your repositories (you do have them?).
The advantages is that your users only have 'one' thing to install and you can easily switch versions or even mixing versions. Actually you can even use different programming languages with this approach without affecting your release management too much.