Develop Reference

For Python2 to Python3 code conversion, Which version of Python & Django best suited?

Currently I am working in big firm where we need to convert python2 old big Django project into python3 version so I have done lots of research related but still not able to find any perfect answer related to which version of Python and Django best suited for conversion.
Currently I am using Python : 2.7.16 & Django : 1.9.13 in my old version.
Anyone can suggest me best suited version of Python & Django for above old version for python2 to python3 conversion.

I thought I'd add a bit to the strategy advocated by Wim's answer - get the appropriate version of Django working on both 2.7 and 3.x first - and outline some tactics that worked for me.
Python 2.7 is your escape pod, until you pull the trigger on 3.x
your tests should run on both
don't use any 3.x specific features, like f-strings
first Python 3.x, then only later Django 2.x which doesn't run on 2.7
start early, don't over analyze, but avoid the big bang approach
file by file at first.
start with the lowest level code, like utility libraries, that you have test suites for.
if possible, try to gradually merge your changes to the 2.7 production branches and keep your 3.x porting code up to date with prod changes.
Which minor version of Django to start with?
My criteria here is that Django migrations can be fairly involved (and actually require more thinking than 2=>3 work). So I would move to the latest and greatest 1.11 that way you're already providing some value to your 2.7 users. There's probably a good number of pre-2.x compatibility shims on 1.11 and you'll be getting its 2.x deprecation warnings.
Which minor version of Python 3.x to start with?
Best to consider all angles, such as the availability of your 3rd party libs, support from your CI/devops suite and availability on your chosen server OS images. You could always install 3.8 and try a pip install of your requirements.txt by itself, for example.
Leverage git (or whatever scm you use) and virtualenv.
separate requirement.txt files, but...
if you have a file-based, git repo, you can point each venv at the same codeline with a pip install -e <your directory>. that means that, in 2 different terminals you can run 2.7 and 3.x against the same unittest(s).
you could even run 2.7 and 3.x Django servers side-by-side on different ports and point say Firefox and Chrome at them.
commit often (on the porting branch at least) and learn about git bisect.
make use of 2to3
Yes, it will break 2.7 code and Django if you let it. So...
run it in preview mode or against a single file. see what it breaks but also see what it did right.
throttle it to only certain conversions that don't break 2.7 or Django. print x=> print (x) and except(Exception) as e are 2 no-brainers.
This is what my throttled command looked like:
2to3 $tgt -w -f except -f raise -f next -f funcattrs -f print
run it file-by-file until you are really confident.
use sed or awk rather than your editor for bulk conversions.
The advantage is that, as you become more aware of your apps' specifics concerns, you can build a suite of changes that can be run on either 1 file or many files and do most of the work without breaking 2.7 or Django. Apply this after your suitably-throttled 2to3 pass. That leaves you with residual cleanups in your editor and getting your tests to pass.
(optional) start running black on 2.7 code.
black which is a code formatter, uses Python 3 ASTs to run its analysis. It doesn't try to run the code, but it will flag syntax errors that prevent it from getting to the AST stage. You will have to work some pip install global magic to get there though and you have to buy into black's usefulness.
Other people have done it - learn from them.
Listening to #155 Practical steps for moving to Python 3 should give you some ideas of the work. Look at the show links for it. They love to talk up the Instagram(?) move which involved a gradual adjustment of running 2.7 code to 3.x syntax on a common codebase, and on the same git branch, until pull-the-trigger day.
See also The Conservative Python 3 Porting Guide
and Instagram Makes a Smooth Move to Python 3 - The New Stack
Conclusion
Your time to Django 1.11 EOL (April 2020) is rather short, so if you have 2+ dev resources to throw at it, I'd consider doing the following in parallel:
DEV#1: start out on a Django 1.11 bump (the theory being that Django 1.11 is probably best positioned as a jump off point to Django 2.x), using 2.7.
DEV#2: get started on Python 3.6/3.7 of your non-Django utility code. Since the code is 2.7 compatible at this point, merge it into #1 as you go.
See how both tasks proceed, assess what the Django related project risk is and what the Python 3 pain looks like. You're already missing the Python 2.7 EOL, but an obsolete web framework is probably more dangerous than legacy Python 2.7, at least for a few months. So I wouldn't wait too long to start migrating off Django 1.9 and your work doing so won't be wasted. As you see the progress, you'll start seeing the project risks better.
Your initial 2to3 progress will be slow, but the tooling and guidance is good enough that you'll quickly pick up speed so don't overthink it before starting to gather experience. The Django side depends on your exposure to breaking changes in the framework which is why I think it's best to start early.
P.S. (controversial/personal opinion) I didn't use six or other canned 2-to-3 bridge libraries much.
It's not because I don't trust it - it's brilliant for 3rd party libs - but rather that I didn't want to add a complex permanent dependency (and I was too lazy to read its doc). I'd been writing 2.7 code in 3.x compatible syntax for a long time so I didn't really feel the need to use them. Your mileage may vary and don't set out on this path if it seems like a lot of work.
Instead, I created a py223.py (57 LOC incl. comments) with this type of content, most of which is concerned with workarounds for deprecations and name changes in the standard library.
try:
basestring_ = basestring
except (NameError,) as e:
basestring_ = str
try:
cmp_ = cmp
except (NameError,) as e:
# from http://portingguide.readthedocs.io/en/latest/comparisons.html
def cmp_(x, y):
"""
Replacement for built-in function cmp that was removed in Python 3
"""
return (x > y) - (x < y)
Then import from that py223 to work around those specific concerns. Later on I will just ditch the import and move those weird isinstance(x, basestr_) to isinstance(x, str) but I know in advance there is little to worry about.

My suggestion is to first upgrade to Django==1.11.26, which is the most recent version of Django that is supporting both Python 2 and Python 3. Stay on your current version of Python 2.7 for now.
Read carefully the release notes for 1.10.x and 1.11.x, checking for deprecations and fixing anything that stopped working from your 1.9.x code. Things WILL break. Django moves fast. For a large Django project, there may be many code changes required, and if you're using a lot of 3rd-party plugins or libraries you may have to juggle their versions around. Some of your 3rd-party dependencies will probably have been abandoned entirely, so you have to find replacements or remove the features.
To find the release notes for each version upgrade, just google "What's new in Django ". The hits will meticulously document all the deprecations and changes:
https://docs.djangoproject.com/en/2.2/releases/1.10/
https://docs.djangoproject.com/en/2.2/releases/1.11/
Once the webapp appears to be working fine on Django 1.11, with all tests passing (you do have a test suite, right?) then you can do the Python 3 conversion, whilst keeping the Django version the same. Django 1.11 supports up to Python 3.7, so that would be a good version to target. Expect unicode all over the place, since the implicit conversions between bytes and text is gone now and many Python 2 webapps relied upon that.
Once the project appears to be working fine on Django 1.11 and Python 3.7, then you can think about upgrading to Django 3.0, following the same process as before - reading the release notes, making the necessary changes, running the test suite, and checking out the webapp in a dev server manually.

I would upgrade to py3 first. You'll need to look at setup.py in the Django repo on the stable/1.9.x branch (https://github.com/django/django/blob/stable/1.9.x/setup.py) to figure out that the py3 versions supported are 3.4 (dead) and 3.5.
Once you're on py3.5 and Django 1.9 you can upgrade one at a time until you get to the version you want to end at. E.g. Django 1.11 supports py3.5 and py3.7, so
py27/dj19 -> py35/dj19 -> py35/dj1.11 -> py37/dj1.11 ... -> py37/dj2.2
dj2.2 is the first version supporting py3.8, but I would probably stop at py37/dj2.2 if you're working in a normally conservative environment.
If you have other packages you'll need to find version combinations that will work together on each step. Having a plan is key, and upgrading only one component at a time will usually end up saving you time.
The future library (https://python-future.org/) will help you with many icky situations while you need code to run on both py27 and 3.x. six is great too. I would avoid rolling your own compatibility layer (why reinvent the wheel?)
If at all possible, try to get your unit test coverage up to 75-85% before starting, and definitely set up automatic testing on both "from" and "to" versions for each upgrade step. Make sure you read and fix all warnings from Django before upgrading to the next version -- Django cares very little about backward compatibility, so I would normally suggest hitting every minor version on the upgrade path (or at least make sure you read the "backwards incompatibilities" and deprecation lists for each minor version).
Good luck (we're upgrading a 300+Kloc code base from py27/dj1.7 right now, so I feel your pain ;-)

I have same kind of issue with my project and I have tried python 3.7.5 with Django version 2.2.7.
You should not go with python latest version 3.8 or Django latest version 3.0 because you there may have been chances that for any kind of bug you may not able to get proper solution for latest versions.

You should try to shoot for the current versions. Python 3.8 and Django 3.0.The Six library will help with some convention changes. Either way you are going to have to do some refactoring so you might as well make it current.

Upgrade Python version while running a Python program

I would like to upgrade my Python version from 2.6.6 to the newest version of 3.6.5 on a server. However, there is a program that has been running for multiple days and will be running for a week or more.
Will my program continue to run until it finishes if I upgrade my Python version?
P.S.: I ran my Python program with the screen command so I wouldn't have to stop it to be able to upgrade my Python version.

Yes, your Python 2 script should continue to run while you install/upgrade Python 3, since Python 2 and 3 are different programs.
However, the installation will be be platform specific and there might be common dependencies, so I would test this on a different machine (e.g. local virtual machine) first.

Several aspects are to be considered here:
In general, a Python program that is running does not access the installed binaries of the interpreter anymore. Also all the modules already loaded will not be read again from disk during the execution time of this program. (In this aspect it differs from e. g. a Bash script which reads its script source while executing.) So you probably are safe in even removing the complete Python installation; of course that's not what you plan on and I wouldn't recommend it for other reasons, but even that should not influence your running Python program. If, however, your running Python program tries to import a module for the first time after the removal or upgrade of your Python installation, this might fail due to incompatibilities between the versions. Most programs do their imports only at the beginning of their runtime, though.
Python 2 and 3 are different languages. Scripts for the one need to be at least adjusted to run with the other. Adjustments aren't too hard to do, though. Most distributions have both installed side-by-side. Most of the time, Python2 is still standard (and accessed by typing python). python2 and python3 are then to make it explicit. This means that you probably should not remove Python 2 from your system, even if you plan on installing Python 3. There probably are scripts relying on the installation of Python 2.

There's a lot going on here that you've not explained/described, so it's hard to give you a definitive answer (such as the operating system you're using). This is also not exactly a programming question, so you might find that the question gets voted to close. That said...
It possible to run multiple versions of Python side by side. Because 2.x and 3.x are very different, it might be much better to have both versions installed. See this answer for some details on doing that:
Official multiple Python versions on the same machine?
It's likely that your code will need some modifications to run on Python 3, so you're going to have to redeploy a new version of your code at some point anyway.

How do I make sure my Python 3.6 code is compatible with other minor versions?

If I'm writing a python package in python 3.6, then how do I ensure my code can be downloaded and ran in other python 3 environments, like python 3.5?
What makes some python packages (e.g. Tensorflow) compatible with all python 3 minor versions, while other python packages (e.g. OpenAI Gym) only compatible with Python 3.5?
Finally: if my goal is to write code that is compatible for Python 3.5 and 3.6, then would it be better to just use a python 3.5 environment?

The glib-but-true answer:
Test your 3.6 code with those other versions of Python.
If you want to see if something runs correctly in environment X, there's no substitute for actually running it in environment X.
Tox
is a Python testing framework designed to do exactly this on your workstation.
Something like this is often part of a larger
continuous integration
framework or service, which might be hosted on a remote server.
Writing your code in the earliest syntax you need to support is always a good idea, but it's not enough.
You still need to test later versions, because functions or classes can change, and even bug-fixes can break your code if you were unwittingly depending on their behavior.
As for why some packages don't work under a specific minor version, the most likely reason is that they use a Python language feature that was introduced later.
(Possibly it's one of their dependencies that requires the language feature.)
This can include language syntax changes like
Python 3.5's
# matrix-multiplication operator,
all the way down to seeming trivia like
Python 3.1's
printing commas as thousands-separators,
which is still enough to raise an exception.
It's actually a bit more complicated than just
"supports version x or greater",
because there are some gaps in Python's history.
The most likely headache is the u'' Unicode literal syntax from Python 2.
It was removed in Python 3.0...
and then
restored in Python 3.3,
after its absence caused more grief than expected.
That one change means any script with a u'Unicode literal' could work under Python 2.7 and 3.3+ while not working under Python 3.0, 3.1, or 3.2.
The Python documentation is very good about keeping track of when a feature was introduced.
For instance, the first thing the
typing module
tells you is:
26.1. typing - Support for type hints
New in version 3.5.
A Ctrl-F style search for "new in" or "changed in" will generally turn up all the most important changes.

Managing Python 3 code with SCons

at work I have the task to convert a large library with Python 2.7 Code to Python 3.x.
This library contains a lot of scripts and extensions made with boost python for C++.
All of this is built with SCons which does not work with a Python 3.x interpreter, but now me and my supervisor want to know if there is a way around this.
The SConstruct file contains expressions with sys.version to determine the correct module-directories to import (numpy etc.). I do not know how to use SCons or the syntax, so I can not give a lot of information about this topic.
Can we use SCons to build Python 3 Code with the given extensions or do we have to wait until SCons is compatible with Python 3?

At the time of writing this, there are plans to support both Python 2.7 and 3.x in a single branch/version. Work on this feature has started, but it will take some more time to reach this goal.
So it looks as if your best bet would be to start right away. SCons itself should run fine under Python 2.7 for compiling the Boost extensions. The problem in your case are the added checks and detection mechanisms for deriving paths and module names from the version of the current Python interpreter.
Since you can't give any more detail about this process, my answer is somewhat vague here, sorry. In principle you'd have to find the place in the SConstructs/SConscripts where the version of the currently running Python interpreter is determined. Just hardcode this to the 3.x version that you have installed on the machine additionally, and keep your fingers crossed that the rest will work automatically.
Note how there is a clear separation here between "compiling code for a Python version" vs "compiling code under a Python version".
In general, a better understanding of SCons internal workings and basic principles might be helpful. If you find the time, check out the UserGuide ( http://scons.org/doc/production/HTML/scons-user.html ) or consult our user mailing list ( see http://scons.org/lists.php ) for larger questions and discussions.

Upgrade from Python 2.2.1 to 3.4

We have got a legacy application that runs on Python 2.2.1. Now we need to upgrade to the latest version of Python (3.4).
I would like to know a few details on the upgrade:
Would that be a direct upgrade for all of the associated files? I mean what libraries that have been used in the older code have to be changed to make it work in latest version?
How do we upgrade the .py files from the older to the newer version?
Everywhere it talks about upgrading 2.6 to 3 and nowhere does it cover 2.2.1 to 3. Is there any direct upgrade possible?

To expand on my comments, the PyPorting docs suggest a seven-step process that I think would be useful here too:
Only worry about supporting Python 2.7 - it will be easier for you to upgrade to 3 if you first make sure your code runs in the latest 2.x branch. If you're lucky, this won't require many changes!
Make sure you have good test coverage - crucial for any major change. If you can't be sure it's working now, how will you be sure it's working after the upgrade?
Learn the differences between Python 2 & 3 - per cdarke's comment, you will probably have to do some manual intervention, so will need to know what's changed. In your case, this may also involve learning the differences between 2.x versions. You can use What's new in Python x.x? to help.
Use Modernize or Futurize to update your code - automated tools to make your code 3.x-ready (the documentation notes that you can use 2to3 if you don't want to retain 2.x compatibility).
Use Pylint to help make sure you don’t regress on your Python 3 support - pylint will give you lots of helpful warnings to help improve the code generally.
Use caniusepython3 to find out which of your dependencies are blocking your use of Python 3 - you ask about updates to libraries; this tool can tell you what's 3.x compatible. You may need to find compatible replacements for some dependencies; see PyPI.
Use continuous integration to make sure you stay compatible with Python 2 & 3 - whatever versions you want to support, good CI can ensure that you stay compatible with all of them as you modify the code.

I would recommend that you take a look at something like: 2to3
The idea is that you can just run the program on your old file, and convert it to 3.4 compatible code. Best of luck!