I've pretty big code base written on Django 1.5, it's about time to upgrade to the newest version (1.11.2) as many stuff don't work well.
I'm wondering if it's best to upgrade step by step: 1.5->1.6->1.7...
or just jump to 1.11.2
what method should be better and make the (hard) process easier? as my project has many dependencies?
Also what are good practices to do? I'm using virtualenv and aware of this Django article about upgrading
The document you found (“Upgrading Django to a newer version”) has a good guide.
An important part, before upgrading, is to have full branch coverage by the automated test suite for your application, before upgrading.
You want to be able to run a full automated test suite, see everything pass and know that all branches are exercised by the test suite.
This means that when you break something by porting to the new Django version, you'll be able to see which parts of your app are not behaving correctly any more.
Read about what changes you need to make by reading the release notes and deprecation timeline, for all of the relevant releases between your current and target Django versions.
See what dependencies you'll need to upgrade for Django; you might need to correct your code if it relies on incompatible features in an outdated third-party library.
All of those should be done before upgrading a single thing, in my opinion.
It depends on two things. How many people are using the Django app, and the extend of your test coverage.
If you are the only user, no worries, upgrade all the way. But if you have a lot of users you will quickly find out by upgrading all the way to 1.11.2 that some edge cases may not be covered by your tests.
Expect a lot of error 500 along the way.
If your coverage is close to 100% on all your apps, you may not have that problem.
Note that a lot has changed since 1.5.
I my case about: Data
I am using SQLite and i am sure that, the lastest version can kill you if you have only 1 conflict.
python manage.py makemigrations and so on had so many chance in it's method.
And if you change Python from 2x to 3x, it is a new case too =))
Related
Currently I am working in big firm where we need to convert python2 old big Django project into python3 version so I have done lots of research related but still not able to find any perfect answer related to which version of Python and Django best suited for conversion.
Currently I am using Python : 2.7.16 & Django : 1.9.13 in my old version.
Anyone can suggest me best suited version of Python & Django for above old version for python2 to python3 conversion.
I thought I'd add a bit to the strategy advocated by Wim's answer - get the appropriate version of Django working on both 2.7 and 3.x first - and outline some tactics that worked for me.
Python 2.7 is your escape pod, until you pull the trigger on 3.x
your tests should run on both
don't use any 3.x specific features, like f-strings
first Python 3.x, then only later Django 2.x which doesn't run on 2.7
start early, don't over analyze, but avoid the big bang approach
file by file at first.
start with the lowest level code, like utility libraries, that you have test suites for.
if possible, try to gradually merge your changes to the 2.7 production branches and keep your 3.x porting code up to date with prod changes.
Which minor version of Django to start with?
My criteria here is that Django migrations can be fairly involved (and actually require more thinking than 2=>3 work). So I would move to the latest and greatest 1.11 that way you're already providing some value to your 2.7 users. There's probably a good number of pre-2.x compatibility shims on 1.11 and you'll be getting its 2.x deprecation warnings.
Which minor version of Python 3.x to start with?
Best to consider all angles, such as the availability of your 3rd party libs, support from your CI/devops suite and availability on your chosen server OS images. You could always install 3.8 and try a pip install of your requirements.txt by itself, for example.
Leverage git (or whatever scm you use) and virtualenv.
separate requirement.txt files, but...
if you have a file-based, git repo, you can point each venv at the same codeline with a pip install -e <your directory>. that means that, in 2 different terminals you can run 2.7 and 3.x against the same unittest(s).
you could even run 2.7 and 3.x Django servers side-by-side on different ports and point say Firefox and Chrome at them.
commit often (on the porting branch at least) and learn about git bisect.
make use of 2to3
Yes, it will break 2.7 code and Django if you let it. So...
run it in preview mode or against a single file. see what it breaks but also see what it did right.
throttle it to only certain conversions that don't break 2.7 or Django. print x=> print (x) and except(Exception) as e are 2 no-brainers.
This is what my throttled command looked like:
2to3 $tgt -w -f except -f raise -f next -f funcattrs -f print
run it file-by-file until you are really confident.
use sed or awk rather than your editor for bulk conversions.
The advantage is that, as you become more aware of your apps' specifics concerns, you can build a suite of changes that can be run on either 1 file or many files and do most of the work without breaking 2.7 or Django. Apply this after your suitably-throttled 2to3 pass. That leaves you with residual cleanups in your editor and getting your tests to pass.
(optional) start running black on 2.7 code.
black which is a code formatter, uses Python 3 ASTs to run its analysis. It doesn't try to run the code, but it will flag syntax errors that prevent it from getting to the AST stage. You will have to work some pip install global magic to get there though and you have to buy into black's usefulness.
Other people have done it - learn from them.
Listening to #155 Practical steps for moving to Python 3 should give you some ideas of the work. Look at the show links for it. They love to talk up the Instagram(?) move which involved a gradual adjustment of running 2.7 code to 3.x syntax on a common codebase, and on the same git branch, until pull-the-trigger day.
See also The Conservative Python 3 Porting Guide
and Instagram Makes a Smooth Move to Python 3 - The New Stack
Conclusion
Your time to Django 1.11 EOL (April 2020) is rather short, so if you have 2+ dev resources to throw at it, I'd consider doing the following in parallel:
DEV#1: start out on a Django 1.11 bump (the theory being that Django 1.11 is probably best positioned as a jump off point to Django 2.x), using 2.7.
DEV#2: get started on Python 3.6/3.7 of your non-Django utility code. Since the code is 2.7 compatible at this point, merge it into #1 as you go.
See how both tasks proceed, assess what the Django related project risk is and what the Python 3 pain looks like. You're already missing the Python 2.7 EOL, but an obsolete web framework is probably more dangerous than legacy Python 2.7, at least for a few months. So I wouldn't wait too long to start migrating off Django 1.9 and your work doing so won't be wasted. As you see the progress, you'll start seeing the project risks better.
Your initial 2to3 progress will be slow, but the tooling and guidance is good enough that you'll quickly pick up speed so don't overthink it before starting to gather experience. The Django side depends on your exposure to breaking changes in the framework which is why I think it's best to start early.
P.S. (controversial/personal opinion) I didn't use six or other canned 2-to-3 bridge libraries much.
It's not because I don't trust it - it's brilliant for 3rd party libs - but rather that I didn't want to add a complex permanent dependency (and I was too lazy to read its doc). I'd been writing 2.7 code in 3.x compatible syntax for a long time so I didn't really feel the need to use them. Your mileage may vary and don't set out on this path if it seems like a lot of work.
Instead, I created a py223.py (57 LOC incl. comments) with this type of content, most of which is concerned with workarounds for deprecations and name changes in the standard library.
try:
basestring_ = basestring
except (NameError,) as e:
basestring_ = str
try:
cmp_ = cmp
except (NameError,) as e:
# from http://portingguide.readthedocs.io/en/latest/comparisons.html
def cmp_(x, y):
"""
Replacement for built-in function cmp that was removed in Python 3
"""
return (x > y) - (x < y)
Then import from that py223 to work around those specific concerns. Later on I will just ditch the import and move those weird isinstance(x, basestr_) to isinstance(x, str) but I know in advance there is little to worry about.
My suggestion is to first upgrade to Django==1.11.26, which is the most recent version of Django that is supporting both Python 2 and Python 3. Stay on your current version of Python 2.7 for now.
Read carefully the release notes for 1.10.x and 1.11.x, checking for deprecations and fixing anything that stopped working from your 1.9.x code. Things WILL break. Django moves fast. For a large Django project, there may be many code changes required, and if you're using a lot of 3rd-party plugins or libraries you may have to juggle their versions around. Some of your 3rd-party dependencies will probably have been abandoned entirely, so you have to find replacements or remove the features.
To find the release notes for each version upgrade, just google "What's new in Django ". The hits will meticulously document all the deprecations and changes:
https://docs.djangoproject.com/en/2.2/releases/1.10/
https://docs.djangoproject.com/en/2.2/releases/1.11/
Once the webapp appears to be working fine on Django 1.11, with all tests passing (you do have a test suite, right?) then you can do the Python 3 conversion, whilst keeping the Django version the same. Django 1.11 supports up to Python 3.7, so that would be a good version to target. Expect unicode all over the place, since the implicit conversions between bytes and text is gone now and many Python 2 webapps relied upon that.
Once the project appears to be working fine on Django 1.11 and Python 3.7, then you can think about upgrading to Django 3.0, following the same process as before - reading the release notes, making the necessary changes, running the test suite, and checking out the webapp in a dev server manually.
I would upgrade to py3 first. You'll need to look at setup.py in the Django repo on the stable/1.9.x branch (https://github.com/django/django/blob/stable/1.9.x/setup.py) to figure out that the py3 versions supported are 3.4 (dead) and 3.5.
Once you're on py3.5 and Django 1.9 you can upgrade one at a time until you get to the version you want to end at. E.g. Django 1.11 supports py3.5 and py3.7, so
py27/dj19 -> py35/dj19 -> py35/dj1.11 -> py37/dj1.11 ... -> py37/dj2.2
dj2.2 is the first version supporting py3.8, but I would probably stop at py37/dj2.2 if you're working in a normally conservative environment.
If you have other packages you'll need to find version combinations that will work together on each step. Having a plan is key, and upgrading only one component at a time will usually end up saving you time.
The future library (https://python-future.org/) will help you with many icky situations while you need code to run on both py27 and 3.x. six is great too. I would avoid rolling your own compatibility layer (why reinvent the wheel?)
If at all possible, try to get your unit test coverage up to 75-85% before starting, and definitely set up automatic testing on both "from" and "to" versions for each upgrade step. Make sure you read and fix all warnings from Django before upgrading to the next version -- Django cares very little about backward compatibility, so I would normally suggest hitting every minor version on the upgrade path (or at least make sure you read the "backwards incompatibilities" and deprecation lists for each minor version).
Good luck (we're upgrading a 300+Kloc code base from py27/dj1.7 right now, so I feel your pain ;-)
I have same kind of issue with my project and I have tried python 3.7.5 with Django version 2.2.7.
You should not go with python latest version 3.8 or Django latest version 3.0 because you there may have been chances that for any kind of bug you may not able to get proper solution for latest versions.
You should try to shoot for the current versions. Python 3.8 and Django 3.0.The Six library will help with some convention changes. Either way you are going to have to do some refactoring so you might as well make it current.
I've a project on Django that needs an upgradation. Should I proceed by making a new project on 1.6 and shifting one by one from Django 1.3 to 1.6 or there's some other easy method?
What the documentation recommends, do it in steps
create a new virtualenv, install the new django version in it with all required dependencies for your app. That way you keep it seperated from the old version
create and run a test suit. Make sure you run the test with the -Wall command to unsilence warnings
When you deploy, make sure to clear your cache
A more personal note:
From my experience, I also recommend you go to the release notes, go one version after the other, and scan the major changes. While some things are still supported, and will work for you, it doesn't mean that one of the newer version didn't present a better method to do the same thing.
One good example would be the render. It was presented as an alternative to render_to_response back in version 1.3, but it didn't catch on that fast. The render_to_response is still supported, and there isn't any deprecation coming. But it's still good habit to get with the times.
Another personal suggestion - recreate the apps with the new django version, and then copy the files. That's because the new versions use different templating for their settings.py and other files, and it might immediatly highlight to you some of the changes that were brought (such as ALLOWED_HOSTS which was presented in 1.5).
Good luck! Hope it wouldn't be too painful
p.s. - consider waiting a little while longer - Django 1.7 is coming soon, and it's expected to be the biggest release since Django 1.0. Though, on the other hand, upgrading from 1.6 to 1.7 will probably be easier than upgrading from 1.3 to 1.7. I'm not sure what's the best course here, but consider it.
As some features get deprecated with new versions of Django, is there a way to check for that on an existing project code say on github.
Could a tool do that. Is there a way to detect that through testcases.
Would it be possible to do the same against a python version.
I guess one way could be to run through a specific version of django / python using tox and then check for errors.
I am just looking for something more elegant or direct, something like which says - "Note this feature has been deprecated", something which can be done in strongly typed language like Java.
If one wanted to build such a tool, what could be starting point for that, if possible.
This is how I got tox to run one project of mine against Django 1.6, 1.7 and 1.8 with deprecation warnings on:
[tox]
envlist = {py27,py34}-{django16,django17,django18}
[testenv]
basepython =
py27: python2.7
py34: python3.4
deps =
django16: Django>=1.6,<1.7
django17: Django>=1.7,<1.8
django18: Django>=1.8,<1.9
commands =
python -Wmodule ./manage.py test
The -Wmodule argument causes Python to output each deprecation warning the first time it occurs in a module, which was good enough for me. I was able to deal with instances where I used from django.core.cache import get_cache, which will be gone in Django 1.9.
In cases where -Wmodule outputs too much, you might want to be more selective. Python's documentation gives the lowdown on how to use this argument. I've purposely not used -Werror because this would not just make the individual tests fail but would make my test suite fail to execute any test because the suite uses deprecated features.
I think this would have to be in the unit tests for your project.
If your tests exercise code that will be deprecated in a future version of Django you will get warnings. If you've jumped to a version of Django where the feature is already deprecated you'll get exceptions and failed tests of course.
You can tell the Python interpreter to promote warnings to exceptions, which would cause tests to fail.
Instructions here to apply the same trick to the popular nosetests test framework.
If you know already (from Django docs) that some code you're writing will need to change depending on Django version it is run under (eg you're distributing a reusable Django app) I would suggest a form of feature detection using try ... except
For example, here I wanted to conditionally use the new transaction.atomic feature from Django >= 1.6: .
As you anticipated, I then run the tests against different versions of Django with the help of Tox.
I'm currently testing frameworks to create a big multiplayers game. I choose Django.
But I have a question about the version of Python. I should to create that project from scratch with Python 3.x or Python 2.x?
Python 3.x and Django compatibly is ok, or not production usable for now?
I wouldn't highly suggest going production with Python 3 with Django or for that matter any other framework that requires you to depend on many third party applications, although many have been ported to Python 3, you still may find bugs, which you will likely have to fix or wait awhile before maintainers get to it.
Also, there aren't many compelling reasons to move to Python 3 at the moment, but that I suspect that will change soon with all the asynchronous work being put into Python 3.
Django is compatible with Python 3.
There's at least one issue with Python3 + Django (1.6) + MySQL.
MySQLdb hasn't been ported to Python3 yet.
The other python-only mysql connector mentioned in the Django documentation (MySQL Connector/Python) has a bug in it which might stop it working with fixtures.
http://bugs.mysql.com/bug.php?id=72001
It looks like Oracle has closed this bug report by documenting that the problem may occur; so a real fix doesn't look likely any time soon.
So if you rely on fixtures and use MySQL, you'll likely have a problem.
Try using the latest GA version of mysql-connector-python.
One issue that comes up during Pinax development is dealing with development versions of external apps. I am trying to come up with a solution that doesn't involve bringing in the version control systems. Reason being I'd rather not have to install all the possible version control systems on my system (or force that upon contributors) and deal the problems that might arise during environment creation.
Take this situation (knowing how Pinax works will be beneficial to understanding):
We are beginning development on a new version of Pinax. The previous version has a pip requirements file with explicit versions set. A bug comes in for an external app that we'd like to get resolved. To get that bug fix in Pinax the current process is to simply make a minor release of the app assuming we have control of the app. Apps we don't have control we just deal with the release cycle of the app author or force them to make releases ;-) I am not too fond of constantly making minor releases for bug fixes as in some cases I'd like to be working on new features for apps as well. Of course branching the older version is what we do and then do backports as we need.
I'd love to hear some thoughts on this.
Could you handle this using the "==dev" version specifier? If the distribution's page on PyPI includes a link to a .tgz of the current dev version (such as both github and bitbucket provide automatically) and you append "#egg=project_name-dev" to the link, both easy_install and pip will use that .tgz if ==dev is requested.
This doesn't allow you to pin to anything more specific than "most recent tip/head", but in a lot of cases that might be good enough?
I meant to mention that the solution I had considered before asking was to put up a Pinax PyPI and make development releases on it. We could put up an instance of chishop. We are already using pip's --find-links to point at pypi.pinaxproject.com for packages we've had to release ourselves.
Most open source distributors (the Debians, Ubuntu's, MacPorts, et al) use some sort of patch management mechanism. So something like: import the base source code for each package as released, as a tar ball, or as a SCM snapshot. Then manage any necessary modifications on top of it using a patch manager, like quilt or Mercurial's Queues. Then bundle up each external package with any applied patches in a consistent format. Or have URLs to the base packages and URLs to the individual patches and have them applied during installation. That's essentially what MacPorts does.
EDIT: To take it one step further, you could then version control the set of patches across all of the external packages and make that available as a unit. That's quite easy to do with Mercurial Queues. Then you've simplified the problem to just publishing one set of patches using one SCM system, with the patches applied locally as above or available for developers to pull and apply to their copies of the base release packages.
EDIT: I am not sure I am reading your question correctly so the following may not answer your question directly.
Something I've considered, but haven't tested, is using pip's freeze bundle feature. Perhaps using that and distributing the bundle with Pinax would work? My only concern would be how different OS's are handled. For example, I've never used pip on Windows, so I wouldn't know how a bundle would interact there.
The full idea I hope to try is creating a paver script that controls management of the bundles, making it easy for users to upgrade to newer versions. This would require a bit of scaffolding though.
One other option may be you keeping a mirror of the apps you don't control, in a consistent vcs, and then distributing your mirrored versions. This would take away the need for "everyone" to have many different programs installed.
Other than that, it seems the only real solution is what you guys are doing, there isn't a hassle-free way that I've been able to find.