Develop Reference

For Python2 to Python3 code conversion, Which version of Python & Django best suited?

Currently I am working in big firm where we need to convert python2 old big Django project into python3 version so I have done lots of research related but still not able to find any perfect answer related to which version of Python and Django best suited for conversion.
Currently I am using Python : 2.7.16 & Django : 1.9.13 in my old version.
Anyone can suggest me best suited version of Python & Django for above old version for python2 to python3 conversion.

I thought I'd add a bit to the strategy advocated by Wim's answer - get the appropriate version of Django working on both 2.7 and 3.x first - and outline some tactics that worked for me.
Python 2.7 is your escape pod, until you pull the trigger on 3.x
your tests should run on both
don't use any 3.x specific features, like f-strings
first Python 3.x, then only later Django 2.x which doesn't run on 2.7
start early, don't over analyze, but avoid the big bang approach
file by file at first.
start with the lowest level code, like utility libraries, that you have test suites for.
if possible, try to gradually merge your changes to the 2.7 production branches and keep your 3.x porting code up to date with prod changes.
Which minor version of Django to start with?
My criteria here is that Django migrations can be fairly involved (and actually require more thinking than 2=>3 work). So I would move to the latest and greatest 1.11 that way you're already providing some value to your 2.7 users. There's probably a good number of pre-2.x compatibility shims on 1.11 and you'll be getting its 2.x deprecation warnings.
Which minor version of Python 3.x to start with?
Best to consider all angles, such as the availability of your 3rd party libs, support from your CI/devops suite and availability on your chosen server OS images. You could always install 3.8 and try a pip install of your requirements.txt by itself, for example.
Leverage git (or whatever scm you use) and virtualenv.
separate requirement.txt files, but...
if you have a file-based, git repo, you can point each venv at the same codeline with a pip install -e <your directory>. that means that, in 2 different terminals you can run 2.7 and 3.x against the same unittest(s).
you could even run 2.7 and 3.x Django servers side-by-side on different ports and point say Firefox and Chrome at them.
commit often (on the porting branch at least) and learn about git bisect.
make use of 2to3
Yes, it will break 2.7 code and Django if you let it. So...
run it in preview mode or against a single file. see what it breaks but also see what it did right.
throttle it to only certain conversions that don't break 2.7 or Django. print x=> print (x) and except(Exception) as e are 2 no-brainers.
This is what my throttled command looked like:
2to3 $tgt -w -f except -f raise -f next -f funcattrs -f print
run it file-by-file until you are really confident.
use sed or awk rather than your editor for bulk conversions.
The advantage is that, as you become more aware of your apps' specifics concerns, you can build a suite of changes that can be run on either 1 file or many files and do most of the work without breaking 2.7 or Django. Apply this after your suitably-throttled 2to3 pass. That leaves you with residual cleanups in your editor and getting your tests to pass.
(optional) start running black on 2.7 code.
black which is a code formatter, uses Python 3 ASTs to run its analysis. It doesn't try to run the code, but it will flag syntax errors that prevent it from getting to the AST stage. You will have to work some pip install global magic to get there though and you have to buy into black's usefulness.
Other people have done it - learn from them.
Listening to #155 Practical steps for moving to Python 3 should give you some ideas of the work. Look at the show links for it. They love to talk up the Instagram(?) move which involved a gradual adjustment of running 2.7 code to 3.x syntax on a common codebase, and on the same git branch, until pull-the-trigger day.
See also The Conservative Python 3 Porting Guide
and Instagram Makes a Smooth Move to Python 3 - The New Stack
Conclusion
Your time to Django 1.11 EOL (April 2020) is rather short, so if you have 2+ dev resources to throw at it, I'd consider doing the following in parallel:
DEV#1: start out on a Django 1.11 bump (the theory being that Django 1.11 is probably best positioned as a jump off point to Django 2.x), using 2.7.
DEV#2: get started on Python 3.6/3.7 of your non-Django utility code. Since the code is 2.7 compatible at this point, merge it into #1 as you go.
See how both tasks proceed, assess what the Django related project risk is and what the Python 3 pain looks like. You're already missing the Python 2.7 EOL, but an obsolete web framework is probably more dangerous than legacy Python 2.7, at least for a few months. So I wouldn't wait too long to start migrating off Django 1.9 and your work doing so won't be wasted. As you see the progress, you'll start seeing the project risks better.
Your initial 2to3 progress will be slow, but the tooling and guidance is good enough that you'll quickly pick up speed so don't overthink it before starting to gather experience. The Django side depends on your exposure to breaking changes in the framework which is why I think it's best to start early.
P.S. (controversial/personal opinion) I didn't use six or other canned 2-to-3 bridge libraries much.
It's not because I don't trust it - it's brilliant for 3rd party libs - but rather that I didn't want to add a complex permanent dependency (and I was too lazy to read its doc). I'd been writing 2.7 code in 3.x compatible syntax for a long time so I didn't really feel the need to use them. Your mileage may vary and don't set out on this path if it seems like a lot of work.
Instead, I created a py223.py (57 LOC incl. comments) with this type of content, most of which is concerned with workarounds for deprecations and name changes in the standard library.
try:
basestring_ = basestring
except (NameError,) as e:
basestring_ = str
try:
cmp_ = cmp
except (NameError,) as e:
# from http://portingguide.readthedocs.io/en/latest/comparisons.html
def cmp_(x, y):
"""
Replacement for built-in function cmp that was removed in Python 3
"""
return (x > y) - (x < y)
Then import from that py223 to work around those specific concerns. Later on I will just ditch the import and move those weird isinstance(x, basestr_) to isinstance(x, str) but I know in advance there is little to worry about.

My suggestion is to first upgrade to Django==1.11.26, which is the most recent version of Django that is supporting both Python 2 and Python 3. Stay on your current version of Python 2.7 for now.
Read carefully the release notes for 1.10.x and 1.11.x, checking for deprecations and fixing anything that stopped working from your 1.9.x code. Things WILL break. Django moves fast. For a large Django project, there may be many code changes required, and if you're using a lot of 3rd-party plugins or libraries you may have to juggle their versions around. Some of your 3rd-party dependencies will probably have been abandoned entirely, so you have to find replacements or remove the features.
To find the release notes for each version upgrade, just google "What's new in Django ". The hits will meticulously document all the deprecations and changes:
https://docs.djangoproject.com/en/2.2/releases/1.10/
https://docs.djangoproject.com/en/2.2/releases/1.11/
Once the webapp appears to be working fine on Django 1.11, with all tests passing (you do have a test suite, right?) then you can do the Python 3 conversion, whilst keeping the Django version the same. Django 1.11 supports up to Python 3.7, so that would be a good version to target. Expect unicode all over the place, since the implicit conversions between bytes and text is gone now and many Python 2 webapps relied upon that.
Once the project appears to be working fine on Django 1.11 and Python 3.7, then you can think about upgrading to Django 3.0, following the same process as before - reading the release notes, making the necessary changes, running the test suite, and checking out the webapp in a dev server manually.

I would upgrade to py3 first. You'll need to look at setup.py in the Django repo on the stable/1.9.x branch (https://github.com/django/django/blob/stable/1.9.x/setup.py) to figure out that the py3 versions supported are 3.4 (dead) and 3.5.
Once you're on py3.5 and Django 1.9 you can upgrade one at a time until you get to the version you want to end at. E.g. Django 1.11 supports py3.5 and py3.7, so
py27/dj19 -> py35/dj19 -> py35/dj1.11 -> py37/dj1.11 ... -> py37/dj2.2
dj2.2 is the first version supporting py3.8, but I would probably stop at py37/dj2.2 if you're working in a normally conservative environment.
If you have other packages you'll need to find version combinations that will work together on each step. Having a plan is key, and upgrading only one component at a time will usually end up saving you time.
The future library (https://python-future.org/) will help you with many icky situations while you need code to run on both py27 and 3.x. six is great too. I would avoid rolling your own compatibility layer (why reinvent the wheel?)
If at all possible, try to get your unit test coverage up to 75-85% before starting, and definitely set up automatic testing on both "from" and "to" versions for each upgrade step. Make sure you read and fix all warnings from Django before upgrading to the next version -- Django cares very little about backward compatibility, so I would normally suggest hitting every minor version on the upgrade path (or at least make sure you read the "backwards incompatibilities" and deprecation lists for each minor version).
Good luck (we're upgrading a 300+Kloc code base from py27/dj1.7 right now, so I feel your pain ;-)

I have same kind of issue with my project and I have tried python 3.7.5 with Django version 2.2.7.
You should not go with python latest version 3.8 or Django latest version 3.0 because you there may have been chances that for any kind of bug you may not able to get proper solution for latest versions.

You should try to shoot for the current versions. Python 3.8 and Django 3.0.The Six library will help with some convention changes. Either way you are going to have to do some refactoring so you might as well make it current.

Upgrading Django Version: Best Practice

I've pretty big code base written on Django 1.5, it's about time to upgrade to the newest version (1.11.2) as many stuff don't work well.
I'm wondering if it's best to upgrade step by step: 1.5->1.6->1.7...
or just jump to 1.11.2
what method should be better and make the (hard) process easier? as my project has many dependencies?
Also what are good practices to do? I'm using virtualenv and aware of this Django article about upgrading

The document you found (“Upgrading Django to a newer version”) has a good guide.
An important part, before upgrading, is to have full branch coverage by the automated test suite for your application, before upgrading.
You want to be able to run a full automated test suite, see everything pass and know that all branches are exercised by the test suite.
This means that when you break something by porting to the new Django version, you'll be able to see which parts of your app are not behaving correctly any more.
Read about what changes you need to make by reading the release notes and deprecation timeline, for all of the relevant releases between your current and target Django versions.
See what dependencies you'll need to upgrade for Django; you might need to correct your code if it relies on incompatible features in an outdated third-party library.
All of those should be done before upgrading a single thing, in my opinion.

It depends on two things. How many people are using the Django app, and the extend of your test coverage.
If you are the only user, no worries, upgrade all the way. But if you have a lot of users you will quickly find out by upgrading all the way to 1.11.2 that some edge cases may not be covered by your tests.
Expect a lot of error 500 along the way.
If your coverage is close to 100% on all your apps, you may not have that problem.
Note that a lot has changed since 1.5.

I my case about: Data
I am using SQLite and i am sure that, the lastest version can kill you if you have only 1 conflict.
python manage.py makemigrations and so on had so many chance in it's method.
And if you change Python from 2x to 3x, it is a new case too =))

Use Pytest with Django without using django-pytest 3rd party app

Is it possible to use Pytest with Django without using django-pytest 3rd party app?
I had tried to set this up, but kept running into random errors, like pytest couldn't find the DJANGO_SETTINGS_MODULE. Then I fixed the path, but the normal python manage.py runserver then couldn't find the DJANGO_SETTINGS_MODULE. I'm running:
Pytest 2.5.4
Python 3.4.0
Django 1.6.2
If it is possible, would you be able to provide a setup example of where to put the tests/ directory, etc... within the project so Pytest works?
Thanks

Hmm, py.test 2.5.4 does not exist afaik.
Anyway, assuming you mean to ask whether it is possible to avoid the pytest-django plugin to test Django using py.test: the short answer is no.
The long answer is yes, but it is extremely difficult to get it all to work and you will basically write at least a minimal version of pytest-django in the conftest.py file. The pytest-django plugin was created specifically to work around all the weirdness which Django does with much global state and other hidden magic.
OTOH looking at the pytest-django source would probably help you kickstart such an effort. However you may consider thinking about what it is of pytest-django you do not like and maybe file an enhancement request to improve it.

How does django-nose differ from the default Django test-runner

I've been seeing and reading about a lot of people using nose to run their Django tests. I haven't been able to figure out the added benefits of using Nose to run my Django tests. If someone could fill me in on what nose is and how it adds more to a Django project, it would be helpful.
I haven't been able to find a good document/article outlining these points.
Thank you

I was curious about this too and it seems that the main advantage of django-nose using the python nose library is "Test Discovery".
In addition, from http://readthedocs.org/docs/nose/en/latest/testing.html
you can also write simple test functions, as well as test classes that are not
subclasses of unittest.TestCase. nose also supplies a number of
helpful functions for writing timed tests, testing for exceptions, and
other common use cases. See Writing tests and Testing tools for more.
From what I understand from other python developers on freenode irc, Trial test runner on Twisted Framework have these similar features like nose.
I am still not entirely convinced about using django-nose for django development but am giving a shot and report back if I find out more!

There are a lot more features overall, but I think one major reason people use nose/djano_nose is that it allows you to very easily do code coverage.
python manage.py test myapp --with-coverage --cover-package=myapp

Using Pylint with Django

I would very much like to integrate pylint into the build process for
my python projects, but I have run into one show-stopper: One of the
error types that I find extremely useful--:E1101: *%s %r has no %r
member*--constantly reports errors when using common django fields,
for example:
E1101:125:get_user_tags: Class 'Tag' has no 'objects' member
which is caused by this code:
def get_user_tags(username):
"""
Gets all the tags that username has used.
Returns a query set.
"""
return Tag.objects.filter( ## This line triggers the error.
tagownership__users__username__exact=username).distinct()
# Here is the Tag class, models.Model is provided by Django:
class Tag(models.Model):
"""
Model for user-defined strings that help categorize Events on
on a per-user basis.
"""
name = models.CharField(max_length=500, null=False, unique=True)
def __unicode__(self):
return self.name
How can I tune Pylint to properly take fields such as objects into account? (I've also looked into the Django source, and I have been unable to find the implementation of objects, so I suspect it is not "just" a class field. On the other hand, I'm fairly new to python, so I may very well have overlooked something.)
Edit: The only way I've found to tell pylint to not warn about these warnings is by blocking all errors of the type (E1101) which is not an acceptable solution, since that is (in my opinion) an extremely useful error. If there is another way, without augmenting the pylint source, please point me to specifics :)
See here for a summary of the problems I've had with pychecker and pyflakes -- they've proven to be far to unstable for general use. (In pychecker's case, the crashes originated in the pychecker code -- not source it was loading/invoking.)

Do not disable or weaken Pylint functionality by adding ignores or generated-members.
Use an actively developed Pylint plugin that understands Django.
This Pylint plugin for Django works quite well:
pip install pylint-django
and when running pylint add the following flag to the command:
--load-plugins pylint_django
Detailed blog post here.

I use the following: pylint --generated-members=objects

If you use Visual Studio Code do this:
pip install pylint-django
And add to VSC config:
"python.linting.pylintArgs": [
"--load-plugins=pylint_django"
],

My ~/.pylintrc contains
[TYPECHECK]
generated-members=REQUEST,acl_users,aq_parent,objects,_meta,id
the last two are specifically for Django.
Note that there is a bug in PyLint 0.21.1 which needs patching to make this work.
Edit: After messing around with this a little more, I decided to hack PyLint just a tiny bit to allow me to expand the above into:
[TYPECHECK]
generated-members=REQUEST,acl_users,aq_parent,objects,_meta,id,[a-zA-Z]+_set
I simply added:
import re
for pattern in self.config.generated_members:
if re.match(pattern, node.attrname):
return
after the fix mentioned in the bug report (i.e., at line 129).
Happy days!

django-lint is a nice tool which wraps pylint with django specific settings : http://chris-lamb.co.uk/projects/django-lint/
github project: https://github.com/lamby/django-lint

Because of how pylint works (it examines the source itself, without letting Python actually execute it) it's very hard for pylint to figure out how metaclasses and complex baseclasses actually affect a class and its instances. The 'pychecker' tool is a bit better in this regard, because it does actually let Python execute the code; it imports the modules and examines the resulting objects. However, that approach has other problems, because it does actually let Python execute the code :-)
You could extend pylint to teach it about the magic Django uses, or to make it understand metaclasses or complex baseclasses better, or to just ignore such cases after detecting one or more features it doesn't quite understand. I don't think it would be particularly easy. You can also just tell pylint to not warn about these things, through special comments in the source, command-line options or a .pylintrc file.

I resigned from using pylint/pychecker in favor of using pyflakes with Django code - it just tries to import module and reports any problem it finds, like unused imports or uninitialized local names.

This is not a solution, but you can add objects = models.Manager() to your Django models without changing any behavior.
I myself only use pyflakes, primarily due to some dumb defaults in pylint and laziness on my part (not wanting to look up how to change the defaults).

Try running pylint with
pylint --ignored-classes=Tags
If that works, add all the other Django classes - possibly using a script, in say, python :P
The documentation for --ignore-classes is:
--ignored-classes=<members names>
List of classes names for which member
attributes should not be checked
(useful for classes with attributes
dynamicaly set). [current: %default]
I should add this is not a particular elegant solution in my view, but it should work.

For neovim & vim8 use w0rp's ale plugin. If you have installed everything correctly including w0rp's ale, pylint & pylint-django. In your vimrc add the following line & have fun developing web apps using django.
Thanks.
let g:ale_python_pylint_options = '--load-plugins pylint_django'

The solution proposed in this other question it to simply add get_attr to your Tag class. Ugly, but works.

So far I have found no real solution to that but work around:
In our company we require a pylint
score > 8. This allows coding
practices pylint doesn't understand
while ensuring that the code isn't
too "unusual". So far we havn't seen
any instance where E1101 kept us
from reaching a score of 8 or
higher.
Our 'make check' targets
filter out "for has no 'objects'
member" messages to remove most of
the distraction caused by pylint not
understanding Django.

For heroku users, you can also use Tal Weiss's answer to this question using the following syntax to run pylint with the pylint-django plugin (replace timekeeping with your app/package):
# run on the entire timekeeping app/package
heroku local:run pylint --load-plugins pylint_django timekeeping
# run on the module timekeeping/report.py
heroku local:run pylint --load-plugins pylint_django timekeeping/report.py
# With temporary command line disables
heroku local:run pylint --disable=invalid-name,missing-function-docstring --load-plugins pylint_django timekeeping/report.py
Note: I was unable to run without specifying project/package directories.
If you have issues with E5110: Django was not configured., you can also invoke as follows to try to work around that (again, change timekeeping to your app/package):
heroku local:run python manage.py shell -c 'from pylint import lint; lint.Run(args=["--load-plugins", "pylint_django", "timekeeping"])'
# With temporary command line disables, specific module
heroku local:run python manage.py shell -c 'from pylint import lint; lint.Run(args=["--load-plugins", "pylint_django", "--disable=invalid-name,missing-function-docstring", "timekeeping/report.py"])'