Related
A Django website I maintain currently uses Python 2.7 but I know that I'll have to upgrade it to Python 3 in a couple of months. If I'm writing code right now that has to work in Python 2, is there a Pythonic way to write it such that it would also work in Python 3 without any changes if I know what the syntax is going to be in Python 3? Ideally I'd like the code to continue to work even after the upgrade without changing it but it would be easy for me to spot where I've done this in the codebase so that I can change the code when I have time. Here's an example of what I'm talking about:
# Python 2 uses 'iteritems'
def log_dict(**kwargs):
for key, value in kwargs.iteritems():
log.info("{0}: {1}".format(key, value))
# Python 3 uses 'items'
def log_dict(**kwargs):
for key, value in kwargs.items():
log.info("{0}: {1}".format(key, value))
There is official documentation suggesting ways to do this. That documentation has changed over time as the situation has changed, so it's worth going directly to the source (especially if you're reading this answer a year or two after it was written).
It's also worth reading the Conservative Python 3 Porting Guide and skimming Nick Coghlan's Python 3 Q&A, especially this section.
Going back in time from the early 2018:
futurize
The current official suggestions are:
Only worry about supporting Python 2.7
Make sure you have good test coverage (coverage.py can help; pip install coverage)
Learn the differences between Python 2 & 3
Use Futurize (or Modernize) to update your code (e.g. pip install future)
Use Pylint to help make sure you don’t regress on your Python 3 support (pip install pylint)
Use caniusepython3 to find out which of your dependencies are blocking your use of Python 3 (pip install caniusepython3)
Once your dependencies are no longer blocking you, use continuous integration to make sure you stay compatible with Python 2 & 3 (tox can help test against multiple versions of Python; pip install tox)
Consider using optional static type checking to make sure your type usage works in both Python 2 & 3 (e.g. use mypy to check your typing under both Python 2 & Python 3).
Notice the last suggestion. Guido and another of the core devs have both been heavily involved in leading large teams to port large 2.7 codebases to 3.x, and found mypy to be very helpful (especially in dealing with bytes-vs.-unicode issues). In fact, that's a large part of the reason gradual static typing is now an official part of the language.
You also almost certainly want to use all of the future statements available in 2.7. This is so obvious that they seem to have forgotten to leave it out of the docs, but, besides making your life easier (e.g., you can write print function calls), futurize and modernize (and six and sixer) require it.
six
The documentation is aimed at people making an irreversible transition to Python 3 in the near future. If you're planning to stick with dual-version code for a long time, you might be better off following the previous recommendations, which largely revolved around using six instead of futurize. Six covers more of the differences between the two languages, and also makes you write code that's explicit about being dual-version instead of being as close to Python 3 as possible while still running in 2.7. But the downside is that you're effectively doing two ports—one from 2.7 to six-based dual-version code, and then, later, from 3.x-only six code to 3.x-only "native" code.
2to3
The original recommended answer was to use 2to3, a tool that can automatically convert Python 2 code to Python 3 code, or guide you in doing so. If you want your code to work in both, you need to deliver Python 2 code, then run 2to3 at installation time to port it to Python 3. Which means you need to test your code both ways, and usually modify it so that it still works in 2.7 but also works in 3.x after 2to3, which isn't always easy to work out. This turns out to not be feasible for most non-trivial projects, so it's no longer recommended by the core devs—but it is still built in with Python 2.7 and 3.x, and getting updates.
There are also two variations on 2to3: sixer auto-ports your Python 2.7 code to dual-version code that uses six, and 3to2 lets you write your code for Python 3 and auto-port back to 2.7 at install time. Both of these were popular for a time, but don't seem to be used much anymore; modernize and futurize, respectively, are their main successors.
For your specific question,
kwargs.items() will work on both, if you don't mind a minor performance cost in 2.7.
2to3 can automatically change that iteritems to items at install time on 3.x.
futurize can be used to do either of the above.
six will allow you to write six.iteritems(kwargs), which will do iteritems in 2.7 and items in 3.x.
six will also allow you to write six.viewitems(kwargs), which will do viewitems in 2.7 (which is identical to what items does in 3.x, rather than just similar).
modernize and sixer will automatically change that kwargs.iteritems() to six.iteritems(kwargs).
3to2 will let you write kwargs.items() and autmatically convert it to viewitems at install time on 2.x.
mypy can verify that you're just using the result as a general iterable (rather than specifically as an iterator), so changing to viewitems or items leaves your code still correctly typed.
you can import the future package
from future import ....
nested_scopes 2.1.0b1 2.2 PEP 227: Statically Nested Scopes
generators 2.2.0a1 2.3 PEP 255: Simple Generators
division 2.2.0a2 3.0 PEP 238: Changing the Division Operator
absolute_import 2.5.0a1 3.0 PEP 328: Imports: Multi-Line and Absolute/Relative
with_statement 2.5.0a1 2.6 PEP 343: The “with” Statement
print_function 2.6.0a2 3.0 PEP 3105: Make print a function
unicode_literals 2.6.0a2 3.0 PEP 3112: Bytes literals in Python 3000
Making your Django project compatible with both Python versions consists of the following steps:
Add from __future__ import unicode_literals at the top of each module and then use usual quotes without a u prefix for Unicode strings and a b prefix for bytestrings.
To ensure that a value is bytestring, use the django.utils.encoding.smart_bytes function. To ensure that a value is Unicode, use the django.utils.encoding.smart_text or django.utils.encoding.force_text function.
In your models use __str__ method instead of __unicode__ and add the python_2_unicode_compatible decorator.
# models.py
# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from django.db import models
from django.utils.translation import ugettext_lazy as _
from django.utils.encoding import python_2_unicode_compatible
#python_2_unicode_compatible
class NewsArticle(models.Model):
title = models.CharField(_("Title"), max_length=200)
content = models.TextField(_("Content"))
def __str__(self):
return self.title
class Meta:
verbose_name = _("News Article")
verbose_name_plural = _("News Articles")
To iterate through dictionaries, use iteritems() , iterkeys() , and itervalues() from django.utils.six . Take a look at the following:
from django.utils.six import iteritems
d = {"imported": 25, "skipped": 12, "deleted": 3}
for k, v in iteritems(d):
print("{0}: {1}".format(k, v))
At the time of capturing exceptions, use the as keyword, as follows:
try:
article = NewsArticle.objects.get(slug="hello-world")
except NewsArticle.DoesNotExist as exc:
pass
except NewsArticle.MultipleObjectsReturned as exc:
pass
Use django.utils.six to check the type of a value as shown in the following:
from django.utils import six
isinstance(val, six.string_types) # previously basestring
isinstance(val, six.text_type) # previously unicode
isinstance(val, bytes) # previously str
isinstance(val, six.integer_types) # previously (int, long)
Use range from django.utils.six.moves ,Instead of xrange , as follows:
from django.utils.six.moves import range
for i in range(1, 11):
print(i)
Source link
In addition to importing from future, there is also the six project that aims to provide api compatibility between Python 2 and Python 3: https://pypi.org/project/six/.
Your example code could be made compatible between version 2 and 3:
import six
for key, value in six.iteritems(dict):
log.info("{0}: {1}".format(key, value))
There are still things that won't be compatible between 2 and 3 like f-strings.
There are a few different tools that will help you make sure you are writing python2/3 compatible code.
If you are interested in porting python2 code into python3, then the 2to3 program that comes with the standard library will try to convert a python 2 program to python 3.
https://docs.python.org/2/library/2to3.html
Another great tool is pylint. pylint is a python linter that will describe issues to you without fixing them. If you pip install pylint on a python3 environment, then it will analyze your code based on python 3's rules. If you use python 2 to install pylint, it will do the same but with python 2's rules.
There are other popular and similar tools like flake8 or autopep8, but I am not familiar with them enough to advertise them.
six and future is a golden rule, enough to make easy a coming migration
add to every python2 file, this as first line:
from __future__ import absolute_import, unicode_literals
use below working with strings, iteration, metaclasses, ...
isinstance(sth, six.string_types)
six.iteritems(dict)
#six.add_metaclass(Meta)
and so on six reference
On Windows 7 with Python 2.7 how can I detect if a path is a symbolic link?
This does not work os.path.islink(), it says it returns false if false or not supported and the path I'm providing is definitely a symbolic link so I'm assuming it's not supported on windows? What can I do?
The root problem is that you're using too old a version of Python. If you want to stick to 2.x, you will not be able to take advantage of new features added after early 2010.
One of those features is handling NTFS symlinks. That functionality was added in 3.2 in late 2010. (See the 3.2, 3.1, and 2.7 source for details.)
The reason Python didn't handle NTFS symlinks before then is that there was no such thing until late 2009. (IIRC, support was included in the 6.0 kernel, but userland support requires a service pack on Vista/2008; only 7/2008R2 and newer come with it built in. Plus, you need a new-enough MSVCRT to be able to access that userland support, and Python has an explicit policy of not upgrading to new Visual Studio versions within a minor release.)
The reason the code wasn't ported back to 2.x is that there will never be a 2.8, and bug fix releases like 2.7.3 (or 2.7.4) don't get new features, only bug fixes.
This has been reported as issue 13143, and the intended fix is to change the 2.7 docs to clarify that islink always returns False on Windows.
So, if you want to read NTFS symlinks under Windows, either upgrade to Python 3.2+, or you have to use win32api, ctypes, etc. to do it yourself.
Or, as Martijn Pieters suggests, instead of doing it yourself, use a third-party library like jaraco.windows that does it and/or borrow their code.
Or, if you really want, borrow the code from the 3.2 source and build a C extension module around it. If you trace down from ntpath to os to nt (which is actually posixmodule.c), I believe the guts of it are in win32_xstat_impl and win32_xstat_impl_w.
This is what I ended up using to determine if a file or a directory is a link in Windows 7:
from subprocess import check_output, CalledProcessError
import os.path
import ctypes
def isLink(path):
if os.path.exists(path):
if os.path.isdir(path):
FILE_ATTRIBUTE_REPARSE_POINT = 0x0400
attributes = ctypes.windll.kernel32.GetFileAttributesW(unicode(path))
return (attributes & FILE_ATTRIBUTE_REPARSE_POINT) > 0
else:
command = ['dir', path]
try:
with open(os.devnull, 'w') as NULL_FILE:
o0 = check_output(command, stderr=NULL_FILE, shell=True)
except CalledProcessError as e:
print e.output
return False
o1 = [s.strip() for s in o0.split('\n')]
if len(o1) < 6:
return False
else:
return 'SYMLINK' in o1[5]
else:
return False
EDIT: Modified code as per suggestions of Zitrax and Annan
EDIT: Added include statements as per the suggestion of shioko
For directories:
import os, ctypes
def IsSymlink(path):
FILE_ATTRIBUTE_REPARSE_POINT = 0x0400
return os.path.isdir(path) and (ctypes.windll.kernel32.GetFileAttributesW(unicode(path)) & FILE_ATTRIBUTE_REPARSE_POINT):
Source
You can also use the pywin32 module: GetFileAttributes is available in the win32api sub-module and FILE_ATTRIBUTE_REPARSE_POINT in the win32con module. For instance, to test if a given path is a symlink to a directory, the code becomes:
import os
import win32api
import win32con
def is_directory_symlink(path):
return bool(os.path.isdir(path)
and (win32api.GetFileAttributes(path) &
win32con.FILE_ATTRIBUTE_REPARSE_POINT))
If using Python 2 and the path may contain non-ascii characters, GetFileAttributes requires a unicode string. However, simply using unicode(path) will generally fail: you should test if path is a str and, if so, use its decode method.
Just using if file[-4:len(file)] != ".lnk": works for me
It's not uncommon for an intro programming class to write a Lisp metacircular evaluator. Has there been any attempt at doing this for Python?
Yes, I know that Lisp's structure and syntax lends itself nicely to a metacircular evaluator, etc etc. Python will most likely be more difficult. I am just curious as to whether such an attempt has been made.
For those who don't know what a meta-circular evaluator is, it is an interpreter which is written in the language to be interpreted. For example: a Lisp interpreter written in Lisp, or in our case, a Python interpreter written in Python. For more information, read this chapter from SICP.
As JBernardo said, PyPy is one. However, PyPy's Python interpreter, the meta-circular evaluator that is, is implemented in a statically typed subset of Python called RPython.
You'll be pleased to know that, as of the 1.5 release, PyPy is fully compliant with the official Python 2.7 specification. Even more so: PyPy nearly always beats Python in performance benchmarks.
For more information see PyPy docs and PyPy extra docs.
I think i wrote one here:
"""
Metacircular Python interpreter with macro feature.
By Cees Timmerman, 14aug13.
"""
import re
re_macros = re.compile("^#define (\S+) ([^\r\n]+)", re.MULTILINE)
def meta_python_exec(code):
# Optional meta feature.
macros = re_macros.findall(code)
code = re_macros.sub("", code)
for m in macros:
code = code.replace(m[0], m[1])
# Run the code.
exec(code)
if __name__ == "__main__":
#code = open("metacircular_overflow.py", "r").read() # Causes a stack overflow in Python 3.2.3, but simply raises "RuntimeError: maximum recursion depth exceeded while calling a Python object" in Python 2.7.3.
code = "#define 1 2\r\nprint(1 + 1)"
meta_python_exec(code)
I would like to see if there is any way of requiring a minimal python version.
I have several python modules that are requiring Python 2.6 due to the new exception handling (as keyword).
It looks that even if I check the python version at the beginning of my script, the code will not run because the interpreter will fail inside the module, throwing an ugly system error instead of telling the user to use a newer python.
You can take advantage of the fact that Python will do the right thing when comparing tuples:
#!/usr/bin/python
import sys
MIN_PYTHON = (2, 6)
if sys.version_info < MIN_PYTHON:
sys.exit("Python %s.%s or later is required.\n" % MIN_PYTHON)
You should not use any Python 2.6 features inside the script itself. Also, you must do your version check before importing any of the modules requiring a new Python version.
E.g. start your script like so:
#!/usr/bin/env python
import sys
if sys.version_info[0] != 2 or sys.version_info[1] < 6:
print("This script requires Python version 2.6")
sys.exit(1)
# rest of script, including real initial imports, here
Starting with version 9.0.0 pip supports Requires-Python field in distribution's metadata which can be written by setuptools starting with version 24-2-0. This feature is available through python_requires keyword argument to setup function.
Example (in setup.py):
setup(
...
python_requires='>=2.5,<2.7',
...
)
To take advantage of this feature one has to package the project/script first if not already done. This is very easy in typical case and should be done nonetheless as it allows users to easily install, use and uninstall given project/script. Please see Python Packaging User Guide for details.
import sys
if sys.hexversion < 0x02060000:
sys.exit("Python 2.6 or newer is required to run this program.")
import module_requiring_26
Also the cool part about this is that it can be included inside the __init__ file or the module.
I used to have a more complicated approach for supporting both Python2 and Python3, but I no longer try to support Python2, so now I just use:
import sys
MIN_PYTHON = (3, 7)
assert sys.version_info >= MIN_PYTHON, f"requires Python {'.'.join([str(n) for n in MIN_PYTHON])} or newer"
If the version check fails, you get a traceback with something like:
AssertionError: requires Python 3.7 or newer
at the bottom.
To complement the existing, helpful answers:
You may want to write scripts that run with both Python 2.x and 3.x, and require a minimum version for each.
For instance, if your code uses the argparse module, you need at least 2.7 (with a 2.x Python) or at least 3.2 (with a 3.x Python).
The following snippet implements such a check; the only thing that needs adapting to a different, but analogous scenario are the MIN_VERSION_PY2=... and MIN_VERSION_PY3=... assignments.
As has been noted: this should be placed at the top of the script, before any other import statements.
import sys
MIN_VERSION_PY2 = (2, 7) # min. 2.x version as major, minor[, micro] tuple
MIN_VERSION_PY3 = (3, 2) # min. 3.x version
# This is generic code that uses the tuples defined above.
if (sys.version_info[0] == 2 and sys.version_info < MIN_VERSION_PY2
or
sys.version_info[0] == 3 and sys.version_info < MIN_VERSION_PY3):
sys.exit(
"ERROR: This script requires Python 2.x >= %s or Python 3.x >= %s;"
" you're running %s." % (
'.'.join(map(str, MIN_VERSION_PY2)),
'.'.join(map(str, MIN_VERSION_PY3)),
'.'.join(map(str, sys.version_info))
)
)
If the version requirements aren't met, something like the following message is printed to stderr and the script exits with exit code 1.
This script requires Python 2.x >= 2.7 or Python 3.x >= 3.2; you're running 2.6.2.final.0.
Note: This is a substantially rewritten version of an earlier, needlessly complicated answer, after realizing - thanks to Arkady's helpful answer - that comparison operators such as > can directly be applied to tuples.
I'm guessing you have something like:
import module_foo
...
import sys
# check sys.version
but module_foo requires a particular version as well? This being the case, it is perfectly valid to rearrange your code thus:
import sys
# check sys.version
import module_foo
Python does not require that imports, aside from from __future__ import [something] be at the top of your code.
I need to make sure I'm using Python 3.5 (or, eventually, higher). I monkeyed around on my own and then I thought to ask SO - but I've not been impressed with the answers (sorry, y'all ::smile::). Rather than giving up, I came up with the approach below. I've tested various combinations of the min_python and max_python specification tuples and it seems to work nicely:
Putting this code into a __init__.py is attractive:
Avoids polluting many modules with a redundant version check
Placing this at the top of a package hierarchy even more further supports the DRY principal, assuming the entire hierarchy abides by the same Python version contraints
Takes advantage of a place (file) where I can use the most portable Python code (e.g. Python 1 ???) for the check logic and still write my real modules in the code version I want
If I have other package-init stuff that is not "All Python Versions Ever" compatible, I can shovel it into another module, e.g. __init_p3__.py as shown in the sample's commented-out final line. Don't forget to replace the pkgname place holder with the appropriate package name.
If you don't want a min (or max), just set it to = ()
If you only care about the major version, just use a "one-ple", e.g. = (3, ) Don't forget the comma, otherwise (3) is just a parenthesized (trivial) expression evaluating to a single int
You can specify finer min/max than just one or two version levels, e.g. = (3, 4, 1)
There will be only one "Consider running as" suggestion when the max isn't actually greater than the min, either because max is an empty tuple (a "none-ple"?), or has fewer elements.
NOTE: I'm not much of a Windoze programmer, so the text_cmd_min and text_cmd_max values are oriented for *Nix systems. If you fix up the code to work in other environments (e.g. Windoze or some particular *Nix variant), then please post. (Ideally, a single super-smartly code block will suffice for all environments, but I'm happy with my *Nix only solution for now.)
PS: I'm somewhat new to Python, and I don't have an interpreter with version less than 2.7.9.final.0, so it's tuff to test my code for earlier variants. On the other hand, does anyone really care? (That's an actual question I have: In what (real-ish) context would I need to deal with the "Graceful Wrong-Python-Version" problem for interpreters prior to 2.7.9?)
__init__.py
'''Verify the Python Interpreter version is in range required by this package'''
min_python = (3, 5)
max_python = (3, )
import sys
if (sys.version_info[:len(min_python)] < min_python) or (sys.version_info[:len(max_python)] > max_python):
text_have = '.'.join("%s" % n for n in sys.version_info)
text_min = '.'.join("%d" % n for n in min_python) if min_python else None
text_max = '.'.join("%d" % n for n in max_python) if max_python else None
text_cmd_min = 'python' + text_min + ' ' + " ".join("'%s'" % a for a in sys.argv) if min_python else None
text_cmd_max = 'python' + text_max + ' ' + " ".join("'%s'" % a for a in sys.argv) if max_python > min_python else None
sys.stderr.write("Using Python version: " + text_have + "\n")
if min_python: sys.stderr.write(" - Min required: " + text_min + "\n")
if max_python: sys.stderr.write(" - Max allowed : " + text_max + "\n")
sys.stderr.write("\n")
sys.stderr.write("Consider running as:\n\n")
if text_cmd_min: sys.stderr.write(text_cmd_min + "\n")
if text_cmd_max: sys.stderr.write(text_cmd_max + "\n")
sys.stderr.write("\n")
sys.exit(9)
# import pkgname.__init_p3__
Rather than indexing you could always do this,
import platform
if platform.python_version() not in ('2.6.6'):
raise RuntimeError('Not right version')
I have a Python script that uses Python version 2.6 syntax (Except error as value:) which version 2.5 complains about. So in my script I have included some code to check for the Python interpreter version before proceeding so that the user doesn't get hit with a nasty error, however, no matter where I place that code, it doesn't work. Once it hits the strange syntax it throws the syntax error, disregarding any attempts of mine of version checking.
I know I could simply place a try/except block over the area that the SyntaxError occurs and generate the message there but I am wondering if there is a more "elegant" way. As I am not very keen on placing try/except blocks all over my code to address the version issue. I looked into using an __ init__.py file, but the user won't be importing/using my code as a package, so I don't think that route will work, unless I am missing something...
Here is my version checking code:
import sys
def isPythonVersion(version):
if float(sys.version[:3]) >= version:
return True
else:
return False
if not isPythonVersion(2.6):
print "You are running Python version", sys.version[:3], ", version 2.6 or 2.7 is required. Please update. Aborting..."
exit()
Create a wrapper script that checks the version and calls your real script -- this gives you a chance to check the version before the interpreter tries to syntax-check the real script.
Something like this in beginning of code?
import sys
if sys.version_info<(2,6):
raise SystemExit('Sorry, this code need Python 2.6 or higher')
In sys.version_info you will find the version information stored in a tuple:
sys.version_info
(2, 6, 6, 'final', 0)
Now you can compare:
def isPythonVersion(version):
return version >= sys.version_info[0] + sys.version_info[1] / 10.
If speed is not a priority, you can avoid this problem entirely by using sys.exc_info to grab the details of the last exception.