Function Annotations: PEP-3107
I ran across a snippet of code demonstrating Python3's function annotations. The concept is simple but I can't think of why these were implemented in Python3 or any good uses for them. Perhaps SO can enlighten me?
How it works:
def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9):
... function body ...
Everything following the colon after an argument is an 'annotation', and the information following the -> is an annotation for the function's return value.
foo.func_annotations would return a dictionary:
{'a': 'x',
'b': 11,
'c': list,
'return': 9}
What's the significance of having this available?
Function annotations are what you make of them.
They can be used for documentation:
def kinetic_energy(mass: 'in kilograms', velocity: 'in meters per second'):
...
They can be used for pre-condition checking:
def validate(func, locals):
for var, test in func.__annotations__.items():
value = locals[var]
msg = 'Var: {0}\tValue: {1}\tTest: {2.__name__}'.format(var, value, test)
assert test(value), msg
def is_int(x):
return isinstance(x, int)
def between(lo, hi):
def _between(x):
return lo <= x <= hi
return _between
def f(x: between(3, 10), y: is_int):
validate(f, locals())
print(x, y)
>>> f(0, 31.1)
Traceback (most recent call last):
...
AssertionError: Var: y Value: 31.1 Test: is_int
Also see http://www.python.org/dev/peps/pep-0362/ for a way to implement type checking.
I think this is actually great.
Coming from an academic background, I can tell you that annotations have proved themselves invaluable for enabling smart static analyzers for languages like Java. For instance, you could define semantics like state restrictions, threads that are allowed to access, architecture limitations, etc., and there are quite a few tools that can then read these and process them to provide assurances beyond what you get from the compilers. You could even write things that check preconditions/postconditions.
I feel something like this is especially needed in Python because of its weaker typing, but there were really no constructs that made this straightforward and part of the official syntax.
There are other uses for annotations beyond assurance. I can see how I could apply my Java-based tools to Python. For instance, I have a tool that lets you assign special warnings to methods, and gives you indications when you call them that you should read their documentation (E.g., imagine you have a method that must not be invoked with a negative value, but it's not intuitive from the name). With annotations, I could technically write something like this for Python. Similarly, a tool that organizes methods in a large class based on tags can be written if there is an official syntax.
This is a way late answer, but AFAICT, the best current use of function annotations is PEP-0484 and MyPy. There's also PyRight from Microsoft which is used by VSCode and also available via CLI.
Mypy is an optional static type checker for Python. You can add type hints to your Python programs using the upcoming standard for type annotations introduced in Python 3.5 beta 1 (PEP 484), and use mypy to type check them statically.
Used like so:
from typing import Iterator
def fib(n: int) -> Iterator[int]:
a, b = 0, 1
while a < n:
yield a
a, b = b, a + b
Just to add a specific example of a good use from my answer here, coupled with decorators a simple mechanism for multimethods can be done.
# This is in the 'mm' module
registry = {}
import inspect
class MultiMethod(object):
def __init__(self, name):
self.name = name
self.typemap = {}
def __call__(self, *args):
types = tuple(arg.__class__ for arg in args) # a generator expression!
function = self.typemap.get(types)
if function is None:
raise TypeError("no match")
return function(*args)
def register(self, types, function):
if types in self.typemap:
raise TypeError("duplicate registration")
self.typemap[types] = function
def multimethod(function):
name = function.__name__
mm = registry.get(name)
if mm is None:
mm = registry[name] = MultiMethod(name)
spec = inspect.getfullargspec(function)
types = tuple(spec.annotations[x] for x in spec.args)
mm.register(types, function)
return mm
and an example of use:
from mm import multimethod
#multimethod
def foo(a: int):
return "an int"
#multimethod
def foo(a: int, b: str):
return "an int and a string"
if __name__ == '__main__':
print("foo(1,'a') = {}".format(foo(1,'a')))
print("foo(7) = {}".format(foo(7)))
This can be done by adding the types to the decorator as Guido's original post shows, but annotating the parameters themselves is better as it avoids the possibility of wrong matching of parameters and types.
Note: In Python you can access the annotations as function.__annotations__ rather than function.func_annotations as the func_* style was removed on Python 3.
Uri has already given a proper answer, so here's a less serious one: So you can make your docstrings shorter.
The first time I saw annotations, I thought "great! Finally I can opt in to some type checking!" Of course, I hadn't noticed that annotations are not actually enforced.
So I decided to write a simple function decorator to enforce them:
def ensure_annotations(f):
from functools import wraps
from inspect import getcallargs
#wraps(f)
def wrapper(*args, **kwargs):
for arg, val in getcallargs(f, *args, **kwargs).items():
if arg in f.__annotations__:
templ = f.__annotations__[arg]
msg = "Argument {arg} to {f} does not match annotation type {t}"
Check(val).is_a(templ).or_raise(EnsureError, msg.format(arg=arg, f=f, t=templ))
return_val = f(*args, **kwargs)
if 'return' in f.__annotations__:
templ = f.__annotations__['return']
msg = "Return value of {f} does not match annotation type {t}"
Check(return_val).is_a(templ).or_raise(EnsureError, msg.format(f=f, t=templ))
return return_val
return wrapper
#ensure_annotations
def f(x: int, y: float) -> float:
return x+y
print(f(1, y=2.2))
>>> 3.2
print(f(1, y=2))
>>> ensure.EnsureError: Argument y to <function f at 0x109b7c710> does not match annotation type <class 'float'>
I added it to the Ensure library.
It a long time since this was asked but the example snippet given in the question is (as stated there as well) from PEP 3107 and at the end of thas PEP example Use cases are also given which might answer the question from the PEPs point of view ;)
The following is quoted from PEP3107
Use Cases
In the course of discussing annotations, a number of use-cases have been raised. Some of these are presented here, grouped by what kind of information they convey. Also included are examples of existing products and packages that could make use of annotations.
Providing typing information
Type checking ([3], [4])
Let IDEs show what types a function expects and returns ([17])
Function overloading / generic functions ([22])
Foreign-language bridges ([18], [19])
Adaptation ([21], [20])
Predicate logic functions
Database query mapping
RPC parameter marshaling ([23])
Other information
Documentation for parameters and return values ([24])
See the PEP for more information on specific points (as well as their references)
Python 3.X (only) also generalizes function definition to allow
arguments and return values to be annotated with object values
for use in extensions.
Its META-data to explain, to be more explicit about the function values.
Annotations are coded as :value after the
argument name and before a default, and as ->value after the
argument list.
They are collected into an __annotations__ attribute of the function, but are not otherwise treated as special by Python itself:
>>> def f(a:99, b:'spam'=None) -> float:
... print(a, b)
...
>>> f(88)
88 None
>>> f.__annotations__
{'a': 99, 'b': 'spam', 'return': <class 'float'>}
Source: Python Pocket Reference, Fifth Edition
EXAMPLE:
The typeannotations module provides a set of tools for type checking and type inference of Python code. It also a provides a set of types useful for annotating functions and objects.
These tools are mainly designed to be used by static analyzers such as linters, code completion libraries and IDEs. Additionally, decorators for making run-time checks are provided. Run-time type checking is not always a good idea in Python, but in some cases it can be very useful.
https://github.com/ceronman/typeannotations
How Typing Helps to Write Better Code
Typing can help you do static code analysis to catch type errors
before you send your code to production and prevent you from some
obvious bugs. There are tools like mypy, which you can add to your
toolbox as part of your software life cycle. mypy can check for
correct types by running against your codebase partially or fully.
mypy also helps you to detect bugs such as checking for the None type
when the value is returned from a function. Typing helps to make your
code cleaner. Instead of documenting your code using comments, where
you specify types in a docstring, you can use types without any
performance cost.
Clean Python: Elegant Coding in Python
ISBN: ISBN-13 (pbk): 978-1-4842-4877-5
PEP 526 -- Syntax for Variable Annotations
https://www.python.org/dev/peps/pep-0526/
https://www.attrs.org/en/stable/types.html
Despite all uses described here, the one enforceable and, most likely, enforced use of annotations will be for type hints.
This is currently not enforced in any way but, judging from PEP 484, future versions of Python will only allow types as the value for annotations.
Quoting What about existing uses of annotations?:
We do hope that type hints will eventually become the sole use for annotations, but this will require additional discussion and a deprecation period after the initial roll-out of the typing module with Python 3.5. The current PEP will have provisional status (see PEP 411 ) until Python 3.6 is released. The fastest conceivable scheme would introduce silent deprecation of non-type-hint annotations in 3.6, full deprecation in 3.7, and declare type hints as the only allowed use of annotations in Python 3.8.
Though I haven't seen any silent deprecations in 3.6 yet, this could very well be bumped to 3.7, instead.
So, even though there might be some other good use-cases, it is best to keep them solely for type hinting if you don't want to go around changing everything in a future where this restriction is in place.
As a bit of a delayed answer, several of my packages (marrow.script, WebCore, etc.) use annotations where available to declare typecasting (i.e. transforming incoming values from the web, detecting which arguments are boolean switches, etc.) as well as to perform additional markup of arguments.
Marrow Script builds a complete command-line interface to arbitrary functions and classes and allows for defining documentation, casting, and callback-derived default values via annotations, with a decorator to support older runtimes. All of my libraries that use annotations support the forms:
any_string # documentation
any_callable # typecast / callback, not called if defaulting
(any_callable, any_string) # combination
AnnotationClass() # package-specific rich annotation object
[AnnotationClass(), AnnotationClass(), …] # cooperative annotation
"Bare" support for docstrings or typecasting functions allows for easier mixing with other libraries that are annotation-aware. (I.e. have a web controller using typecasting that also happens to be exposed as a command-line script.)
Edited to add: I've also begun making use of the TypeGuard package using development-time assertions for validation. Benefit: when run with "optimizations" enabled (-O / PYTHONOPTIMIZE env var) the checks, which may be expensive (e.g. recursive) are omitted, with the idea that you've properly tested your app in development so the checks should be unnecessary in production.
Annotations can be used for easily modularizing code. E.g. a module for a program which I'm maintaining could just define a method like:
def run(param1: int):
"""
Does things.
:param param1: Needed for counting.
"""
pass
and we could ask the user for a thing named "param1" which is "Needed for counting" and should be an "int". In the end we can even convert the string given by the user to the desired type to get the most hassle free experience.
See our function metadata object for an open source class which helps with this and can automatically retrieve needed values and convert them to any desired type (because the annotation is a conversion method). Even IDEs show autocompletions right and assume that types are according to annotations - a perfect fit.
If you look at the list of benefits of Cython, a major one is the ability to tell the compiler which type a Python object is.
I can envision a future where Cython (or similar tools that compile some of your Python code) will use the annotation syntax to do their magic.
Related
One of the most talked-about features in Python 3.5 is type hints.
An example of type hints is mentioned in this article and this one while also mentioning to use type hints responsibly. Can someone explain more about them and when they should be used and when not?
I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them TypeErrors.
Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the . and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
Find bugs sooner: This is self-evident, I believe.
The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that
dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from
a behavioral aspect) you require.
Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function
annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and
hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
Function annotations (PEP 3107).
Stub files for built-in/user modules.
Special # type: type comments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for # type: type comments)
Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).
For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints
the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy, on the other hand, we get the following response:
(Python3)jimmi#jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).
Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__ attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi#jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the
PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
You need to type check a module for which you do not want to directly alter the function signatures
You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been
mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
Mypy: as described here.
PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:
class Counter(Dict[_T, int], Generic[_T]):
#overload
def __init__(self) -> None: ...
#overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
#overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.
Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Adding to Jim's elaborate answer:
Check the typing module -- this module supports type hints as specified by PEP 484.
For example, the function below takes and returns values of type str and is annotated as follows:
def greeting(name: str) -> str:
return 'Hello ' + name
The typing module also supports:
Type aliasing.
Type hinting for callback functions.
Generics - Abstract base classes have been extended to support subscription to denote expected types for container elements.
User-defined generic types - A user-defined class can be defined as a generic class.
Any type - Every type is a subtype of Any.
The newly released PyCharm 5 supports type hinting. In their blog post about it (see Python 3.5 type hinting in PyCharm 5) they offer a great explanation of what type hints are and aren't along with several examples and illustrations for how to use them in your code.
Additionally, it is supported in Python 2.7, as explained in this comment:
PyCharm supports the typing module from PyPI for Python 2.7, Python 3.2-3.4. For 2.7 you have to put type hints in *.pyi stub files since function annotations were added in Python 3.0.
Type hints are for maintainability and don't get interpreted by Python. In the code below, the line def add(self, ic:int) doesn't result in an error until the next return... line:
class C1:
def __init__(self):
self.idn = 1
def add(self, ic: int):
return self.idn + ic
c1 = C1()
c1.add(2)
c1.add(c1)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 5, in add
TypeError: unsupported operand type(s) for +: 'int' and 'C1'
Python has dynamic type checking, hence the types are known at runtime and not compile time (as is the case in static type checked languages like C#).
With TypeHints, Python supports type annotation for the basic variable types supported by the language str, int, float, bool and None. It also comes with a typing library batteries included; this typing libraries provides us with means to use more special types.
from typing import List
name: str = 'Tommy'
age: int = 24
height_in_meters: float = 1.7
Read more: https://tomisin.dev/blog/improving-your-python-projects-with-type-hints
One of the most talked-about features in Python 3.5 is type hints.
An example of type hints is mentioned in this article and this one while also mentioning to use type hints responsibly. Can someone explain more about them and when they should be used and when not?
I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them TypeErrors.
Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the . and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
Find bugs sooner: This is self-evident, I believe.
The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that
dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from
a behavioral aspect) you require.
Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function
annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and
hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
Function annotations (PEP 3107).
Stub files for built-in/user modules.
Special # type: type comments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for # type: type comments)
Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).
For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints
the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy, on the other hand, we get the following response:
(Python3)jimmi#jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).
Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__ attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi#jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the
PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
You need to type check a module for which you do not want to directly alter the function signatures
You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been
mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
Mypy: as described here.
PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:
class Counter(Dict[_T, int], Generic[_T]):
#overload
def __init__(self) -> None: ...
#overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
#overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.
Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Adding to Jim's elaborate answer:
Check the typing module -- this module supports type hints as specified by PEP 484.
For example, the function below takes and returns values of type str and is annotated as follows:
def greeting(name: str) -> str:
return 'Hello ' + name
The typing module also supports:
Type aliasing.
Type hinting for callback functions.
Generics - Abstract base classes have been extended to support subscription to denote expected types for container elements.
User-defined generic types - A user-defined class can be defined as a generic class.
Any type - Every type is a subtype of Any.
The newly released PyCharm 5 supports type hinting. In their blog post about it (see Python 3.5 type hinting in PyCharm 5) they offer a great explanation of what type hints are and aren't along with several examples and illustrations for how to use them in your code.
Additionally, it is supported in Python 2.7, as explained in this comment:
PyCharm supports the typing module from PyPI for Python 2.7, Python 3.2-3.4. For 2.7 you have to put type hints in *.pyi stub files since function annotations were added in Python 3.0.
Type hints are for maintainability and don't get interpreted by Python. In the code below, the line def add(self, ic:int) doesn't result in an error until the next return... line:
class C1:
def __init__(self):
self.idn = 1
def add(self, ic: int):
return self.idn + ic
c1 = C1()
c1.add(2)
c1.add(c1)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 5, in add
TypeError: unsupported operand type(s) for +: 'int' and 'C1'
Python has dynamic type checking, hence the types are known at runtime and not compile time (as is the case in static type checked languages like C#).
With TypeHints, Python supports type annotation for the basic variable types supported by the language str, int, float, bool and None. It also comes with a typing library batteries included; this typing libraries provides us with means to use more special types.
from typing import List
name: str = 'Tommy'
age: int = 24
height_in_meters: float = 1.7
Read more: https://tomisin.dev/blog/improving-your-python-projects-with-type-hints
One of the most talked-about features in Python 3.5 is type hints.
An example of type hints is mentioned in this article and this one while also mentioning to use type hints responsibly. Can someone explain more about them and when they should be used and when not?
I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them TypeErrors.
Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the . and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
Find bugs sooner: This is self-evident, I believe.
The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that
dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from
a behavioral aspect) you require.
Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function
annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and
hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
Function annotations (PEP 3107).
Stub files for built-in/user modules.
Special # type: type comments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for # type: type comments)
Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).
For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints
the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy, on the other hand, we get the following response:
(Python3)jimmi#jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).
Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__ attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi#jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the
PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
You need to type check a module for which you do not want to directly alter the function signatures
You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been
mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
Mypy: as described here.
PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:
class Counter(Dict[_T, int], Generic[_T]):
#overload
def __init__(self) -> None: ...
#overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
#overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.
Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Adding to Jim's elaborate answer:
Check the typing module -- this module supports type hints as specified by PEP 484.
For example, the function below takes and returns values of type str and is annotated as follows:
def greeting(name: str) -> str:
return 'Hello ' + name
The typing module also supports:
Type aliasing.
Type hinting for callback functions.
Generics - Abstract base classes have been extended to support subscription to denote expected types for container elements.
User-defined generic types - A user-defined class can be defined as a generic class.
Any type - Every type is a subtype of Any.
The newly released PyCharm 5 supports type hinting. In their blog post about it (see Python 3.5 type hinting in PyCharm 5) they offer a great explanation of what type hints are and aren't along with several examples and illustrations for how to use them in your code.
Additionally, it is supported in Python 2.7, as explained in this comment:
PyCharm supports the typing module from PyPI for Python 2.7, Python 3.2-3.4. For 2.7 you have to put type hints in *.pyi stub files since function annotations were added in Python 3.0.
Type hints are for maintainability and don't get interpreted by Python. In the code below, the line def add(self, ic:int) doesn't result in an error until the next return... line:
class C1:
def __init__(self):
self.idn = 1
def add(self, ic: int):
return self.idn + ic
c1 = C1()
c1.add(2)
c1.add(c1)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 5, in add
TypeError: unsupported operand type(s) for +: 'int' and 'C1'
Python has dynamic type checking, hence the types are known at runtime and not compile time (as is the case in static type checked languages like C#).
With TypeHints, Python supports type annotation for the basic variable types supported by the language str, int, float, bool and None. It also comes with a typing library batteries included; this typing libraries provides us with means to use more special types.
from typing import List
name: str = 'Tommy'
age: int = 24
height_in_meters: float = 1.7
Read more: https://tomisin.dev/blog/improving-your-python-projects-with-type-hints
One of the most talked-about features in Python 3.5 is type hints.
An example of type hints is mentioned in this article and this one while also mentioning to use type hints responsibly. Can someone explain more about them and when they should be used and when not?
I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them TypeErrors.
Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the . and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
Find bugs sooner: This is self-evident, I believe.
The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that
dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from
a behavioral aspect) you require.
Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function
annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and
hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
Function annotations (PEP 3107).
Stub files for built-in/user modules.
Special # type: type comments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for # type: type comments)
Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).
For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints
the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy, on the other hand, we get the following response:
(Python3)jimmi#jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).
Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__ attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi#jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the
PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
You need to type check a module for which you do not want to directly alter the function signatures
You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been
mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
Mypy: as described here.
PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:
class Counter(Dict[_T, int], Generic[_T]):
#overload
def __init__(self) -> None: ...
#overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
#overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.
Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Adding to Jim's elaborate answer:
Check the typing module -- this module supports type hints as specified by PEP 484.
For example, the function below takes and returns values of type str and is annotated as follows:
def greeting(name: str) -> str:
return 'Hello ' + name
The typing module also supports:
Type aliasing.
Type hinting for callback functions.
Generics - Abstract base classes have been extended to support subscription to denote expected types for container elements.
User-defined generic types - A user-defined class can be defined as a generic class.
Any type - Every type is a subtype of Any.
The newly released PyCharm 5 supports type hinting. In their blog post about it (see Python 3.5 type hinting in PyCharm 5) they offer a great explanation of what type hints are and aren't along with several examples and illustrations for how to use them in your code.
Additionally, it is supported in Python 2.7, as explained in this comment:
PyCharm supports the typing module from PyPI for Python 2.7, Python 3.2-3.4. For 2.7 you have to put type hints in *.pyi stub files since function annotations were added in Python 3.0.
Type hints are for maintainability and don't get interpreted by Python. In the code below, the line def add(self, ic:int) doesn't result in an error until the next return... line:
class C1:
def __init__(self):
self.idn = 1
def add(self, ic: int):
return self.idn + ic
c1 = C1()
c1.add(2)
c1.add(c1)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 5, in add
TypeError: unsupported operand type(s) for +: 'int' and 'C1'
Python has dynamic type checking, hence the types are known at runtime and not compile time (as is the case in static type checked languages like C#).
With TypeHints, Python supports type annotation for the basic variable types supported by the language str, int, float, bool and None. It also comes with a typing library batteries included; this typing libraries provides us with means to use more special types.
from typing import List
name: str = 'Tommy'
age: int = 24
height_in_meters: float = 1.7
Read more: https://tomisin.dev/blog/improving-your-python-projects-with-type-hints
One of the most talked-about features in Python 3.5 is type hints.
An example of type hints is mentioned in this article and this one while also mentioning to use type hints responsibly. Can someone explain more about them and when they should be used and when not?
I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them TypeErrors.
Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the . and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
Find bugs sooner: This is self-evident, I believe.
The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that
dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from
a behavioral aspect) you require.
Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function
annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and
hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
Function annotations (PEP 3107).
Stub files for built-in/user modules.
Special # type: type comments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for # type: type comments)
Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).
For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints
the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy, on the other hand, we get the following response:
(Python3)jimmi#jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).
Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__ attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi#jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the
PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
You need to type check a module for which you do not want to directly alter the function signatures
You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been
mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
Mypy: as described here.
PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:
class Counter(Dict[_T, int], Generic[_T]):
#overload
def __init__(self) -> None: ...
#overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
#overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.
Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Adding to Jim's elaborate answer:
Check the typing module -- this module supports type hints as specified by PEP 484.
For example, the function below takes and returns values of type str and is annotated as follows:
def greeting(name: str) -> str:
return 'Hello ' + name
The typing module also supports:
Type aliasing.
Type hinting for callback functions.
Generics - Abstract base classes have been extended to support subscription to denote expected types for container elements.
User-defined generic types - A user-defined class can be defined as a generic class.
Any type - Every type is a subtype of Any.
The newly released PyCharm 5 supports type hinting. In their blog post about it (see Python 3.5 type hinting in PyCharm 5) they offer a great explanation of what type hints are and aren't along with several examples and illustrations for how to use them in your code.
Additionally, it is supported in Python 2.7, as explained in this comment:
PyCharm supports the typing module from PyPI for Python 2.7, Python 3.2-3.4. For 2.7 you have to put type hints in *.pyi stub files since function annotations were added in Python 3.0.
Type hints are for maintainability and don't get interpreted by Python. In the code below, the line def add(self, ic:int) doesn't result in an error until the next return... line:
class C1:
def __init__(self):
self.idn = 1
def add(self, ic: int):
return self.idn + ic
c1 = C1()
c1.add(2)
c1.add(c1)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 5, in add
TypeError: unsupported operand type(s) for +: 'int' and 'C1'
Python has dynamic type checking, hence the types are known at runtime and not compile time (as is the case in static type checked languages like C#).
With TypeHints, Python supports type annotation for the basic variable types supported by the language str, int, float, bool and None. It also comes with a typing library batteries included; this typing libraries provides us with means to use more special types.
from typing import List
name: str = 'Tommy'
age: int = 24
height_in_meters: float = 1.7
Read more: https://tomisin.dev/blog/improving-your-python-projects-with-type-hints