I'm using type hints and mypy more and more. I however have some questions about when I should explicitly annotate a declaration, and when the type can be determined automatically by mypy.
Ex:
def assign_volume(self, volume: float) -> None:
self._volume = volume * 1000
Should I write
self._volume: float = volume *1000
In this case?
Now if I have the following function:
def return_volume(self) -> float:
return self._volume
and somewhere in my code:
my_volume = return_volume()
Should I write:
my_volume: float = return_volume()
Mypy (and PEP 484 in general) is designed so that in the most ideal case, you only need to add type annotations to the "boundaries" or "interfaces" of your code.
For example, you basically must add annotations/type metadata in the following places:
The parameter and return types of functions and methods.
Any object fields (assuming the types of your fields are not inferrable just by looking at your constructor)
When you inherit a class. For example, if you specifically want to subclass a dict of ints to strs, you should do class MyClass(Dict[int, str]): ..., not class MyClass(dict): ....
These are all examples of "boundaries" of your code. Type hints on parameter/return types let the caller of the function make sure they're calling it correctly, type hints on fields let the caller know they're using the object correctly, etc...
Mypy (and other PEP 484 compliant tools) will then use that information and try to infer the types of everything else. This behavior is designed to roughly mimic how humans read code: once you know what types are being passed in, for example, it's usually pretty easy to understand what the rest of the code does.
After all, Python is a language that was designed from the start to be readable! We don't need to scatter type hints everywhere to enhance our understanding of what the code does.
Of course, mypy (and other PEP 484-compliant tools) aren't perfect, and sometimes they might not correctly infer what the type of some local variable will be. In that case, you might need to add a type hint to help mypy along. Ethan's answer gives a good overview of some common cases to watch out for. (Interestingly, these cases also tend to be examples of where a human reader might struggle to understand your code!)
So, to put everything together, the general recommendation is to:
Add type hints to all of the "boundaries" of your code, like function parameters and return types.
Default to not annotating variables. If mypy is unable to infer what type some variable should be, add an annotation to help it.
If you find yourself needing to annotate lots of variables to make mypy happy, consider refactoring your code. If mypy is getting confused easily, a human reader is also likely to get confused easily.
So, to go back to your examples, you would not add type hints in either case. Both a human reader and mypy can tell that your _volume field must be a float: it's immediately obvious that must be the case since the parameter is a float and multiplying a float by an int will always produce another float.
Similarly, you would not add an annotation to your my_volume variable. Since return_volume() has type hints, it's trivially easy to see what type it's returning and understand that my_volume is of type float. (And if you make a mistake and accidentally think it's something other then a float, then mypy will catch that for you.)
Mypy does some pretty advanced type inference. Usually, you do not need to annotate variables. The mypy documentation [1] says this about inference:
Mypy considers the initial assignment as the definition of a variable. If you do not explicitly specify the type of the variable, mypy infers the type based on the static type of the value expression
The general rule of thumb then is "annotate variables whose types are not inferrable at their initial assignment".
Here are some examples:
Empty containers. If I define a as a = [], mypy will not know what types are valid in the list a.
Optional types. Oftentimes, if I define an Optional type, I will assign the variable to None. For example, if I do a = None, mypy will infer that a has type NoneType, if you want to assign a to 5 later on, you need to annotate it: a: Optional[int] = None.
Complex nested containers. For example, if you have a dictionary with both list and string values, mypy might, for example, infer Dict[str, Any]. You may need to annotate it to be more accurate.
Of course there are many more cases.
In your examples, mypy can infer the types of the expressions.
[1] https://mypy.readthedocs.io/en/latest/type_inference_and_annotations.html
For myself it started to write type hints everywhere, where it is possible. It isn't slower at all and it makes it easier if you will go back to your old code in the feature. So there is now negative aspect on using them as much as possible except of the size of your python file.
Related
One of the most talked-about features in Python 3.5 is type hints.
An example of type hints is mentioned in this article and this one while also mentioning to use type hints responsibly. Can someone explain more about them and when they should be used and when not?
I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them TypeErrors.
Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the . and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
Find bugs sooner: This is self-evident, I believe.
The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that
dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from
a behavioral aspect) you require.
Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function
annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and
hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
Function annotations (PEP 3107).
Stub files for built-in/user modules.
Special # type: type comments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for # type: type comments)
Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).
For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints
the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy, on the other hand, we get the following response:
(Python3)jimmi#jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).
Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__ attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi#jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the
PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
You need to type check a module for which you do not want to directly alter the function signatures
You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been
mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
Mypy: as described here.
PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:
class Counter(Dict[_T, int], Generic[_T]):
#overload
def __init__(self) -> None: ...
#overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
#overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.
Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Adding to Jim's elaborate answer:
Check the typing module -- this module supports type hints as specified by PEP 484.
For example, the function below takes and returns values of type str and is annotated as follows:
def greeting(name: str) -> str:
return 'Hello ' + name
The typing module also supports:
Type aliasing.
Type hinting for callback functions.
Generics - Abstract base classes have been extended to support subscription to denote expected types for container elements.
User-defined generic types - A user-defined class can be defined as a generic class.
Any type - Every type is a subtype of Any.
The newly released PyCharm 5 supports type hinting. In their blog post about it (see Python 3.5 type hinting in PyCharm 5) they offer a great explanation of what type hints are and aren't along with several examples and illustrations for how to use them in your code.
Additionally, it is supported in Python 2.7, as explained in this comment:
PyCharm supports the typing module from PyPI for Python 2.7, Python 3.2-3.4. For 2.7 you have to put type hints in *.pyi stub files since function annotations were added in Python 3.0.
Type hints are for maintainability and don't get interpreted by Python. In the code below, the line def add(self, ic:int) doesn't result in an error until the next return... line:
class C1:
def __init__(self):
self.idn = 1
def add(self, ic: int):
return self.idn + ic
c1 = C1()
c1.add(2)
c1.add(c1)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 5, in add
TypeError: unsupported operand type(s) for +: 'int' and 'C1'
Python has dynamic type checking, hence the types are known at runtime and not compile time (as is the case in static type checked languages like C#).
With TypeHints, Python supports type annotation for the basic variable types supported by the language str, int, float, bool and None. It also comes with a typing library batteries included; this typing libraries provides us with means to use more special types.
from typing import List
name: str = 'Tommy'
age: int = 24
height_in_meters: float = 1.7
Read more: https://tomisin.dev/blog/improving-your-python-projects-with-type-hints
One of the most talked-about features in Python 3.5 is type hints.
An example of type hints is mentioned in this article and this one while also mentioning to use type hints responsibly. Can someone explain more about them and when they should be used and when not?
I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them TypeErrors.
Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the . and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
Find bugs sooner: This is self-evident, I believe.
The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that
dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from
a behavioral aspect) you require.
Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function
annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and
hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
Function annotations (PEP 3107).
Stub files for built-in/user modules.
Special # type: type comments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for # type: type comments)
Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).
For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints
the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy, on the other hand, we get the following response:
(Python3)jimmi#jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).
Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__ attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi#jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the
PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
You need to type check a module for which you do not want to directly alter the function signatures
You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been
mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
Mypy: as described here.
PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:
class Counter(Dict[_T, int], Generic[_T]):
#overload
def __init__(self) -> None: ...
#overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
#overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.
Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Adding to Jim's elaborate answer:
Check the typing module -- this module supports type hints as specified by PEP 484.
For example, the function below takes and returns values of type str and is annotated as follows:
def greeting(name: str) -> str:
return 'Hello ' + name
The typing module also supports:
Type aliasing.
Type hinting for callback functions.
Generics - Abstract base classes have been extended to support subscription to denote expected types for container elements.
User-defined generic types - A user-defined class can be defined as a generic class.
Any type - Every type is a subtype of Any.
The newly released PyCharm 5 supports type hinting. In their blog post about it (see Python 3.5 type hinting in PyCharm 5) they offer a great explanation of what type hints are and aren't along with several examples and illustrations for how to use them in your code.
Additionally, it is supported in Python 2.7, as explained in this comment:
PyCharm supports the typing module from PyPI for Python 2.7, Python 3.2-3.4. For 2.7 you have to put type hints in *.pyi stub files since function annotations were added in Python 3.0.
Type hints are for maintainability and don't get interpreted by Python. In the code below, the line def add(self, ic:int) doesn't result in an error until the next return... line:
class C1:
def __init__(self):
self.idn = 1
def add(self, ic: int):
return self.idn + ic
c1 = C1()
c1.add(2)
c1.add(c1)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 5, in add
TypeError: unsupported operand type(s) for +: 'int' and 'C1'
Python has dynamic type checking, hence the types are known at runtime and not compile time (as is the case in static type checked languages like C#).
With TypeHints, Python supports type annotation for the basic variable types supported by the language str, int, float, bool and None. It also comes with a typing library batteries included; this typing libraries provides us with means to use more special types.
from typing import List
name: str = 'Tommy'
age: int = 24
height_in_meters: float = 1.7
Read more: https://tomisin.dev/blog/improving-your-python-projects-with-type-hints
One of the most talked-about features in Python 3.5 is type hints.
An example of type hints is mentioned in this article and this one while also mentioning to use type hints responsibly. Can someone explain more about them and when they should be used and when not?
I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them TypeErrors.
Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the . and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
Find bugs sooner: This is self-evident, I believe.
The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that
dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from
a behavioral aspect) you require.
Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function
annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and
hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
Function annotations (PEP 3107).
Stub files for built-in/user modules.
Special # type: type comments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for # type: type comments)
Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).
For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints
the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy, on the other hand, we get the following response:
(Python3)jimmi#jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).
Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__ attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi#jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the
PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
You need to type check a module for which you do not want to directly alter the function signatures
You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been
mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
Mypy: as described here.
PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:
class Counter(Dict[_T, int], Generic[_T]):
#overload
def __init__(self) -> None: ...
#overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
#overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.
Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Adding to Jim's elaborate answer:
Check the typing module -- this module supports type hints as specified by PEP 484.
For example, the function below takes and returns values of type str and is annotated as follows:
def greeting(name: str) -> str:
return 'Hello ' + name
The typing module also supports:
Type aliasing.
Type hinting for callback functions.
Generics - Abstract base classes have been extended to support subscription to denote expected types for container elements.
User-defined generic types - A user-defined class can be defined as a generic class.
Any type - Every type is a subtype of Any.
The newly released PyCharm 5 supports type hinting. In their blog post about it (see Python 3.5 type hinting in PyCharm 5) they offer a great explanation of what type hints are and aren't along with several examples and illustrations for how to use them in your code.
Additionally, it is supported in Python 2.7, as explained in this comment:
PyCharm supports the typing module from PyPI for Python 2.7, Python 3.2-3.4. For 2.7 you have to put type hints in *.pyi stub files since function annotations were added in Python 3.0.
Type hints are for maintainability and don't get interpreted by Python. In the code below, the line def add(self, ic:int) doesn't result in an error until the next return... line:
class C1:
def __init__(self):
self.idn = 1
def add(self, ic: int):
return self.idn + ic
c1 = C1()
c1.add(2)
c1.add(c1)
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 5, in add
TypeError: unsupported operand type(s) for +: 'int' and 'C1'
Python has dynamic type checking, hence the types are known at runtime and not compile time (as is the case in static type checked languages like C#).
With TypeHints, Python supports type annotation for the basic variable types supported by the language str, int, float, bool and None. It also comes with a typing library batteries included; this typing libraries provides us with means to use more special types.
from typing import List
name: str = 'Tommy'
age: int = 24
height_in_meters: float = 1.7
Read more: https://tomisin.dev/blog/improving-your-python-projects-with-type-hints
Does python have a stance on the PEP-484 type annotation for a property setter's argument? I see two options, both of which seem valid (according to me, and to mypy).
Consider:
from dataclasses import dataclass
from typing import Any
#dataclass
class Foo:
_bar: int = 1
#property
def bar(self) -> int:
return self._bar
#bar.setter
def bar(self, value) -> None:
self._bar = value
The question is:
Should #bar.setter's value argument be typed with typing.Any or with int?
On one hand, within the setter, having the expected type hint would be nice for performing validations, but on the other hand, the incoming value could be of any type.
One thing of note, though; mypy does warn about the incorrect assignment to a property setter:
f = Foo()
f.bar = 2 # Ok
f.bar = "baz" # Incompatible types in assignment (expression has type "str", variable has type "int")
I believe this comes from the revealed type of Foo.bar being an int, not from the type of the value argument of #bar.setter.
I searched through the python/cpython and python/typeshed projects for examples, but didn't come up with anything definitive.
I'm very experienced with modern python, and am comfortable reading cpython sources (beit in C or python itself). An answer that references a PEP, or includes input from a cpython or mypy maintainer would be ideal.
This question lends itself very well to being strongly opinion based, however, I think there may be a stronger argument for mutators annotated with the expected type (ie def bar(self, value: int) -> None:). First, annotations were implemented to aid in static analysis rather than prviding any real runtime benefit (they currently do not to my knowledge. From PEP 484 rationale:
Of these goals, static analysis is the most important. This includes support for off-line type checkers such as mypy, as well as providing a standard notation that can be used by IDEs for code completion and refactoring.
If type annotations are largely meant to benefit in static analysis, linting, etc it would make sense that you would want to be able to check that you are passing in the wrong type rather than potentially discover at runtime that you have not handled the parameter properly with type checks using isinstance for example.
This would also mean that we can do more with less, since the more specific int annotation would remove the need for us to add those type guards:
def bigger_fun(n: Any) -> None:
if isinstance(n, float):
# do something...
else
# dosomething else...
def smaller_fun(n: int) -> None:
# do something
You will know exactly what type you will receive and how to handle it, rather than needing to implement different multiple conditional branches to first cast the parameter to an expected value before operating on it. This will allow allow you to make your mutators as slim as possible with only minimal internal logic / processing.
If you were to pass it the wrong type, your IDE or static analysis tool will at the very least warn you when passing a float for smaller_fun for example. On the other hand, using Any might produce unexpected behavior for some types, which introduces runtime bugs which could be difficult to track down.
Now more specifically to your question, the same PEP touches upon the use of #property annotations in The Meaning of Annotations
Type checkers are expected to attempt to infer as much information as necessary. The minimum requirement is to handle the builtin decorators #property, #staticmethod and #classmethod.
This means that you can expect the #property annotation should function normally as you'd expect. Without any special treatment.
While python is at heart a dynamically typed language, methods like a mutator are very strongly tied to a specific value (and therfore type) and should only really do one thing rather than one of many things. So while it probably makes since for a comparison method like __gt__, which will likely perform different operations for different types, to take an Any value, a mutator should take as narrow a scope as possible.
Finally, even though type hints are not and probably should never be mandatory, all of the most popular python IDEs such as Pycharm automatically support type hints. They will often give warnings even when another programmer may not be annotating types, but the type can be safely inferred. This means that even when using a library with types hints, mutators with an int annotation, will still be more informative and useful to the end-user than an Any annotation.
I'd like to be able to say
def f(param) -> type(param): return param
but I get the NameError: name 'param' is not defined. Key thing here is that the return type is a function of a function parameter. I have glanced through the https://www.python.org/dev/peps/pep-3107/, but I don't see any precise description of what comprises a valid annotation expression.
I would accept an answer which explains why exactly is this not possible at the moment, i.e., does it not fit into current annotation paradigm or is there a technical problem with this?
There are a few issues with the type(param) method.
First off, as Oleh mentioned in his answer, all annotations must be valid at the time of the function's definition. In an example like yours, you could potentially have problems due to variable shadowing.
param = 10
def f(param) -> type(param):
return param
f('a')
Since the variable param is of type int, the function's annotation is essentially read as f(param: Any) -> int. So when you pass in the argument param with the value 'a', which means f will return a str, this makes it inconsistent with the annotation. Admittedly this example is contrived, but from a language design stand point, it is something to be careful.
Instead, as jonrsharpe mentioned, often the best way to reference the generic types of parameters (as jonrsharpe) mentioned is with type variables.
This can be done using the typing.TypeVar class.
from typing import TypeVar
def f(param: T) -> T:
return param
This means that static checkers won't need to actually access the type of param, just check that at check-time that there is a way to consider both param and the return value of the same type. I say consider the same type because you will sometimes only assert that they both implement the same abstract base class/interface, like numbers.Real.
And then can use typevars in generic types
from typing import List, TypeVar
T = TypeVar('T')
def total(items: List[T]) -> List[T]:
return [f(item) for item in items]
Using type variables and generics can be better because it adds additional information and allows for a little bit more flexibility (as explained in the example with numbers.Real). For instance, the ability to use List[T] is really important. In your case of using type(param), it would only return list, not list of like List[T] would. So using type(param) would actually lose information, not add it.
Therefore, it is a better idea to stick to using type variables and generic types instead.
TL;DR:
Due to variable shadowing, type(param) could lead to inconsistent annotations.
Since sometimes when thinking of the types of your system you are thinking in terms of interfaces (abstract base classes in Python) instead of concrete types, it can be better to rely on ABC's and type variables
Using type(param) could lose information that would be provided by generics.
Let's take a glance at PEP-484 - Type Hints # Acceptable type hints.
Annotations must be valid expressions that evaluate without raising exceptions at the time the function is defined (but see below for forward references).
Annotations should be kept simple or static analysis tools may not be able to interpret the values. For example, dynamically computed types are unlikely to be understood. (This is an intentionally somewhat vague requirement, specific inclusions and exclusions may be added to future versions of this PEP as warranted by the discussion.)
I'd say that your approach is quite interesting and may be useful for static analysis. But if we accept PEPs as a source of an explanation for the current annotation paradigm, the highlighted text explains why return type can't be defined dynamically at the time the function is called.