I recently ran into Ellipsis (...) that is used in function parameters in aiohttp code and then in that function's body:
def make_mocked_request(method, path, headers=None, *,
match_info=sentinel,
version=HttpVersion(1, 1), closing=False,
app=None,
writer=sentinel,
protocol=sentinel,
transport=sentinel,
payload=sentinel,
sslcontext=None,
client_max_size=1024**2,
loop=...):
"""Creates mocked web.Request testing purposes.
Useful in unit tests, when spinning full web server is overkill or
specific conditions and errors are hard to trigger.
"""
task = mock.Mock()
if loop is ...:
loop = mock.Mock()
loop.create_future.return_value = ()
Can you explain this new python 3 feature?
Ellipsis is a Built-in constant in Python. In Python 3 it has a literal syntax ... so it can be used like any other literal. This was accepted by Guido for Python 3 because some folks thought it would be cute.
The code you have found (use as function argument default) is apparently one such "cute" usage. Later in that code you'll see:
if loop is ...:
loop = mock.Mock()
loop.create_future.return_value = ()
It's just being used as a sentinel, here, and may as well be object() or anything else - there is nothing specific to Ellipsis. Perhaps the usual sentinel, None, has some other specific meaning in this context, although I can not see any evidence of that in the commit (it seems like None will have worked just as well).
Another use-case for an ellipsis literal sometimes seen in the wild is a placeholder for code not written yet, similar to pass statement:
class Todo:
...
For the more typical use-cases, which are involving the extended slice syntax, see What does the Python Ellipsis object do?
Building on https://stackoverflow.com/a/54406084/1988486, using ... vs. None has some validity when dealing with type annotations. Consider the following:
def foo(map: dict = None):
if not map:
map = {}
return map
Granted, this doesn't do much, but this is causing type hints to conflict in a modern editor such as VS Code. This is due to the fact that you're claiming that map is a dict and yet the default is a NoneType (not a dict).
The Ellipsis is its own type, so VS Code will not complain, but you can interact with it in similar ways because it is a sentinel value.
def food(map: dict = ...):
if map is ...:
map = {}
return map
VS Code will not raise any concerns about the type hinting and the various types this variable can become in the course of running the code.
I can't say for sure why the aiohttp code uses it since they don't appear to be using type annotations, but it's at least a viable reason to do so.
I run into the following two issues at the same time fairly frequently
I have a function with an argument that's expected to be a container of strings
I would like to simplify calls by passing the function either a container of strings or a single string that is not a singleton list
I usually handle this with something like the following, which seems somehow not pythonic to me (I don't know why). Is there a more pythonic way to handle the situation? Is this actually a bad habit and it's most appropriate to require function calls like my_func(['just_deal_with_it'])?
Note the function iterable_not_string below is from sorin's answer to another question
from collections.abc import Iterable
def iterable_not_string(x):
is_iter = isinstance(x, Iterable)
is_not_str = (not isinstance(x, str))
return (is_iter and is_not_str)
def my_func(list_or_string):
if iterable_not_string(list_or_string):
do_stuff(list_or_string)
else:
do_stuff([list_or_string])
I use the following idiom, which works with any flexibly typed language:
def my_func(arg):
"""arg can be a list, or a single string"""
if isinstance(arg, str):
arg = [ arg ]
# the rest of the code can just treat `arg` as a list
do_stuff_with_a_list(arg)
By normalizing the argument to a list at the start, you avoid code duplication and type-checking later... and the attendant bugs if you forget a check.
Another options is to accept arbitrary arguments
def my_func(*strings):
do_stuff(strings)
my_func('string')
l = ['list', 'of', 'strings']
my_func(*l)
However, I advise to only do this, if the amount of elements is expected to be small, since the unpacking of the iterable may take some time and consume a lot of memory (i.e. on long generators).
You can do this with the python library typing
Typing provids type hinting for python
For example:
from typing import List, Union
def my_func(list_or_string: Union[List, str]):
...
Python 3.10 provides a cleaner approach to this:
from typing import List
def my_func(list_or_string: List | str):
...
In this case Union is replaced by the | (pipe)
I'd like to be able to say
def f(param) -> type(param): return param
but I get the NameError: name 'param' is not defined. Key thing here is that the return type is a function of a function parameter. I have glanced through the https://www.python.org/dev/peps/pep-3107/, but I don't see any precise description of what comprises a valid annotation expression.
I would accept an answer which explains why exactly is this not possible at the moment, i.e., does it not fit into current annotation paradigm or is there a technical problem with this?
There are a few issues with the type(param) method.
First off, as Oleh mentioned in his answer, all annotations must be valid at the time of the function's definition. In an example like yours, you could potentially have problems due to variable shadowing.
param = 10
def f(param) -> type(param):
return param
f('a')
Since the variable param is of type int, the function's annotation is essentially read as f(param: Any) -> int. So when you pass in the argument param with the value 'a', which means f will return a str, this makes it inconsistent with the annotation. Admittedly this example is contrived, but from a language design stand point, it is something to be careful.
Instead, as jonrsharpe mentioned, often the best way to reference the generic types of parameters (as jonrsharpe) mentioned is with type variables.
This can be done using the typing.TypeVar class.
from typing import TypeVar
def f(param: T) -> T:
return param
This means that static checkers won't need to actually access the type of param, just check that at check-time that there is a way to consider both param and the return value of the same type. I say consider the same type because you will sometimes only assert that they both implement the same abstract base class/interface, like numbers.Real.
And then can use typevars in generic types
from typing import List, TypeVar
T = TypeVar('T')
def total(items: List[T]) -> List[T]:
return [f(item) for item in items]
Using type variables and generics can be better because it adds additional information and allows for a little bit more flexibility (as explained in the example with numbers.Real). For instance, the ability to use List[T] is really important. In your case of using type(param), it would only return list, not list of like List[T] would. So using type(param) would actually lose information, not add it.
Therefore, it is a better idea to stick to using type variables and generic types instead.
TL;DR:
Due to variable shadowing, type(param) could lead to inconsistent annotations.
Since sometimes when thinking of the types of your system you are thinking in terms of interfaces (abstract base classes in Python) instead of concrete types, it can be better to rely on ABC's and type variables
Using type(param) could lose information that would be provided by generics.
Let's take a glance at PEP-484 - Type Hints # Acceptable type hints.
Annotations must be valid expressions that evaluate without raising exceptions at the time the function is defined (but see below for forward references).
Annotations should be kept simple or static analysis tools may not be able to interpret the values. For example, dynamically computed types are unlikely to be understood. (This is an intentionally somewhat vague requirement, specific inclusions and exclusions may be added to future versions of this PEP as warranted by the discussion.)
I'd say that your approach is quite interesting and may be useful for static analysis. But if we accept PEPs as a source of an explanation for the current annotation paradigm, the highlighted text explains why return type can't be defined dynamically at the time the function is called.
I use the following format to document my Python code:
def my_function(param: str) -> dict:
some code
I can't figure out how to document a function passed to another function.
For example:
def my_function(my_other_function: ???) -> dict:
some code
How do I make a function annotation?
First Thoughts: "Everything in python is an object"
I couldn't find anything in the docs, but as everything in python is an object i would shoot for object.
def my_function(my_other_function: object) -> dict:
some code
To proof it:
if isinstance(my_function, my_function, object):
print("yes")
>yes
Anyhow, this might not be too explicit, therefore:
Seconds thoughts: Using proper type hints
Based on what COLDSPEED commented, a more explicit type hint would be using typing
import typing
def my_function(my_other_function:typing.Callable):->dict:
pass
"The only way that annotations take on meaning is when they are interpreted by third-party libraries". Which means, for your source-code itself, it doesn't change anything. Just wanted to mention it.
Function Annotations: PEP-3107
I ran across a snippet of code demonstrating Python3's function annotations. The concept is simple but I can't think of why these were implemented in Python3 or any good uses for them. Perhaps SO can enlighten me?
How it works:
def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9):
... function body ...
Everything following the colon after an argument is an 'annotation', and the information following the -> is an annotation for the function's return value.
foo.func_annotations would return a dictionary:
{'a': 'x',
'b': 11,
'c': list,
'return': 9}
What's the significance of having this available?
Function annotations are what you make of them.
They can be used for documentation:
def kinetic_energy(mass: 'in kilograms', velocity: 'in meters per second'):
...
They can be used for pre-condition checking:
def validate(func, locals):
for var, test in func.__annotations__.items():
value = locals[var]
msg = 'Var: {0}\tValue: {1}\tTest: {2.__name__}'.format(var, value, test)
assert test(value), msg
def is_int(x):
return isinstance(x, int)
def between(lo, hi):
def _between(x):
return lo <= x <= hi
return _between
def f(x: between(3, 10), y: is_int):
validate(f, locals())
print(x, y)
>>> f(0, 31.1)
Traceback (most recent call last):
...
AssertionError: Var: y Value: 31.1 Test: is_int
Also see http://www.python.org/dev/peps/pep-0362/ for a way to implement type checking.
I think this is actually great.
Coming from an academic background, I can tell you that annotations have proved themselves invaluable for enabling smart static analyzers for languages like Java. For instance, you could define semantics like state restrictions, threads that are allowed to access, architecture limitations, etc., and there are quite a few tools that can then read these and process them to provide assurances beyond what you get from the compilers. You could even write things that check preconditions/postconditions.
I feel something like this is especially needed in Python because of its weaker typing, but there were really no constructs that made this straightforward and part of the official syntax.
There are other uses for annotations beyond assurance. I can see how I could apply my Java-based tools to Python. For instance, I have a tool that lets you assign special warnings to methods, and gives you indications when you call them that you should read their documentation (E.g., imagine you have a method that must not be invoked with a negative value, but it's not intuitive from the name). With annotations, I could technically write something like this for Python. Similarly, a tool that organizes methods in a large class based on tags can be written if there is an official syntax.
This is a way late answer, but AFAICT, the best current use of function annotations is PEP-0484 and MyPy. There's also PyRight from Microsoft which is used by VSCode and also available via CLI.
Mypy is an optional static type checker for Python. You can add type hints to your Python programs using the upcoming standard for type annotations introduced in Python 3.5 beta 1 (PEP 484), and use mypy to type check them statically.
Used like so:
from typing import Iterator
def fib(n: int) -> Iterator[int]:
a, b = 0, 1
while a < n:
yield a
a, b = b, a + b
Just to add a specific example of a good use from my answer here, coupled with decorators a simple mechanism for multimethods can be done.
# This is in the 'mm' module
registry = {}
import inspect
class MultiMethod(object):
def __init__(self, name):
self.name = name
self.typemap = {}
def __call__(self, *args):
types = tuple(arg.__class__ for arg in args) # a generator expression!
function = self.typemap.get(types)
if function is None:
raise TypeError("no match")
return function(*args)
def register(self, types, function):
if types in self.typemap:
raise TypeError("duplicate registration")
self.typemap[types] = function
def multimethod(function):
name = function.__name__
mm = registry.get(name)
if mm is None:
mm = registry[name] = MultiMethod(name)
spec = inspect.getfullargspec(function)
types = tuple(spec.annotations[x] for x in spec.args)
mm.register(types, function)
return mm
and an example of use:
from mm import multimethod
#multimethod
def foo(a: int):
return "an int"
#multimethod
def foo(a: int, b: str):
return "an int and a string"
if __name__ == '__main__':
print("foo(1,'a') = {}".format(foo(1,'a')))
print("foo(7) = {}".format(foo(7)))
This can be done by adding the types to the decorator as Guido's original post shows, but annotating the parameters themselves is better as it avoids the possibility of wrong matching of parameters and types.
Note: In Python you can access the annotations as function.__annotations__ rather than function.func_annotations as the func_* style was removed on Python 3.
Uri has already given a proper answer, so here's a less serious one: So you can make your docstrings shorter.
The first time I saw annotations, I thought "great! Finally I can opt in to some type checking!" Of course, I hadn't noticed that annotations are not actually enforced.
So I decided to write a simple function decorator to enforce them:
def ensure_annotations(f):
from functools import wraps
from inspect import getcallargs
#wraps(f)
def wrapper(*args, **kwargs):
for arg, val in getcallargs(f, *args, **kwargs).items():
if arg in f.__annotations__:
templ = f.__annotations__[arg]
msg = "Argument {arg} to {f} does not match annotation type {t}"
Check(val).is_a(templ).or_raise(EnsureError, msg.format(arg=arg, f=f, t=templ))
return_val = f(*args, **kwargs)
if 'return' in f.__annotations__:
templ = f.__annotations__['return']
msg = "Return value of {f} does not match annotation type {t}"
Check(return_val).is_a(templ).or_raise(EnsureError, msg.format(f=f, t=templ))
return return_val
return wrapper
#ensure_annotations
def f(x: int, y: float) -> float:
return x+y
print(f(1, y=2.2))
>>> 3.2
print(f(1, y=2))
>>> ensure.EnsureError: Argument y to <function f at 0x109b7c710> does not match annotation type <class 'float'>
I added it to the Ensure library.
It a long time since this was asked but the example snippet given in the question is (as stated there as well) from PEP 3107 and at the end of thas PEP example Use cases are also given which might answer the question from the PEPs point of view ;)
The following is quoted from PEP3107
Use Cases
In the course of discussing annotations, a number of use-cases have been raised. Some of these are presented here, grouped by what kind of information they convey. Also included are examples of existing products and packages that could make use of annotations.
Providing typing information
Type checking ([3], [4])
Let IDEs show what types a function expects and returns ([17])
Function overloading / generic functions ([22])
Foreign-language bridges ([18], [19])
Adaptation ([21], [20])
Predicate logic functions
Database query mapping
RPC parameter marshaling ([23])
Other information
Documentation for parameters and return values ([24])
See the PEP for more information on specific points (as well as their references)
Python 3.X (only) also generalizes function definition to allow
arguments and return values to be annotated with object values
for use in extensions.
Its META-data to explain, to be more explicit about the function values.
Annotations are coded as :value after the
argument name and before a default, and as ->value after the
argument list.
They are collected into an __annotations__ attribute of the function, but are not otherwise treated as special by Python itself:
>>> def f(a:99, b:'spam'=None) -> float:
... print(a, b)
...
>>> f(88)
88 None
>>> f.__annotations__
{'a': 99, 'b': 'spam', 'return': <class 'float'>}
Source: Python Pocket Reference, Fifth Edition
EXAMPLE:
The typeannotations module provides a set of tools for type checking and type inference of Python code. It also a provides a set of types useful for annotating functions and objects.
These tools are mainly designed to be used by static analyzers such as linters, code completion libraries and IDEs. Additionally, decorators for making run-time checks are provided. Run-time type checking is not always a good idea in Python, but in some cases it can be very useful.
https://github.com/ceronman/typeannotations
How Typing Helps to Write Better Code
Typing can help you do static code analysis to catch type errors
before you send your code to production and prevent you from some
obvious bugs. There are tools like mypy, which you can add to your
toolbox as part of your software life cycle. mypy can check for
correct types by running against your codebase partially or fully.
mypy also helps you to detect bugs such as checking for the None type
when the value is returned from a function. Typing helps to make your
code cleaner. Instead of documenting your code using comments, where
you specify types in a docstring, you can use types without any
performance cost.
Clean Python: Elegant Coding in Python
ISBN: ISBN-13 (pbk): 978-1-4842-4877-5
PEP 526 -- Syntax for Variable Annotations
https://www.python.org/dev/peps/pep-0526/
https://www.attrs.org/en/stable/types.html
Despite all uses described here, the one enforceable and, most likely, enforced use of annotations will be for type hints.
This is currently not enforced in any way but, judging from PEP 484, future versions of Python will only allow types as the value for annotations.
Quoting What about existing uses of annotations?:
We do hope that type hints will eventually become the sole use for annotations, but this will require additional discussion and a deprecation period after the initial roll-out of the typing module with Python 3.5. The current PEP will have provisional status (see PEP 411 ) until Python 3.6 is released. The fastest conceivable scheme would introduce silent deprecation of non-type-hint annotations in 3.6, full deprecation in 3.7, and declare type hints as the only allowed use of annotations in Python 3.8.
Though I haven't seen any silent deprecations in 3.6 yet, this could very well be bumped to 3.7, instead.
So, even though there might be some other good use-cases, it is best to keep them solely for type hinting if you don't want to go around changing everything in a future where this restriction is in place.
As a bit of a delayed answer, several of my packages (marrow.script, WebCore, etc.) use annotations where available to declare typecasting (i.e. transforming incoming values from the web, detecting which arguments are boolean switches, etc.) as well as to perform additional markup of arguments.
Marrow Script builds a complete command-line interface to arbitrary functions and classes and allows for defining documentation, casting, and callback-derived default values via annotations, with a decorator to support older runtimes. All of my libraries that use annotations support the forms:
any_string # documentation
any_callable # typecast / callback, not called if defaulting
(any_callable, any_string) # combination
AnnotationClass() # package-specific rich annotation object
[AnnotationClass(), AnnotationClass(), …] # cooperative annotation
"Bare" support for docstrings or typecasting functions allows for easier mixing with other libraries that are annotation-aware. (I.e. have a web controller using typecasting that also happens to be exposed as a command-line script.)
Edited to add: I've also begun making use of the TypeGuard package using development-time assertions for validation. Benefit: when run with "optimizations" enabled (-O / PYTHONOPTIMIZE env var) the checks, which may be expensive (e.g. recursive) are omitted, with the idea that you've properly tested your app in development so the checks should be unnecessary in production.
Annotations can be used for easily modularizing code. E.g. a module for a program which I'm maintaining could just define a method like:
def run(param1: int):
"""
Does things.
:param param1: Needed for counting.
"""
pass
and we could ask the user for a thing named "param1" which is "Needed for counting" and should be an "int". In the end we can even convert the string given by the user to the desired type to get the most hassle free experience.
See our function metadata object for an open source class which helps with this and can automatically retrieve needed values and convert them to any desired type (because the annotation is a conversion method). Even IDEs show autocompletions right and assume that types are according to annotations - a perfect fit.
If you look at the list of benefits of Cython, a major one is the ability to tell the compiler which type a Python object is.
I can envision a future where Cython (or similar tools that compile some of your Python code) will use the annotation syntax to do their magic.