I was looking at the PEP 484 section on Forward References and noticed the statement:
...that definition may be expressed as a string literal, to be resolved later.
And that got me wondering, when is "later" and by what? The interpreter doesn't try to resolve it as a literal later, so what does? Is it just if a third party tool is written to do that?
Small example to demonstrate the interpreter result:
class A:
def test(self, a: 'A') -> None:
pass
class B:
def test(self, a: A) -> None:
pass
>>> A().test.__annotations__
{'a': 'A', 'return': None}
>>> B().test.__annotations__
{'a': <class '__main__.A'>, 'return': None}
If my understanding of function annotations and type hints is correct, Python doesn't really do anything with them at runtime to improve performance, but rather the introspective use allows strictly third party applications such as linters, IDEs and static analysis tools (such as mypy) to take advantage of their availability. So would those tools try to resolve the type hint of 'A' rather than having that be a job given to the interpreter and if so, how do they accomplish this?
Update:
By using the typing module, user code can perform the following:
>>> typing.get_type_hints(A().test)
{'a': <class '__main__.A'>, 'return': <class 'NoneType'>}
>>> typing.get_type_hints(B().test)
{'a': <class '__main__.A'>, 'return': <class 'NoneType'>}
However, my question is aimed at whether or not Python has any responsibility in updating the __annotations__ of a function from a string literal, that is to say at runtime change:
>>> A().test.__annotations__
{'a': 'A', 'return': None}
to...
>>> A().test.__annotations__
{'a': <class '__main__.A'>, 'return': None}
If Python doesn't do it, then why would I want a string literal as a type hint other than for self-documented code? What value does the first form give to me, a user or a third party tool?
Consider the following code:
class Foo:
def bar(self) -> Foo:
return Foo()
This program will actually crash at runtime if you try running it with Python: when the interpreter sees the definition of bar, the definition of Foo is not yet finished. So, since Foo has not yet been added to the global namespace, we can't use it as a type hint yet.
Similarly, consider this program:
class Foo:
def bar(self) -> Bar:
return Bar()
class Bar:
def foo(self) -> Foo:
return Foo()
This mutually dependent definition suffers from the same problem: while we're evaluating Foo, Bar hasn't been evaluated yet so the interpreter throws an exception.
There are three solutions to this problem. The first is to make some of your type hints strings, effectively "forward declaring" them:
class Foo:
def bar(self) -> "Foo":
return Foo()
This satisfies the Python interpreter, and won't disrupt third party tools like mypy: they can just remove the quotes before parsing the type. The main disadvantage is that this syntax looks sort of ugly and clunky.
The second solution is to use type comments syntax:
class Foo:
def bar(self):
# type: () -> Foo
return Foo()
This has the same benefits and disadvantages as the first solution: it satisfies the interpreter and tooling, but looks hacky and ugly. It also has the additional benefit that it keeps your code backwards-compatibile with Python 2.7.
The third solution is Python 3.7+ only -- use the from __future__ import annotations directive:
from __future__ import annotations
class Foo:
def bar(self) -> Foo:
return Foo()
This will automatically make all annotations be represented as strings. So we get the benefit of the first solution, but without the ugliness.
This behavior will eventually become the default in future versions of Python.
It also turns out that automatically making all annotations strings can come with some performance improvements. Constructing types like List[Dict[str, int]] can be surprisingly expensive: they're just regular expressions at runtime and evaluated as if they were written as List.__getitem__(Dict.__getitem__((str, int)).
Evaluating this expression is somewhat expensive: we end up performing two method calls, constructing a tuple, and constructing two objects. This isn't counting any additional work that happens in the __getitem__ methods themselves, of course -- and the work that happens in those methods ends up being non-trivial out of necessity.
(In short, they need to construct special objects that ensure types like List[int] can't be used in inappropriate ways at runtime -- e.g. in isinstance checks and the like.)
Related
I'm having trouble expressing the precise type hints for a pair of parameters where one needs to be a value that is an instance of some type, and the other needs to be the type itself, or some super-type.
One case where you see this in Python is in the __exit__ method for a context manager (class).
import typing as t
from types import TracebackType
from contextlib import AbstractContextManager
class Acme(AbstractContextManager):
def __exit__(self, exc_type: t.Type[Exception], exc: Exception,
tb: Tracebacktype) -> None:
...
This specific case may not matter, because context managers are handled internally by Python, but I'd still like to understand how to express two arguments where the first one is the (super-)type of the second one.
Conceptually, my problem is I want to express that the value exc has the type exc_type or some subtype. In the expression above, I guess mypy would be perfectly happy with arguments like LookupError, RuntimeError('foo') even though a RuntimeError isn't a type of LookupError. Is there a better way to express this where mypy would catch that kind of mistake?
Update
Trying a test case here using TypeVars:
import typing as t
C = t.TypeVar('C', bound=t.Collection)
def f(t: t.Type[C], c: C) -> None:
pass
f(t.Dict, [])
I'd expect mypy to complain about this code, because even though the empty list is a type of Collection, it's not a dictionary.
This kind of use of Type[some_type_var] is seemingly discouraged by a seciton in the PEP where only a subset of this support is provided (notably for binding TypeVars to class types). It appears this discouragement is not phillisophical (this would be a common use of generics in a language where types and values are both first-class) - but more related to the practical implementation of type checkers. The use of TypeVar for classmethod signatures - a special case of your question - was even a late addition to the pep.
Possible solutions that do not work with mypy (as of now):
Standard TypeVar
There is no type checking in function signatures of Type[some_type_var]. This is not variable based on if some_type_var is contravariant or covariant (nor should it be).
TypeVar of Type[TypeVar]
T = TypeVar("T")
Meta = TypeVar("Meta", bound=Type[T])
def foo(the_type: Meta, the_value: T):
....
T and Meta can not "bind" together .
To my surprise, in some cases you can squeeze out a bit of this behavior (but this is a special case and this seems undefined to me)
T = TypeVar('T', str, int, float, bytes)
class Foo:
def bar(self, the_type: Type[T], the_value: T):
print(isinstance(the_value, the_type))
f = Foo()
f.bar(str, "baz")
f.bar(str, b"baz") # wrong!
f.bar(int, 1)
f.bar(int, 3.15159) # wrong! (but no complaint)
f.bar(float, 1.0)
f.bar(float, 1) # wrong! (but no complaint)
f.bar(float, b'1.0') # wrong!
Giving
so.py:nn: error: Value of type variable "T" of "bar" of "Foo" cannot be "object"
so.py:nn: error: Value of type variable "T" of "bar" of "Foo" cannot be "object"
Found 2 errors in 1 file (checked 1 source file)
But only seemingly for python's primitive types (this will not work with user-space types) (and again, only some of python's primitive types as shown (see the error-miss with float and int). I think this is a clear "nice-to-have" extension of TypeVar (and makes more Generic use-cases possible).
There are some related issues on mypy about this:
about class annotation behavior
TypeVar of a TypeVar
This is what TypeVars are for! You want something like:
from types import TracebackType
from typing import Type, TypeVar
from contextlib import AbstractContextManager
_E = TypeVar('_E', bound=Exception)
class Acme(AbstractContextManager):
def __exit__(
self,
exc_type: Type[_E],
exc: _E,
tb: Tracebacktype
) -> None:
...
What are the drawbacks or benefits to using types.FunctionType vs typing.Callable as a type-hint annotation?
Consider the following code...
import types
import typing
def functionA(func: types.FunctionType):
rt = func()
print(func.__name__)
return rt
def functionB(func: typing.Callable):
rt = func()
print(func.__name__)
return rt
The only difference I can see is Callable could be any sort of callable object (function, method, class, etc) while FunctionType is limited to only functions.
Am I overlooking something? Is there a benefit to using FunctionType over Callable in certain situations?
The types module predates PEP 484 annotations and was created mostly to make runtime introspection of objects easier. For example, to determine if some value is a function, you can run isinstance(my_var, types.FunctionType).
The typing module contains type hints that are specifically intended to assist static analysis tools such as mypy. For example, suppose you want to indicate that a parameter must be a function that accepts two ints and returns a str. You can do so like this:
def handle(f: Callable[[int, int], str]) -> None: ...
There is no way to use FunctionType in a similar manner: it simply was not designed for this purpose.
This function signature is also more flexible: it can also accept things like objects with a __call__ since such objects are indeed callable.
The contents of the typing module can also sometimes be used for runtime checks in a manner similar to the contents of types as a convenience: for example, doing isinstance(f, Callable) works. However, this functionality is deliberately limited: doing isinstance(f, Callable[[int, int], str]) is intentionally disallowed. Attempting to perform that check will raise an exception at runtime.
That said, I don't think it's a good style to perform runtime checks using anything from typing: the typing module is meant first and foremost for static analysis.
I would not use anything from the types module within type hints for similar reasons. The only exception is if your function is written in a way such that it's critical that the value you need to receive is specifically an instance of FunctionType, rather than being any arbitrary callable.
I was reading Ian Goodfellow's GAN source code in Github (link https://github.com/goodfeli/adversarial/blob/master/deconv.py). In particular, at line 40/41, the code is:
#functools.wraps(Model.get_lr_scalers)
def get_lr_scalers(self):
It's a rather unfamiliar way of using wraps, and it seems the goal is to replace the get_lr_scalers with a user defined function. But in that case, we don't really need a wrapper for that, right? I don't really know the purpose of wraps in this case.
wraps copies a number of attributes from another function onto this function—by default, __module__, __name__, __qualname__, __annotations__ and __doc__.
The most obviously useful one to copy over is the __doc__. Consider this simpler example:1
class Base:
def spam(self, breakfast):
"""spam(self, breakfast) -> breakfast with added spam
<29 lines of detailed information here>
"""
class Child:
#functools.wraps(Base.spam)
def spam(self, breakfast):
newbreakfast = breakfast.copy()
newbreakfast.meats['spam'] + 30
return newbreakfast
Now if someone wants to use help(mychild.spam), they'll get the 29 lines of useful information. (Or, if they autocomplete mychild.spam in PyCharm, it'll pop up the overlay with the documentation, etc.) All without me having to manually copy and paste it. And, even better, if Base came from some framework that I didn't write, and my user upgrades from 1.2.3 to 1.2.4 of that framework, and there's a better docstring, they'll see that better docstring.
In the most common case, Child would be a subclass of Base, and spam would be an override.2 But that isn't actually required—wraps doesn't care whether you're subtyping via inheritance, or duck typing by just implementing an implicit protocol; it's equally useful for both cases. As long as Child is intended to implement the spam protocol from Base, it makes sense for Child.spam to have the same docstring (and maybe other metadata attributes).
Others attributes probably aren't quite as useful as docstrings. For example, if you're using type annotations, their benefit in reading the code is probably at least as high as their benefit in being able to run Mypy for static type checking, so just copying them over dynamically from another method often isn't all that useful. And __module__ and __qualname__ are primarily used for reflection/inspection, and are more likely to be misleading than helpful in this case (although you could probably come up with an example of a framework where you'd want people to read the code in Base instead of the code in Child, that isn't true for the default obvious example). But, unless they're actively harmful, the readability cost of using #functools.wraps(Base.spam, assigned=('__doc__',)) instead of just the defaults may not be worth it.
1. If you're using Python 2, change these classes to inherit from object; otherwise they'll be old-style classes, which just complicates things in an irrelevant way. If Python 3, there are no old-style classes, so this issue can't even arise.
2. Or maybe a "virtual subclass" of an ABC, declared via a register call, or via a subclass hook.
The purpose of #wraps is to copy meta information of one function to another function. This is usually done when replacing the original function by wrapping it, which is often done by decorators.
But in general case, here is what it does in an example:
def f1():
"""Function named f1. Prints 'f1'."""
print('f1')
#functools.wraps(f1)
def f2():
print('f2')
Now, you can test what happened:
>>> f1
<function f1 at 0x006AD8E8>
>>> f2
<function f1 at 0x006AD978>
>>> f1()
f1
>>> f2()
f2
>>> f1.__doc__
"Function named f1. Prints 'f1'."
>>> f2.__doc__
"Function named f1. Prints 'f1'."
When you call f2, it is obvious that it is actually f2, but when you inspect it, it behaves like f1 - it has the same doc string and the same name.
What is that good for? For this:
f1 = f2
Now the original f1 is replaced with a new functionality, but it still looks like f1 from the outside.
It is usually done in a decorator:
def replace(func):
#functools.wraps(func)
def replacement():
print('replacement')
return replacement
#replace
def f1():
"""Function named f1. Prints 'f1'."""
print('f1')
And it behaves like this:
>>> f1()
replacement
>>> f1
<function f1 at 0x006AD930>
>>> f1.__name__
'f1'
>>> f1.__doc__
"Function named f1. Prints 'f1'."
This question already has answers here:
How do I type hint a method with the type of the enclosing class?
(7 answers)
Closed 4 years ago.
In python 3 I can make arguments and return type annotations. Example:
class Graph:
def __init__(self, V: int, E: int, edges: list):
pass
#classmethod
def fromfile(cls, readobj: type(sys.stdin)):
pass
def V(self) -> int:
pass
def E(self) -> int:
pass
The problem is I can't make an annotation with return type of the current class (Graph), which is not defined yet.
Example:
class Graph:
def reverse(self) -> Graph:
pass
This code goes with error
def reverse(self) -> Graph:
NameError: name 'Graph' is not defined
These annotations are really useful both for documenting and allowing IDE to recognize argument and return types => enable autocomplete
UPD:
So what I came up is this is either impossible or requires some hacks I don't like, so I decided to use just def reverse (self) -> 'Graph':
which is understandable for documentation although breaks the rule. The downside is that it doesn't work for IDE autocomplete.
In python-3.7 this issue has been resolved by not evaluating the annotations at function definition time. Instead, they are preserved in __annotations__ in string form. This is called Postponed Evaluation of Annotations, introduced in PEP 563.
Also note:
Deprecation policy
Starting with Python 3.7, a __future__ import is required to use the
described functionality. No warnings are raised.
In Python 3.8 a PendingDeprecationWarning is raised by the compiler in
the presence of type annotations in modules without the __future__
import.
Starting with Python 3.9 the warning becomes a DeprecationWarning.
In Python 4.0 this will become the default behavior. Use of
annotations incompatible with this PEP is no longer supported.
Here is an example:
In [7]: from __future__ import annotations
In [8]: class C:
...: def func(cls, arg:str) -> C:
...: pass
...:
In [9]: c = C()
So now after a while I can say that decision I took was using -> 'Graph' instead of -> Graph. It does not make my IDE (PyCharm) able to recognize a type this way but it just works well enough for documentation purposes.
Another possible solution I could use was changing annotation at runtime but that doesn't solve the problem with documentation - you won't want to look for type declarations somewhere in the middle of sources...
The problem has roots in recognizing class object before the class was actually defined. That is simply impossible to do in python.
In python I can create a class without class statement:
MyClass = type('X', (object,), dict(a=1))
Is there a way to create a function without 'def'?
Thats as far as i got...
d={} # func from string
exec'''\
def synthetics(s):
return s*s+1
''' in d
>>> d.keys()
['__builtins__', 'synthetics']
>>> d['synthetics']
<function synthetics at 0x00D09E70>
>>> foo = d['synthetics']
>>> foo(1)
2
Technically, yes, this is possible. The type of a function is, like all other types, a constructor for instances of that type:
FunctionType = type(lambda: 0)
help(FunctionType)
As you can see from the help, you need at minimum code and globals. The former is a compiled bytecode object; the latter is a dictionary.
To make the code object, you can use the code type's constructor:
CodeType = type((lambda: 0).func_code)
help(CodeType)
The help says this is "not for the faint of heart" and that's true. You need to pass bytecode and a bunch of other stuff to this constructor. So the easiest way to get a code object is from another function, or using the compile() function. But it is technically possible to generate code objects completely synthetically if you understand Python bytecode well enough. (I have done this, on a very limited basis, to construct signature-preserving wrapper functions for use in decorators.)
PS -- FunctionType and CodeType are also available via the types module.
There might be a more direct way than the following, but here's a full-blown function without def. First, use a trivial lambda expression to get a function object:
>>> func = lambda: None
Then, compile some source code to get a code object and use that to replace the lambda's code:
>>> func.__code__ = compile("print('Hello, world!')", "<no file>", "exec")
>>> func()
Hello, world!