I use a Generic class in python, whose instances are little but wrappers around instances of other types:
import random
from typing import Generic
from typing import TypeVar
from typing import Union
T = TypeVar("T", covariant=True)
class Wrapper(Generic[T]):
def __init__(self, value: T) -> None:
self._value = value
#property
def unwrap(self) -> T:
return self._value
def test_union_wrapper() -> None:
def wrapper_union() -> Wrapper[Union[str, int]]:
if random.random() >= 0.5:
return Wrapper("foo")
else:
return Wrapper(42)
# mypy will give an error for this line
w_u: Union[Wrapper[str], Wrapper[int]] = wrapper_union()
Running mypy on the above code will result in:
error: Incompatible types in assignment (expression has type "Wrapper[Union[str, int]]", variable has type "Union[Wrapper[str], Wrapper[int]]")
This might seem reasonable, because
Wrapper[Union[str, int]] ≮: Wrapper[str], and
Wrapper[Union[str, int]] ≮: Wrapper[int]
and, as can be read in PEP483:
Union behaves covariantly in all its arguments. Indeed, as discussed above, Union[t1, t2, ...] is a subtype of Union[u1, u2, ...], if t1 is a subtype of u1, etc.
But I take issue with it, because I know that ∀w ∈ Wrapper[Union[str, int]], w ∈ Wrapper[str] or w ∈ Wrapper[int], hence, Wrapper[Union[str, int]] <: Union[Wrapper[str], Wrapper[int]], regardless. I would like to get mypy to recognize the same fact, but I don't know how.
There is even an example of such a recognition using the standard library. If I replace Wrapper with Type - another covariant Generic - in the code above, we get:
def test_union_type() -> None:
def type_union() -> Type[Union[str, int]]:
if random.random() >= 0.5:
return str
else:
return int
# mypy has no problem with this
w_u: Union[Type[str], Type[int]] = type_union()
Here, mypy recognizes that the function return type, Type[Union[str, int]] is equivalent to the variable type, Union[Type[str], Type[int]].
I thus have two questions:
How do I tell a type checker that Wrapper behaves as Type does with regards to Union?
Does this behavior with regards to Union have a name? If we treat Wrapper and Unions as functions on types, we could say that they commute with each other, but not sure what the right term is in the context of type theory, or Python specifically.
I found an unsatisfying solution to Q1, using a mypy pluging with the following type analyzer hook:
def commute_with_union_hook(ctx: AnalyzeTypeContext) -> Type:
sym = ctx.api.lookup_qualified(ctx.type.name, ctx.type)
node = sym.node
if not ctx.type.args and len(ctx.type.args) == 1:
return ctx.api.analyze_type_with_type_info(node, ctx.type.args, ctx.type)
t: Type = ctx.type.args[0]
t = ctx.api.anal_type(t)
if not isinstance(t, UnionType):
wrapper = ctx.api.analyze_type_with_type_info(node, ctx.type.args, ctx.type)
return wrapper
else:
union_wrapper = UnionType.make_union(
items=[Instance(typ=node, args=[item]) for item in t.items]
)
return union_wrapper
Feels overkill, but it might be the only way.
Related
Similar to Python type hint Callable with one known positional type and then *args and **kwargs, I want to type hint a Callable for which is known:
It must have at least 1 positional input.
The first positional input must be int.
It must return None.
Apart from that, any signature is valid. I tried to do the following, but it doesn't work. So, is it possible to do it in python 3.10/3.11 at all?
from typing import TypeAlias, ParamSpec, Concatenate, Callable
P = ParamSpec("P")
intfun: TypeAlias = Callable[Concatenate[int, P], None]
def foo(i: int) -> None:
pass
a: intfun = foo # ✘ Incompatible types in assignment
# expression has type "Callable[[int], None]",
# variable has type "Callable[[int, VarArg(Any), KwArg(Any)], None]")
https://mypy-play.net/?mypy=latest&python=3.11&gist=f4c26907bfc0ae0118b90c1fa5a79fe8
I am using mypy==1.0.0.
Context: I want to type hint a dict hat holds key-value pairs where the value could be any Callable satisfying properties 1,2,3.
The somewhat cryptic notation used by mypy for the type you annotated a with indicates that it does not implicitly specify the generic Callable you defined to intfun[...] (literal ellipses here), when you omit type arguments. As pointed out here, there is a case to be made for mypy to simply assume that like it assumes an Any argument for "normal" generics.
But that is beside the point here because when you are dealing with generics you should treat them as such. Essentially the P is the type variable here, so whenever you want to use intfun for annotation, you should either specify it or bind another type variable, depending on the context.
In this case, you can just do a: intfun[[]] = foo and the code will pass the type check. That annotation specifies intfun to callables that have no other parameters (besides the mandatory int). Thus the following will cause an error in the last line:
from typing import TypeAlias, ParamSpec, Concatenate, Callable
P = ParamSpec("P")
intfun: TypeAlias = Callable[Concatenate[int, P], None]
def foo(i: int) -> None:
pass
def bar(i: int, s: str) -> None:
pass
a: intfun[[]] = foo # passes
a = bar # error
Depending on the context, in which you want to use that type alias, you may want to provide a different type argument. Here is another example, where you use intfun in its generic form:
from typing import TypeAlias, ParamSpec, Concatenate, Callable
P = ParamSpec("P")
intfun: TypeAlias = Callable[Concatenate[int, P], None]
def foo(i: int) -> None:
pass
def baz(f: intfun[P]) -> Callable[Concatenate[int, P], int]:
def wrapper(i: int, /, *args: P.args, **kwargs: P.kwargs) -> int:
f(i, *args, **kwargs)
return i
return wrapper
fooz = baz(foo) # passes
I have a generic lookup function, that mostly returns TypeA, but sometimes can return TypeB:
Types = Union[TypeA,TypeB]
def get_hashed_value(
key:str, table: Dict[str,Types]
) -> Types:
return table.get(key)
and I use it in two less-generic functions:
def get_valueA(key: str) -> TypeA:
return get_hashed_value(key, A_dict) # A_dict: Dict[str, TypeA]
and
def get_valueB(key: str) -> TypeB:
return get_hashed_value(key, B_dict) # B_dict: Dict[str, TypeB]
what is the best way to handle typing on this?
since get_hashed_value can return either TypeA or TypeB, the return statement in the get_* functions throws a typing exception (during my linting)
there’s more logic in these methods, and I need the separate get_* functions, so I can’t just collapse all the usages
it would be really nice to have explicit return types on the get_* functions
it feels like a bad practice to duplicate get_hashed_value, just to get around the typing issue
it feels bad to just ignore type everything get_hashed_value is called
Thanks for your help! Also I am sure this has been asked before, but I had trouble finding the answer. :\
Interestingly, this doesn't return a type warning for me (in Pycharm). I'm not sure why it isn't warning on what's comparable to a "downcast", but Pycharm isn't perfect.
Regardless, this seems like a job that's more suited for a TypeVar than a Union:
from typing import TypeVar, Dict
T = TypeVar("T", TypeA, TypeB) # A generic type that can only be a TypeA or TypeB
# And the T stays consistent from the input to the output
def get_hashed_value(key: str, table: Dict[str, T]) -> T:
return table.get(key)
# Which means that if you feed it a Dict[str, TypeA], it will expect a TypeA return
def get_valueA(key: str) -> TypeA:
return get_hashed_value(key, A_dict)
# And if you feed it a Dict[str, TypeB], it will expect an TypeB return
def get_valueB(key: str) -> TypeB:
return get_hashed_value(key, B_dict)
Suppose I have function that takes type as argument and returns instance of that type:
def fun(t):
return t(42)
Then I can call it and get objects of provided types:
fun(int) # 42
fun(float) # 42.0
fun(complex) # (42+0j)
fun(str) # "42"
fun(MyCustomType) # something
That list is not exhaustive, I'd like to be able to use any type with appropriate constructor.
Then, I'd like to add type hints for that function. What should be the type hint for return value of that function?
I've tried using simply t, as t is a type:
def fun(t: type) -> t:
return t(42)
but that doesn't work:
main.py:1: error: Name 't' is not defined
This answer suggests using a TypeVar:
from typing import TypeVar
T = TypeVar("T")
def fun(t: T) -> T:
return t(42)
But that doesn't seem to be right, as T denotes a type, so it suggests that type itself is returned, not its instance. Mypy rejects it:
main.py:6: error: "object" not callable
Using Any obviously work, but I feel it's too vague, it doesn't convey the intent:
from typing import Any
def fun(t: type) -> Any:
return t(42)
TLDR: You need a TypeVar for the return type of calling t:
def fun(t: Callable[[int], R]) -> R:
...
Constraining on a type is too restrictive here. The function accepts any Callable that takes an integer, and the return type of the function is that of the Callable. This can be specified using a TypeVar for the return type:
from typing import Callable, TypeVar
R = TypeVar('R') # the variable return type
def fun(t: Callable[[int], R]) -> R:
return t(42)
fun(int) # Revealed type is 'builtins.int*'
fun(float) # Revealed type is 'builtins.float*'
reveal_type(fun(lambda x: str(x))) # Revealed type is 'builtins.str*'
This works for types as well, because type instantiation is a call.
If a more complex signature, e.g. with keyword arguments, is needed, use Protocol (from typing or typing_extensions).
Note that if one explicitly wants to pass only 42 to the Callable, Literal (from typing or typing_extensions) can be used to specify that.
R = TypeVar('R')
def fun(t: Callable[[Literal[42]], R]) -> R:
return t(42)
Note that any function of the type Callable[[int], R] also satisfies Callable[[Literal[42]], R].
You are looking for typing.Type, so something to the effect of:
from typing import TypeVar, Type
T = TypeVar("T", str, complex, float, int)
def fun(t: Type[T]) -> T:
return t(42)
fun(int)
fun(float)
fun(complex)
fun(str)
Note, your type variable needs to be constrained, because not all Type objects accept arguments, but you can constrain it to a few that do like your example.
In the following example, how can I properly annotate the return type of the sum_two function?
from typing import Any, TypeVar
T = TypeVar('T')
S = TypeVar('S')
def sum_two(first: T, second: S):
return first + second
Assuming the __add__ operator is properly annotated for all possible arguments that will be passed to this function, is there some way to express the return type as the return type of calling __add__ on objects of type T and S?
I would like to avoid using typing's overload decorator to identify all possible cases as there could be dozens of cases.
You can theoretically accomplish a part of his by making first a generic protocol, which lets you "capture" the return type of __add__. For example:
# If you are using Python 3.7 or earlier, you'll need to pip-install
# the typing_extensions module and import Protocol from there.
from typing import TypeVar, Protocol, Generic
TOther = TypeVar('TOther', contravariant=True)
TSum = TypeVar('TSum', covariant=True)
class SupportsAdd(Protocol, Generic[TOther, TSum]):
def __add__(self, other: TOther) -> TSum: ...
Then, you could do the following:
S = TypeVar('S')
R = TypeVar('R')
# Due to how we defined the protocol, R will correspond to the
# return type of `__add__`.
def sum_two(first: SupportsAdd[S, R], second: S) -> R:
return first + second
# Type checks
reveal_type(sum_two("foo", "bar")) # Revealed type is str
reveal_type(sum_two(1, 2)) # Revealed type is int
reveal_type(sum_two(1.0, 2)) # Revealed type is float
# Does not type check, since float's __radd__ is ignored
sum_two(1, 2.0)
class Custom:
def __add__(self, x: int) -> int:
return x
# Type checks
reveal_type(sum_two(Custom(), 3)) # Revealed type is int
# Does not type check
reveal_type(sum_two(Custom(), "bad"))
This approach does have a few limitations, however:
It does not handle cases where there's no matching __add__ in 'first' but do have a matching __radd__ in 'second'.
You might get some weird results if you modify Custom so __add__ is an overload. I think at least mypy currently has a bug where it doesn't know how to handle complicated cases involving subtypes and overloads properly.
Consider following code sample:
from typing import Dict, Union
def count_chars(string) -> Dict[str, Union[str, bool, int]]:
result = {} # type: Dict[str, Union[str, bool, int]]
if isinstance(string, str) is False:
result["success"] = False
result["message"] = "Inavlid argument"
else:
result["success"] = True
result["result"] = len(string)
return result
def get_square(integer: int) -> int:
return integer * integer
def validate_str(string: str) -> bool:
check_count = count_chars(string)
if check_count["success"] is False:
print(check_count["message"])
return False
str_len_square = get_square(check_count["result"])
return bool(str_len_square > 42)
result = validate_str("Lorem ipsum")
When running mypy against this code, following error is returned:
error: Argument 1 to "get_square" has incompatible type "Union[str, bool, int]"; expected "int"
and I'm not sure how I could avoid this error without using Dict[str, Any] as returned type in the first function or installing 'TypedDict' mypy extension. Is mypy actually 'right', any my code isn't type safe or is this should be considered as mypy bug?
Mypy is correct here -- if the values in your dict can be strs, ints, or bools, then strictly speaking we can't assume check_count["result"] will always evaluate to exactly an int.
You have a few ways of resolving this. The first way is to actually just check the type of check_count["result"] to see if it's an int. You can do this using an assert:
assert isinstance(check_count["result"], int)
str_len_square = get_square(check_count["result"])
...or perhaps an if statement:
if isinstance(check_count["result"], int):
str_len_square = get_square(check_count["result"])
else:
# Throw some kind of exception here?
Mypy understands type checks of this form in asserts and if statements (to a limited extent).
However, it can get tedious scattering these checks throughout your code. So, it might be best to actually just give up on using dicts and switch to using classes.
That is, define a class:
class Result:
def __init__(self, success: bool, message: str) -> None:
self.success = success
self.message = message
...and return an instance of that instead.
This is slightly more inconvenient in that if your goal is to ultimately return/manipulate json, you now need to write code to convert this class from/to json, but it does let you avoid type-related errors.
Defining a custom class can get slightly tedious, so you can try using the NamedTuple type instead:
from typing import NamedTuple
Result = NamedTuple('Result', [('success', bool), ('message', str)])
# Use Result as a regular class
You still need to write the tuple -> json code, and iirc namedtuples (both the regular version from the collections module and this typed variant) are less performant then classes, but perhaps that doesn't matter for your use case.