Develop Reference

Type hints with map like function?

Suppose I've got a map like function:
def generate(data, per_element):
for element in data:
per_element(element)
How can I add type-hints so that if I call generate(some_data, some_function) where some_data: List[SomeClass], I get a warning if SomeClass is missing a field used by some_function?
As an example - with the following code:
def some_function(x):
print(x.value)
some_data: List[int] = [1, 2, 3]
generate(some_data, some_function)
I would like to get a warning that int does not have the attribute value.

Use a type variable to make generate generic in the type of object that data contains and that per_element expects as an argument.
from typing import TypeVar, List, Callable
T = TypeVar('T')
def generate(data: List[T], per_element: Callable[[T], Any]):
for element in data:
per_element(element)
class Foo:
def __init__(self):
self.value = 3
def foo(x: Foo):
print(x.value)
def bar(x: int):
pass
generate([Foo(), Foo()], foo) # OK
# Argument 2 to "generate" has incompatible type "Callable[[Foo], Any]"; expected "Callable[[int], Any]"
generate([1,2,3], foo)
Whatever T is, it has to be the same type for both the list and the function, to ensure that per_element can, in fact, be called on every value in data. The error produced by the second call to generate isn't exactly what you asked for, but it essentially catches the same problem: the list establishes what type T is bound to, and the function doesn't accept the correct type.
If you specifically want to require that T be a type whose instances have a value attribute, it's a bit trickier. It's similar to the use case for Protocol, but that only supports methods (or class attributes in general?), not instance attributes, as far as I know. Perhaps someone else can provide a better answer.

Seems like you're searching for:
def generate(data: List[AClass], per_element):
for element in data:
per_element(element)
So that AClass implements the method you need.

Your class needs the value attribute:
class SomeClass:
value: Any # I used any but use whatever type hint is appropriate
Then using typing.Callable in your function as well as the builtin types. starting with python 3.7 and finally fully implemented in python 3.9 you can use the builtins themselves as well as in python 3.9 you can use parameter specifications
from typing import ParamSpec, TypeVar, Callable
P = ParamSpec("P")
R = TypeVar("R")
def generate(data: list[SomeClass], per_element: Callable[P, R]) -> None:
for element in data:
per_element(element)
Then in some_function using the class type hint and None return variable:
def some_function(x: SomeClass) -> None:
print(x.value)

Generic function parameters in Python

I'm trying to write a Python function that constructs a list with intercepted methods that's reasonably type safe. It intercepts the methods by subclassing the list that's passed.
from typing import Type, TypeVar, List
V = TypeVar("V")
T = TypeVar("T", bound=List[V])
def build_interceptor(list_cls: Type[T]) -> T:
class LImpl(list_cls):
def append(self, v: V) -> None:
print(v)
super().append(v)
return LImpl()
l: List[int] = build_interceptor(List[int])
l.append(10)
MyPy isn't happy with this, but the code does work.
main.py:4: error: Type variable "__main__.V" is unbound
main.py:4: note: (Hint: Use "Generic[V]" or "Protocol[V]" base class to bind "V" inside a class)
main.py:4: note: (Hint: Use "V" in function signature to bind "V" inside a function)
main.py:8: error: Variable "list_cls" is not valid as a type
main.py:8: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
main.py:8: error: Invalid base class "list_cls"
I'm not sure what the fixes are. Yes, V is unbound, but I don't really care what it is beyond getting the right return type. I also think there's an issue with making both the list and its contents generic, but I'm not sure how to express that.

I think the problem with 'V' is that it cant be used in the context of a TypeVar, but when defining your new class:
from typing import List, Type, TypeVar
T = TypeVar("T", bound="List")
V = TypeVar("V")
def build_interceptor(list_cls: Type[T]) -> T:
class LImpl(list_cls[V]): # type: ignore
def append(self, v: V) -> None:
print(v)
super().append(v)
return LImpl()
l: List[int] = build_interceptor(List[int])
l.append(10)
This still produces 'Variable "list_cls" is not valid as a type' which is likely related to mypy.
It seems to work after adding a type ignore comment.

Generic type interplay with Union type

I use a Generic class in python, whose instances are little but wrappers around instances of other types:
import random
from typing import Generic
from typing import TypeVar
from typing import Union
T = TypeVar("T", covariant=True)
class Wrapper(Generic[T]):
def __init__(self, value: T) -> None:
self._value = value
#property
def unwrap(self) -> T:
return self._value
def test_union_wrapper() -> None:
def wrapper_union() -> Wrapper[Union[str, int]]:
if random.random() >= 0.5:
return Wrapper("foo")
else:
return Wrapper(42)
# mypy will give an error for this line
w_u: Union[Wrapper[str], Wrapper[int]] = wrapper_union()
Running mypy on the above code will result in:
error: Incompatible types in assignment (expression has type "Wrapper[Union[str, int]]", variable has type "Union[Wrapper[str], Wrapper[int]]")
This might seem reasonable, because
Wrapper[Union[str, int]] ≮: Wrapper[str], and
Wrapper[Union[str, int]] ≮: Wrapper[int]
and, as can be read in PEP483:
Union behaves covariantly in all its arguments. Indeed, as discussed above, Union[t1, t2, ...] is a subtype of Union[u1, u2, ...], if t1 is a subtype of u1, etc.
But I take issue with it, because I know that ∀w ∈ Wrapper[Union[str, int]], w ∈ Wrapper[str] or w ∈ Wrapper[int], hence, Wrapper[Union[str, int]] <: Union[Wrapper[str], Wrapper[int]], regardless. I would like to get mypy to recognize the same fact, but I don't know how.
There is even an example of such a recognition using the standard library. If I replace Wrapper with Type - another covariant Generic - in the code above, we get:
def test_union_type() -> None:
def type_union() -> Type[Union[str, int]]:
if random.random() >= 0.5:
return str
else:
return int
# mypy has no problem with this
w_u: Union[Type[str], Type[int]] = type_union()
Here, mypy recognizes that the function return type, Type[Union[str, int]] is equivalent to the variable type, Union[Type[str], Type[int]].
I thus have two questions:
How do I tell a type checker that Wrapper behaves as Type does with regards to Union?
Does this behavior with regards to Union have a name? If we treat Wrapper and Unions as functions on types, we could say that they commute with each other, but not sure what the right term is in the context of type theory, or Python specifically.

I found an unsatisfying solution to Q1, using a mypy pluging with the following type analyzer hook:
def commute_with_union_hook(ctx: AnalyzeTypeContext) -> Type:
sym = ctx.api.lookup_qualified(ctx.type.name, ctx.type)
node = sym.node
if not ctx.type.args and len(ctx.type.args) == 1:
return ctx.api.analyze_type_with_type_info(node, ctx.type.args, ctx.type)
t: Type = ctx.type.args[0]
t = ctx.api.anal_type(t)
if not isinstance(t, UnionType):
wrapper = ctx.api.analyze_type_with_type_info(node, ctx.type.args, ctx.type)
return wrapper
else:
union_wrapper = UnionType.make_union(
items=[Instance(typ=node, args=[item]) for item in t.items]
)
return union_wrapper
Feels overkill, but it might be the only way.

Python building complex mypy types

In a perfect world, I could just do this:
ScoreBaseType = Union[bool, int, float]
ScoreComplexType = Union[ScoreBaseType, Dict[str, ScoreBaseType]]
But, that says a ScoreComplexType is either a ScoreBaseType or a dictionary which allows multiple types of values... not what I want.
The following looks like it should work to me, but it doesn't:
ScoreBaseTypeList = [bool, int, float]
ScoreBaseType = Union[*ScoreBaseTypeList] # pycharm says "can't use starred expression here"
ScoreDictType = reduce(lambda lhs,rhs: Union[lhs, rhs], map(lambda x: Dict[str, x], ScoreBaseTypeList))
ScoreComplexType = Union[ScoreBaseType, ScoreDictType]
Is there any way I can do something like the above without having to go through this tedium?
ScoreComplexType = Union[bool, int, float,
                     Dict[str, bool],
                     Dict[str, int],
                     Dict[str, float]]
Edit: More fleshed out desired usage example:
# these strings are completely arbitrary and determined at runtime. Used as keys in nested dictionaries.
CatalogStr = NewType('CatalogStr', str)
DatasetStr = NewType('DatasetStr', str)
ScoreTypeStr = NewType('ScoreTypeStr', str)
ScoreBaseType = Union[bool, int, float]
ScoreDictType = Dict[ScoreTypeStr, 'ScoreBaseTypeVar']
ScoreComplexType = Union['ScoreBaseTypeVar', ScoreDictType]
ScoreBaseTypeVar = TypeVar('ScoreBaseTypeVar', bound=ScoreBaseType)
ScoreComplexTypeVar = TypeVar('ScoreComplexTypeVar', bound=ScoreComplexType) # errors: "constraints cannot be parameterized by type variables"
class EvalBase(ABC, Generic[ScoreComplexTypeVar]):
def __init__(self) -> None:
self.scores: Dict[CatalogStr,
Dict[DatasetStr,
ScoreComplexTypeVar]
] = {}
class EvalExample(EvalBase[Dict[float]]): # can't do this either
...
Edit 2:
It occurs to me that I could simplify a LOT of my type hinting if I used tuples instead of nested dictionaries. This seems to maybe work? I've only tried it in the below toy example and haven't yet tried adapting all my code.
# These are used to make typing hints easier to understand
CatalogStr = NewType('CatalogStr', str) # A str corresponding to the name of a catalog
DatasetStr = NewType('DatasetStr', str) # A str corresponding to the name of a dataset
ScoreTypeStr = NewType('ScoreTypeStr', str) # A str corresponding to the label for a ScoreType
ScoreBaseType = Union[bool, int, float]
SimpleScoreDictKey = Tuple[CatalogStr, DatasetStr]
ComplexScoreDictKey = Tuple[CatalogStr, DatasetStr, ScoreTypeStr]
ScoreKey = Union[SimpleScoreDictKey, ComplexScoreDictKey]
ScoreKeyTypeVar = TypeVar('ScoreKeyTypeVar', bound=ScoreKey)
ScoreDictType = Dict[ScoreKey, ScoreBaseType]
# These are used for Generics in classes
DatasetTypeVar = TypeVar('DatasetTypeVar', bound='Dataset') # Must match a type inherited from Dataset
ScoreBaseTypeVar = TypeVar('ScoreBaseTypeVar', bound=ScoreBaseType)
class EvalBase(ABC, Generic[ScoreBaseTypeVar, ScoreKeyTypeVar]):
def __init__(self):
self.score: ScoreDictType = {}
class EvalExample(EvalBase[float, ComplexScoreDictKey]):
...
Although then what would the equivalent of this be? Seems like I might have to store a couple lists of keys in order to iterate?
for catalog_name in self.catalog_list:
for dataset_name in self.scores[catalog_name]:
for score in self.scores[catalog_name][dataset_name]:

You may need to use TypeVars to express this, but without an example of how you intend to use it, it's hard to say.
An example of how this would be used for typing a return value dependent on input:
ScoreBaseType = Union[bool, int, float]
ScoreTypeVar = TypeVar('ScoreTypeVar', bound=ScoreBaseType)
ScoreDictType = Union[ScoreTypeVar, Dict[str, ScoreTypeVar]]
def scoring_func(Iterable[ScoreTypeVar]) -> ScoreDictType:
...
If you're not doing this based on input values though, you probably want
ScoreBaseType = Union[bool, int, float]
ScoreDictTypes = Union[Dict[str, bool], Dict[str, int], Dict[str, float]]
ScoreComplexType = Union[ScoreBaseType, ScoreDictTypes]
Depending on how you are handling the types, you may also be able to use SupportsInt or SupportsFloat types rather than both int and float
Edit: (Additional Info Based on the edited OP below)
Since you are typing an ABC with this, it may be sufficient to type the base class using Dict[str, Any] and constrain subclasses further.
If it isn't, you are going to have very verbose type definitions, and there isn't much alternative, as mypy currently has some issues resolving some classes of programmatically generated types, even when operating on constants.
mypy also doesn't have support for recursive type aliases at this time (though there is a potential of support for them being added, it's not currently planned), so for readability, you'd need to define the allowed types for each potential level of nesting, and then collect those into a type representing the full nested structure.

mypy error - incompatible type despite using 'Union'

Consider following code sample:
from typing import Dict, Union
def count_chars(string) -> Dict[str, Union[str, bool, int]]:
result = {} # type: Dict[str, Union[str, bool, int]]
if isinstance(string, str) is False:
result["success"] = False
result["message"] = "Inavlid argument"
else:
result["success"] = True
result["result"] = len(string)
return result
def get_square(integer: int) -> int:
return integer * integer
def validate_str(string: str) -> bool:
check_count = count_chars(string)
if check_count["success"] is False:
print(check_count["message"])
return False
str_len_square = get_square(check_count["result"])
return bool(str_len_square > 42)
result = validate_str("Lorem ipsum")
When running mypy against this code, following error is returned:
error: Argument 1 to "get_square" has incompatible type "Union[str, bool, int]"; expected "int"
and I'm not sure how I could avoid this error without using Dict[str, Any] as returned type in the first function or installing 'TypedDict' mypy extension. Is mypy actually 'right', any my code isn't type safe or is this should be considered as mypy bug?

Mypy is correct here -- if the values in your dict can be strs, ints, or bools, then strictly speaking we can't assume check_count["result"] will always evaluate to exactly an int.
You have a few ways of resolving this. The first way is to actually just check the type of check_count["result"] to see if it's an int. You can do this using an assert:
assert isinstance(check_count["result"], int)
str_len_square = get_square(check_count["result"])
...or perhaps an if statement:
if isinstance(check_count["result"], int):
str_len_square = get_square(check_count["result"])
else:
# Throw some kind of exception here?
Mypy understands type checks of this form in asserts and if statements (to a limited extent).
However, it can get tedious scattering these checks throughout your code. So, it might be best to actually just give up on using dicts and switch to using classes.
That is, define a class:
class Result:
def __init__(self, success: bool, message: str) -> None:
self.success = success
self.message = message
...and return an instance of that instead.
This is slightly more inconvenient in that if your goal is to ultimately return/manipulate json, you now need to write code to convert this class from/to json, but it does let you avoid type-related errors.
Defining a custom class can get slightly tedious, so you can try using the NamedTuple type instead:
from typing import NamedTuple
Result = NamedTuple('Result', [('success', bool), ('message', str)])
# Use Result as a regular class
You still need to write the tuple -> json code, and iirc namedtuples (both the regular version from the collections module and this typed variant) are less performant then classes, but perhaps that doesn't matter for your use case.