I have an inherited class member that is an optional callable and I want to type hint it.
import typing
class BytesDecoder(typing.Protocol):
def __call__(self,data:bytes)->None: ...
class BaseClass:
_decodeBytes:typing.Optional[BytesDecoder]=None
#typing.final
def decode(self,data:bytes):
if self._decodeBytes is not None:
self._decodeBytes(data)
class DerivedClass(BaseClass):
def _decodeBytes(self,data:bytes)->None:
...
Mypy complains about the derived _decodeBytes method:
Signature of "_decodeBytes" incompatible with supertype "BaseClass"
I've also tried defining BytesDecoder like BytesDecoder=typing.Callable[[bytes],None] but that does the same thing.
I stumbled upon one solution, though it's not ideal. In the base class I can create a dummy _decodeBytes() function and then forcibly smash it back to None.
class BaseClass:
def _decodeBytes(self,data:bytes)->None: ...
_decodeBytes=None # type: ignore
It's weird and non-intuitive, but it does seem to make type checking work as expected for derived classes.
A better solution may be to make _decodeBytes required, but provide a dummy implementation that behaves the same as not calling it at all.
import typing
class BaseClass:
def _decodeBytes(self, data: bytes) -> None:
pass
#typing.final
def decode(self,data:bytes):
self._decodeBytes(data)
class DerivedClass(BaseClass):
def _decodeBytes(self,data:bytes)->None:
...
The original problem was that mypy couldn't verify something like the following was safe:
decoders: list[BaseClass] = [BaseClass(), DerivedClass()]
for x in decoders:
x._decodeBytes = None
The assignment is legal according to the static type of decoders and for the runtime type of decoders[0], but illegal for the runtime type of decoders[1]. Getting rid of the Optional status of _decodeBytes takes away the possibility of assigning None to any object's _decodeBytes attribute.
Related
Context
Say we want to define a custom generic (base) class that inherits from typing.Generic.
For the sake of simplicity, we want it to be parameterized by a single type variable T. So the class definition starts like this:
from typing import Generic, TypeVar
T = TypeVar("T")
class GenericBase(Generic[T]):
...
Question
Is there a way to access the type argument T in any specific subclass of GenericBase?
The solution should be universal enough to work in a subclass with additional bases besides GenericBase and be independent of instantiation (i.e. work on the class level).
The desired outcome is a class-method like this:
class GenericBase(Generic[T]):
#classmethod
def get_type_arg(cls) -> Type[T]:
...
Usage
class Foo:
pass
class Bar:
pass
class Specific(Foo, GenericBase[str], Bar):
pass
print(Specific.get_type_arg())
The output should be <class 'str'>.
Bonus
It would be nice if all relevant type annotations were made such that static type checkers could correctly infer the specific class returned by get_type_arg.
Related questions
Generic[T] base class - how to get type of T from within instance? - This question focuses on direct instances of the custom generic class itself, not on specified subclasses.
How can I access T from a Generic[T] instance early in its lifecycle? - This is a variation on the previous one.
How to access the type arguments of typing.Generic? - This is very close, but does not cover the possibility of other base classes.
TL;DR
Grab the GenericBase from the subclass' __orig_bases__ tuple, pass it to typing.get_args, grab the first element from the tuple it returns, and make sure what you have is a concrete type.
1) Starting with get_args
As pointed out in this post, the typing module for Python 3.8+ provides the get_args function. It is convenient because given a specialization of a generic type, get_args returns its type arguments (as a tuple).
Demonstration:
from typing import Generic, TypeVar, get_args
T = TypeVar("T")
class GenericBase(Generic[T]):
pass
print(get_args(GenericBase[int]))
Output:
(<class 'int'>,)
This means that once we have access to a specialized GenericBase type, we can easily extract its type argument.
2) Continuing with __orig_bases__
As further pointed out in the aforementioned post, there is this handy little class attribute __orig_bases__ that is set by the type metaclass when a new class is created. It is mentioned here in PEP 560, but is otherwise hardly documented.
This attribute contains (as the name suggests) the original bases as they were passed to the metaclass constructor in the form of a tuple. This distinguishes it from __bases__, which contains the already resolved bases as returned by types.resolve_bases.
Demonstration:
from typing import Generic, TypeVar
T = TypeVar("T")
class GenericBase(Generic[T]):
pass
class Specific(GenericBase[int]):
pass
print(Specific.__bases__)
print(Specific.__orig_bases__)
Output:
(<class '__main__.GenericBase'>,)
(__main__.GenericBase[int],)
We are interested in the original base because that is the specialization of our generic class, meaning it is the one that "knows" about the type argument (int in this example), whereas the resolved base class is just an instance of type.
3) Simplistic solution
If we put these two together, we can quickly construct a simplistic solution like this:
from typing import Generic, TypeVar, get_args
T = TypeVar("T")
class GenericBase(Generic[T]):
#classmethod
def get_type_arg_simple(cls):
return get_args(cls.__orig_bases__[0])[0]
class Specific(GenericBase[int]):
pass
print(Specific.get_type_arg_simple())
Output:
<class 'int'>
But this will break as soon as we introduce another base class on top of our GenericBase.
from typing import Generic, TypeVar, get_args
T = TypeVar("T")
class GenericBase(Generic[T]):
#classmethod
def get_type_arg_simple(cls):
return get_args(cls.__orig_bases__[0])[0]
class Mixin:
pass
class Specific(Mixin, GenericBase[int]):
pass
print(Specific.get_type_arg_simple())
Output:
Traceback (most recent call last):
...
return get_args(cls.__orig_bases__[0])[0]
IndexError: tuple index out of range
This happens because cls.__orig_bases__[0] now happens to be Mixin, which is not a parameterized type, so get_args returns an empty tuple ().
So what we need is a way to unambiguously identify the GenericBase from the __orig_bases__ tuple.
4) Identifying with get_origin
Just like typing.get_args gives us the type arguments for a generic type, typing.get_origin gives us the unspecified version of a generic type.
Demonstration:
from typing import Generic, TypeVar, get_origin
T = TypeVar("T")
class GenericBase(Generic[T]):
pass
print(get_origin(GenericBase[int]))
print(get_origin(GenericBase[str]) is GenericBase)
Output:
<class '__main__.GenericBase'>
True
5) Putting them together
With these components, we can now write a function get_type_arg that takes a class as an argument and -- if that class is specialized form of our GenericBase -- returns its type argument:
from typing import Generic, TypeVar, get_origin, get_args
T = TypeVar("T")
class GenericBase(Generic[T]):
pass
class Specific(GenericBase[int]):
pass
def get_type_arg(cls):
for base in cls.__orig_bases__:
origin = get_origin(base)
if origin is None or not issubclass(origin, GenericBase):
continue
return get_args(base)[0]
print(get_type_arg(Specific))
Output:
<class 'int'>
Now all that is left to do is embed this directly as a class-method of GenericBase, optimize it a little bit and fix the type annotations.
One thing we can do to optimize this, is only run this algorithm only once for any given subclass of GenericBase, namely when it is defined, and then save the type in a class-attribute. Since the type argument presumably never changes for a specific class, there is no need to compute this every time we want to access the type argument. To accomplish this, we can hook into __init_subclass__ and do our loop there.
We should also define a proper response for when get_type_arg is called on a (unspecified) generic class. An AttributeError seems appropriate.
6) Full working example
from typing import Any, Generic, Optional, Type, TypeVar, get_args, get_origin
# The `GenericBase` must be parameterized with exactly one type variable.
T = TypeVar("T")
class GenericBase(Generic[T]):
_type_arg: Optional[Type[T]] = None # set in specified subclasses
#classmethod
def __init_subclass__(cls, **kwargs: Any) -> None:
"""
Initializes a subclass of `GenericBase`.
Identifies the specified `GenericBase` among all base classes and
saves the provided type argument in the `_type_arg` class attribute
"""
super().__init_subclass__(**kwargs)
for base in cls.__orig_bases__: # type: ignore[attr-defined]
origin = get_origin(base)
if origin is None or not issubclass(origin, GenericBase):
continue
type_arg = get_args(base)[0]
# Do not set the attribute for GENERIC subclasses!
if not isinstance(type_arg, TypeVar):
cls._type_arg = type_arg
return
#classmethod
def get_type_arg(cls) -> Type[T]:
if cls._type_arg is None:
raise AttributeError(
f"{cls.__name__} is generic; type argument unspecified"
)
return cls._type_arg
def demo_a() -> None:
class SpecificA(GenericBase[int]):
pass
print(SpecificA.get_type_arg())
def demo_b() -> None:
class Foo:
pass
class Bar:
pass
class GenericSubclass(GenericBase[T]):
pass
class SpecificB(Foo, GenericSubclass[str], Bar):
pass
type_b = SpecificB.get_type_arg()
print(type_b)
e = type_b.lower("E") # static type checkers correctly infer `str` type
assert e == "e"
if __name__ == '__main__':
demo_a()
demo_b()
Output:
<class 'int'>
<class 'str'>
An IDE like PyCharm even provides the correct auto-suggestions for whatever type is returned by get_type_arg, which is really nice. 🎉
7) Caveats
The __orig_bases__ attribute is not well documented. I am not sure it should be considered entirely stable. Although it doesn't appear to be "just an implementation detail" either. I would suggest keeping an eye on that.
mypy seems to agree with this caution and raises a no attribute error in the place where you access __orig_bases__. Thus a type: ignore was placed in that line.
The entire setup is for one single type parameter for our generic class. It can be adapted relatively easily to multiple parameters, though annotations for type checkers might become more tricky.
This method does not work when called directly from a specialized GenericBase class, i.e. GenericBase[str].get_type_arg(). But for that one just needs to call typing.get_args on it as shown in the very beginning.
I have a use case where I would like to define a common function in a base class, but annotate the return type from an overridden property in a subclass. This is a silly example, but clearly demonstrates what I would like to do:
from typing import Any
class BaseCls:
RETURNS_TYPE: Any = NotImplemented
# This is what I am not sure how to do, this does not work
# Have tried a few different things, but hoping someone just knows
def cast_to_return_type(self: 'BaseCls', value: Any) -> 'BaseCls.RETURNS_TYPE':
return some_work_that_returns_self_RETURNS_TYPE()
class IntCls(BaseCls):
RETURNS_TYPE = int
class StrCls(BaseCls):
RETURNS_TYPE = str
How can I type hint the return value of cast_to_return_type so that a a regular old IDE would be able to detect what the return type should be when doing StrCls().cast_to_return_type(some_var) (tried on pycharm and vscode, neither knew what to do any decided Any was the return type, which makes sense if it cant infer the child class value)
The closest thing that typing has to type variables is typing.TypeVar. Their use is to create a generic class, in the same way that you write list[int] or Mapping[str, str].
So we can rewrite your code so that BaseCls is a generic type, with a type variable ReturnsType. (You should only use all caps for constants.)
The subclasses IntCls and StrCls "assign" to the type variable by including the type inside square braces after the type they are subclassing.
from typing import TypeVar, Generic, Any
ReturnsType = TypeVar('ReturnsType')
class BaseCls(Generic[ReturnsType]):
def cast_to_return_type(self, value: Any) -> ReturnsType:
return some_work_that_returns_self_RETURNS_TYPE()
class IntCls(BaseCls[int]):
pass
class StrCls(BaseCls[str]):
pass
I am writing a CustomEnum class in which I want to add some helper methods, that would then be available by the classes subclassing my CustomEnum. One of the methods is to return a random enum value, and this is where I am stuck. The function works as expected, but on the type-hinting side, I cannot figure out a way of saying "the return type is the same type of cls".
I am fairly sure there's some TypeVar or similar magic involved, but since I never had to use them I never took the time to figure them out.
class CustomEnum(Enum):
#classmethod
def random(cls) -> ???:
return random.choice(list(cls))
class SubclassingEnum(CustomEnum):
A = "a"
B = "b"
random_subclassing_enum: SubclassingEnum
random_subclassing_enum = SubclassingEnum.random() # Incompatible types in assignment (expression has type "CustomEnum", variable has type "SubclassingEnum")
Can somebody help me or give me a hint on how to proceed?
Thanks!
The syntax here is kind of horrible, but I don't think there's a cleaner way to do this. The following passes MyPy:
from typing import TypeVar
from enum import Enum
import random
T = TypeVar("T", bound="CustomEnum")
class CustomEnum(Enum):
#classmethod
def random(cls: type[T]) -> T:
return random.choice(list(cls))
(In python versions <= 3.8, you have to use typing.Type rather than the builtin type if you want to parameterise it.)
What's going on here?
T is defined at the top as being a type variable that is "bound" to the CustomEnum class. This means that a variable annotated with T can only be an instance of CustomEnum or an instance of a class inheriting from CustomEnum.
In the classmethod above, we're actually using this type-variable to define the type of the cls parameter with respect to the return type. Usually we do the opposite — we usually define a function's return types with respect to the types of that function's input parameters. So it's understandable if this feels a little mind-bending!
We're saying: this method leads to instances of a class — we don't know what the class will be, but we know it will either be CustomEnum or a class inheriting from CustomEnum. We also know that whatever class is returned, we can guarantee that the type of the cls parameter in the function will be "one level up" in the type heirarchy from the type of the return value.
In a lot of situations, we might know that type[cls] will always be a fixed value. In those situations, it would be possible to hardcode that into the type annotations. However, it's best not to do so, and instead to use this method, which clearly shows the relationship between the type of the input and the return type (even if it uses horrible syntax to do so!).
Further reading: the MyPy documentation on the type of class objects.
Further explanation and examples
For the vast majority of classes (not with Enums, they use metaclasses, but let's leave that aside for the moment), the following will hold true:
Example 1
Class A:
pass
instance_of_a = A()
type(instance_of_a) == A # True
type(A) == type # True
Example 2
class B:
pass
instance_of_b = B()
type(instance_of_b) == B # True
type(B) == type # True
For the cls parameter of your CustomEnum.random() method, we're annotating the equivalent of A rather than instance_of_a in my Example 1 above.
The type of instance_of_a is A.
But the type of A is not A — A is a class, not an instance of a class.
Classes are not instances of classes; they are either instances of type or instances of custom metaclasses that inherit from type.
No metaclasses are being used here; ergo, the type of A is type.
The rule is as follows:
The type of all python class instances will be the class they're an instance of.
The type of all python classes will be either type or (if you're being too clever for your own good) a custom metaclass that inherits from type.
With your CustomEnum class, we could annotate the cls parameter with the metaclass that the enum module uses (enum.EnumType, if you want to know). But, as I say — best not to. The solution I've suggested illustrates the relationship between the input type and the return type more clearly.
Starting in Python 3.11, the correct return annotation for this code is Self:
from typing import Self
class CustomEnum(Enum):
#classmethod
def random(cls) -> Self:
return random.choice(list(cls))
Quoting from the PEP:
This PEP introduces a simple and intuitive way to annotate methods that return an instance of their class. This behaves the same as the TypeVar-based approach specified in PEP 484 but is more concise and easier to follow.
The current workaround for this is unintuitive and error-prone:
Self = TypeVar("Self", bound="Shape")
class Shape:
#classmethod
def from_config(cls: type[Self], config: dict[str, float]) -> Self:
return cls(config["scale"])
We propose using Self directly:
from typing import Self
class Shape:
#classmethod
def from_config(cls, config: dict[str, float]) -> Self:
return cls(config["scale"])
This avoids the complicated cls: type[Self] annotation and the TypeVar declaration with a bound. Once again, the latter code behaves equivalently to the former code.
I'am trying to define a return value of method foo as a list of AbstractChild sub-class instances, but mypy keeps giving me an error.
class AbstractParent(ABC):
#abstractmethod
def foo(self) -> List["AbstractChild"]: pass
class AbstractChild(ABC):
pass
class Parent(AbstractParent):
def foo(self) -> List["Child"]: pass
# ^ mypy: error Return type "List[Child]" of "foo" incompatible with return type "List[AbstractChild]" in supertype "AbstractParent"
class Child(AbstractChild):
pass
Changing the return type from the list to a single value will make mypy to stop complaining which I find quite strange, but I'm still getting used to python type system so I might be missing something.
mypy is correct here because your Parent doesn't implement AbstractParent correctly - to do that, it should define a method foo that returns a list of AbstractChildren, not Children. This is because collections are not polymorphic (and this is true for other languages too, e.g. Java): List[AbstractChild] is not the same type as List[Child], and List[Child] doesn't inherit from List[AbstractChild] just because Child does. If we wouldn't have this restriction, errors like this would be possible:
class AbstractChild(ABC):
pass
class Child(AbstractChild):
pass
class GrandChild(AbstractChild):
pass
grandchildren: List[GrandChild] = [GrandChild()]
all_children: List[AbstractChild] = grandchildren
all_children.append(Child())
grandchild: GrandChild = grandchildren[0] # would pass typechecks but is a Child, actually
(this is a rephrased example of Jon Skeet's answer for a similar question in Java).
Java, for example, catches this type of errors at compilation and requires explicit covariance, e.g. List<? extends Child> for reading from and List<? super Child> for writing to the list.
In your case, you would introduce a generic type as well. In the example below, I change AbstractParent to return a List of elements having the same type C that can be anything that subclasses AbstractChild, and Parent is a concrete implementation of the generic AbstractChild with concrete child type Child:
from typing import List, TypeVar, Generic
C = TypeVar('C', bound='AbstractChild')
class AbstractParent(ABC, Generic[C]):
#abstractmethod
def foo(self) -> List[C]: pass
class Parent(AbstractParent["Child"]):
def foo(self) -> List["Child"]:
return []
For more examples, check out the Generics chapter from mypy docs, in particular the Variance of generic types section.
When I run mypy over the following code I see several errors:
from typing import Callable, Type
def class_creator(outside_reference: Callable[[str], None]) -> Type[object]:
class SomeClass():
reference: Callable[[str], None]
def __init__(self) -> None:
self.reference = outside_reference
super().__init__()
def __str__(self):
self.reference("SomeClass instance")
return SomeClass
def callback(string: str) -> None:
print("Prepping: " + string)
instance = class_creator(callback)()
print(instance)
Here are the errors:
test.py:9: error: Cannot assign to a method
test.py:9: error: Invalid self argument "SomeClass" to attribute function "reference" with type "Callable[[str], None]"
test.py:9: error: Incompatible types in assignment (expression has type "Callable[[str], None]", variable has type "Callable[[], None]")
Line #9 is self.reference = outside_reference.
I'm basically positive that I'm just misunderstanding something, but I just can't see where I'm going wrong.
This is the minimal reproducible reference. If I change the types from Callable[[str], None] to int (and don't actually call it), then it runs just fine without showing any errors. It's only when I switch to Callable that it starts showing these errors.
What should my annotations be here?
Until the issue in https://github.com/python/mypy/issues/708 is fixed, one clean way to work around this is to make the callable attribute optional, and wrap it in a method with an assert:
from typing import Any, Callable, Optional
class SomeClass:
_reference: Optional[Callable[[], Any]]
def reference(self) -> Any:
assert self._reference is not None
return self._reference()
def __init__(self, reference):
self.reference = reference
c = SomeClass(lambda: 42)
print(c.reference())
$ mypy test.py
Success: no issues found in 1 source file
A similar but shorter workaround is to annotate the member with a Union type, duplicating the Callable type:
from typing import Callable, Union
class SomeClass:
reference: Union[Callable[[], int], Callable[[], int]]
def __init__(self, reference: Callable[[], int]):
self.reference = reference
c = SomeClass(lambda: 42)
print(c.reference())
For the time being, MyPy just doesn't support you doing this. Support for this pattern is tracked in GitHub issue 708: https://github.com/python/mypy/issues/708
The closest pattern in most cases will be to define an abstract class with a method like execute, then have callers subclass this with their implementation, instantiate this, and supply the instance as the argument instead of the callback. You can see this approach in older Java codebases (before Java 8), as a common use case for anonymous inner classes. This is, of course, tedious.
Alternatively, you could simply ask mypy to ignore the violations.