I have a use case where I would like to define a common function in a base class, but annotate the return type from an overridden property in a subclass. This is a silly example, but clearly demonstrates what I would like to do:
from typing import Any
class BaseCls:
RETURNS_TYPE: Any = NotImplemented
# This is what I am not sure how to do, this does not work
# Have tried a few different things, but hoping someone just knows
def cast_to_return_type(self: 'BaseCls', value: Any) -> 'BaseCls.RETURNS_TYPE':
return some_work_that_returns_self_RETURNS_TYPE()
class IntCls(BaseCls):
RETURNS_TYPE = int
class StrCls(BaseCls):
RETURNS_TYPE = str
How can I type hint the return value of cast_to_return_type so that a a regular old IDE would be able to detect what the return type should be when doing StrCls().cast_to_return_type(some_var) (tried on pycharm and vscode, neither knew what to do any decided Any was the return type, which makes sense if it cant infer the child class value)
The closest thing that typing has to type variables is typing.TypeVar. Their use is to create a generic class, in the same way that you write list[int] or Mapping[str, str].
So we can rewrite your code so that BaseCls is a generic type, with a type variable ReturnsType. (You should only use all caps for constants.)
The subclasses IntCls and StrCls "assign" to the type variable by including the type inside square braces after the type they are subclassing.
from typing import TypeVar, Generic, Any
ReturnsType = TypeVar('ReturnsType')
class BaseCls(Generic[ReturnsType]):
def cast_to_return_type(self, value: Any) -> ReturnsType:
return some_work_that_returns_self_RETURNS_TYPE()
class IntCls(BaseCls[int]):
pass
class StrCls(BaseCls[str]):
pass
Related
What's a type that accepts an instantiation of any custom class, but not for example, instatiations of builtins like int, str, dict, etc.
In other words, what would MyType be here:
def func(x: MyType) -> None:
pass
func([]) # should have type error
func(5) # should have type error
class Dog:
pass
func(Dog()) # no type error
There is no such type. Python does not distinguish built-in and custom types that way.
Python does not enforce strict typing. Even if you chose not to provide MyType in the function definition, your code will still work fine.
If you would like to anyway have MyType in function definition for better readability, MyType should be Dog. Here's a screenshot from PyCharm which points the same fact:
Edit:
For your function to accept any data type, you'd need to import Any from typing module:
from typing import Any
def myfunc(x: Any):
pass
I have a class structure that looks something like this:
class Base:
class Nested:
pass
def __init__(self):
self.nestedInstance = self.Nested()
where subclasses of Base each have their own Nested class extending the original, like this:
class Sub(Base):
class Nested(Base.Nested):
pass
This works perfectly, and instances of Sub have their nestedInstance attributes set to instances of Sub.Nested.
However, in my IDE the nestedInstance attribute is always treated as an instance of Base.Nested, not the inherited Sub.Nested. How can I make it so that nestedInstance will be inferred to be Sub.Nested rather than Base.Nested? (Without having to add extra code to every subclass; preferably, this would all be done in Base.)
(By the way, I'm aware that this is an odd structure, and I can go into detail about why I chose it if necessary, but I think it's pretty elegant for my situation, and I'm hoping there's a solution to this problem.)
I don't agree with the statement that you were trying to violate the Liskov substitution principle. You were merely looking for a way to let a static type checker infer the type of nested_instance for classes inheriting from Base to be their respective Nested class. Obviously this wasn't possible with the code you had; otherwise there would be no question.
There actually is a way to minimize repetition and accomplish what you want.
Generics to the rescue!
You can define your Base class as generic over a type variable with the upper bound of Base.Nested. When you define Sub as a subclass Base, you provide a reference to Sub.Nested as the concrete type argument. Here is the setup:
from typing import Generic, TypeVar, cast
N = TypeVar("N", bound="Base.Nested")
class Base(Generic[N]):
nested_instance: N
class Nested:
pass
def __init__(self) -> None:
self.nested_instance = cast(N, self.Nested())
class Sub(Base["Sub.Nested"]):
class Nested(Base.Nested):
pass
This is actually all you need. For more info about generics I recommend the relevant section of PEP 484. A few things to note:
Why do we need the bound?
If we were to just use N = TypeVar("N"), the type checker would have no problem if we wanted do define a subclass like this:
class Broken(Base[int]):
class Nested(Base.Nested):
pass
But this would be a problem since now the nested_instance attribute would be expected to be of the type int, which is not what we want. That upper bound on N will prevent this causing mypy to complain:
error: Type argument "int" of "Base" must be a subtype of "Nested" [type-var]
Why explicitly declare nested_instance?
The whole point of making a class generic is to bind some type variable (like N) to it and then indicate that some associated type inside that class is in fact N (or even multiple). We essentially tell the type checker to expect nested_instance to always be of the type N, which must be provided, whenever Base is used to annotate something.
However, now the type checker will always complain, if we ever omit the type argument for Base and tried an annotation like this: x: Base. Again, mypy would tell us:
error: Missing type parameters for generic type "Base" [type-arg]
This may be the only "downside" to the use of generics in this fashion.
Why cast?
The problem is that inside Base, the nested_instance attribute is declared as a generic type N, whereas in Base.__init__, we assign an instance of the specific type Base.Nested. Even though it may seem like this should work, it does not. Omitting the cast call results in the following mypy error:
error: Incompatible types in assignment (expression has type "Nested", variable has type "N") [assignment]
Are the quotes necessary?
Yes, and importing __future__.annotations does not help here. I am not entirely sure why that is, but I believe in case of the Base[...] usage the reason is that __class_getitem__ is actually called and you cannot provide Sub.Nested to it because it is not even defined at that point.
Full working example
from typing import Generic, TypeVar, cast
N = TypeVar("N", bound="Base.Nested")
class Base(Generic[N]):
nested_instance: N
class Nested:
pass
def __init__(self) -> None:
self.nested_instance = cast(N, self.Nested())
class Sub(Base["Sub.Nested"]):
class Nested(Base.Nested):
pass
def get_nested(obj: Base[N]) -> N:
return obj.nested_instance
def assert_instance_of_nested(nested_obj: N, cls: type[Base[N]]) -> None:
assert isinstance(nested_obj, cls.Nested)
if __name__ == '__main__':
sub = Sub()
nested = get_nested(sub)
assert_instance_of_nested(nested, Sub)
This script works "as is" and mypy is perfectly happy with it.
The two functions are just for demonstration purposes, so that you see how you could leverage the generic Base.
Additional sanity checks
To assure you even more, you can for example add reveal_type(sub.nested_instance) at the bottom and mypy will tell you:
note: Revealed type is "[...].Sub.Nested"
This is what we wanted.
If we declare a new subclass
class AnotherSub(Base["AnotherSub.Nested"]):
class Nested(Base.Nested):
pass
and try this
a: AnotherSub.Nested
a = Sub().nested_instance
we are again correctly reprimanded by mypy:
error: Incompatible types in assignment (expression has type "[...].Sub.Nested", variable has type "[...].AnotherSub.Nested") [assignment]
Hope this helps.
PS
To be clear, you can still inherit from Base without specifying the type argument. This has no runtime implications either way. It's just that a strict type checker will complain about it because it is generic, just as it would complain if you annotate something with list without specifying the type argument. (Yes, list is generic.)
Also, whether or not your IDE actually infers this correctly of course depends on how consistent their internal type checker is with the typing rules in Python. PyCharm for example seems to deal with this setup as expected.
It gets typed as Base.Nested because that's what's in scope at initialisation. If you want to declare that nestedInstance is actually something else, then you'll probably need to actually type-hint it.
class Sub(Base):
class Nested(Base.Nested):
pass
nestedInstance: Nested
Context
Say we want to define a custom generic (base) class that inherits from typing.Generic.
For the sake of simplicity, we want it to be parameterized by a single type variable T. So the class definition starts like this:
from typing import Generic, TypeVar
T = TypeVar("T")
class GenericBase(Generic[T]):
...
Question
Is there a way to access the type argument T in any specific subclass of GenericBase?
The solution should be universal enough to work in a subclass with additional bases besides GenericBase and be independent of instantiation (i.e. work on the class level).
The desired outcome is a class-method like this:
class GenericBase(Generic[T]):
#classmethod
def get_type_arg(cls) -> Type[T]:
...
Usage
class Foo:
pass
class Bar:
pass
class Specific(Foo, GenericBase[str], Bar):
pass
print(Specific.get_type_arg())
The output should be <class 'str'>.
Bonus
It would be nice if all relevant type annotations were made such that static type checkers could correctly infer the specific class returned by get_type_arg.
Related questions
Generic[T] base class - how to get type of T from within instance? - This question focuses on direct instances of the custom generic class itself, not on specified subclasses.
How can I access T from a Generic[T] instance early in its lifecycle? - This is a variation on the previous one.
How to access the type arguments of typing.Generic? - This is very close, but does not cover the possibility of other base classes.
TL;DR
Grab the GenericBase from the subclass' __orig_bases__ tuple, pass it to typing.get_args, grab the first element from the tuple it returns, and make sure what you have is a concrete type.
1) Starting with get_args
As pointed out in this post, the typing module for Python 3.8+ provides the get_args function. It is convenient because given a specialization of a generic type, get_args returns its type arguments (as a tuple).
Demonstration:
from typing import Generic, TypeVar, get_args
T = TypeVar("T")
class GenericBase(Generic[T]):
pass
print(get_args(GenericBase[int]))
Output:
(<class 'int'>,)
This means that once we have access to a specialized GenericBase type, we can easily extract its type argument.
2) Continuing with __orig_bases__
As further pointed out in the aforementioned post, there is this handy little class attribute __orig_bases__ that is set by the type metaclass when a new class is created. It is mentioned here in PEP 560, but is otherwise hardly documented.
This attribute contains (as the name suggests) the original bases as they were passed to the metaclass constructor in the form of a tuple. This distinguishes it from __bases__, which contains the already resolved bases as returned by types.resolve_bases.
Demonstration:
from typing import Generic, TypeVar
T = TypeVar("T")
class GenericBase(Generic[T]):
pass
class Specific(GenericBase[int]):
pass
print(Specific.__bases__)
print(Specific.__orig_bases__)
Output:
(<class '__main__.GenericBase'>,)
(__main__.GenericBase[int],)
We are interested in the original base because that is the specialization of our generic class, meaning it is the one that "knows" about the type argument (int in this example), whereas the resolved base class is just an instance of type.
3) Simplistic solution
If we put these two together, we can quickly construct a simplistic solution like this:
from typing import Generic, TypeVar, get_args
T = TypeVar("T")
class GenericBase(Generic[T]):
#classmethod
def get_type_arg_simple(cls):
return get_args(cls.__orig_bases__[0])[0]
class Specific(GenericBase[int]):
pass
print(Specific.get_type_arg_simple())
Output:
<class 'int'>
But this will break as soon as we introduce another base class on top of our GenericBase.
from typing import Generic, TypeVar, get_args
T = TypeVar("T")
class GenericBase(Generic[T]):
#classmethod
def get_type_arg_simple(cls):
return get_args(cls.__orig_bases__[0])[0]
class Mixin:
pass
class Specific(Mixin, GenericBase[int]):
pass
print(Specific.get_type_arg_simple())
Output:
Traceback (most recent call last):
...
return get_args(cls.__orig_bases__[0])[0]
IndexError: tuple index out of range
This happens because cls.__orig_bases__[0] now happens to be Mixin, which is not a parameterized type, so get_args returns an empty tuple ().
So what we need is a way to unambiguously identify the GenericBase from the __orig_bases__ tuple.
4) Identifying with get_origin
Just like typing.get_args gives us the type arguments for a generic type, typing.get_origin gives us the unspecified version of a generic type.
Demonstration:
from typing import Generic, TypeVar, get_origin
T = TypeVar("T")
class GenericBase(Generic[T]):
pass
print(get_origin(GenericBase[int]))
print(get_origin(GenericBase[str]) is GenericBase)
Output:
<class '__main__.GenericBase'>
True
5) Putting them together
With these components, we can now write a function get_type_arg that takes a class as an argument and -- if that class is specialized form of our GenericBase -- returns its type argument:
from typing import Generic, TypeVar, get_origin, get_args
T = TypeVar("T")
class GenericBase(Generic[T]):
pass
class Specific(GenericBase[int]):
pass
def get_type_arg(cls):
for base in cls.__orig_bases__:
origin = get_origin(base)
if origin is None or not issubclass(origin, GenericBase):
continue
return get_args(base)[0]
print(get_type_arg(Specific))
Output:
<class 'int'>
Now all that is left to do is embed this directly as a class-method of GenericBase, optimize it a little bit and fix the type annotations.
One thing we can do to optimize this, is only run this algorithm only once for any given subclass of GenericBase, namely when it is defined, and then save the type in a class-attribute. Since the type argument presumably never changes for a specific class, there is no need to compute this every time we want to access the type argument. To accomplish this, we can hook into __init_subclass__ and do our loop there.
We should also define a proper response for when get_type_arg is called on a (unspecified) generic class. An AttributeError seems appropriate.
6) Full working example
from typing import Any, Generic, Optional, Type, TypeVar, get_args, get_origin
# The `GenericBase` must be parameterized with exactly one type variable.
T = TypeVar("T")
class GenericBase(Generic[T]):
_type_arg: Optional[Type[T]] = None # set in specified subclasses
#classmethod
def __init_subclass__(cls, **kwargs: Any) -> None:
"""
Initializes a subclass of `GenericBase`.
Identifies the specified `GenericBase` among all base classes and
saves the provided type argument in the `_type_arg` class attribute
"""
super().__init_subclass__(**kwargs)
for base in cls.__orig_bases__: # type: ignore[attr-defined]
origin = get_origin(base)
if origin is None or not issubclass(origin, GenericBase):
continue
type_arg = get_args(base)[0]
# Do not set the attribute for GENERIC subclasses!
if not isinstance(type_arg, TypeVar):
cls._type_arg = type_arg
return
#classmethod
def get_type_arg(cls) -> Type[T]:
if cls._type_arg is None:
raise AttributeError(
f"{cls.__name__} is generic; type argument unspecified"
)
return cls._type_arg
def demo_a() -> None:
class SpecificA(GenericBase[int]):
pass
print(SpecificA.get_type_arg())
def demo_b() -> None:
class Foo:
pass
class Bar:
pass
class GenericSubclass(GenericBase[T]):
pass
class SpecificB(Foo, GenericSubclass[str], Bar):
pass
type_b = SpecificB.get_type_arg()
print(type_b)
e = type_b.lower("E") # static type checkers correctly infer `str` type
assert e == "e"
if __name__ == '__main__':
demo_a()
demo_b()
Output:
<class 'int'>
<class 'str'>
An IDE like PyCharm even provides the correct auto-suggestions for whatever type is returned by get_type_arg, which is really nice. 🎉
7) Caveats
The __orig_bases__ attribute is not well documented. I am not sure it should be considered entirely stable. Although it doesn't appear to be "just an implementation detail" either. I would suggest keeping an eye on that.
mypy seems to agree with this caution and raises a no attribute error in the place where you access __orig_bases__. Thus a type: ignore was placed in that line.
The entire setup is for one single type parameter for our generic class. It can be adapted relatively easily to multiple parameters, though annotations for type checkers might become more tricky.
This method does not work when called directly from a specialized GenericBase class, i.e. GenericBase[str].get_type_arg(). But for that one just needs to call typing.get_args on it as shown in the very beginning.
I am writing a CustomEnum class in which I want to add some helper methods, that would then be available by the classes subclassing my CustomEnum. One of the methods is to return a random enum value, and this is where I am stuck. The function works as expected, but on the type-hinting side, I cannot figure out a way of saying "the return type is the same type of cls".
I am fairly sure there's some TypeVar or similar magic involved, but since I never had to use them I never took the time to figure them out.
class CustomEnum(Enum):
#classmethod
def random(cls) -> ???:
return random.choice(list(cls))
class SubclassingEnum(CustomEnum):
A = "a"
B = "b"
random_subclassing_enum: SubclassingEnum
random_subclassing_enum = SubclassingEnum.random() # Incompatible types in assignment (expression has type "CustomEnum", variable has type "SubclassingEnum")
Can somebody help me or give me a hint on how to proceed?
Thanks!
The syntax here is kind of horrible, but I don't think there's a cleaner way to do this. The following passes MyPy:
from typing import TypeVar
from enum import Enum
import random
T = TypeVar("T", bound="CustomEnum")
class CustomEnum(Enum):
#classmethod
def random(cls: type[T]) -> T:
return random.choice(list(cls))
(In python versions <= 3.8, you have to use typing.Type rather than the builtin type if you want to parameterise it.)
What's going on here?
T is defined at the top as being a type variable that is "bound" to the CustomEnum class. This means that a variable annotated with T can only be an instance of CustomEnum or an instance of a class inheriting from CustomEnum.
In the classmethod above, we're actually using this type-variable to define the type of the cls parameter with respect to the return type. Usually we do the opposite — we usually define a function's return types with respect to the types of that function's input parameters. So it's understandable if this feels a little mind-bending!
We're saying: this method leads to instances of a class — we don't know what the class will be, but we know it will either be CustomEnum or a class inheriting from CustomEnum. We also know that whatever class is returned, we can guarantee that the type of the cls parameter in the function will be "one level up" in the type heirarchy from the type of the return value.
In a lot of situations, we might know that type[cls] will always be a fixed value. In those situations, it would be possible to hardcode that into the type annotations. However, it's best not to do so, and instead to use this method, which clearly shows the relationship between the type of the input and the return type (even if it uses horrible syntax to do so!).
Further reading: the MyPy documentation on the type of class objects.
Further explanation and examples
For the vast majority of classes (not with Enums, they use metaclasses, but let's leave that aside for the moment), the following will hold true:
Example 1
Class A:
pass
instance_of_a = A()
type(instance_of_a) == A # True
type(A) == type # True
Example 2
class B:
pass
instance_of_b = B()
type(instance_of_b) == B # True
type(B) == type # True
For the cls parameter of your CustomEnum.random() method, we're annotating the equivalent of A rather than instance_of_a in my Example 1 above.
The type of instance_of_a is A.
But the type of A is not A — A is a class, not an instance of a class.
Classes are not instances of classes; they are either instances of type or instances of custom metaclasses that inherit from type.
No metaclasses are being used here; ergo, the type of A is type.
The rule is as follows:
The type of all python class instances will be the class they're an instance of.
The type of all python classes will be either type or (if you're being too clever for your own good) a custom metaclass that inherits from type.
With your CustomEnum class, we could annotate the cls parameter with the metaclass that the enum module uses (enum.EnumType, if you want to know). But, as I say — best not to. The solution I've suggested illustrates the relationship between the input type and the return type more clearly.
Starting in Python 3.11, the correct return annotation for this code is Self:
from typing import Self
class CustomEnum(Enum):
#classmethod
def random(cls) -> Self:
return random.choice(list(cls))
Quoting from the PEP:
This PEP introduces a simple and intuitive way to annotate methods that return an instance of their class. This behaves the same as the TypeVar-based approach specified in PEP 484 but is more concise and easier to follow.
The current workaround for this is unintuitive and error-prone:
Self = TypeVar("Self", bound="Shape")
class Shape:
#classmethod
def from_config(cls: type[Self], config: dict[str, float]) -> Self:
return cls(config["scale"])
We propose using Self directly:
from typing import Self
class Shape:
#classmethod
def from_config(cls, config: dict[str, float]) -> Self:
return cls(config["scale"])
This avoids the complicated cls: type[Self] annotation and the TypeVar declaration with a bound. Once again, the latter code behaves equivalently to the former code.
How can one define a function in a class such that the return type of the function is the 'current class' - and not the base class. For example:
Class Parent:
def set_common_properties_from_string( input : str ) -> <WHAT SHOULD BE HERE>
# Do some stuff you want to do in all classes
return self
Class Child( Parent ):
pass
def from_file( filename : str ) -> 'Child'
return Child().set_common_properties_from_string() # The return type of set_common must be Child
Or should one cast it somehow? If the return type is baseclass, then it will give an error.
I know one could just drop it to two lines and add temporary variable for holding the Child(), but i think the one liner is much nicer looking.
I use mypy for type checking.
You can use the newly implemented (and still experimental) generic self feature, which is a mechanism designed to help solve exactly the problem you're encountering.
Mypy supports the "generic self" feature as of version 0.4.6 (note: the latest version of mypy as of time of writing is 0.470). Unfortunately, I don't remember off the top of my head if other PEP 484 compliant type checkers support this feature yet.
In short, what you need to do is create a new TypeVar, explicitly annotate your self variable to have that type, and also have that TypeVar be the return value.
So in your case, you'd need to modify your code to the following:
from typing import TypeVar
T = TypeVar('T', bound='Parent')
class Parent:
def set_common_properties(self: T, input: str) -> T:
# Do some stuff you want to do in all classes
return self
class Child(Parent):
def from_file(self, filename: str) -> 'Child':
# More code here
return Child().set_common_properties(...)
Note that we need to set our TypeVar to be bounded by the Parent class -- that way, within the set_common_properties method, we'd be able to call any other methods that live within Parent.
You can find a little more information on mypy's website and within PEP 484:
http://mypy.readthedocs.io/en/latest/generics.html#generic-methods-and-generic-self
https://www.python.org/dev/peps/pep-0484/#annotating-instance-and-class-methods