I have a class like the one below.
from enum import Enum, unique
from typing import Tuple
#unique
class Test(Enum):
v1: Tuple = (1, "value_1")
v2: Tuple = (2, "value_2")
def __init__(self, value_number: int, value_name: str) -> None:
self.value_number = value_number
self.value_name = value_name
#classmethod
def get_value_name_by_value_number(cls, number: int) -> str:
for member in cls.__members__.values():
if member.value_number == number:
return member.value_name
else:
raise NotImplementedError()
if __name__ == '__main__':
name = Test.get_value_name_by_value_number(number=2)
print(name)
When I run the code, the output is correct. But I don't understand why it's working with no problem. As far as I know the __init__ method is called on the creation of an instance. The method I'm calling is classmethod and thus I'm not creating an object and calling it directly from the class. So the __init__ is not invoked. But how does my class know what member.value_number and member.value_name are?
When creating the Test class, the Enum internals will create all enum members and manually call your __init__ method:
enum_member.__init__(*args)
You'd think it'd just instantiate your class normally somewhere, but with how much weird magic goes on in the Enum implementation, it turns out that wouldn't work.
Related
I'm building an API that deals with serializing data, and I'd prefer to add support for as much static analysis as possible. I'm inspired by Django's Meta pattern for declaring metadata on a class, and use an internal library similar to pydantic for inspecting type annotations for serialization.
I'd like the API to work something like:
class Base:
Data: type
def __init__(self):
self.data = self.Data()
class Derived(Base):
class Data:
# Actually, more like a pydantic model.
# Used for serialization.
attr: str = None
obj = Derived()
type(obj.data) # Derived.Data
obj.data.attr = 'content'
This works well and is readable, however it doesn't seem to support static analysis at all. How do I annotate self.data in Base so that I have proper type information on obj?
reveal_type(obj.data) # Derived.Data
reveal_type(obj.data.attr) # str
obj.data.attr = 7 # Should be static error
obj.data.other = 7 # Should be static error
I might write self.data: typing.Self.Data but this obviously doesn't work.
I was able to get something close with typing.Generic and forward references:
import typing
T = typing.TypeVar('T')
class Base(typing.Generic[T]):
Data: type[T]
def __init__(self):
self.data: T = self.Data()
class Derived(Base['Derived.Data']):
class Data:
attr: str = None
But it's not DRY and it doesn't enforce that the annotation and runtime type actually match. For example:
class Derived(Base[SomeOtherType]):
class Data: # Should be static error
attr: str = None
type(obj.data) # Derived.Data
reveal_type(obj.data) # SomeOtherType
I could also require the derived class provide an annotation for data, but this suffers similar issues as typing.Generic.
class Derived(Base):
data: SomeOtherClass # should be 'Data'
class Data: # should be a static error
attr: str = None
To attempt to fix this I tried writing some validation logic in __init_subclass__ to ensure T matches cls.data; however this is brittle and doesn't work in all cases. It also forbids creating any abstract derived class which doesn't define Data.
This is actually non-trivial because you run into the classic problem of wanting to dynamically create types, while simultaneously having static type checkers understand them. An obvious contradiction in terms.
Quick Pydantic digression
Since you mentioned Pydantic, I'll pick up on it. The way they solve it, greatly simplified, is by never actually instantiating the inner Config class. Instead, the __config__ attribute is set on your class, whenever you subclass BaseModel and this attribute holds itself a class (meaning an instance of type).
That class referenced by __config__ inherits from BaseConfig and is dynamically created by the ModelMetaclass constructor. In the process it inherits all the attributes set by the model's base classes and overrides them with whatever you set in the inner Config.
You can see the consequences in this example:
from pydantic import BaseConfig, BaseModel
class Model(BaseModel):
class Config:
frozen = True
a = BaseModel()
b = Model()
a_conf = a.__config__
b_conf = b.__config__
assert isinstance(a_conf, type) and issubclass(a_conf, BaseConfig)
assert isinstance(b_conf, type) and issubclass(b_conf, BaseConfig)
assert not a_conf.frozen
assert b_conf.frozen
By the way, this is why you should not refer to the inner Config directly in your code. It will only have the attributes you set on that one class explicitly and nothing inherited, not even the defaults from BaseConfig. The documented way to access the full model config is via __config__.
This is also why there is no such thing as model instance config. Change an attribute of __config__ and you'll change it for the entire class/model:
from pydantic import BaseModel
foo = BaseModel()
bar = BaseModel()
assert not foo.__config__.frozen
bar.__config__.frozen = True
assert foo.__config__.frozen
Possible solutions
An important constraint of this approach is that it only really makes sense, when you have some fixed type that all these dynamically created classes can inherit from. In the case of Pydantic it is BaseConfig and the __config__ attribute is annotated accordingly, namely with type[BaseConfig], which allows a static type checker to infer the interface of that __config__ class.
You could of course go the opposite way and allow literally any inner class to be defined for Data on your classes, but this probably defeats the purpose of your design. It would work fine though and you could hook into class creation via the meta class to enforce that Data is set and a class. You could even enforce that specific attributes on that inner class are set, but at that point you might as well have a common base class for that.
If you wanted to replicate the Pydantic approach, I can give you a very crude example of how this can be accomplished, with the basic ideas shamelessly stolen from (or inspired by) the Pydantic code.
You can set up a BaseData class and fully define its attributes for the annotations and type inferences down the line. Then you set up your custom meta class. In its __new__ method you perform the inheritance loop to dynamically build the new BaseData subclass and assign the result to the __data__ attribute of the new outer class:
from __future__ import annotations
from typing import ClassVar, cast
class BaseData:
foo: str = "abc"
bar: int = 1
class CustomMeta(type):
def __new__(
mcs,
name: str,
bases: tuple[type],
namespace: dict[str, object],
**kwargs: object,
) -> CustomMeta:
data = BaseData
for base in reversed(bases):
if issubclass(base, Base):
data = inherit_data(base.__data__, data)
own_data = cast(type[BaseData], namespace.get('Data'))
data = inherit_data(own_data, data)
namespace["__data__"] = data
cls = super().__new__(mcs, name, bases, namespace, **kwargs)
return cls
def inherit_data(
own_data: type[BaseData] | None,
parent_data: type[BaseData],
) -> type[BaseData]:
if own_data is None:
base_classes: tuple[type[BaseData], ...] = (parent_data,)
elif own_data == parent_data:
base_classes = (own_data,)
else:
base_classes = own_data, parent_data
return type('Data', base_classes, {})
... # more code below...
With this you can now define your Base class, annotate __data__ in its namespace with type[BaseData], and assign BaseData to its Data attribute. The inner Data classes on all derived classes can now define just those attributes that are different from their parents' Data. To demonstrate that this works, try this:
... # Code from above
class Base(metaclass=CustomMeta):
__data__: ClassVar[type[BaseData]]
Data = BaseData
class Derived1(Base):
class Data:
foo = "xyz"
class Derived2(Derived1):
class Data:
bar = 42
if __name__ == "__main__":
obj0 = Base()
obj1 = Derived1()
obj2 = Derived2()
print(obj0.__data__.foo, obj0.__data__.bar) # abc 1
print(obj1.__data__.foo, obj1.__data__.bar) # xyz 1
print(obj2.__data__.foo, obj2.__data__.bar) # xyz 42
Static type checkers will of course also know what to expect from the __data__ attribute and IDEs should give proper auto-suggestions for it. If you add reveal_type(obj2.__data__.foo) and reveal_type(obj2.__data__.bar) at the bottom and run mypy over the code, it will output that the revealed types are str and int respectively.
Caveat
An important drawback of this approach is that the inheritance is abstracted away in such a way that the inner Data class is treated as its own class unrelated to BaseData in any way by a static type checker, which makes sense because that is what it is; it just inherits from object.
Thus, you will not get any suggestions about the attributes you can override on Data by your IDE. This is the same deal with Pydantic, which is one of the reasons they roll their own custom plugins for mypy and PyCharm for example. The latter allows PyCharm to suggest you the BaseConfig attributes, when you are writing the inner Data class on any derived class.
I know I already provided one answer, but after the little back-and-forth, I thought of another possible solution involving an entirely different design from what I proposed earlier. I think this improves readability, if I post it as a second answer.
No inner classes; just a single type argument
See here for the details about how you can access the type argument provided during subclassing.
from typing import Generic, TypeVar, get_args, get_origin
D = TypeVar("D", bound="BaseData")
class BaseData:
foo: str = "abc"
bar: int = 1
class Base(Generic[D]):
__data__: type[D]
#classmethod
def __init_subclass__(cls, **kwargs: object) -> None:
super().__init_subclass__(**kwargs)
for base in cls.__orig_bases__: # type: ignore[attr-defined]
origin = get_origin(base)
if origin is None or not issubclass(origin, Base):
continue
type_arg = get_args(base)[0]
# Do not set the attribute for GENERIC subclasses!
if not isinstance(type_arg, TypeVar):
cls.__data__ = type_arg
return
Usage:
class Derived1Data(BaseData):
foo = "xyz"
class Derived1(Base[Derived1Data]):
pass
class Derived2Data(Derived1Data):
bar = 42
baz = True
class Derived2(Base[Derived2Data]):
pass
if __name__ == "__main__":
obj1 = Derived1()
obj2 = Derived2()
assert "xyz" == obj1.__data__.foo == obj2.__data__.foo
assert 42 == obj2.__data__.bar
assert not hasattr(obj1.__data__, "baz")
assert obj2.__data__.baz
Adding reveal_type(obj1.__data__) and reveal_type(obj2.__data__) for mypy will show type[Derived1Data] and type[Derived2Data] respectively.
The downside is obvious: It is not the "inner class"-design you had in mind.
The upside is that is entirely type safe, while requiring minimal code. The user merely needs to provide his own BaseData subclass as a type argument, when subclassing Base.
Adding the instance (optional)
If you want to have __data__ be an instance attribute and actual instance of the specified BaseData subclass, this is also easily accomplished. Here is a crude but working example:
from typing import Generic, TypeVar, get_args, get_origin
D = TypeVar("D", bound="BaseData")
class BaseData:
foo: str = "abc"
bar: int = 1
def __init__(self, **kwargs: object) -> None:
self.__dict__.update(kwargs)
class Base(Generic[D]):
__data_cls__: type[D]
__data__: D
#classmethod
def __init_subclass__(cls, **kwargs: object) -> None:
super().__init_subclass__(**kwargs)
for base in cls.__orig_bases__: # type: ignore[attr-defined]
origin = get_origin(base)
if origin is None or not issubclass(origin, Base):
continue
type_arg = get_args(base)[0]
# Do not set the attribute for GENERIC subclasses!
if not isinstance(type_arg, TypeVar):
cls.__data_cls__ = type_arg
return
def __init__(self, **data_kwargs: object) -> None:
self.__data__ = self.__data_cls__(**data_kwargs)
Usage:
class DerivedData(BaseData):
foo = "xyz"
baz = True
class Derived(Base[DerivedData]):
pass
if __name__ == "__main__":
obj = Derived(baz=False)
print(obj.__data__.foo) # xyz
print(obj.__data__.bar) # 1
print(obj.__data__.baz) # False
Again, a static type checker will know that __data__ is of the DerivedData type.
Though, I suppose at that point you might as well just have the user provide his own instance of a BaseData subclass during initialization of Derived. Maybe this is a cleaner and more intuitive design anyway.
I think your initial idea will only work, if you roll your own plugins for static type checkers.
It is not completely DRY, but given advice from #daniil-fajnberg I think this is probably preferable. Explicit is better than implicit, right?
The idea is to require derived classes to specify a type annotation for data; type checkers will be happy since the derived classes all annotate with the correct type, and the base class only needs to inspect that single annotation to determine the runtime type.
from typing import ClassVar, TypeVar, get_type_hints
class Base:
__data_cls__: ClassVar[type]
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
hints = get_type_hints(cls)
if 'data' in hints:
if isinstance(hints['data'], TypeVar):
raise TypeError('Cannot infer __data_cls__ from TypeVar.')
cls.__data_cls__ = hints['data']
def __init__(self):
self.data = self.__data_cls__()
Usage looks like this. Note the name of the data type and the data attribute are no longer coupled.
class Derived1(Base):
class TheDataType:
foo: str = ''
bar: int = 77
data: TheDataType
print('Derived1:')
obj1 = Derived1()
reveal_type(obj1.data) # Derived1.TheDataType
reveal_type(obj1.data.foo) # str
reveal_type(obj1.data.bar) # int
And that decoupling means you are not required to use an inner type.
class Derived2(Base):
data: Derived1.TheDataType
print('Derived3:')
obj2 = Derived2()
reveal_type(obj2.data) # Derived1.TheDataType
reveal_type(obj2.data.foo) # str
reveal_type(obj2.data.bar) # int
I don't think it's possible to support generic subclasses in this solution. It might be possible to adapt the code in https://stackoverflow.com/a/74788026/4672189 to fetch the runtime type in certain situations.
I need to initialise an object of type TypeVar described in an generic class
class TDict(TypedDict):
a: int
T = TypeVar("TType", bound=TDict)
class TestClass(Generic[T]):
d: T
def __init__(self) -> None:
self.d = TDict(a=1)
This cause an error
Incompatible types in assignment (expression has type "TDict", variable has type "T")
At the same time, I cannot create an object with type T. How can I create an object with type T?
upd.
I need to inherit typed dict from TDict and use it in my program. So I heed TestClass to create a dict object of inherited class
Something like this
class ExpandedTDict(TDict):
b: int
t_class: TestClass[ExpandedTDict] = TestClass[ExpandedTDict]()
assert t_class.d[a] == 1
t_class.d[b] = 2
class OtherExpandedTDict(TDict):
c: int
other_t_class: TestClass[OtherExpandedTDict] = TestClass[OtherExpandedTDict]()
assert other_t_class.d[a] == 1
other_t_class.d[c] = 2
other_t_class.d[b] = 2 # -> error
In Python, type variables are "erased" at runtime -- you can't access T at runtime.
You can instantiate the class without providing a type to the class at all:
my_class: TestClass[SubclassOfTDict] = TestClass()
You don't necessarily have a concrete class as the parameter. For example, T could be assigned to Any or to a union of classes. What do you do in that case?
What you can do is pass the dictionary into __init__:
class TestClass(Generic[T]):
d: T
def __init__(self, d: T):
self.d = d
However, I think you're experiencing some kind of XY problem. I don't see how it's useful to instantiate a subclass of a TypedDict class, given that a subclass will have additional fields. Could you explain more about your probleM?
You can wrap the constructor call in a generic function. That lets you specify the type without the redundancy of also specifying it as a generic type to the class.
from typing import TypeVar, Type, Generic, TypedDict
class TDict(TypedDict):
a: int
_T = TypeVar("_T", bound="TDict")
class TestClass(Generic[_T]):
def __init__(self, type_: Type[_T]):
self.d: _T = type_(a=1)
# This is the trick you're looking for
def new_test_class(type_: Type[_T]):
return TestClass[_T](type_)
class ExpandedTDict(TDict):
b: int
class OtherExpandedTDict(TDict):
c: int
t_class = new_test_class(ExpandedTDict)
assert t_class.d['a'] == 1
t_class.d['b'] = 2
other_t_class = new_test_class(OtherExpandedTDict)
assert other_t_class.d['a'] == 1
other_t_class.d['c'] = 2
print(other_t_class.d['b']) # -> error. reading fails as desired. assignment as in your example won't fail no matter what.
This also works with regular classes and attributes.
I'm trying to define a couple of dataclasses and an abstract class that manipulates those classes. Eventually, the my_class_handler types could be dealing with say: json, xml or sqlite files as concrete instance types.
Can someone please explain to me what this message means?
<bound method my_class_handler.class_name of <__main__.my_class_handler object at 0x000001A55FB96580>>
Here's the source code that generates the error for me.
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import List
#dataclass
class column:
name: str
heading: str
#dataclass
class my_class:
class_name: str
class_description: str
columns: List[column]
class iclass_handler(ABC):
#abstractmethod
def class_name(self) -> str:
pass
#abstractmethod
def class_name(self, value: str):
pass
class my_class_handler(iclass_handler):
obj: my_class
def __init__(self):
self.obj = my_class("test-class", "", None)
def class_name(self) -> str:
return self.obj.class_names
def class_name(self, value: str):
if (value != self.obj.class_name):
self.obj.class_name = value
if __name__ == '__main__':
handler = my_class_handler()
print(handler.class_name)
If this is not the proper way of doing this, please point me in the direction where I might learn the proper way.
Thanks for your time,
Python does not allow overloading like Java, so remove methods that overlap.
#khelwood pointed out the answer to the original question. Thanks
As for the #property approach, I tried that and was having nothing but problems and couldn't find any useful examples of inherited properties so I just rewrote the function to take an additional parameter:
# I'm working from memory here but I believe this is the jist...
def class_name(self, new_value: str = None) -> str:
if (new_value is None)
return self.obj.class_name
if (isinstance(new_value, str)):
if (new_value != self.obj.class_name):
self.obj.class_name = new_value
return None
Anyhow, I have since refactored and have completely removed the whole class_name() method as a result of a redesign that dropped the whole concept of data-handlers.
Thanks again for the comments.
The goal is to have the following pseudocode valid in Python 3.7+ and have static analysis tools understand it.
class VariadicType(MaybeASpecialBaseClass, metaclass=MaybeASpecialMetaClass):
#classmethod
def method(cls)->Union[???]:
pass # some irrelevant code
assert(VariadicType[Type1, Type2, Type3, Type4].method.__annotations__["return"] == Union[Type1, Type2, Type3, Type4])
assert(VariadicType[Type1, Type2, Type3, Type4, Type5].method.__annotations__["return"] == Union[Type1, Type2, Type3, Type4, Type5])
Is it possible to support some kind of class VariadicType(Generic[...]) but then get all the passed generic types?
I was considering a C# approach of having
class VariadicType(Generic[T1]):
...
class VariadicType(Generic[T1, T2]):
...
class VariadicType(Generic[T1, T2, T3]):
...
class VariadicType(Generic[T1, T2, T3, T4]):
...
class VariadicType(Generic[T1, T2, T3, T4, T5]):
...
but that it not a valid code - VariadicType should only be defined once.
PS. the irrelevant part of code should be checking the __annotations__["return"] and returning results accordingly. It is applying mixins. If the return type is not a union of all applied mixins, then static analysis complains on missing fields and methods. Having a non-hinted code where types are given as method arguments but return type is Any is the last resort.
I already faced this problem, so maybe i can put some light on it.
The problem
Suppose we have the next class definition:
T = TypeVar('T')
S = TypeVar('S')
class VaradicType(Generic[T, S]):
pass
The issue is that VaradicType[T, S] invokes VaradicType.__class_getitem__((T, S)) which returns an object of the class _GenericAlias.
Then, if you do cls = VaradicType[int, float], you can introspect the arguments used as indices with
cls.__args__.
However, if you instantiate an object like obj = cls(), you cannot do obj.__class__.__args__.
This is because _GenericAlias implements the method __call__ that returns directly an object of VaradicType which dont have any class in its MRO that contains information about the arguments supplied.
class VaradicType(Generic[T, S]):
pass
cls = VaradicType[int, float]().__class__
print('__args__' in cls) # False
One solution
One possible approach to solve this issue could be adding information about the generic arguments to the objects of the class VaradicType when they are instantiated.
First (following the previous code snippets), we will add a metaclass to VaradicType:
class VaradicType(Generic[T, S], metaclass=GenericMixin):
pass
We can use the fact that if __getitem__ its defined on the metaclass, has priority over __class_getitem__ in order to bypass Generic.__class_getitem__
class GenericMixin(type):
def __getitem__(cls, items):
return GenericAliasWrapper(cls.__class_getitem__(items))
Now, VaradicType[int, float] is equivalent to GenericMixin.__getitem__(VaradicType, (int, float)) and it will return an object of the class GenericAliasWrapper (it is used to "wrap" typing._GenericAlias instances):
class GenericAliasWrapper:
def __init__(self, x):
self.wrapped = x
def __call__(self, *args, **kwargs):
obj = self.wrapped.__call__(*args, **kwargs)
obj.__dict__['__args__'] = self.wrapped.__args__
return obj
Now, if you have cls=VaradicType[int, float], the code cls() will be equivalent to GenericAliasWrapper( VaradicType.__class_getitem__((int, float)) ).__call__() which creates a new instance of the class VaradicType and also adds the attribute __args__ to its dictionary.
e.g:
VaradicType[int, float]().__args__ # (<class int>, <class float>)
I want to allow type hinting using Python 3 to accept sub classes of a certain class. E.g.:
class A:
pass
class B(A):
pass
class C(A):
pass
def process_any_subclass_type_of_A(cls: A):
if cls == B:
# do something
elif cls == C:
# do something else
Now when typing the following code:
process_any_subclass_type_of_A(B)
I get an PyCharm IDE hint 'Expected type A, got Type[B] instead.'
How can I change type hinting here to accept any subtypes of A?
According to PEP 484 ("Expressions whose type is a subtype of a specific argument type are also accepted for that argument."), I understand that my solution (cls: A) should work?
When you specify cls: A, you're saying that cls expects an instance of type A. The type hint to specify cls as a class object for the type A (or its subtypes) uses typing.Type.
from typing import Type
def process_any_subclass_type_of_A(cls: Type[A]):
pass
From The type of class objects
:
Sometimes you want to talk about class objects that inherit from a
given class. This can be spelled as Type[C] where C is a class. In
other words, when C is the name of a class, using C to annotate an
argument declares that the argument is an instance of C (or of a
subclass of C), but using Type[C] as an argument annotation declares
that the argument is a class object deriving from C (or C itself).
If we look at the Type description from the typing module, then we see these docs:
A special construct usable to annotate class objects.
For example, suppose we have the following classes::
class User: ... # Abstract base for User classes
class BasicUser(User): ...
class ProUser(User): ...
class TeamUser(User): ...
And a function that takes a class argument that's a subclass of
User and returns an instance of the corresponding class::
U = TypeVar('U', bound=User)
def new_user(user_class: Type[U]) -> U:
user = user_class()
# (Here we could write the user object to a database)
return user
joe = new_user(BasicUser)
At this point the type checker knows that joe has type BasicUser.
Based on this, I can imagine a synthetic example that reproduces the problem with type hinting errors in PyCharm.
from typing import Type, Tuple
class BaseClass: ...
class SubClass(BaseClass): ...
class SubSubClass(SubClass): ...
def process(model_instance: BaseClass, model_class: Type[BaseClass]) -> Tuple[BaseClass, BaseClass]:
""" Accepts all of the above classes """
return model_instance, model_class()
class ProcessorA:
#staticmethod
def proc() -> Tuple[SubClass, SubClass]:
""" PyCharm will show an error
`Expected type 'tuple[SubClass, SubClass]', got 'tuple[BaseClass, BaseClass]' instead` """
return process(SubClass(), SubClass)
class ProcessorB:
#staticmethod
def proc() -> Tuple[SubSubClass, SubSubClass]:
""" PyCharm will show an error
`Expected type 'tuple[SubSubClass, SubSubClass]', got 'tuple[BaseClass, BaseClass]' instead` """
return process(SubSubClass(), SubSubClass)
But we see in docs for Type that the situation can be corrected by using TypeVar with the bound argument. Then use it in places where BaseClass is declared as a type.
from typing import TypeVar, Type, Tuple
class BaseClass: ...
B = TypeVar('B', bound=BaseClass)
class SubClass(BaseClass): ...
class SubSubClass(SubClass): ...
def process(model_instance: B, model_class: Type[B]) -> Tuple[B, B]:
""" Accepts all of the above classes """
return model_instance, model_class()
class ProcessorA:
#staticmethod
def proc() -> Tuple[SubClass, SubClass]:
return process(SubClass(), SubClass)
class ProcessorB:
#staticmethod
def proc() -> Tuple[SubSubClass, SubSubClass]:
return process(SubSubClass(), SubSubClass)
Hope this will be helpful.
Type[A] accepts also the class itself, which is not always needed.
If you want your function to accept only subclasses, you should go with NewType, like
class A:
pass
B = NewType('B', A)
def foo(cls: Type[B]):
...