First, a project that I am working using the plugin systme similar to PEP-0478 using a PriortyQueue for insuring plugins run in the right order. I am going to leave out the how this works:
It is well documented in PEP-487
Not directly required for understanding this issue.
First, I have basic classes as below:
# abstract.py
import abc
from dataclasses import data, field
#dataclass(order=True)
class PluginBase(abc.ABC):
value: int
def __eq__(self, __o: object) -> bool:
print("PluginBase __eq__ called")
return self.value == __o.value
# plugin_a.py
from abstract import PluginBase
class PluginA(PluginBase):
pass
# plugin_b.py
from abstract import PluginBase
class PluginB(PluginBase):
pass
First, after the plugins are discovered and loaded they are wrapped in a dataclass this is to ensure min amount of code for each plugin.
Problem: __eq__() is never passed to subclasses.
It does not matter how I setup the classes or even use funtools.total_ordering.
One thing I noticed, is that if you do assert PluginA(1) == PluginB(1) always returns false and assert PluginA(1) < PluginB(2) always returns TypeError: '<' not supported between instances with the current classes.
This this always intended?
To fix the first issue, where == doesn't work you need to add:
def __eq__(self, __o: object) -> bool:
print("Plugin __eq__ called")
return super().__eq__(__o)
To one or both subclasses which addes more boiler plate code to the plugin system which I do not want.
To fix the '<' issue, you need to change PluginBase to:
#functools.total_ordering
#dataclass
class PluginBase(abc.ABC):
value: int
def __eq__(self, __o: object) -> bool:
print("PluginBase __eq__ called")
return self.value == __o.value
def __lt__(self, __o: object) -> bool:
print("PluginBase __lt__ called")
return self.value < __o.value
This allows you to do PluginA(1) < PluginB(2) which will be true. However, == still does not work here.
I think this is due to the differences in class instances and Python is enforcing __eq__() to check the instance type before anything else. Is there anyway to get this to work?
One solution is to wrap each of the Plugins into a comparable object and use that into the Queue.
#chepner was on the right track here.
Change code to here works:
# abstract.py
import abc
from dataclasses import dataclass, field
#functools.total_ordering
#dataclass
class PluginBase(abc.ABC):
value: int
def __eq__(self, __o: object) -> bool:
print("PluginBase __eq__ called")
return self.value == __o.value
def __lt__(self, __o: object) -> bool:
print("PluginBase __lt__ called")
return self.value < __o.value
# plugin_a.py
from dataclasses import dataclass
from abstract import PluginBase
#dataclass(eq=False)
class PluginA(PluginBase):
pass
# plugin_b.py
from dataclasses import dataclass
from abstract import PluginBase
#dataclass(eq=False)
class PluginB(PluginBase):
pass
Adding the eq=False forces it not to genereate the interclass eq() functions which then defaults to the base class.
assert PluginA(1) < PluginB(2) # True
PluginBase __lt__ called
assert PluginA(1) == PluginB(1) # True
PluginBase __eq__ called
Some reason I just missed trying this combonation of options.
Related
I'm trying to define a couple of dataclasses and an abstract class that manipulates those classes. Eventually, the my_class_handler types could be dealing with say: json, xml or sqlite files as concrete instance types.
Can someone please explain to me what this message means?
<bound method my_class_handler.class_name of <__main__.my_class_handler object at 0x000001A55FB96580>>
Here's the source code that generates the error for me.
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import List
#dataclass
class column:
name: str
heading: str
#dataclass
class my_class:
class_name: str
class_description: str
columns: List[column]
class iclass_handler(ABC):
#abstractmethod
def class_name(self) -> str:
pass
#abstractmethod
def class_name(self, value: str):
pass
class my_class_handler(iclass_handler):
obj: my_class
def __init__(self):
self.obj = my_class("test-class", "", None)
def class_name(self) -> str:
return self.obj.class_names
def class_name(self, value: str):
if (value != self.obj.class_name):
self.obj.class_name = value
if __name__ == '__main__':
handler = my_class_handler()
print(handler.class_name)
If this is not the proper way of doing this, please point me in the direction where I might learn the proper way.
Thanks for your time,
Python does not allow overloading like Java, so remove methods that overlap.
#khelwood pointed out the answer to the original question. Thanks
As for the #property approach, I tried that and was having nothing but problems and couldn't find any useful examples of inherited properties so I just rewrote the function to take an additional parameter:
# I'm working from memory here but I believe this is the jist...
def class_name(self, new_value: str = None) -> str:
if (new_value is None)
return self.obj.class_name
if (isinstance(new_value, str)):
if (new_value != self.obj.class_name):
self.obj.class_name = new_value
return None
Anyhow, I have since refactored and have completely removed the whole class_name() method as a result of a redesign that dropped the whole concept of data-handlers.
Thanks again for the comments.
Problem
Suppose I want to implement a class decorator that adds some attributes and functions to an existing class.
In particular, let's say I have a protocol called HasNumber, and I need a decorator can_add that adds the missing methods to convert HasNumber class to CanAdd.
class HasNumber(Protocol):
num: int
class CanAdd(HasNumber):
def add(self, num: int) -> int: ...
Implementation
I implement the decorator as follows:
_HasNumberT = TypeVar("_HasNumberT", bound=HasNumber)
def can_add(cls: Type[_HasNumberT]) -> Type[CanAdd]:
def add(self: _HasNumberT, num: int) -> int:
return self.num + num
setattr(cls, "add", add)
return cast(Type[CanAdd], cls)
#can_add
class Foo:
num: int = 12
Error
The code works just fine when I run it, but mypy is unhappy about it for some reason.
It gives the error "Foo" has no attribute "add" [attr-defined], as if it doesn't take the return value (annotated as Type[CanAdd]) of the can_add decorator into account.
foo = Foo()
print(foo.add(4)) # "Foo" has no attribute "add" [attr-defined]
reveal_type(foo) # note: Revealed type is "test.Foo"
Question
In this issue, someone demonstrated a way of annotating this with Intersection. However, is there a way to achieve it without Intersection? (Supposing that I don't care about other attributes in Foo except the ones defined in the protocols)
Or, is it a limitation of mypy itself?
Related posts that don't solve my problem:
Mypy annotation on a class decorator
Class Decorator Compatible for Mypy
cast tells mypy that cls (with or without an add attribute) is safe to use as the return value for can_add. It does not guarantee that the protocol holds.
As a result, mypy cannot tell whether Foo has been given an add attribute, only that it's OK to use the can_add decorator. The fact that can_add has a side effect of defining the add attribute isn't visible to mypy.
You can, however, replace the decorator with direct inheritance, something like
class HasNumber(Protocol):
num: int
_HasNumberT = TypeVar("_HasNumberT", bound=HasNumber)
class Adder(HasNumber):
def add(self, num: int) -> int:
return self.num + num
class Foo(Adder):
num: int = 12
foo = Foo()
print(foo.add(4))
So I have a situation where one module writes some code that processes records, so lets say it does this:
from datetime import datetime
def is_record_today(rec: Record) -> bool:
return (rec.date == str(datetime.now().date()))
def is_record_valid(rec: Record) -> bool:
return record.is_valid()
so at this time I need to define:
import abc
class Record(abc.ABC):
#abc.abstractproperty
def date(self) -> str:
print('I am abstract so should never be called')
#abc.abstractmethod
def is_valid(self) -> bool:
print('I am abstract so should never be called')
now when I am processing a record in another module I want to inherit from this abstract class so I raise an error if I don't have date and is_valid. However, I also want to inherit from another class to get the default properties, lets say I want my record to be manipulatable like a dictionary. Then:
class ValidRecord(Record, dict):
#property
def date(self) -> str:
return self['date']
def is_valid(self) -> bool:
return self['valid']
class InvalidRecord(Record, dict):
#property
def date(self) -> str:
return self['date']
we would expect that ValidRecord should create without issue and InvalidRecord should throw TypeError, however both seem fine, and I can even call the missing abstractmethod from the Record class which as far as I understand abstract methods should not be possible:
data = {
'date': '2021-05-01',
'valid': False
}
InValidRecord(data).is_valid()
>>> "I am abstract so should never be called"
If I take away the dictionary inheritance I get the expected error so, what is going on here and how can I get the desired behavior of inheriting from one class but requiring additional methods be added?
I have a class split into mixins:
class MyObject(MyObjectFilesMixin, MyObjectProcessingMixin, ...):
def __init__(self, value):
self.value = self.preprocess(value)
A mixin looks like this:
class MyObjectFilesMixin:
def load_from_file(cls, filename):
return ...
Now I'd like to add typing to the class and mixins:
class MyObjectFilesMixin:
def load_from_file(cls, filename: str) -> MyObject:
return ...
class MyObjectProcessingMixin:
def preprocess(self: MyObject, value: bytes):
return value # logic is omitted
def append(self: MyObject, other: MyObject):
self.value += other.value
But it leads to cyclic links. Of course I can create some MyObjectBase (following dependency inversion principle), so that MyObject will also inherit this class, and mixins will use it as argument/return type, but this will lead to wrong types anyway. Is it possible to fix??
I miss so much header+source files from C++
In the case of using mixin inheritance, protocols and structural subtyping and TYPE_CHECKING can help to successfully type classes by avoiding circular imports, it might look like this:
# my_object.py
from mixins import ProcessingMixin
class MyObject(ProcessingMixin):
def __init__(self, value):
self.value = self.preprocess(value)
def some_process(self, value) -> bytes:
...
# mixins.py
from typing import Protocol, TYPE_CHECKING
if TYPE_CHECKING:
from my_object import MyObject
class ValueObjectProto(Protocol):
"""Protocol for value processing methods."""
value: bytes
def preprocess(self, value: bytes) -> bytes: ...
def some_process(self, value) -> bytes: ...
class MyObjectFilesMixin:
def load_from_file(cls, filename: str) -> MyObject:
return cast(Type[MyObject], cls)(1)
class ProcessingMixin:
def preprocess(self: ValueObjectProto, value: bytes) -> bytes:
value = self.some_process(value)
return value
def append(self: ValueObjectProto, other: ValueObjectProto) -> None:
self.value += other.value
I found how to do the trick without protocols or other unnecessary code, DRY! My solution is simple:
from typing import TYPE_CHECKING, Type
if TYPE_CHECKING:
from my_object import MyObject
MO = Type('MyObject')
class MyObjectFilesMixin:
def load_from_file(cls: MO, filename: str) -> 'MyObject':
# Now I'm able to access all the methods of MyObject by cls
# with correct IDE support and MyPy checking
return ...
class MyObjectProcessingMixin:
def preprocess(self: 'MyObject', value: bytes):
return value # logic is omitted
def append(self: 'MyObject', other: 'MyObject'):
self.value += other.value
However, I get another warning, probably because MyPy doesn't expect a child to be used as a type in parent:
Mypy: The erased type of self "Type[... MyObject]" is not a supertype of its class "Type[...MyObjectFilesMixin]"
But it's a little cost for the methods and types to be seen and understood correctly by IDE and MyPy!
I've read in the abc python module docs that a Sequence is something that implements the following: __getitem__, __len__, __contains__, __iter__, __reversed__, index, and count.
Yet, when I run the following example it yields false:
from collections import abc
class Sequence():
def __getitem__(self):
pass
def __len__(self):
pass
def index(self):
pass
def count(self):
pass
def __contains__(self):
pass
def __iter__(self):
pass
def __reversed__(self):
pass
print(isinstance(Sequence(), abc.Sequence)) # False
When I was doing similar stuff for abc.Collection or abc.Reversed to play around I got the results I expected, for example I created a dummy class that implements __contains__, __iter__, __len__ and it was in fact detected correctly as an instance of abc.Collection.
Do you have any idea what's wrong with Sequence?
EDIT 1:
from collections import abc
class CustomIterable:
def __iter__(self):
pass
print(isinstance(CustomIterable(), abc.Iterable)) # True
even though my custom iterable is not from abc it still is recognized as Iterable because it implements __iter__ special method.
I had a similar question today.
From what I can gather, collections.abc.Iterable implements a custom __subclasshook__() method, whereas collections.abc.Sequence does not:
# _collections_abc.py
class Iterable(metaclass=ABCMeta):
__slots__ = ()
#abstractmethod
def __iter__(self):
while False:
yield None
#classmethod
def __subclasshook__(cls, C):
if cls is Iterable:
return _check_methods(C, "__iter__") # True if __iter__ implemented
return NotImplemented
What this means is that if a class Foo defines the required __iter__ method then isinstance(Foo(), collections.abc.Iterable) will return True:
from collections.abc import Iterable
class Foo:
def __iter__(self):
return []
assert isinstance(Foo(), Iterable)
I'm not sure why collections.abc.Iterable implements a custom __subclasshook__() method but collections.abc.Sequence does not.
Your custom Sequence class is different from the abc.Sequence class so isinstance will return false.
If you're looking for True, your custom class needs to inherit from abc.Sequence:
class Sequence(abc.Sequence):
.....