I'd like to submit a code pattern I often see in my python code, here at work, and I'm not satisfy with it and I'd like a better solution.
Here is what we have today:
class AbstractClass(object):
....
def method1(self,...)
raise NotImplementedError
...
class FirstImplementationMixin(object)
def method1(self,....)
...
class SecondImplementationMixin(object)
def method1(self...)
....
class InstanciatedClass1(FirstImplementationMixin, AbstractClass)
....
class InstanciatedClass2(SecondImplementationMixin, AbstractClass)
....
Do you see the trick? I have to add the mixin at first position in the list
of the inheritance, and I don't like this solution. If we add it at the second position, the interpreter will use AbstractClass.method1 and so raise the exception.
In this trivial situation, replacing the mixin by intermediate class is possible, but in case of complex inheritance with already a base class, the solution might not be obvious.
What is for you the best design pattern?
Well, as I noted in my comment this is just how Python works, and I don't think that changing it is a good idea.
Nevertheless, metaclasses to the rescue!
You can create your own metaclass and make it as complex as you need. Here I'll show you a very basic example.
def mixed(name, bases, dct):
return type(name, bases[1:] + (bases[0],), dct)
It simply moves the first baseclasses to the end, so now you put your actual base-class first, followed by all the mixins. That means that you are limited to a single base class. You can lift this limitation by some convention, for example, you'll list base classes first, then None, then mixins; and your metaclass will look for this None and reorder the tuple accordingly.
class MixinA:
pass
class MixinB:
pass
class Base(object):
pass
class Test(Base, MixinA, MixinB):
__metaclass__ = mixed
>>> Test.__mro__
>
(<class '__main__.Test'>,
<class __main__.MixinA at 0x6f8850db48>,
<class __main__.MixinB at 0x6f8850dae0>,
<class '__main__.Base'>,
<type 'object'>)
Of course you won't be specifying __metaclass__ each time, because of the rules that are used to chose a metaclass. You'll probably set __metaclass__ on the module level or, maybe, in your mixins. That's your choice, whatever fits your code structure better.
Related
Let's say I want to create a registry of subclasses of a certain class. Now there are two approaches I can think of and while I'm aware of (some of) their differences, I'd love to learn more about the topic.
class Base:
pass
class DerivedA(Base):
pass
class DerivedB(Base):
pass
__subclasses__()
If I have the situation above, I can simply get the list of subclasses of Base like this:
>>> [cmd.__name__ for cmd in Base.__subclasses__()]
['DerivedA', 'DerivedB']
Now I'm aware that if I add a third class that is not directly subclassing Base like this:
class DerivedC(DerivedA):
pass
I will not see this one in the list:
>>> [cmd.__name__ for cmd in Base.__subclasses__()]
['DerivedA', 'DerivedB']
Also I can't filter the subclasses and for example ignore a particular subclass for any reason.
__init_subclass__()
Since Python 3.6 there is a nice hook into class creating process and more advanced things can be done without writing one's own metaclass. Thus I can also do something like this...
_registry = []
class Base:
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
_registry.append(cls.__name__)
class DerivedA(Base):
pass
class DerivedB(Base):
pass
class DerivedC(DerivedA):
pass
And then simply access _registry:
>>> _registry
['DerivedA', 'DerivedB', 'DerivedC']
I can also modify Base to ignore certain subclasses if I wanted:
_registry = []
class Base:
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
if cls.__name__ != 'DerivedB':
_registry.append(cls.__name__)
class DerivedA(Base):
pass
class DerivedB(Base):
pass
class DerivedC(DerivedA):
pass
>>> _registry
['DerivedA', 'DerivedC']
Why use the latter?
Now let's say that I don't want to filter the subclasses and I'm only interested in direct subclasses. The former approach seems to be simpler (subjective, I know). What are other differences and maybe what are the advantages of the latter approach?
Thanks!
The obvious gain of writing __init_subclass__ in a base class in this case is that you can automatically get to the subclasses that do not inherit directly from your base class, as you put it.
If you only need the classes that inherit directly from your base, then it is ready in the __subclasses__ method, and the major advantage is that you don't need to write a single line of code, and not even keep a separate registry, as the __subclasses__ will do that for you.
However, unless you are writing a relatively small app, or are dealing with a feature that just needs a small fixed number of these subclasses to be looked-up, relying in __subclasses__ is not enough - if you simply need, or want, another level of classes in your hierarchy, it will stop working, and you have to resort to a true registry anyway.
Prior to having the __init_subclass__ hook, one would have to write a proper metaclass to keep this registry, feeding it on the metaclass __init__ method, or do a complicated recursive query like:
def check_subclass(base, candidate):
for cls in base.__subclasses__():
if cls is candidate:
return True
if check_subclass(cls, candidate):
return True
return False
And, although it should go without saying, the __init_subclass__ method can do a lot more than simply keep a registry - as it can run any code. It could check against the DB layer if the fields mapped to that subclass are up to date, and warn of a needed migration - or even perform the DB migration itself, or initialise any resources that instances of the class will need to find ready when they are created, such as logger-handlers, thread-pools, db-connection pools, you name it.
TL;DR: If you just need the direct subclasses of a class, go with __subclasses__. The catch is exactly that it just annotates the direct subclasses.
In Java, for example, you can make a class MyClass with certain methods that are specified but not implemented in MyClass, but must be implemented in any class MySubClass that inherits from MyClass. So basically there is some common functionality among all subclasses you want, so you put it in MyClass, and there is some functionality unique (but required) for each subclass, so you want it in each subclass. How can this behavior be achieved in Python?
(I know there are concise terms to describe what I'm asking, so feel free to let me know what these are and how I can better describe my question.)
A very basic example but the abc docs provide a few more
import abc
class Foo():
__metaclass__ = abc.ABCMeta
#abc.abstractmethod
def bar(self):
raise NotImplemented
class FooBar(Foo):
pass
f = FooBar()
TypeError: Can't instantiate abstract class FooBar with abstract methods bar
You can't require the implementation of a method in a subclass in a way that will break at compile-time, but the convention on writing a method on the base class that must be implemented in the subclasses is to raise NotImplementedError.
Something like this:
class MyBase(object):
def my_method(self, *args, **kwargs):
raise NotImplementedError("You should implement this method on a subclass of MyBase")
Then your subclasses can implement my_method, but this will break only when the method is called. If you have comprehensive unit tests, as you should, this won't be a problem.
I have a base class from which other classes should inherit:
class AppToolbar(wx.ToolBar):
''' Base class for the Canary toolbars '''
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# ... a few common implementation details that work as expected...
self._PopulateToolbar()
self.Realize()
The base class does not (and cannot) implement _PopulateToolbar(); it should be an abstract method. As such, I figured using abc was a good plan, so I tried this:
class AppToolbar(wx.ToolBar, metaclass=abc.ABCMeta):
# ... as above, but with the following added
#abc.abstractmethod
def _PopulateToolbar():
pass
Perhaps unsurprisingly, attempting to run this led to TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases. I thought, "Oh, right, I'll just use a mixin":
class PopulateToolbarMixin(metaclass=ABCMeta):
#abstractmethod
def _PopulateToolbar(self):
pass
PopulateToolbarMixin.register(wx.ToolBar)
PopulateToolbarMixin.register(AppToolbar)
No change: still the same TypeError message. I suspect I'm missing something obvious with the use of ABCMeta here; this doesn't look like an error specific to wxPython. What am I doing wrong? Is there a better way to approach the same issue?
Edit: it has been pointed out to me in a conversation with a colleague that one cannot mix metaclasses. Since wx.ToolBar apparently derives from sip.wrappertype, it looks like there is no way to do this. What is another, still Pythonic way to handle the "abstract method" approach here?
In your first example, where you inherit from wx.ToolBar and abc.ABCMeta, you don't want AppToolbar to be a subclass of abc.ABCMeta, you want AppToolbar to be an instance of it. Try this:
class AppToolbar(wx.ToolBar, metaclass=abc.ABCMeta):
# ... as above, but with the following added
#abc.abstractmethod
def _PopulateToolbar():
pass
Though looking at this a bit closer, it seems that you can't define a subclass of wx.Toolbar with abc.ABCMeta as its metaclass, as wx.Toolbar is an instance of a metaclass other than bultins.type. You can, however, get abstract-like behavior out of AppToolbar._PopulateToolbar:
class AppToolbar(wx.ToolBar):
def _PopulateToolbar():
''' This is an abstract method; subclasses must override it. '''
raise NotImplementedError('Abstract method "_PopulateToolbar" must be overridden before it can be called.')
class MType(type):
pass
class MClass(object):
__metaclass__ = MType
a = MType
From the above snippet is there anyway I can create an instance of MClass given a?
(It's probably important to note that I won't be able to simply contruct MClass())
Here's a horrific way of doing it. Although, I'll be honest.. I'm not sure what the difference between setting __metaclass__ as a module property is vs. inheritance.
class MType(type):
pass
class MClass(MType):
pass
a = MType
print a.__mro__[0].__subclasses__(a)
Yields:
[<class '__main__.MClass'>]
There are two straightforward ways I can think of to find the instances of a metaclass. One is to have the metaclass keep track of a list of its instances as they are created, as follows:
class MType(type):
instances = [] #list of instances of the metaclass
def __init__(cls, name, bases, dct):
MType.instances.append(cls) #append to list of instances
super(MType, cls).__init__(name, bases, dct)
class MClass(object):
__metaclass__ = MType
Now MType.instances is [<class '__main__.MClass'>].
Another way is to use the
inspect module to look through the classes in, say, a given module, and use isinstance to check if they are instances of MType. This isn't necessarily a crazy thing to do, either - for example, various unit testing suites work by looking through a module for classes and functions that appear to be tests, based on their names and/or inherited classes - I'm not really sure why you would want to do this with a metaclass, though.
No, if you have b = MClass, you can simply do obj = b(), but the metaclass does not "know" about the class(es) it is set as the metaclass of.
You could call __new__ on a if you have the class in a variable, whether b or MClass.
I have a number of atomic classes (Components/Mixins, not really sure what to call them) in a library I'm developing, which are meant to be subclassed by applications. This atomicity was created so that applications can only use the features that they need, and combine the components through multiple inheritance.
However, sometimes this atomicity cannot be ensured because some component may depend on another one. For example, imagine I have a component that gives a graphical representation to an object, and another component which uses this graphical representation to perform some collision checking. The first is purely atomic, however the latter requires that the current object already subclassed this graphical representation component, so that its methods are available to it. This is a problem, because we have to somehow tell the users of this library, that in order to use a certain Component, they also have to subclass this other one. We could make this collision component sub class the visual component, but if the user also subclasses this visual component, it wouldn't work because the class is not on the same level (unlike a simple diamond relationship, which is desired), and would give the cryptic meta class errors which are hard to understand for the programmer.
Therefore, I would like to know if there is any cool way, through maybe metaclass redefinition or using class decorators, to mark these unatomic components, and when they are subclassed, the additional dependency would be injected into the current object, if its not yet available. Example:
class AtomicComponent(object):
pass
#depends(AtomicComponent) # <- something like this?
class UnAtomicComponent(object):
pass
class UserClass(UnAtomicComponent): #automatically includes AtomicComponent
pass
class UserClass2(AtomicComponent, UnAtomicComponent): #also works without problem
pass
Can someone give me an hint on how I can do this? or if it is even possible...
edit:
Since it is debatable that the meta class solution is the best one, I'll leave this unaccepted for 2 days.
Other solutions might be to improve error messages, for example, doing something like UserClass2 would give an error saying that UnAtomicComponent already extends this component. This however creates the problem that it is impossible to use two UnAtomicComponents, given that they would subclass object on different levels.
"Metaclasses"
This is what they are for! At time of class creation, the class parameters run through the
metaclass code, where you can check the bases and change then, for example.
This runs without error - though it does not preserve the order of needed classes
marked with the "depends" decorator:
class AutoSubclass(type):
def __new__(metacls, name, bases, dct):
new_bases = set()
for base in bases:
if hasattr(base, "_depends"):
for dependence in base._depends:
if not dependence in bases:
new_bases.add(dependence)
bases = bases + tuple(new_bases)
return type.__new__(metacls, name, bases, dct)
__metaclass__ = AutoSubclass
def depends(*args):
def decorator(cls):
cls._depends = args
return cls
return decorator
class AtomicComponent:
pass
#depends(AtomicComponent) # <- something like this?
class UnAtomicComponent:
pass
class UserClass(UnAtomicComponent): #automatically includes AtomicComponent
pass
class UserClass2(AtomicComponent, UnAtomicComponent): #also works without problem
pass
(I removed inheritance from "object", as I declared a global __metaclass__ variable. All classs will still be new style class and have this metaclass. Inheriting from object or another class does override the global __metaclass__variable, and a class level __metclass__ will have to be declared)
-- edit --
Without metaclasses, the way to go is to have your classes to properly inherit from their dependencies. Tehy will no longer be that "atomic", but, since they could not work being that atomic, it may be no matter.
In the example bellow, classes C and D would be your User classes:
>>> class A(object): pass
...
>>> class B(A, object): pass
...
>>>
>>> class C(B): pass
...
>>> class D(B,A): pass
...
>>>