Is there a reasonable way in Python to implement mixin behavior similar to that found in Ruby -- that is, without using inheritance?
class Mixin(object):
def b(self): print "b()"
def c(self): print "c()"
class Foo(object):
# Somehow mix in the behavior of the Mixin class,
# so that all of the methods below will run and
# the issubclass() test will be False.
def a(self): print "a()"
f = Foo()
f.a()
f.b()
f.c()
print issubclass(Foo, Mixin)
I had a vague idea to do this with a class decorator, but my attempts led to confusion. Most of my searches on the topic have led in the direction of using inheritance (or in more complex scenarios, multiple inheritance) to achieve mixin behavior.
def mixer(*args):
"""Decorator for mixing mixins"""
def inner(cls):
for a,k in ((a,k) for a in args for k,v in vars(a).items() if callable(v)):
setattr(cls, k, getattr(a, k).im_func)
return cls
return inner
class Mixin(object):
def b(self): print "b()"
def c(self): print "c()"
class Mixin2(object):
def d(self): print "d()"
def e(self): print "e()"
#mixer(Mixin, Mixin2)
class Foo(object):
# Somehow mix in the behavior of the Mixin class,
# so that all of the methods below will run and
# the issubclass() test will be False.
def a(self): print "a()"
f = Foo()
f.a()
f.b()
f.c()
f.d()
f.e()
print issubclass(Foo, Mixin)
output:
a()
b()
c()
d()
e()
False
You can add the methods as functions:
Foo.b = Mixin.b.im_func
Foo.c = Mixin.c.im_func
I am not that familiar with Python, but from what I know about Python metaprogramming, you could actually do it pretty much the same way it is done in Ruby.
In Ruby, a module basically consists of two things: a pointer to a method dictionary and a pointer to a constant dictionary. A class consists of three things: a pointer to a method dictionary, a pointer to a constant dictionary and a pointer to the superclass.
When you mix in a module M into a class C, the following happens:
an anonymous class α is created (this is called an include class)
α's method dictionary and constant dictionary pointers are set equal to M's
α's superclass pointer is set equal to C's
C's superclass pointer is set to α
In other words: a fake class which shares its behavior with the mixin is injected into the inheritance hierarchy. So, Ruby actually does use inheritance for mixin composition.
I left out a couple of subleties above: first off, the module doesn't actually get inserted as C's superclass, it gets inserted as C's superclasses' (which is C's singleton class) superclass. And secondly, if the mixin itself has mixed in other mixins, then those also get wrapped into fake classes which get inserted directly above α, and this process is applied recursively, in case the mixed in mixins in turn have mixins.
Basically, the whole mixin hierarchy gets flattened into a straight line and spliced into the inheritance chain.
AFAIK, Python actually allows you to change a class's superclass(es) after the fact (something which Ruby does not allow you to do), and it also gives you access to a class's dict (again, something that is impossible in Ruby), so you should be able to implement this yourself.
EDIT: Fixed what could (and probably should) be construed as a bug. Now it builds a new dict and then updates that from the class's dict. This prevents mixins from overwriting methods that are defined directly on the class. The code is still untested but should work. I'm busy ATM so I'll test it later. It worked fine except for a syntax error. In retrospect, I decided that I don't like it (even after my further improvements) and much prefer my other solution even if it is more complicated. The test code for that one applies here as well but I wont duplicate it.
You could use a metaclass factory:
import inspect
def add_mixins(*mixins):
Dummy = type('Dummy', mixins, {})
d = {}
for mixin in reversed(inspect.getmro(Dummy)):
d.update(mixin.__dict__)
class WithMixins(type):
def __new__(meta, classname, bases, classdict):
d.update(classdict)
return super(WithMixins, meta).__new__(meta, classname, bases, d)
return WithMixins
then use it like:
class Foo(object):
__metaclass__ = add_mixins(Mixin1, Mixin2)
# rest of the stuff
This one is based on the way it's done in ruby as explained by Jörg W Mittag. All of the wall of code after if __name__=='__main__' is test/demo code. There's actually only 13 lines of real code to it.
import inspect
def add_mixins(*mixins):
Dummy = type('Dummy', mixins, {})
d = {}
# Now get all the class attributes. Use reversed so that conflicts
# are resolved with the proper priority. This rules out the possibility
# of the mixins calling methods from their base classes that get overridden
# using super but is necessary for the subclass check to fail. If that wasn't a
# requirement, we would just use Dummy above (or use MI directly and
# forget all the metaclass stuff).
for base in reversed(inspect.getmro(Dummy)):
d.update(base.__dict__)
# Create the mixin class. This should be equivalent to creating the
# anonymous class in Ruby.
Mixin = type('Mixin', (object,), d)
class WithMixins(type):
def __new__(meta, classname, bases, classdict):
# The check below prevents an inheritance cycle from forming which
# leads to a TypeError when trying to inherit from the resulting
# class.
if not any(issubclass(base, Mixin) for base in bases):
# This should be the the equivalent of setting the superclass
# pointers in Ruby.
bases = (Mixin,) + bases
return super(WithMixins, meta).__new__(meta, classname, bases,
classdict)
return WithMixins
if __name__ == '__main__':
class Mixin1(object):
def b(self): print "b()"
def c(self): print "c()"
class Mixin2(object):
def d(self): print "d()"
def e(self): print "e()"
class Mixin3Base(object):
def f(self): print "f()"
class Mixin3(Mixin3Base): pass
class Foo(object):
__metaclass__ = add_mixins(Mixin1, Mixin2, Mixin3)
def a(self): print "a()"
class Bar(Foo):
def f(self): print "Bar.f()"
def test_class(cls):
print "Testing {0}".format(cls.__name__)
f = cls()
f.a()
f.b()
f.c()
f.d()
f.e()
f.f()
print (issubclass(cls, Mixin1) or
issubclass(cls, Mixin2) or
issubclass(cls, Mixin3))
test_class(Foo)
test_class(Bar)
You could decorate the classes __getattr__ to check in the mixin. The problem is that all methods of the mixin would always require an object the type of the mixin as their first parameter, so you would have to decorate __init__ as well to create a mixin-object. I believe you could achieve this using a class decorator.
from functools import partial
class Mixin(object):
#staticmethod
def b(self): print "b()"
#staticmethod
def c(self): print "c()"
class Foo(object):
def __init__(self, mixin_cls):
self.delegate_cls = mixin_cls
def __getattr__(self, attr):
if hasattr(self.delegate_cls, attr):
return partial(getattr(self.delegate_cls, attr), self)
def a(self): print "a()"
f = Foo(Mixin)
f.a()
f.b()
f.c()
print issubclass(Foo, Mixin)
This basically uses the Mixin class as a container to hold ad-hoc functions (not methods) that behave like methods by taking an object instance (self) as the first argument. __getattr__ will redirect missing calls to these methods-alike functions.
This passes your simple tests as shown below. But I cannot guarantee it will do all the things you want. Make more thorough test to make sure.
$ python mixin.py
a()
b()
c()
False
Composition? It seems like that would be the simplest way to handle this: either wrap your object in a decorator or just import the methods as an object into your class definition itself. This is what I usually do: put the methods that I want to share between classes in a file and then import the file. If I want to override some behavior I import a modified file with the same method names as the same object name. It's a little sloppy, but it works.
For example, if I want the init_covers behavior from this file (bedg.py)
import cove as cov
def init_covers(n):
n.covers.append(cov.Cover((set([n.id]))))
id_list = []
for a in n.neighbors:
id_list.append(a.id)
n.covers.append(cov.Cover((set(id_list))))
def update_degree(n):
for a in n.covers:
a.degree = 0
for b in n.covers:
if a != b:
a.degree += len(a.node_list.intersection(b.node_list))
In my bar class file I would do: import bedg as foo
and then if I want to change my foo behaviors in another class that inherited bar, I write
import bild as foo
Like I say, it is sloppy.
Related
Suppose I have different classes providing access to different subsystems but with a common interface. They all provide the same set of methods but each class implements them in a different way (think using foo.write() to write to a file or send data via socket, etc)
Since the interface is the same, I wanted to make a single class that is able to pick the correct class but only based on the constructor/initializer parameters.
On code, it would look like
class Foo(object):
def who_am_i(self):
print "Foo"
class Bar(object):
def who_am_i(self):
print "Bar"
# Class that decides which one to use and adds some methods that are common to both
class SomeClass(Foo, Bar):
def __init__(self, use_foo):
# How inherit methods from Foo -OR- Bar?
How can the SomeClass inherit methods from Foo or Bar given the __init__ and/or __new__ arguments?
The goal should be something like
>>> some_object = SomeClass(use_foo=True)
>>> some_object.who_am_i()
Foo
>>> another_object = SomeClass(use_foo=False)
>>> another_object.who_am_i()
Bar
Is there some clean "pythonic" way to achieve this? I didn't wanted to use a function to dynamically define SomeClass, but I'm not finding another way to do this.
Thanks!
As mentioned in the comments, this can be done with a factory function (a function that pretends to be a class):
def SomeClass(use_foo):
if use_foo:
return Foo()
else:
return Bar()
As far as I can see, you have your inheritance completely backwards; instead of the multiple inheritance you're proposing:
Foo Bar
- foo code - bar code
\ /
SomeClass(Foo, Bar)
- common code
you could use a much simpler single inheritance model:
SomeClass
- common code
/ \
Foo(SomeClass) Bar(SomeClass)
- foo code - bar code
This then makes your problem one of choosing which subclass to instantiate (a decision that only needs to be made once) rather than which superclass method to call (which potentially needs to be made on every method call). This could be solved with as little as:
thing = Foo() if use_foo else Bar()
A class factory can be used here. Note the use of dictionary to make sure that the same subclass instance is used for each base class.
def makeclass(baseclass, classes={}):
if baseclass not in classes:
class Class(baseclass):
pass # define your methods here
classes[baseclass] = Class
return classes[baseclass]
obj1 = makeclass(Foo)(...)
obj2 = makeclass(Bar)(...)
isinstance(obj1, makeclass(Foo)) # True
isinstance(obj1, Foo) # True
issubclass(makeclass(Foo), Foo) # True
issubclass(type(obj1), Foo) # True
You could also make a dict subclass with a __missing__ method to do essentially the same thing; it makes it more explicit that you've got a container that stores classes, but creates them on demand:
class ClassDict(dict):
def __missing__(self, baseclass):
class Class(baseclass):
pass # define your methods here
self[baseclass] = Class
return Class
subclasses = ClassDict()
obj1 = subclasses[Foo]
obj2 = subclasses[Bar]
Judging by the lack of agreement upon the answer, maybe the problem is the question. jonrsharpe's comment gave an interesting insight on the problem: this should not be solved via inheritance.
Consider SomeClass defined as follows:
# Class that uses Foo or Bar depending on the environment
# Notice it doesn't subclasses either Foo or Bar
class SomeClass(object):
def __init__(self, use_foo):
if use_foo:
self.handler = Foo()
else:
self.handler = Bar()
# Makes more sense asking 'Who implements?' instead of 'Who am I?'
def who_implements(self):
return self.handler
# Explicitly expose methods from the handler
def some_handler_method(self, *args, **kwargs):
return self.handler.some_handler_method(*args, **kwargs)
def another_handler_method(self, *args, **kwargs):
return self.handler.another_handler_method(*args, **kwargs)
Should we need to get details on the handler implementation, just get the handler attribute. Other classes that subclass SomeClass won't even see the handler directly, which actually makes sense.
One could use a __new__ method for this purpose:
_foos = {}
_bars = {}
class SomeClass(object):
def __new__(cls,use_foo,*args,**kwargs):
if use_foo:
if cls not in _foos:
class specialized(cls,Foo):pass
_foos[cls] = specialized
else:
specialized = _foos[cls]
else:
if cls not in _bars:
class specialized(cls,Bar):pass
_bars[cls] = specialized
else:
specialized = _bars[cls]
specialized.__name__ = cls.__name__
return object.__new__(specialized,*args,**kwargs)
#common methods to both go here
pass
The advantage of this over a factory function is that isinstance(SomeClass(True),SomeClass) works, and that SomeClass can be subclassed.
Consider two classes that each have an existing shared function, but separate inheritance paths:
class ClazzA(SuperClazzX):
def foo(self):
return goo(super(SuperClazzX, self).foo())
class ClazzB(SuperClazzY):
def foo(self):
return goo(super(SuperClazzY, self).foo())
foo is clearly a common function which can be extracted to a mixin, what is the right way to do so, such that the functionality remains even though a different super foo is to be called?
Edit: I removed the other mixin, it was confusing and irrelevant.
EDIT: simpler code
Mixin has access to other bases of (future) child class, here C::Mixin::foo has access to the other base of C, namely C::Base::foo. Authoritative explanation here.
class Base(object): # must be new-style class in py 2.x
def foo(self):
print "base foo called"
class Mixin(object):
def foo(self):
rv = super(Mixin, self).foo() # call actual implementation
# use rv
class C(Mixin, Base):
pass
C().foo()
base foo called
What this does:
self is instance of C, it's __mro__ is (Mixin, Base)
when Mixin calls super(Mixin, self), the resulting object retains bases (Base,)
when .foo attribute is resolved, this object finds it in Base
thus Base.foo is invoked with original self
If you want custom control over implementation, you have access to your own bases, e.g.:
class Foo(...):
def foo(self):
print self.__class__.__bases__
Your mixin could look something like this, super-manual approach:
class Mixin(object):
def foo(self):
assert self.__class__ is not Mixin # no point calling directly
# find the other base
other = [b for b in self.__class__.__bases__ if b is not Mixin]
# what to do if there's more than 1 real base?
# pick base explicitly
base = other[1]
# call it,
return something(base.foo(self, some_args))
How can I access a class's __dict__ within its own "class initializer" (class definition) code? Failing that, how can I access the class's-to-be-defined attributes as strings (so that one can generate their names, and set their values, programmatically)
class A: # from a library by a stubborn maintainer
def __init__ (self):
self.hello = "Hello"
# + some huge misguided crap that I do not control
def f1 (self):
self.f2 ()
def f2 (self):
print self.hello
class B:
f1 = A.__dict__ ['f1'] # works
# how to loop over 2..10 and set B.f$i <- A.f$i ?
# this doesn't work:
#B.__dict__ ['f2'] = A.__dict__ ['f2'] # B not yet defined
def __init__ (self):
#self.hello = A ().hello # the actual code has to be copied manually
# misguided crap removed
a = B ()
a.f2()
It would be good to also not copy/paste the self.hello initialization just to bypass the misguided stuff, but I don't think that can be helped easily short of refactoring tools.
Sorry, you can't really do that while the class is being defined. In other words, you can't access a class's__dict__in its own definition because the results of executing the definition will become the dictionary's content (that's why the class name hasn't been bound to anything yet). Probably the simplest workaround for this would be to give your class a __metaclass__.
A metaclass is the class-of-a-class, so instances of them are classes (hence the name). For simplicity I've put the definition of one forBinside of it, since its use and scope will be limited to that class (this, however, can't be done in Python 3).
The code in__metaclass__.__new__()is only executed when instance of it are created, namely during classB's definition, not whenever instances of classBare created, so the overhead of copying all the methods is will incurred only when that happens -- usually once. Note that because class methods are also descriptor objects, it's necessary to call the__get__method of each one in order to get the proper value to bind them to the another class object. See the section titled Descriptor HowTo Guide by Raymond Hettinger in the documentation for more details.
class A(object):
def __init__(self):
self.hello = "Hello"
# + some huge misguided crap that I do not control
def f1(self):
self.f2()
def f2(self):
print self.hello
class B(object):
class __metaclass__(type):
A_METHODS = ['f{}'.format(i) for i in xrange(1, 11)] # 'f1'...'f10'
def __new__(mcls, classname, bases, classdict):
classobj = type.__new__(mcls, classname, bases, classdict)
classobj.hello = A().hello
for fn in mcls.A_METHODS:
if fn in A.__dict__:
setattr(classobj, fn, A.__dict__[fn].__get__(classobj, A))
return classobj
def __init__(self):
pass
a = B()
a.f2() # --> Hello
There may be a better approach entirely (such as forking the class A and removing the misguided code, or having each B instance hold or share an A instance, or modifying B.__dict__ after the definition of B is complete).
That said, you could replace:
B.__dict__ ['f2'] = A.__dict__ ['f2'] # B not yet defined
with:
locals()['f2'] = A.__dict__ ['f2']
This is not guaranteed to work because you aren't allowed to modify locals(), but it does appear to work in CPython 2.7.5 at least:
class Foo:
a = locals()
locals()['b'] = 0
print Foo.a is Foo.__dict__, Foo.b
outputs True 0. Interestingly, it outputs False 0 if Foo inherits from object, which indicates the difficult water I'm in.
Of course this is a horrible abuse of implementation-specific behaviour, is not recommended, will probably go wrong somewhere down the line, etc :-)
When you decorate a method, it is not bound yet to the class, and therefor doesn't have the im_class attribute yet. I looking for a way to get the information about the class inside the decorator. I tried this:
import types
def decorator(method):
def set_signal(self, name, value):
print name
if name == 'im_class':
print "I got the class"
method.__setattr__ = types.MethodType(set_signal, method)
return method
class Test(object):
#decorator
def bar(self, foo):
print foo
But it doesn't print anything.
I can imagine doing this:
class Test(object):
#decorator(klass=Test)
def bar(self, foo):
print foo
But if I can avoid it, it would make my day.
__setattr__ is only called on explicit object.attribute = assignments; building a class does not use attribute assignment but builds a dictionary (Test.__dict__) instead.
To access the class you have a few different options though:
Use a class decorator instead; it'll be passed the completed class after building it, you could decorate individual methods on that class by replacing them (decorated) in the class. You could use a combination of a function decorator and a class decorator to mark which methods are to be decorated:
def methoddecoratormarker(func):
func._decorate_me = True
return func
def realmethoddecorator(func):
# do something with func.
# Note: it is still an unbound function here, not a method!
return func
def classdecorator(klass):
for name, item in klass.__dict__.iteritems():
if getattr(item, '_decorate_me', False):
klass.__dict__[name] = realmethoddecorator(item)
You could use a metaclass instead of a class decorator to achieve the same, of course.
Cheat, and use sys._getframe() to retrieve the class from the calling frame:
import sys
def methoddecorator(func):
callingframe = sys._getframe(1)
classname = callingframe.f_code.co_name
Note that all you can retrieve is the name of the class; the class itself is still being built at this time. You can add items to callingframe.f_locals (a mapping) and they'll be made part of the new class object.
Access self whenever the method is called. self is a reference to the instance after all, and self.__class__ is going to be, at the very least, a sub-class of the original class the function was defined in.
My strict answer would be: It's not possible, because the class does not yet exist when the decorator is executed.
The longer answer would depend on your very exact requirements. As I wrote, you cannot access the class if it does not yet exists. One solution would be, to mark the decorated method to be "transformed" later. Then use a metaclass or class decorator to apply your modifications after the class has been created.
Another option involves some magic. Look for the implementation of the implements method in zope.interfaces. It has some access to the information about the class which is just been parsed. Don't know if it will be enough for your use case.
You might want to take a look at descriptors. They let you implement a __get__ that is used when an attribute is accessed, and can return different things depending on the object and its type.
Use method decorators to add some marker attributes to the interesting methods, and use a metaclass which iterates over the methods, finds the marker attributes, and does the logic. The metaclass code is run when the class is created, so it has a reference to the newly created class.
class MyMeta(object):
def __new__(...):
...
cls = ...
... iterate over dir(cls), find methods having .is_decorated, act on them
return cls
def decorator(f):
f.is_decorated = True
return f
class MyBase(object):
__metaclass__ = MyMeta
class MyClass(MyBase):
#decorator
def bar(self, foo):
print foo
If you worry about that the programmer of MyClass forgets to use MyBase, you can forcibly set the metaclass in decorator, by exampining the globals dicitionary of the caller stack frame (sys._getframe()).
Take the following minimal example:
import abc
class FooClass(object):
__metaclass__ = abc.ABCMeta
#abc.abstractmethod
def FooMethod(self):
raise NotImplementedError()
def main():
derived_type = type('Derived', (FooClass,), {})
def BarOverride(self):
print 'Hello, world!'
derived_type.FooMethod = BarOverride
instance = derived_type()
Running main() gets you:
TypeError: Can't instantiate abstract class Derived with abstract methods FooMethod
(The exception occurs on the instance = derived_type() line.)
But FooMethod shouldn't be abstract: I've overridden it with BarOverride. So, why is this raising exceptions?
Disclaimer: Yes, I could use the explicit class syntax, and accomplish the exact same thing. (And even better, I can make it work!) But this is a minimal test case, and the larger example is dynamically creating classes. :-) And I'm curious as to why this doesn't work.
Edit: And to prevent the other obvious non-answer: I don't want to pass BarOverride in the third argument to type: In the real example, BarOverride needs to have derived_type bound to it. It is easier to do this if I can define BarOverride after the creation of derived_type. (If I can't do this, then why?)
Because the docs say so:
Dynamically adding abstract methods to a class, or attempting to
modify the abstraction status of a method or class once it is created,
are not supported. The abstractmethod() only affects subclasses
derived using regular inheritance; “virtual subclasses” registered
with the ABC’s register() method are not affected.
A metaclass is only called when a class is defined. When abstractmethod has marked a class as abstract that status won't change later.
Jochen is right; the abstract methods are set at class creation and won't me modified just because you reassign an attribute.
You can manually remove it from the list of abstract methods by doing
DerivedType.__abstractmethods__ = frozenset()
or
DerivedType.__abstractmethods__ = frozenset(
elem for elem in DerivedType.__abstractmethods__ if elem != 'FooMethod')
as well as setattr, so it doesn't still think that FooMethod is abstract.
I know this topic is really old but... That is really a nice question.
It doesn't work because abc can only check for abstract methods during instatiation of types, that is, when type('Derived', (FooClass,), {}) is running. Any setattr done after that is not accessible from abc.
So, setattr wont work, buuut...
Your problem of addressing the name of a class that wasn't previously declared or defined looks solvable:
I wrote a little metaclass that lets you use a placeholder "clazz" for accessing any class that will eventually get the method you are writing outside a class definition.
That way you won't get TypeError from abc anymore, since you can now define your method BEFORE instatiating your type, and then pass it to type at the dict argument. Then abc will see it as a proper method override.
Aaand, with the new metaclass you can refer to the class object during that method.
And this is super, because now you can use super! =P
I can guess you were worried about that too...
Take a look:
import abc
import inspect
clazz = type('clazz', (object,), {})()
def clazzRef(func_obj):
func_obj.__hasclazzref__ = True
return func_obj
class MetaClazzRef(type):
"""Makes the clazz placeholder work.
Checks which of your functions or methods use the decorator clazzRef
and swaps its global reference so that "clazz" resolves to the
desired class, that is, the one where the method is set or defined.
"""
methods = {}
def __new__(mcs, name, bases, dict):
ret = super(MetaClazzRef, mcs).__new__(mcs, name, bases, dict)
for (k,f) in dict.items():
if getattr(f, '__hasclazzref__', False):
if inspect.ismethod(f):
f = f.im_func
if inspect.isfunction(f):
for (var,value) in f.func_globals.items():
if value is clazz:
f.func_globals[var] = ret
return ret
class MetaMix(abc.ABCMeta, MetaClazzRef):
pass
class FooClass(object):
__metaclass__ = MetaMix
#abc.abstractmethod
def FooMethod(self):
print 'Ooops...'
#raise NotImplementedError()
def main():
#clazzRef
def BarOverride(self):
print "Hello, world! I'm a %s but this method is from class %s!" % (type(self), clazz)
super(clazz, self).FooMethod() # Now I have SUPER!!!
derived_type = type('Derived', (FooClass,), {'FooMethod': BarOverride})
instance = derived_type()
instance.FooMethod()
class derivedDerived(derived_type):
def FooMethod(self):
print 'I inherit from derived.'
super(derivedDerived,self).FooMethod()
instance = derivedDerived()
instance.FooMethod()
main()
The output is:
Hello, world! I'm a <class 'clazz.Derived'> but this method is from class <class 'clazz.Derived'>!
Ooops...
I inherit from derived.
Hello, world! I'm a <class 'clazz.derivedDerived'> but this method is from class <class 'clazz.Derived'>!
Ooops...
Well, if you must do it this way, then you could just pass a dummy dict {'FooMethod':None} as the third argument to type. This allows derived_type to satisfy ABCMeta's requirement that all abstract methods be overridden. Later on you can supply the real FooMethod:
def main():
derived_type = type('Derived', (FooClass,), {'FooMethod':None})
def BarOverride(self):
print 'Hello, world!'
setattr(derived_type, 'FooMethod', BarOverride)
instance = derived_type()