EDIT: My assumptions were wrong; the code below does in fact work like I wanted it to. The behaviour I was observing turned out to be caused by another bit of the code I was working with that I overlooked because it looked completely unrelated.
So I have some code structured like this:
class B(object):
def foo(self):
print "spam"
def bar(self):
do_something()
self.foo()
do_something_else()
class C(B):
def foo(self):
print "ham"
def bar(self):
super(C, self).bar()
print "eggs"
c = C()
c.bar()
This would do_something(), print "spam", do_something_else(), and then print "eggs". However, what I want to do is do_something(), print "ham", do_something_else(), and then print "eggs". In other words, I want to call B's bar method from C's bar method, but I want it to call C's foo method, not B's.
Is there a way to do this? Note that in the actual code I'm dealing with, both the B class and the code that actually calls c.bar() are part of an evolving third-party library, so mucking with that would be a last resort.
The code you have posted does what you want.
When B.bar calls self.foo() when self is an object of type C, it will call C.foo.
This works because self.foo is looked up first on self then on self's actual type's method-resolution-order C3-linearised class-and-base-classes tuple (here (C, B, object)). This will return C.foo before B.foo.
Related
class parent:
def fun1(self):
print("foo")
fun2 = fun1
class child(parent):
def fun1(self):
print("bar")
child().fun2()
This code outputs "foo". I'm pretty sure I understand why this is happening, but I'm wondering if there is a straightforward way to implement what I want (inheritable aliases that behave according to MRO, so it outputs "bar") or a reason why this is bad programming practice.
There is no way to directly do what you want.
To understand why, you need to understand the way method lookup works. It's explained in detail in the Descriptor HOWTO, and I tried to write a less expert-level explanation here, so I'll assume you read both of those and just show the effects:
>>> child.fun1
<function __main__.child.fun1>
>>> child.fun2
<function __main__.parent.fun1>
>>> child().fun2
<bound method parent.fun1 of <__main__.child object at 0x10b735ba8>>
Notice that child.fun2 is parent.fun1, not child.fun1. Which explains why child().fun2 ends up as a bound method around parent.fun1, even though the self ends up as a child instance. Why? Well, obviously fun2 is not in child.__dict__ (or we wouldn't need an MRO). And in parent.__dict__, it can't be anything but parent.fun1.
So, what are the workarounds?
You could make fun2 into a property that forwards to fun2, as Patrick Haugh suggested. Or you can just do this:
def fun2(self):
return self.fun1()
This solution has the benefit of being dead simple—anyone who can Python will understand why it works.
But Patrick's solution has the advantage of making not just child().fun2(), but also child().fun2 (as an object you can pass around, compare for identity, etc.) work the way you want it to.
Meanwhile, there is an idiom closely related to what you're asking for, where a set of public methods that you don't expect to override call a "protected" implementation method that you do. For example, a simple 1D array-math class might do this:
def _math(lhs, rhs, op):
# not actually a useful implementation...
return [op(x, y) for x, y in zip(lhs, rhs)]
def __add__(self, other):
return self._math(other, add)
def __radd__(self, other):
return self._math(other, add)
# etc.
And now there's no asymmetry between __add__ and __radd__, only between the "public" interface (__add__ and __radd__) and the "protected" one (_math).
Here's a very straightforward way—that's perfectly OK:
class Parent:
def fun1(self):
print("foo")
def fun2(self):
return self.fun1()
class Child(Parent):
def fun1(self):
print("bar")
Child().fun2() # -> bar
You could have fun2 be a property that returns self.fun1
class Parent:
def fun1(self):
print('foo')
#property
def fun2(self):
return self.fun1
class Child(Parent):
def fun1(self):
print('bar')
Child().fun2()
# bar
I'm making a program in python in which specific instances of an object must be decorated with new functions built at runtime.
I've seen very simple examples of adding functions to objects through MethodType:
import types
def foo():
print("foo")
class A:
bar = "bar"
a = A()
a.foo = types.MethodType(foo, a)
But none of the examples I've seen show how a function added in this manner can reference to the new owner's attributes. As far as I know, even though this binds the foo() function to the instance a, foo() must still be a pure function, and cannot contain references to anything local.
In my case, I need functions to change attributes of the object they are added to. Here are two examples of the kind of thing I need to be able to do:
class A:
foo = "foo"
def printme():
print(foo)
def nofoo():
foo = "bar"
def printBar():
if foo != "foo"
self.printme()
I would then need a way to add a copy of a nofoo() or printBar() to an A object in such a way that they can access the object attributes named foo and the function named printme() correctly.
So, is this possible? Is there a way to do this kind of programming in vanilla Python? or at least Is there a programming pattern that achieves this kind of behavior?
P.S.: In my system, I also add attributes dynamically to objects. Your first thought then might be "How can I ever be sure that the object I'm adding the nofoo() function to actually has an attribute named foo?", but I also have a fairly robust tag system that makes sure that I never try to add a nofoo() function to an object that hasn't a foo variable. The reason I mention this is that solutions that look at the class definition aren't very useful to me.
As said in the comments, your function actually must take at least one parameter: self, the instance the method is being called on. The self parameter can be used as it would be used in a normal instance method. Here is an example:
>>> from types import MethodType
>>>
>>> class Class:
def method(self):
print('method run')
>>> cls = Class()
>>>
>>> def func(self): # must accept one argument, `self`
self.method()
>>> cls.func = MethodType(func, cls)
>>> cls.func()
method run
>>>
Without your function accepting self, an exception would be raised:
>>> def func():
self.method()
>>> cls.func = MethodType(func, cls)
>>> cls.func()
Traceback (most recent call last):
File "<pyshell#21>", line 1, in <module>
cls.func()
TypeError: func() takes 0 positional arguments but 1 was given
>>>
class A:
def __init__(self):
self.foo = "foo"
def printme(self):
print(self.foo)
def nofoo(self):
self.foo = "bar"
a.nofoo = types.MethodType(nofoo, a)
a.nofoo()
a.printme()
prints
bar
It's not entirely clear what you're trying to do, and I'm worried that whatever it is may be a bad idea. However, I can explain how to do what you're asking, even if it isn't what you want, or should want. I'll point out that it's very uncommon to want to do the second version below, and even rarer to want to do the third version, but Python does allow them both, because "even rarer than very uncommon" still isn't "never". And, in the same spirit…
The short answer is "yes". A dynamically-added method can access the owner object exactly the same way a normal method can.
First, here's a normal, non-dynamic method:
class C:
def meth(self):
return self.x
c = C()
c.x = 3
c.meth()
Obviously, with a normal method like this, when you call c.meth(), the c ends up as the value of the self parameter, so self.x is c.x, which is 3.
Now, here's how you dynamically add a method to a class:
class C:
pass
c = C()
c.x = 3
def meth(self):
print(self.x)
C.meth = meth
c.meth()
This is actually doing exactly the same thing. (Well, we've left another name for the same function object sitting around in globals, but that's the only difference) If C.meth is the same function it was in the first version, then obviously whatever magic made c.meth() work in the first version will do the exact same thing here.
(This used to be slightly more complicated in Python 2, because of unbound methods, and classic classes too… but fortunately you don't have to worry about that.)
Finally, here's how you dynamically add a method to an instance:
class C:
pass
c = C()
c.x = 3
def meth(self):
print(self.x)
c.meth = types.MethodType(meth, c)
c.meth()
Here, you actually have to know the magic that makes c.meth() work in the first two cases. So read the Descriptor HOWTO. After that, it should be obvious.
But if you just want to pretend that Guido is a wizard (Raymond definitely is a wizard) and it's magic… Well, in the first two versions, Guido's magic wand creates a special bound method object whenever you ask for c.meth, but even he isn't magical enough to do that when C.meth doesn't exist. But we can painstakingly create that same bound method object and store it as c.meth. After that, we're going to get the same thing we stored whenever we ask for c.meth, which we explicitly built as the same thing we got in the first two examples, so it'll obviously do the same thing.
But what if we did this:
class C:
pass
c = C()
c.x = 3
def meth(self):
print(self.x)
c.meth = meth
c.meth(c)
Here, you're not letting Guido do his descriptor magic to create c.meth, and you're not doing it manually, you're just sticking a regular function there. Which means if you want anything to show up as the self parameter, you have to explicitly pass it as an argument, as in that silly c.meth(c) line at the end. But if you're willing to do that, then even this one works. No matter how self ends up as c, self.x is going to be c.x.
From a famous example, I learned the difference between method, classmethod and staticmethod in a Python class.
Source:
What is the difference between #staticmethod and #classmethod in Python?
class A(object):
def foo(self,x):
print "executing foo(%s,%s)"%(self,x)
#classmethod
def class_foo(cls,x):
print "executing class_foo(%s,%s)"%(cls,x)
#staticmethod
def static_foo(x):
print "executing static_foo(%s)"%x
# My Guesses
def My_Question(self,x):
self.foo(x)
A.class_foo(x)
A.static_foo(x)
a=A()
Now I am wondering, how to call a method, #classmethod, and #staticmethod inside the class.
I put my guesses in the My_Question function above, please correct me if I am wrong with any of these.
Yes, your guesses will work. Note that it is also possible/normal to call staticmethods and classmethods outside the class:
class A():
...
A.class_foo()
A.static_foo()
Also note that inside regular instance methods, it's customary to call the staticmethods and class methods directly on the instance (self) rather than the class (A):
class A():
def instance_method(self):
self.class_foo()
self.static_foo()
This allow for inheritance to work as you might expect -- If I create a B subclass from A, if I call B.instance_method(), my class_foo function will get B instead of A as the cls argument -- And possibly, if I override static_foo on B to do something slightly different than A.static_foo, this will allow the overridden version to be called as well.
Some examples might make this more clear:
class A(object):
#staticmethod
def static():
print("Static, in A")
#staticmethod
def staticoverride():
print("Static, in A, overrideable")
#classmethod
def clsmethod(cls):
print("class, in A", cls)
#classmethod
def clsmethodoverrideable(cls):
print("class, in A, overridable", cls)
def instance_method(self):
self.static()
self.staticoverride()
self.clsmethod()
self.clsmethodoverride()
class B(A):
#classmethod
def clsmethodoverrideable(cls):
print("class, in B, overridable", cls)
#staticmethod
def staticoverride():
print("Static, in B, overrideable")
a = A()
b = B()
a.instance_method()
b.instance_method()
...
After you've run that, try it by changing all of the self. to A. inside instance_method. Rerun and compare. You'll see that all of the references to B have gone (even when you're calling b.instance_method()). This is why you want to use self rather than the class.
As #wim said, what you have is right. Here's the output when My_Question is called.
>>> a.My_Question("My_Answer=D")
executing foo(<__main__.A object at 0x0000015790FF4668>,My_Answer=D)
executing class_foo(<class '__main__.A'>,My_Answer=D)
executing static_foo(My_Answer=D)
There are a lot of great resources on super(), including this great blog post that pops up a lot, as well as many questions on Stack Overflow. However I feel like they all stop short of explaining how it works in the most general case (with arbitrary inheritance graphs), as well as what is going on under the hood.
Consider this basic example of diamond inheritance:
class A(object):
def foo(self):
print 'A foo'
class B(A):
def foo(self):
print 'B foo before'
super(B, self).foo()
print 'B foo after'
class C(A):
def foo(self):
print 'C foo before'
super(C, self).foo()
print 'C foo after'
class D(B, C):
def foo(self):
print 'D foo before'
super(D, self).foo()
print 'D foo after'
If you read up on Python's rules for method resolution order from sources like this or look up the wikipedia page for C3 linearization, you will see that the MRO must be (D, B, C, A, object). This is of course confirmed by D.__mro__:
(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <type 'object'>)
And
d = D()
d.foo()
prints
D foo before
B foo before
C foo before
A foo
C foo after
B foo after
D foo after
which matches the MRO. However, consider that above super(B, self).foo() in B actually calls C.foo, whereas in b = B(); b.foo() it would simply go straight to A.foo. Clearly using super(B, self).foo() is not simply a shortcut for A.foo(self) as is sometimes taught.
super() is then obviously aware of the previous calls before it and the overall MRO the chain is trying to follow. I can see two ways this might be accomplished. The first is to do something like passing the super object itself as the self argument to the next method in the chain, which would act like the original object but also contain this information. However this also seems like it would break a lot of things (super(D, d) is d is false) and by doing a little experimenting I can see this isn't the case.
The other option is to have some sort of global context that stores the MRO and the current position in it. I imagine the algorithm for super goes something like:
Is there currently a context we are working in? If not, create one which contains a queue. Get the MRO for the class argument, push all elements except for the first into the queue.
Pop the next element from the current context's MRO queue, use it as the current class when constructing the super instance.
When a method is accessed from the super instance, look it up in the current class and call it using the same context.
However, this doesn't account for weird things like using a different base class as the first argument to a call to super, or even calling a different method on it. I would like to know the general algorithm for this. Also, if this context exists somewhere, can I inspect it? Can I muck with it? Terrible idea of course, but Python typically expects you to be a mature adult even if you're not.
This also introduces a lot of design considerations. If I wrote B thinking only of its relation to A, then later someone else writes C and a third person writes D, my B.foo() method has to call super in a way that is compatible with C.foo() even though it didn't exist at the time I wrote it! If I want my class to be easily extensible I will need to account for this, but I am not sure if it is more complicated than simply making sure all versions of foo have identical signatures. There is also the question of when to put code before or after the call to super, even if it does not make any difference considering B's base classes only.
super() is then obviously aware of the previous calls before it
It's not. When you do super(B, self).foo, super knows the MRO because that's just type(self).__mro__, and it knows that it should start looking for foo at the point in the MRO immediately after B. A rough pure-Python equivalent would be
class super(object):
def __init__(self, klass, obj):
self.klass = klass
self.obj = obj
def __getattr__(self, attrname):
classes = iter(type(self.obj).__mro__)
# search the MRO to find self.klass
for klass in classes:
if klass is self.klass:
break
# start searching for attrname at the next class after self.klass
for klass in classes:
if attrname in klass.__dict__:
attr = klass.__dict__[attrname]
break
else:
raise AttributeError
# handle methods and other descriptors
try:
return attr.__get__(self.obj, type(self.obj))
except AttributeError:
return attr
If I wrote B thinking only of its relation to A, then later someone else writes C and a third person writes D, my B.foo() method has to call super in a way that is compatible with C.foo() even though it didn't exist at the time I wrote it!
There's no expectation that you should be able to multiple-inherit from arbitrary classes. Unless foo is specifically designed to be overloaded by sibling classes in a multiple-inheritance situation, D should not exist.
How can I access a class's __dict__ within its own "class initializer" (class definition) code? Failing that, how can I access the class's-to-be-defined attributes as strings (so that one can generate their names, and set their values, programmatically)
class A: # from a library by a stubborn maintainer
def __init__ (self):
self.hello = "Hello"
# + some huge misguided crap that I do not control
def f1 (self):
self.f2 ()
def f2 (self):
print self.hello
class B:
f1 = A.__dict__ ['f1'] # works
# how to loop over 2..10 and set B.f$i <- A.f$i ?
# this doesn't work:
#B.__dict__ ['f2'] = A.__dict__ ['f2'] # B not yet defined
def __init__ (self):
#self.hello = A ().hello # the actual code has to be copied manually
# misguided crap removed
a = B ()
a.f2()
It would be good to also not copy/paste the self.hello initialization just to bypass the misguided stuff, but I don't think that can be helped easily short of refactoring tools.
Sorry, you can't really do that while the class is being defined. In other words, you can't access a class's__dict__in its own definition because the results of executing the definition will become the dictionary's content (that's why the class name hasn't been bound to anything yet). Probably the simplest workaround for this would be to give your class a __metaclass__.
A metaclass is the class-of-a-class, so instances of them are classes (hence the name). For simplicity I've put the definition of one forBinside of it, since its use and scope will be limited to that class (this, however, can't be done in Python 3).
The code in__metaclass__.__new__()is only executed when instance of it are created, namely during classB's definition, not whenever instances of classBare created, so the overhead of copying all the methods is will incurred only when that happens -- usually once. Note that because class methods are also descriptor objects, it's necessary to call the__get__method of each one in order to get the proper value to bind them to the another class object. See the section titled Descriptor HowTo Guide by Raymond Hettinger in the documentation for more details.
class A(object):
def __init__(self):
self.hello = "Hello"
# + some huge misguided crap that I do not control
def f1(self):
self.f2()
def f2(self):
print self.hello
class B(object):
class __metaclass__(type):
A_METHODS = ['f{}'.format(i) for i in xrange(1, 11)] # 'f1'...'f10'
def __new__(mcls, classname, bases, classdict):
classobj = type.__new__(mcls, classname, bases, classdict)
classobj.hello = A().hello
for fn in mcls.A_METHODS:
if fn in A.__dict__:
setattr(classobj, fn, A.__dict__[fn].__get__(classobj, A))
return classobj
def __init__(self):
pass
a = B()
a.f2() # --> Hello
There may be a better approach entirely (such as forking the class A and removing the misguided code, or having each B instance hold or share an A instance, or modifying B.__dict__ after the definition of B is complete).
That said, you could replace:
B.__dict__ ['f2'] = A.__dict__ ['f2'] # B not yet defined
with:
locals()['f2'] = A.__dict__ ['f2']
This is not guaranteed to work because you aren't allowed to modify locals(), but it does appear to work in CPython 2.7.5 at least:
class Foo:
a = locals()
locals()['b'] = 0
print Foo.a is Foo.__dict__, Foo.b
outputs True 0. Interestingly, it outputs False 0 if Foo inherits from object, which indicates the difficult water I'm in.
Of course this is a horrible abuse of implementation-specific behaviour, is not recommended, will probably go wrong somewhere down the line, etc :-)