Given the following code that emulates a staticmethod:
class StaticMethod(object):
"Emulate PyStaticMethod_Type() in Objects/funcobject.c"
def __init__(self, f):
self.f = f
def __get__(self, obj, objtype=None):
print('getting')
return self.f
class A:
def func2():
print('hello')
func2 = StaticMethod(func2)
When I call:
A.func2
i get what i expect:
getting
<function __main__.A.func2>
When I call: A.func2() I get:
getting
hello
Does this mean then that whenever you call a Descriptor Decorator method Python first retrieves it from the Descriptor's __get__ method?
If, yes, how then does the method actually get called? What exactly is happening under the hood?
Does this mean then that whenever you call a Descriptor Decorator method Python first retrieves it from the Descriptor's get method?
When you access an attribute on an object where its class has a class attribute that is a descriptor, the object calls __get__ and returns the result.
A "descriptor decorator" is not really a thing, the fact that a decorator is used to set the descriptor is irrelevant to its functionality.
If, yes, how then does the method actually get called? What exactly is happening under the hood?
If you mean the underlying function that was decorated with staticmethod, then the method gets called whenever you call it, the descriptor/staticmethod itself doesn't prescribe that, it just returns the function.
Related
This question already has answers here:
How does `super` interacts with a class's `__mro__` attribute in multiple inheritance?
(2 answers)
Closed 4 years ago.
From Python3's documentation super() "returns a proxy object that delegates method calls to a parent or sibling class of type." What does that mean?
Suppose I have the following code:
class SuperClass():
def __init__(self):
print("__init__ from SuperClass.")
print("self object id from SuperClass: " + str(id(self)))
class SubClass(SuperClass):
def __init__(self):
print("__init__ from SubClass.")
print("self object id from SubClass: " + str(id(self)))
super().__init__()
sc = SubClass()
The output I get from this is:
__init__ from SubClass.
self object id from SubClass: 140690611849200
__init__ from SuperClass.
self object id from SuperClass: 140690611849200
This means that in the line super().__init__(), super() is returning the current object which is then implicitly passed to the superclass' __init__() method. Is this accurate or am I missing something here?
To put it simply, I want to understand the following:
When super().__init__() is run,
What exactly is being passed to __init__() and how? We are calling it on super() so whatever this is returning should be getting passed to the __init__() method from what I understand about Python so far.
Why don't we have to pass in self to super().__init__()?
returns a proxy object that delegates method calls to a parent or
sibling class of type.
This proxy is an object that acts as the method-calling portion of the parent class. It is not the class itself; rather, it's just enough information so that you can use it to call the parent class methods.
If you call __init__(), you get your own, local, sub-class __init__ function. When you call super(), you get that proxy object, which will redirect you to the parent-class methods. Thus, when you call super().__init__(), that proxy redirects the call to the parent-class __init__ method.
Similarly, if you were to call super().foo, you would get the foo method from the parent class -- again, re-routed by that proxy.
Is that clear to you?
Responses to OP comments
But that must mean that this proxy object is being passed to
__init__() when running super().__init__() right?
Wrong. The proxy object is like a package name, such as calling math.sqrt(). You're not passing math to sqrt, you're using it to denote which sqrt you're using. If you wanted to pass the proxy to __init__, the call would be __init__(super()). That call would be semantically ridiculous, of course.
When we have to actually pass in self which is the sc object in my example.
No, you are not passing in sc; that is the result of the object creation call (internal method __new__), which includes an invocation of init. For __init__, the self object is a new item created for you by the Python run-time system. For most class methods, that first argument (called self out of convention, this in other languages) is the object that invoked the method.
This means that in the line super().__init__(), super() is returning the current object which is then implicitly passed to the superclass' __init__() method. Is this accurate or am I missing something here?
>>> help(super)
super() -> same as super(__class__, <first argument>)
super call returns a proxy/wrapper object which remembers:
The instance invoking super()
The class of the calling object
The class that's invoking super()
This is perfectly sound. super always fetches the attribute of the next class in the hierarchy ( really the MRO) that has the attribute that you're looking for. So it's not returning the current object, but rather and more accurately, it returns an object that remembers enough information to search for attributes higher in the class hierarchy.
What exactly is being passed to __init__() and how? We are calling it on super() so whatever this is returning should be getting passed to the __init__() method from what I understand about Python so far.
You're almost right. But super loves to play tricks on us. super class defines __getattribute__, this method is responsible for attribute search. When you do something like: super().y(), super.__getattribute__ gets called searching for y. Once it finds y it passes the instance that's invoking the super call to y. Also, super has __get__ method, which makes it a descriptor, I'll omit the details of descriptors here, refer to the documentation to know more. This answers your second question as well, as to why self isn't passed explicitly.
*Note: super is a little bit different and relies on some magic. Almost for all other classes, the behavior is the same. That is:
a = A() # A is a class
a.y() # same as A.y(a), self is a
But super is different:
class A:
def y(self):
return self
class B(A):
def y(self)
return super().y() # equivalent to: A.y(self)
b = B()
b.y() is b # True: returns b not super(), self is b not super()
I wrote a simple test to investigate what CPython does for super:
class A:
pass
class B(A):
def f(self):
return super()
#classmethod
def g(cls):
return super()
def h(selfish):
selfish = B()
return super()
class C(B):
pass
c = C()
for method in 'fgh':
super_object = getattr(c, method)()
print(super_object, super_object.__self__, super_object.__self_class__, super_object.__thisclass__) # (These methods were found using dir.)
The zero-argument super call returns an object that stores three things:
__self__ stores the object whose name matches the first parameter of the method—even if that name has been reassigned.
__self_class__ stores its type, or itself in the case of a class method.
__thisclass__ stores the class in which the method is defined.
(It is unfortunate that __thisclass__ was implemented this way rather than fetching an attribute on the method because it makes it impossible to use the zero-argument form of super with meta-programming.)
The object returned by super implements getattribute, which forwards method calls to the type found in the __mro__ of __self_class__ one step after __thisclass__.
I'm trying to understand how methods are working as descriptors.
There is written something like: this:
To support method calls, functions include the __get__() method for binding methods during attribute access. This means that all functions are non-data descriptors which return bound methods when they are invoked from an object.
Basing on this description and the fact that "Dotted access from a class calls __get__()" I assumed that it is possible to decorate __get__ of class method.
I'm trying to wrap someFunction.__get__:
def wrap(func):
def wrapped(*args, **kwargs):
print("Wrapping...")
return func(*args, **kwargs)
return wrapped
class SomeClass():
def someFunction(self):
print("someFunction called")
someFunction.__get__ = wrap(someFunction.__get__)
obj = SomeClass()
obj.someFunction()
The output is just "someFunction called" as if nothing was not changed.
So my question is: why Wrapping... log is not printed?
Does the assignement:
someFunction.__get__ = wrap(someFunction.__get__)
anything?
From the descriptor documentation:
The following methods only apply when an instance of the class containing the method (a so-called descriptor class) appears in an owner class (the descriptor must be in either the owner’s class dictionary or in the class dictionary for one of its parents).
I interpret that to mean that there's some kind of magic going on that requires __get__ to be defined in the class (that of functions, in this case). It just won't work if you add it to an instance. I guess Python has to do some kind of initialization when a descriptor class is defined.
Why are python instance methods callable, but static methods and class methods not callable?
I did the following:
class Test():
class_var = 42
#classmethod
def class_method(cls):
pass
#staticmethod
def static_method():
pass
def instance_method(self):
pass
for attr, val in vars(Test).items():
if not attr.startswith("__"):
print (attr, "is %s callable" % ("" if callable(val) else "NOT"))
The result is:
static_method is NOT callable
instance_method is callable
class_method is NOT callable
class_var is NOT callable
Technically this may be because instance method object might have a particular attribute (not) set in a particular way (possibly __call__). Why such asymmetry, or what purpose does it serve?
I came across this while learning python inspection tools.
Additional remarks from comments:
The SO answer linked in the comments says that the static/class methods are descriptors , which are not callable. Now I am curious, why are descriptors made not callable, since descriptors are class with particular attributes (one of __get__, __set__, __del___) defined.
Why are descriptors not callable? Basically because they don't need to be. Not every descriptor represents a callable either.
As you correctly note, the descriptor protocol consists of __get__, __set__ and __del__. Note no __call__, that's the technical reason why it's not callable. The actual callable is the return value of your static_method.__get__(...).
As for the philosophical reason, let's look at the class. The contents of the __dict__, or in your case results of vars(), are basically locals() of the class block. If you define a function, it gets dumped as a plain function. If you use a decorator, such as #staticmethod, it's equivalent to something like:
def _this_is_not_stored_anywhere():
pass
static_method = staticmethod(_this_is_not_stored_anywhere)
I.e., static_method is assigned a return value of the staticmethod() function.
Now, function objects actually implement the descriptor protocol - every function has a __get__ method on it. This is where the special self and the bound-method behavior comes from. See:
def xyz(what):
print(what)
repr(xyz) # '<function xyz at 0x7f8f924bdea0>'
repr(xyz.__get__("hello")) # "<bound method str.xyz of 'hello'>"
xyz.__get__("hello")() # "hello"
Because of how the class calls __get__, your test.instance_method binds to the instance and gets it pre-filled as it first argument.
But the whole point of #classmethod and #staticmethod is that they do something special to avoid the default "bound method" behavior! So they can't return a plain function. Instead they return a descriptor object with a custom __get__ implementation.
Of course, you could put a __call__ method on this descriptor object, but why? It's code that you don't need in practice; you can almost never touch the descriptor object itself. If you do (in code similar to yours), you still need special handling for descriptors, because a general descriptor doesn't have to be(have like a) callable - properties are descriptors too. So you don't want __call__ in the descriptor protocol. So if a third party "forgets" to implement __call__ on something you consider a "callable", your code will miss it.
Also, the object is a descriptor, not a function. Putting a __call__ method on it would be masking its true nature :) I mean, it's not wrong per se, it's just ... something that you should never need for anything.
BTW, in case of classmethod/staticmethod, you can get back the original function from their __func__ attribute.
I want to assign a static method to a class variable in Python, and below is what my code looks like.
class Klass:
classVariable = None
#staticmethod
def method():
print "method called"
Klass.classVariable = Klass.method
Klass.method()
Klass.classVariable()
This gave me an error at the last line,
TypeError: unbound method method() must be called with Klass instance as first argument (got nothing instead).
But when I change the static method to class method it works. Can anyone give me any idea of why this is the case?
Backstory (descriptor protocol)
First, we need to know a little about python descriptors...
For this answer, it should be enough to know the following:
Functions are descriptors.
Binding behavior of methods (i.e. how a method knows what self to pass) is implemented via the function's __get__ method and the built-in descriptor protocol.
When you put a descriptor foo on a class, accessing the descriptor actually calls the .__get__ method. (This is really just a generalization of statement 2)
In other words:
class Foo(object):
val = some_descriptor
When I do:
result = Foo.val
Python actually does:
Foo.val.__get__(None, Foo)
When I do:
f = Foo()
f.val
python does:
f = Foo()
type(f).val.__get__(f, type(f))
Now the good stuff.
It looks like (on python2.x), staticmethod is implemented such that it's __get__ method returns a regular function. You can see this by printing the type of Klass.method:
print type(Klass.method) # <type 'function'>
So what we've learned is that the method returned by Klass.method.__get__ is just a regular function.
When you put that regular function onto a class, it's __get__ method returns an instancemethod (which expects a self argument). This isn't surprising ... We do it all the time:
class Foo(object):
def bar(self):
print self
Is no different to python than:
def bar(self):
print self
class Foo(object):
pass
Foo.bar = bar
except that the first version is a lot easier to read and doesn't clutter your module namespace.
So now we've explained how your staticmethod turned into an instance method. Is there anything we can do about it?
Solution
When you put the method onto the class, designate it as a staticmethod again and it will work out Ok.
class Klass(object): # inheriting from object is a good idea.
classVariable = None
#staticmethod
def method():
print("method called")
Klass.classVariable = staticmethod(Klass.method) # Note extra staticmethod
Klass.method()
Klass.classVariable()
Appendix -- Re-implementation of #staticmethod
If you're a little but curious how you might implement staticmethod to not have this problem -- Here's an example:
class StaticMethod(object):
def __init__(self, fn):
self.fn = fn
def __get__(self, inst, cls):
return self
def __call__(self, *args, **kwargs):
return self.fn(*args, **kwargs)
class Klass(object):
classVariable = None
#StaticMethod
def method():
print("method called")
Klass.classVariable = Klass.method
Klass.method()
Klass.classVariable()
Klass().method()
Klass().classVariable()
The trick here is that my __get__ doesn't return a function. It returns itself. When you put it on a different class (or the same class), it's __get__ will still just return itself. Since it is returning itself from __get__, it needs to pretend to be a function (so it can be called after "__gotten__") so I implement a custom __call__ method to do the right thing (pass through to the delegate function and return the result).
Please note, I'm not advocating that you use this StaticMethod instead of staticmethod. It'll be less efficient and not as introspectible (and probably confusing for your code readers). This is only for educational purposes.
I need to decorate a object's method. It needs to be at runtime because the decorators applied on the object depends on the arguments that the user gave when calling the program (arguments supplied with argv), so a same object could be decorated 3 times, 2 times, or not be decorated at all.
Here is some context, the program is a puzzle solver, the main behavior is to find a solution for the puzzle automatically, by automatically I mean without user intervention. And here is where the decoration gets to play, one of the things I want to is draw a graph of what happened during the execution, but I want to do so only when the flag --draw-graph is used.
Here is what I've tried:
class GraphDecorator(object):
def __init__(self, wrappee):
self.wrappee = wrappee
def method(self):
# do my stuff here
self.wrappee.method()
# do more of stuff here
def __getattr__(self,attr):
return getattr(self.wrappee,attr)
And why it did NOT work:
It did not work because of the way I built the application, when a method that did not exist in my Decorator class was called it felt back to the implementation of the decorated class, the problem is that the application always started invoking the method run that did not need to be decorated, so the undecorated fall back was used and from inside the undecorated form it always called undecorated methods, what I needed was to replace the method from the object, not to proxy it:
# method responsible to replace the undecorated form by the decorated one
def graphDecorator(obj):
old_method = obj.method
def method(self):
# do my stuff here
old_method()
# do more of my stuff
setattr(obj,'method',method) # replace with the decorated form
And here is my problem, the decorated form does not receive self when it is called resulting on a TypeError because of the wrong number of arguments.
The problem was that I couldn't use func(self) as a method. The reason is that setattr() method does not bound the function, and the function acts like it a static method - not a class method -, thanks to the introspective nature of python I've able to come up with this solution:
def decorator(obj):
old_func = obj.func # can't call 'by name' because of recursion
def decorated_func(self):
# do my stuff here
old_func() # does not need pass obj
# do some othere stuff here
# here is the magic, this get the type of a 'normal method' of a class
method = type(obj.func)
# this bounds the method to the object, so self is passed by default
obj.func = method(decorated_func, obj)
I think this is the best way to decorate a object's method at runtime, though it would be nice to find a way to call method() directly, without the line method = type(obj.func)
You might want to use __getattribute__ instead of __getattr__ (the latter being only called if "standard" lookup fails):
class GraphDecorator(object):
def __init__(self, wrappee):
self.__wrappee = wrappee
def method(self):
# do my stuff here
self.wrappe.method()
# do more of stuff here
def __getattribute__(self, name):
try:
wrappee = object.__getattribute__(self, "_GraphDecorator__wrappee")
return getattr(wrappee, name)
except AttributeError:
return object.__getattribute__(self, name)
I need to decorate a object's method. It needs to be at runtime because the decorators applied on the object depends on the arguments that the user gave when calling the program (arguments supplied with argv), so a same object could be decorated 3 times, 2 times, or not be decorated at all.
The above is unfortunately incorrect, and what you are trying to do is unnecessary.
You can do this at runtime like so. Example:
import sys
args = sys.argv[1:]
class MyClass(object):
pass
if args[0]=='--decorateWithFoo':
MyClass = decoratorFoo(MyClass)
if args[1]=='--decorateWithBar'
MyClass = decoratorBar(MyClass)
The syntax:
#deco
define something
Is the same thing as:
define something
something = deco(something)
You could also make a decorator factory #makeDecorator(command_line_arguments)
"It needs to be at runtime because the decorators applied on the object depends on the arguments that the user gave when calling the program"
The don't use decorators. Decorators are only syntactical support for wrappers, you can just as well use normal function/method calls instead.