Develop Reference

Python: Why do functools.partial functions not become bound methods when set as class attributes?

I was reading about how functions become bound methods when being set as class atrributes. I then observed that this is not the case for functions that are wrapped by functools.partial. What is the explanation for this?
Simple example:
from functools import partial
def func1():
print("foo")
func1_partial = partial(func1)
class A:
f = func1
g = func1_partial
a = A()
a.f() # TypeError: func1() takes 0 positional arguments but 1 was given
a.g() # prints "foo"
I kind of expected them both to behave in the same way.

The trick that allows functions to become bound methods is the __get__ magic method.
To very briefly summarize that page, when you access a field on an instance, say foo.bar, Python first checks whether bar exists in foo's __dict__ (or __slots__, if it has one). If it does, we return it, no harm done. If not, then we look on type(foo). However, when we access the field Foo.bar on the class Foo through an instance, something magical happens. When we write foo.bar, assuming there is no bar on foo's __dict__ (resp. __slots__), then we actually call Foo.bar.__get__(foo, Foo). That is, Python calls a magic method asking the object how it would like to be retrieved.
This is how properties are implemented, and it's also how bound methods are implemented. Somewhere deep down (probably written in C), there's a __get__ function on the type function that binds the method when accessed through an instance.
functools.partial, despite looking a lot like a function, is not an instance of the type function. It's just a random class that happens to implement __call__, and it doesn't implement __get__. Why doesn't it? Well, they probably just didn't think it was worth it, or it's possible nobody even considered it. Regardless, the "bound method" trick applies to the type called function, not to all callable objects.
Another useful resource on magic methods, and __get__ in particular: https://rszalski.github.io/magicmethods/#descriptor

The type function implements the __get__ method:
>>> import types
>>> types.FunctionType.__get__
<slot wrapper '__get__' of 'function' objects>
partial does not.
>>> from functools import partial
>>> partial.__get__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'functools.partial' has no attribute '__get__'. Did you mean: '__ge__'?
The __get__ method is what makes a.f evaluate to a value of type method instead of A.f. Without __get__, a.g is equivalent to A.g.

Do we really need #staticmethod decorator in python to declare static method

I am curious about why we need the #staticmethod decorator to declare method as static. I was reading about static methods in Python, and I came to know that static method can be callable without instantiating its class.
So I tried the two examples below, but both do the same:
class StatMethod:
def stat():
print("without Decorator")
class StatMethod_with_decorator:
#staticmethod
def stat():
print("With Decorator")
If I call the stat() method on the class directly, both print/show the values below:
>> StatMethod.stat()
without Decorator
>> StatMethod_with_decorator.stat()
With Decorator

You need the decorator if you intend to try to call the #staticmethod from the instance of the class instead of of the class directly
class Foo():
def bar(x):
return x + 5
>>> f = Foo()
>>> f.bar(4)
Traceback (most recent call last):
File "<pyshell#7>", line 1, in <module>
f.bar(4)
TypeError: bar() takes 1 positional argument but 2 were given
Now if I declare #staticmethod the self argument isn't passed implicitly as the first argument
class Foo():
#staticmethod
def bar(x):
return x + 5
>>> f = Foo()
>>> f.bar(4)
9

The documentation describes some transformations that are done when calling a user defined method:
Note that the transformation from function object to (unbound or
bound) method object happens each time the attribute is retrieved from
the class or instance. In some cases, a fruitful optimization is to
assign the attribute to a local variable and call that local variable.
Also notice that this transformation only happens for user-defined
functions; other callable objects (and all non-callable objects) are
retrieved without transformation. It is also important to note that
user-defined functions which are attributes of a class instance are
not converted to bound methods; this only happens when the function is
an attribute of the class.
For methods marked as staticmethod this is different:
Static method objects provide a way of defeating the transformation of
function objects to method objects described above. A static method
object is a wrapper around any other object, usually a user-defined
method object. When a static method object is retrieved from a class
or a class instance, the object actually returned is the wrapped
object, which is not subject to any further transformation. Static
method objects are not themselves callable, although the objects they
wrap usually are. Static method objects are created by the built-in
staticmethod() constructor.

if function has some parameters, then call non static method would be failed
and static method didn't use the local variables in the class, but the class method will be

Update: In python 3.10 you dont need the decorator any more

Just my simple approach here. Forget the decorators. Use the class directly (Python 3.8):
class MyClass:
def myMethod(self, myValue):
print(myValue)
MyClass.myMethod(None, "hi")
Or both:
MyClass.myMethod(None, "hi from class")
myInstance = myClass()
myInstance.myMethod2("hi from instance")

Why are python static/class method not callable?

Why are python instance methods callable, but static methods and class methods not callable?
I did the following:
class Test():
class_var = 42
#classmethod
def class_method(cls):
pass
#staticmethod
def static_method():
pass
def instance_method(self):
pass
for attr, val in vars(Test).items():
if not attr.startswith("__"):
print (attr, "is %s callable" % ("" if callable(val) else "NOT"))
The result is:
static_method is NOT callable
instance_method is callable
class_method is NOT callable
class_var is NOT callable
Technically this may be because instance method object might have a particular attribute (not) set in a particular way (possibly __call__). Why such asymmetry, or what purpose does it serve?
I came across this while learning python inspection tools.
Additional remarks from comments:
The SO answer linked in the comments says that the static/class methods are descriptors , which are not callable. Now I am curious, why are descriptors made not callable, since descriptors are class with particular attributes (one of __get__, __set__, __del___) defined.

Why are descriptors not callable? Basically because they don't need to be. Not every descriptor represents a callable either.
As you correctly note, the descriptor protocol consists of __get__, __set__ and __del__. Note no __call__, that's the technical reason why it's not callable. The actual callable is the return value of your static_method.__get__(...).
As for the philosophical reason, let's look at the class. The contents of the __dict__, or in your case results of vars(), are basically locals() of the class block. If you define a function, it gets dumped as a plain function. If you use a decorator, such as #staticmethod, it's equivalent to something like:
def _this_is_not_stored_anywhere():
pass
static_method = staticmethod(_this_is_not_stored_anywhere)
I.e., static_method is assigned a return value of the staticmethod() function.
Now, function objects actually implement the descriptor protocol - every function has a __get__ method on it. This is where the special self and the bound-method behavior comes from. See:
def xyz(what):
print(what)
repr(xyz) # '<function xyz at 0x7f8f924bdea0>'
repr(xyz.__get__("hello")) # "<bound method str.xyz of 'hello'>"
xyz.__get__("hello")() # "hello"
Because of how the class calls __get__, your test.instance_method binds to the instance and gets it pre-filled as it first argument.
But the whole point of #classmethod and #staticmethod is that they do something special to avoid the default "bound method" behavior! So they can't return a plain function. Instead they return a descriptor object with a custom __get__ implementation.
Of course, you could put a __call__ method on this descriptor object, but why? It's code that you don't need in practice; you can almost never touch the descriptor object itself. If you do (in code similar to yours), you still need special handling for descriptors, because a general descriptor doesn't have to be(have like a) callable - properties are descriptors too. So you don't want __call__ in the descriptor protocol. So if a third party "forgets" to implement __call__ on something you consider a "callable", your code will miss it.
Also, the object is a descriptor, not a function. Putting a __call__ method on it would be masking its true nature :) I mean, it's not wrong per se, it's just ... something that you should never need for anything.
BTW, in case of classmethod/staticmethod, you can get back the original function from their __func__ attribute.

Why do people default owner parameter to None in __get__?

I've seen this quite often:
def __get__(self, instance, owner=None):
Why do some people use the default value of None for the the owner parameter?
This is even done in the Python docs:
descr.__get__(self, obj, type=None) --> value

Because the owner can easily be derived from the instance, the second argument is optional. Only when there is no instance to derive an owner from, is the owner argument needed.
This is described in the proposal that introduced descriptors, PEP 252 - Making Types Look More Like Classes:
__get__: a function callable with one or two arguments that
retrieves the attribute value from an object. This is also
referred to as a "binding" operation, because it may return a
"bound method" object in the case of method descriptors. The
first argument, X, is the object from which the attribute must
be retrieved or to which it must be bound. When X is None,
the optional second argument, T, should be meta-object and the
binding operation may return an unbound method restricted to
instances of T.
(Bold emphasis mine).
Binding, from day one, was meant to be applicable to the instance alone, with the type being optional. Methods don't need it, for example, since they can be bound to the instance alone:
>>> class Foo: pass
...
>>> def bar(self): return self
...
>>> foo = Foo()
>>> foo.bar = bar.__get__(foo) # look ma! no class!
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10a0c2710>>
>>> foo.bar()
<__main__.Foo object at 0x10a0c2710>
Besides, the second argument can easily be derived from the first argument; witness a classmethod still binding to the class even though we did not pass one in:
>>> classmethod(bar).__get__(foo)
<bound method type.bar of <class '__main__.Foo'>>
>>> classmethod(bar).__get__(foo)()
<class '__main__.Foo'>
The only reason the argument is there in the first place is to support binding to class, e.g. when there is no instance to bind to. The class method again; binding to None as the instance won't work, it only works if we actually pass in the class:
>>> classmethod(bar).__get__(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __get__(None, None) is invalid
>>> classmethod(bar).__get__(None, Foo)
<bound method type.bar of <class '__main__.Foo'>>

This is the standard way to do it; all Python built-in descriptors I've seen do it, including functions, properties, staticmethods, etc. I know of no case in the descriptor protocol where __get__ will be called without the owner argument, but if you want to call __get__ manually, it can be useful not to have to pass an owner. The owner argument usually doesn't do much.
As an example, you might want a cleaner way to give individual objects new methods. The following decorator cleans up the syntax and lets the methods have access to self:
def method_of(instance):
def method_adder(function):
setattr(instance, function.__name__, function.__get__(instance))
return function
return method_adder
#method_of(a)
def foo(self, arg1, arg2):
stuff()
Now a has a foo method. We manually used the __get__ method of the foo function to create a bound method object like any other, except that since this method isn't associated with a class, we didn't pass __get__ a class. Pretty much the only difference is that when you print the method object, you see ?.foo instead of SomeClassName.foo.

Because that's how the descriptor protocol is specified:
descr.__get__(self, obj, type=None) --> value
cf https://docs.python.org/2/howto/descriptor.html#descriptor-protocol
The type argument allows access to the class on which the descriptor is looked up when it's looked up on a class instead of an instance. Since you can get the class from the instance, it's somehow redundant when the descriptor is looked up on an instance, so it has been made optional to allow the less verbose desc.__get__(obj) call (instead of desc.__get__(obj, type(obj))).

How does assignment of a function as a class attribute become a method in Python?

>>> class A(object): pass
>>> def func(cls): pass
>>> A.func = func
>>> A.func
<unbound method A.func>
How does this assignment create a method? It seems unintuitive that assignment does the following for classes:
Turn functions into unbound instance methods
Turn functions wrapped in classmethod() into class methods (actually, this is pretty intuitive)
Turn functions wrapped in staticmethod() into functions
It seems that for the first, there should be an instancemethod(), and for the last one, there shouldn't be a wrapper function at all. I understand that these are for uses within a class block, but why should they apply outside of it?
But more importantly, how exactly does assignment of the function into a class work? What magic happens that resolves those 3 things?
Even more confusing with this:
>>> A.func
<unbound method A.func>
>>> A.__dict__['func']
<function func at 0x...>
But I think this is something to do with descriptors, when retrieving attributes. I don't think it has much to do with the setting of attributes here.

You're right that this has something to do with descriptor protocol. Descriptors are how passing the receiver object as the first parameter of a method is implemented in Python. You can read more detail about Python attribute lookup from here. The following shows on a bit lower level, what is happening when you do A.func = func; A.func:
# A.func = func
A.__dict__['func'] = func # This just sets the attribute
# A.func
# The __getattribute__ method of a type object calls the __get__ method with
# None as the first parameter and the type as the second.
A.__dict__['func'].__get__(None, A) # The __get__ method of a function object
# returns an unbound method object if the
# first parameter is None.
a = A()
# a.func()
# The __getattribute__ method of object finds an attribute on the type object
# and calls the __get__ method of it with the instance as its first parameter.
a.__class__.__dict__['func'].__get__(a, a.__class__)
# This returns a bound method object that is actually just a proxy for
# inserting the object as the first parameter to the function call.
So it's the looking up of the function on a class or an instance that turns it into a method, not assigning it to a class attribute.
classmethod and staticmethod are just slightly different descriptors, classmethod returning a bound method object bound to a type object and staticmethod just returns the original function.

Descriptors are the magic1 that turns an ordinary function into a bound or unbound method when you retrieve it from an instance or class, since they’re all just functions that need different binding strategies. The classmethod and staticmethod decorators implement other binding strategies, and staticmethod actually just returns the raw function, which is the same behavior you get from a non-function callable object.
See “User-defined methods” for some gory details, but note this:
Also notice that this transformation only happens for user-defined functions; other callable objects (and all non-callable objects) are retrieved without transformation.
So if you wanted this transformation for your own callable object, you could just wrap it in a function, but you could also write a descriptor to implement your own binding strategy.
Here’s the staticmethod decorator in action, returning the underlying function when it’s accessed.
>>> #staticmethod
... def f(): pass
>>> class A(object): pass
>>> A.f = f
>>> A.f
<function f at 0x100479398>
>>> f
<staticmethod object at 0x100492750>
Whereas a normal object with a __call__ method doesn’t get transformed:
>>> class C(object):
... def __call__(self): pass
>>> c = C()
>>> A.c = c
>>> A.c
<__main__.C object at 0x10048b890>
>>> c
<__main__.C object at 0x10048b890>
1 The specific function is func_descr_get in Objects/funcobject.c.

What you have to consider is that in Python everything is an object. By establishing that it is easier to understand what is happening. If you have a function def foo(bar): print bar, you can do spam = foo and call spam(1), getting of course, 1.
Objects in Python keep their instance attributes in a dictionary called __dict__ with a "pointer" to other objects. As functions in Python are objects as well, they can be assigned and manipulated as simple variables, passed around to other functions, etc. Python's implementation of object orientation takes advantage of this, and treats methods as attributes, as functions that are in the __dict__ of the object.
Instance methods' first parameter is always the instance object itself, generally called self (but this could be called this or banana). When a method is called directly on the class, it is unbound to any instance, so you have to give it an instance object as the first parameter (A.func(A())). When you call a bound function (A().func()), the first parameter of the method, self, is implicit, but behind the curtains Python does exactly the same as calling directly on the unbound function and passing the instance object as the first parameter.
If this is understood, the fact that assigning A.func = func (which behind the curtains is doing A.__dict__["func"] = func) leaves you with an unbound method, is unsurprising.
In your example the cls in def func(cls): pass actually what will be passed on is the instance (self) of type A. When you apply the classmethod or staticmethod decorators do nothing more than take the first argument obtained during the call of the function/method, and transform it into something else, before calling the function.
classmethod takes the first argument, gets the class object of the instance, and passes that as the first argument to the function call, while staticmethod simply discards the first parameter and calls the function without it.

Point 1: The function func you defined exists as a First-Class Object in Python.
Point 2: Classes in Python store their attributes in their __dict__.
So what happens when you pass a function as the value of a class attribute in Python? That function is stored in the class' __dict__, making it a method of that class accessed by calling the attribute name you assigned it to.

Relating to MTsoul's comment to Gabriel Hurley's answer:
What is different is that func has a __call__() method, making it "callable", i.e. you can apply the () operator to it. Check out the Python docs (search for __call__ on that page).