Develop Reference

Python: Why do functools.partial functions not become bound methods when set as class attributes?

I was reading about how functions become bound methods when being set as class atrributes. I then observed that this is not the case for functions that are wrapped by functools.partial. What is the explanation for this?
Simple example:
from functools import partial
def func1():
print("foo")
func1_partial = partial(func1)
class A:
f = func1
g = func1_partial
a = A()
a.f() # TypeError: func1() takes 0 positional arguments but 1 was given
a.g() # prints "foo"
I kind of expected them both to behave in the same way.

The trick that allows functions to become bound methods is the __get__ magic method.
To very briefly summarize that page, when you access a field on an instance, say foo.bar, Python first checks whether bar exists in foo's __dict__ (or __slots__, if it has one). If it does, we return it, no harm done. If not, then we look on type(foo). However, when we access the field Foo.bar on the class Foo through an instance, something magical happens. When we write foo.bar, assuming there is no bar on foo's __dict__ (resp. __slots__), then we actually call Foo.bar.__get__(foo, Foo). That is, Python calls a magic method asking the object how it would like to be retrieved.
This is how properties are implemented, and it's also how bound methods are implemented. Somewhere deep down (probably written in C), there's a __get__ function on the type function that binds the method when accessed through an instance.
functools.partial, despite looking a lot like a function, is not an instance of the type function. It's just a random class that happens to implement __call__, and it doesn't implement __get__. Why doesn't it? Well, they probably just didn't think it was worth it, or it's possible nobody even considered it. Regardless, the "bound method" trick applies to the type called function, not to all callable objects.
Another useful resource on magic methods, and __get__ in particular: https://rszalski.github.io/magicmethods/#descriptor

The type function implements the __get__ method:
>>> import types
>>> types.FunctionType.__get__
<slot wrapper '__get__' of 'function' objects>
partial does not.
>>> from functools import partial
>>> partial.__get__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'functools.partial' has no attribute '__get__'. Did you mean: '__ge__'?
The __get__ method is what makes a.f evaluate to a value of type method instead of A.f. Without __get__, a.g is equivalent to A.g.

Difference between having #staticmethod and not having it

In Python 3.x what can I do with bar() that I cannot do with foo()?
class A:
def foo():
print("some code")
#staticmethod
def bar():
print("some code")
Note: I initially forgot to specify self as an argument to foo(), but I'm leaving the mistake there, since the answers andress that.

a staticmethod is a method that does not require an object as its first parameter. This means it is a method useful to the class itself and all instantiations of it, rather than just instances of it (object initialized as A().
What this means in practical terms, is that Python does not implicitly send the object itself as a parameter. Your first method will break once you call it:
>>> a.foo()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: foo() takes 0 positional arguments but 1 was given
This is because Python supplies object methods with the object itself as a first parameter. Hence the ubiquitous self argument:
def foo(self): #Proper signature
On the other hand,
A.bar()
will work just fine, and so will
a.bar()
The object is not supplied as a first argument. Use staticmethods for methods that are supposed to be helpful to the class and its instances, but do not require knowledge of either. Usually I use these as utility functions.
Note there is a third version, a classmethod, which is similar to a regular method in that it accepts a first parameter by default - the class of the caller. In this case the minimal signature is
#classmethod
def operateOnClass(cls):
Use this to make changes that affect all instances, such as changing class variables.

bar() can be called from an uninstantiated class object. foo() needs to be fed self as an argument, and as such can only be called from an object already declared as an instance of class A

Why do I need to pass the class instance or object to a function definition inside a class? [duplicate]

What is the difference between the following class methods?
Is it that one is static and the other is not?
class Test(object):
def method_one(self):
print "Called method_one"
def method_two():
print "Called method_two"
a_test = Test()
a_test.method_one()
a_test.method_two()

In Python, there is a distinction between bound and unbound methods.
Basically, a call to a member function (like method_one), a bound function
a_test.method_one()
is translated to
Test.method_one(a_test)
i.e. a call to an unbound method. Because of that, a call to your version of method_two will fail with a TypeError
>>> a_test = Test()
>>> a_test.method_two()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method_two() takes no arguments (1 given)
You can change the behavior of a method using a decorator
class Test(object):
def method_one(self):
print "Called method_one"
#staticmethod
def method_two():
print "Called method two"
The decorator tells the built-in default metaclass type (the class of a class, cf. this question) to not create bound methods for method_two.
Now, you can invoke static method both on an instance or on the class directly:
>>> a_test = Test()
>>> a_test.method_one()
Called method_one
>>> a_test.method_two()
Called method_two
>>> Test.method_two()
Called method_two

Methods in Python are a very, very simple thing once you understood the basics of the descriptor system. Imagine the following class:
class C(object):
def foo(self):
pass
Now let's have a look at that class in the shell:
>>> C.foo
<unbound method C.foo>
>>> C.__dict__['foo']
<function foo at 0x17d05b0>
As you can see if you access the foo attribute on the class you get back an unbound method, however inside the class storage (the dict) there is a function. Why's that? The reason for this is that the class of your class implements a __getattribute__ that resolves descriptors. Sounds complex, but is not. C.foo is roughly equivalent to this code in that special case:
>>> C.__dict__['foo'].__get__(None, C)
<unbound method C.foo>
That's because functions have a __get__ method which makes them descriptors. If you have an instance of a class it's nearly the same, just that None is the class instance:
>>> c = C()
>>> C.__dict__['foo'].__get__(c, C)
<bound method C.foo of <__main__.C object at 0x17bd4d0>>
Now why does Python do that? Because the method object binds the first parameter of a function to the instance of the class. That's where self comes from. Now sometimes you don't want your class to make a function a method, that's where staticmethod comes into play:
class C(object):
#staticmethod
def foo():
pass
The staticmethod decorator wraps your class and implements a dummy __get__ that returns the wrapped function as function and not as a method:
>>> C.__dict__['foo'].__get__(None, C)
<function foo at 0x17d0c30>
Hope that explains it.

When you call a class member, Python automatically uses a reference to the object as the first parameter. The variable self actually means nothing, it's just a coding convention. You could call it gargaloo if you wanted. That said, the call to method_two would raise a TypeError, because Python is automatically trying to pass a parameter (the reference to its parent object) to a method that was defined as having no parameters.
To actually make it work, you could append this to your class definition:
method_two = staticmethod(method_two)
or you could use the #staticmethod function decorator.

>>> class Class(object):
... def __init__(self):
... self.i = 0
... def instance_method(self):
... self.i += 1
... print self.i
... c = 0
... #classmethod
... def class_method(cls):
... cls.c += 1
... print cls.c
... #staticmethod
... def static_method(s):
... s += 1
... print s
...
>>> a = Class()
>>> a.class_method()
1
>>> Class.class_method() # The class shares this value across instances
2
>>> a.instance_method()
1
>>> Class.instance_method() # The class cannot use an instance method
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unbound method instance_method() must be called with Class instance as first argument (got nothing instead)
>>> Class.instance_method(a)
2
>>> b = 0
>>> a.static_method(b)
1
>>> a.static_method(a.c) # Static method does not have direct access to
>>> # class or instance properties.
3
>>> Class.c # a.c above was passed by value and not by reference.
2
>>> a.c
2
>>> a.c = 5 # The connection between the instance
>>> Class.c # and its class is weak as seen here.
2
>>> Class.class_method()
3
>>> a.c
5

method_two won't work because you're defining a member function but not telling it what the function is a member of. If you execute the last line you'll get:
>>> a_test.method_two()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method_two() takes no arguments (1 given)
If you're defining member functions for a class the first argument must always be 'self'.

Accurate explanation from Armin Ronacher above, expanding on his answers so that beginners like me understand it well:
Difference in the methods defined in a class, whether static or instance method(there is yet another type - class method - not discussed here so skipping it), lay in the fact whether they are somehow bound to the class instance or not. For example, say whether the method receives a reference to the class instance during runtime
class C:
a = []
def foo(self):
pass
C # this is the class object
C.a # is a list object (class property object)
C.foo # is a function object (class property object)
c = C()
c # this is the class instance
The __dict__ dictionary property of the class object holds the reference to all the properties and methods of a class object and thus
>>> C.__dict__['foo']
<function foo at 0x17d05b0>
the method foo is accessible as above. An important point to note here is that everything in python is an object and so references in the dictionary above are themselves pointing to other objects. Let me call them Class Property Objects - or as CPO within the scope of my answer for brevity.
If a CPO is a descriptor, then python interpretor calls the __get__() method of the CPO to access the value it contains.
In order to determine if a CPO is a descriptor, python interpretor checks if it implements the descriptor protocol. To implement descriptor protocol is to implement 3 methods
def __get__(self, instance, owner)
def __set__(self, instance, value)
def __delete__(self, instance)
for e.g.
>>> C.__dict__['foo'].__get__(c, C)
where
self is the CPO (it could be an instance of list, str, function etc) and is supplied by the runtime
instance is the instance of the class where this CPO is defined (the object 'c' above) and needs to be explicity supplied by us
owner is the class where this CPO is defined(the class object 'C' above) and needs to be supplied by us. However this is because we are calling it on the CPO. when we call it on the instance, we dont need to supply this since the runtime can supply the instance or its class(polymorphism)
value is the intended value for the CPO and needs to be supplied by us
Not all CPO are descriptors. For example
>>> C.__dict__['foo'].__get__(None, C)
<function C.foo at 0x10a72f510>
>>> C.__dict__['a'].__get__(None, C)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute '__get__'
This is because the list class doesnt implement the descriptor protocol.
Thus the argument self in c.foo(self) is required because its method signature is actually this C.__dict__['foo'].__get__(c, C) (as explained above, C is not needed as it can be found out or polymorphed)
And this is also why you get a TypeError if you dont pass that required instance argument.
If you notice the method is still referenced via the class Object C and the binding with the class instance is achieved via passing a context in the form of the instance object into this function.
This is pretty awesome since if you chose to keep no context or no binding to the instance, all that was needed was to write a class to wrap the descriptor CPO and override its __get__() method to require no context.
This new class is what we call a decorator and is applied via the keyword #staticmethod
class C(object):
#staticmethod
def foo():
pass
The absence of context in the new wrapped CPO foo doesnt throw an error and can be verified as follows:
>>> C.__dict__['foo'].__get__(None, C)
<function foo at 0x17d0c30>
Use case of a static method is more of a namespacing and code maintainability one(taking it out of a class and making it available throughout the module etc).
It maybe better to write static methods rather than instance methods whenever possible, unless ofcourse you need to contexualise the methods(like access instance variables, class variables etc). One reason is to ease garbage collection by not keeping unwanted reference to objects.

that is an error.
first of all, first line should be like this (be careful of capitals)
class Test(object):
Whenever you call a method of a class, it gets itself as the first argument (hence the name self) and method_two gives this error
>>> a.method_two()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method_two() takes no arguments (1 given)

The second one won't work because when you call it like that python internally tries to call it with the a_test instance as the first argument, but your method_two doesn't accept any arguments, so it wont work, you'll get a runtime error.
If you want the equivalent of a static method you can use a class method.
There's much less need for class methods in Python than static methods in languages like Java or C#. Most often the best solution is to use a method in the module, outside a class definition, those work more efficiently than class methods.

The call to method_two will throw an exception for not accepting the self parameter the Python runtime will automatically pass it.
If you want to create a static method in a Python class, decorate it with the staticmethod decorator.
Class Test(Object):
#staticmethod
def method_two():
print "Called method_two"
Test.method_two()

Please read this docs from the Guido First Class everything Clearly explained how Unbound, Bound methods are born.

Bound method = instance method
Unbound method = static method.

The definition of method_two is invalid. When you call method_two, you'll get TypeError: method_two() takes 0 positional arguments but 1 was given from the interpreter.
An instance method is a bounded function when you call it like a_test.method_two(). It automatically accepts self, which points to an instance of Test, as its first parameter. Through the self parameter, an instance method can freely access attributes and modify them on the same object.

Unbound Methods
Unbound methods are methods that are not bound to any particular class instance yet.
Bound Methods
Bound methods are the ones which are bound to a specific instance of a class.
As its documented here, self can refer to different things depending on the function is bound, unbound or static.
Take a look at the following example:
class MyClass:
def some_method(self):
return self # For the sake of the example
>>> MyClass().some_method()
<__main__.MyClass object at 0x10e8e43a0># This can also be written as:>>> obj = MyClass()
>>> obj.some_method()
<__main__.MyClass object at 0x10ea12bb0>
# Bound method call:
>>> obj.some_method(10)
TypeError: some_method() takes 1 positional argument but 2 were given
# WHY IT DIDN'T WORK?
# obj.some_method(10) bound call translated as
# MyClass.some_method(obj, 10) unbound method and it takes 2
# arguments now instead of 1
# ----- USING THE UNBOUND METHOD ------
>>> MyClass.some_method(10)
10
Since we did not use the class instance — obj — on the last call, we can kinda say it looks like a static method.
If so, what is the difference between MyClass.some_method(10) call and a call to a static function decorated with a #staticmethod decorator?
By using the decorator, we explicitly make it clear that the method will be used without creating an instance for it first. Normally one would not expect the class member methods to be used without the instance and accesing them can cause possible errors depending on the structure of the method.
Also, by adding the #staticmethod decorator, we are making it possible to be reached through an object as well.
class MyClass:
def some_method(self):
return self
#staticmethod
def some_static_method(number):
return number
>>> MyClass.some_static_method(10) # without an instance
10
>>> MyClass().some_static_method(10) # Calling through an instance
10
You can’t do the above example with the instance methods. You may survive the first one (as we did before) but the second one will be translated into an unbound call MyClass.some_method(obj, 10) which will raise a TypeError since the instance method takes one argument and you unintentionally tried to pass two.
Then, you might say, “if I can call static methods through both an instance and a class, MyClass.some_static_method and MyClass().some_static_method should be the same methods.” Yes!

Why do people default owner parameter to None in __get__?

I've seen this quite often:
def __get__(self, instance, owner=None):
Why do some people use the default value of None for the the owner parameter?
This is even done in the Python docs:
descr.__get__(self, obj, type=None) --> value

Because the owner can easily be derived from the instance, the second argument is optional. Only when there is no instance to derive an owner from, is the owner argument needed.
This is described in the proposal that introduced descriptors, PEP 252 - Making Types Look More Like Classes:
__get__: a function callable with one or two arguments that
retrieves the attribute value from an object. This is also
referred to as a "binding" operation, because it may return a
"bound method" object in the case of method descriptors. The
first argument, X, is the object from which the attribute must
be retrieved or to which it must be bound. When X is None,
the optional second argument, T, should be meta-object and the
binding operation may return an unbound method restricted to
instances of T.
(Bold emphasis mine).
Binding, from day one, was meant to be applicable to the instance alone, with the type being optional. Methods don't need it, for example, since they can be bound to the instance alone:
>>> class Foo: pass
...
>>> def bar(self): return self
...
>>> foo = Foo()
>>> foo.bar = bar.__get__(foo) # look ma! no class!
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x10a0c2710>>
>>> foo.bar()
<__main__.Foo object at 0x10a0c2710>
Besides, the second argument can easily be derived from the first argument; witness a classmethod still binding to the class even though we did not pass one in:
>>> classmethod(bar).__get__(foo)
<bound method type.bar of <class '__main__.Foo'>>
>>> classmethod(bar).__get__(foo)()
<class '__main__.Foo'>
The only reason the argument is there in the first place is to support binding to class, e.g. when there is no instance to bind to. The class method again; binding to None as the instance won't work, it only works if we actually pass in the class:
>>> classmethod(bar).__get__(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __get__(None, None) is invalid
>>> classmethod(bar).__get__(None, Foo)
<bound method type.bar of <class '__main__.Foo'>>

This is the standard way to do it; all Python built-in descriptors I've seen do it, including functions, properties, staticmethods, etc. I know of no case in the descriptor protocol where __get__ will be called without the owner argument, but if you want to call __get__ manually, it can be useful not to have to pass an owner. The owner argument usually doesn't do much.
As an example, you might want a cleaner way to give individual objects new methods. The following decorator cleans up the syntax and lets the methods have access to self:
def method_of(instance):
def method_adder(function):
setattr(instance, function.__name__, function.__get__(instance))
return function
return method_adder
#method_of(a)
def foo(self, arg1, arg2):
stuff()
Now a has a foo method. We manually used the __get__ method of the foo function to create a bound method object like any other, except that since this method isn't associated with a class, we didn't pass __get__ a class. Pretty much the only difference is that when you print the method object, you see ?.foo instead of SomeClassName.foo.

Because that's how the descriptor protocol is specified:
descr.__get__(self, obj, type=None) --> value
cf https://docs.python.org/2/howto/descriptor.html#descriptor-protocol
The type argument allows access to the class on which the descriptor is looked up when it's looked up on a class instead of an instance. Since you can get the class from the instance, it's somehow redundant when the descriptor is looked up on an instance, so it has been made optional to allow the less verbose desc.__get__(obj) call (instead of desc.__get__(obj, type(obj))).

Descriptors and direct access: Python reference

The python 3.3 documentation tells me that direct access to a property descriptor should be possible, although I'm skeptical of its syntax x.__get__(a). But the example that I constructed below fails. Am I missing something?
class MyDescriptor(object):
"""Descriptor"""
def __get__(self, instance, owner):
print "hello"
return 42
class Owner(object):
x = MyDescriptor()
def do_direct_access(self):
self.x.__get__(self)
if __name__ == '__main__':
my_instance = Owner()
print my_instance.x
my_instance.do_direct_access()
Here's the error I get in Python 2.7 (and also Python 3.2 after porting the snippet of code). The error message makes sense to me, but that doesn't seem to be how the documentation said it would work.
Traceback (most recent call last):
File "descriptor_test.py", line 15, in <module>
my_instance.do_direct_access()
File "descriptor_test.py", line 10, in do_direct_access
self.x.__get__(self)
AttributeError: 'int' object has no attribute '__get__'
shell returned 1

By accessing the descriptor on self you invoked __get__ already. The value 42 is being returned.
For any attribute access, Python will look to the type of the object (so type(self) here) to see if there is a descriptor object there (an object with a .__get__() method, for example), and will then invoke that descriptor.
That's how methods work; a function object is found, which has a .__get__() method, which is invoked and returns a method object bound to self.
If you wanted to access the descriptor directly, you'd have to bypass this mechanism; access x in the __dict__ dictionary of Owner:
>>> Owner.__dict__['x']
<__main__.MyDescriptor object at 0x100e48e10>
>>> Owner.__dict__['x'].__get__(None, Owner)
hello
42
This behaviour is documented right above where you saw the x.__get__(a) direct call:
The default behavior for attribute access is to get, set, or delete the attribute from an object’s dictionary. For instance, a.x has a lookup chain starting with a.__dict__['x'], then type(a).__dict__['x'], and continuing through the base classes of type(a) excluding metaclasses.
The Direct Call scenario in the documentation only applies when you have a direct reference to the descriptor object (not invoked); the Owner.__dict__['x'] expression is such a reference.
Your code on the other hand, is an example of the Instance Binding scenario:
Instance Binding
If binding to an object instance, a.x is transformed into the call: type(a).__dict__['x'].__get__(a, type(a)).