While reading up about some python module I encountered this decorator class:
# this decorator lets me use methods as both static and instance methods
class omnimethod(object):
def __init__(self, func):
self.func = func
def __get__(self, instance, owner):
return functools.partial(self.func, instance)
What I know about decorators, is that the can extend functionality (e.g. for a function). Could someone be so kind and explain to me why the class above is useful and how it exactly works?
It is used in the code this way:
#omnimethod:
def some_function(...):
pass
Another question:
I encountered this piece of code in the same file:
#property
def some_other_function(...):
pass
#property is not defined anywhere in the file. Is this some standard decorator? If yes, what does it do? Google couldn't help me on this case.
By the way, here is the source where I found the code: http://code.xster.net/pygeocoder/src/c9460febdbd1/pygeocoder.py
that omnimethod is very clever. It uses some very subtle tricks to do it's job. Let's start at the beginning.
You probably already know that decorator syntax is just sugar for function application, ie:
#somedecorator
def somefunc(...):
pass
# is the same thing as
def somefunc(...):
pass
somefunc = somedecorator(somefunc)
so somefunc is actually an omnimethod instance, not the function that had been defined. what's interesting about this is that omnimethod also implements the descriptor interface. If a class attribute defines a __get__ method, then whenever that attribute is mentioned, the interpreter instead calls __get__ on that object, and returns that instead of returning the attribute itself.
the __get__ method is always called with instance as first argument, and the class of that instance as second argument. If the attribute was actually looked up from the class itself, then instance will be None.
The last bit of trickery is functools.partial, which is the python way of function currying. when you use partial, you pass it a function and some arguments, and it returns a new function, that when called, will call the original function with the original arguments in addition to whichever arguments you passed in later. omnimethod uses this technique to populate the self parameter to the function it wraps.
Here's what that looks like. a regular method can be called when you read it from an instance but you can't use it from the class itself. you get an unbound TypeError
>>> class Foo(object):
... def bar(self, baz):
... print self, baz
...
>>> f = Foo()
>>> f.bar('apples')
<__main__.Foo object at 0x7fe81ab52f90> apples
>>> Foo.bar('quux')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unbound method bar() must be called with
Foo instance as first argument (got str instance instead)
>>> Foo.bar(None, 'quux')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unbound method bar() must be called with
Foo instance as first argument (got NoneType instance instead)
>>>
Python provides a bultin decorator classmethod (and also staticmethod, but nevermind that), that will allow you to use it at the class level, but it never gets to see the instance. it always recieves the class as it's first argument.
>>> class Foo(object):
... #classmethod
... def bar(cls, baz):
... print cls, baz
...
>>> f = Foo()
>>> Foo.bar('abc')
<class '__main__.Foo'> abc
>>> f.bar('def')
<class '__main__.Foo'> def
>>>
By it's bit of cleverness, omnimethod gives you a little bit of both.
>>> class Foo(object):
... #omnimethod
... def bar(self, baz):
... print self, baz
...
>>> f = Foo()
>>> Foo.bar('bananas')
None bananas
>>> f.bar('apples')
<__main__.Foo object at 0x7fe81ab52f90> apples
>>>
omnimethod does what it says in the comment; it will let you call some_function as either a 'static function' on a class or as a function on an instance of the class. #property is a standard decorator (see the python docs) that exposes a function in a way that makes it look like a simple instance variable.
class B:
#omnimethod
def test(self):
print 1
#property
def prop(self):
return 2
>>> b = B()
>>> b.test()
1
>>> B.test()
1
>>> b.prop
2
Related
I have inherited code that looks something like this:
class A:
def __init__(self):
print('A')
def foo(self, *args):
print('foo')
a = A()
setattr(a,'foo',types.MethodType(foo,a,A))
For that last line, I want to make the code 2to3 compatible, but MethodType only takes two arguments in python3.
Simplest option is probably to let it break intelligently and
try:
setattr(a,'foo',types.MethodType(foo,a,A))
except TypeError:
setattr(a,'foo',types.MethodType(foo,a))
But then I realized that I don't understand why I'm adding the third argument in python2, because setattr(a,'foo',types.MethodType(foo,a)) works across languages.
In Python2, what is the third argument buying me, to bind it to the class vs not?
>>> types.MethodType(foo,a)
<bound method ?.foo of <__main__.A instance at 0x1>>
>>> types.MethodType(foo,a,A)
<bound method A.foo of <__main__.A instance at 0x1>>
In Python 2, the third argument to the method type constructor was mostly used for unbound method objects:
>>> class Foo(object):
... def bar(self):
... pass
...
>>> Foo.bar
<unbound method Foo.bar>
A direct constructor call to create one of these would have looked like types.MethodType(bar, None, Foo), where bar is the function. Unbound method objects did a bit of type checking to ensure they weren't used for objects of the wrong type, but they were deemed not useful enough to justify their existence, so they got taken out in Python 3. With no more unbound method objects, there wasn't much reason for the method constructor to take a third argument, so that was removed too.
A python2 / python3 compatible way to accomplish what you want is to use the descriptor protocol (__get__):
class A(object):
def __init__(self):
print('A')
def foo(self, *args):
print('foo')
a = A()
setattr(a,'foo', foo.__get__(a, A))
.__get__ will return a bound method instance
What is the difference between the following class methods?
Is it that one is static and the other is not?
class Test(object):
def method_one(self):
print "Called method_one"
def method_two():
print "Called method_two"
a_test = Test()
a_test.method_one()
a_test.method_two()
In Python, there is a distinction between bound and unbound methods.
Basically, a call to a member function (like method_one), a bound function
a_test.method_one()
is translated to
Test.method_one(a_test)
i.e. a call to an unbound method. Because of that, a call to your version of method_two will fail with a TypeError
>>> a_test = Test()
>>> a_test.method_two()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method_two() takes no arguments (1 given)
You can change the behavior of a method using a decorator
class Test(object):
def method_one(self):
print "Called method_one"
#staticmethod
def method_two():
print "Called method two"
The decorator tells the built-in default metaclass type (the class of a class, cf. this question) to not create bound methods for method_two.
Now, you can invoke static method both on an instance or on the class directly:
>>> a_test = Test()
>>> a_test.method_one()
Called method_one
>>> a_test.method_two()
Called method_two
>>> Test.method_two()
Called method_two
Methods in Python are a very, very simple thing once you understood the basics of the descriptor system. Imagine the following class:
class C(object):
def foo(self):
pass
Now let's have a look at that class in the shell:
>>> C.foo
<unbound method C.foo>
>>> C.__dict__['foo']
<function foo at 0x17d05b0>
As you can see if you access the foo attribute on the class you get back an unbound method, however inside the class storage (the dict) there is a function. Why's that? The reason for this is that the class of your class implements a __getattribute__ that resolves descriptors. Sounds complex, but is not. C.foo is roughly equivalent to this code in that special case:
>>> C.__dict__['foo'].__get__(None, C)
<unbound method C.foo>
That's because functions have a __get__ method which makes them descriptors. If you have an instance of a class it's nearly the same, just that None is the class instance:
>>> c = C()
>>> C.__dict__['foo'].__get__(c, C)
<bound method C.foo of <__main__.C object at 0x17bd4d0>>
Now why does Python do that? Because the method object binds the first parameter of a function to the instance of the class. That's where self comes from. Now sometimes you don't want your class to make a function a method, that's where staticmethod comes into play:
class C(object):
#staticmethod
def foo():
pass
The staticmethod decorator wraps your class and implements a dummy __get__ that returns the wrapped function as function and not as a method:
>>> C.__dict__['foo'].__get__(None, C)
<function foo at 0x17d0c30>
Hope that explains it.
When you call a class member, Python automatically uses a reference to the object as the first parameter. The variable self actually means nothing, it's just a coding convention. You could call it gargaloo if you wanted. That said, the call to method_two would raise a TypeError, because Python is automatically trying to pass a parameter (the reference to its parent object) to a method that was defined as having no parameters.
To actually make it work, you could append this to your class definition:
method_two = staticmethod(method_two)
or you could use the #staticmethod function decorator.
>>> class Class(object):
... def __init__(self):
... self.i = 0
... def instance_method(self):
... self.i += 1
... print self.i
... c = 0
... #classmethod
... def class_method(cls):
... cls.c += 1
... print cls.c
... #staticmethod
... def static_method(s):
... s += 1
... print s
...
>>> a = Class()
>>> a.class_method()
1
>>> Class.class_method() # The class shares this value across instances
2
>>> a.instance_method()
1
>>> Class.instance_method() # The class cannot use an instance method
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unbound method instance_method() must be called with Class instance as first argument (got nothing instead)
>>> Class.instance_method(a)
2
>>> b = 0
>>> a.static_method(b)
1
>>> a.static_method(a.c) # Static method does not have direct access to
>>> # class or instance properties.
3
>>> Class.c # a.c above was passed by value and not by reference.
2
>>> a.c
2
>>> a.c = 5 # The connection between the instance
>>> Class.c # and its class is weak as seen here.
2
>>> Class.class_method()
3
>>> a.c
5
method_two won't work because you're defining a member function but not telling it what the function is a member of. If you execute the last line you'll get:
>>> a_test.method_two()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method_two() takes no arguments (1 given)
If you're defining member functions for a class the first argument must always be 'self'.
Accurate explanation from Armin Ronacher above, expanding on his answers so that beginners like me understand it well:
Difference in the methods defined in a class, whether static or instance method(there is yet another type - class method - not discussed here so skipping it), lay in the fact whether they are somehow bound to the class instance or not. For example, say whether the method receives a reference to the class instance during runtime
class C:
a = []
def foo(self):
pass
C # this is the class object
C.a # is a list object (class property object)
C.foo # is a function object (class property object)
c = C()
c # this is the class instance
The __dict__ dictionary property of the class object holds the reference to all the properties and methods of a class object and thus
>>> C.__dict__['foo']
<function foo at 0x17d05b0>
the method foo is accessible as above. An important point to note here is that everything in python is an object and so references in the dictionary above are themselves pointing to other objects. Let me call them Class Property Objects - or as CPO within the scope of my answer for brevity.
If a CPO is a descriptor, then python interpretor calls the __get__() method of the CPO to access the value it contains.
In order to determine if a CPO is a descriptor, python interpretor checks if it implements the descriptor protocol. To implement descriptor protocol is to implement 3 methods
def __get__(self, instance, owner)
def __set__(self, instance, value)
def __delete__(self, instance)
for e.g.
>>> C.__dict__['foo'].__get__(c, C)
where
self is the CPO (it could be an instance of list, str, function etc) and is supplied by the runtime
instance is the instance of the class where this CPO is defined (the object 'c' above) and needs to be explicity supplied by us
owner is the class where this CPO is defined(the class object 'C' above) and needs to be supplied by us. However this is because we are calling it on the CPO. when we call it on the instance, we dont need to supply this since the runtime can supply the instance or its class(polymorphism)
value is the intended value for the CPO and needs to be supplied by us
Not all CPO are descriptors. For example
>>> C.__dict__['foo'].__get__(None, C)
<function C.foo at 0x10a72f510>
>>> C.__dict__['a'].__get__(None, C)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute '__get__'
This is because the list class doesnt implement the descriptor protocol.
Thus the argument self in c.foo(self) is required because its method signature is actually this C.__dict__['foo'].__get__(c, C) (as explained above, C is not needed as it can be found out or polymorphed)
And this is also why you get a TypeError if you dont pass that required instance argument.
If you notice the method is still referenced via the class Object C and the binding with the class instance is achieved via passing a context in the form of the instance object into this function.
This is pretty awesome since if you chose to keep no context or no binding to the instance, all that was needed was to write a class to wrap the descriptor CPO and override its __get__() method to require no context.
This new class is what we call a decorator and is applied via the keyword #staticmethod
class C(object):
#staticmethod
def foo():
pass
The absence of context in the new wrapped CPO foo doesnt throw an error and can be verified as follows:
>>> C.__dict__['foo'].__get__(None, C)
<function foo at 0x17d0c30>
Use case of a static method is more of a namespacing and code maintainability one(taking it out of a class and making it available throughout the module etc).
It maybe better to write static methods rather than instance methods whenever possible, unless ofcourse you need to contexualise the methods(like access instance variables, class variables etc). One reason is to ease garbage collection by not keeping unwanted reference to objects.
that is an error.
first of all, first line should be like this (be careful of capitals)
class Test(object):
Whenever you call a method of a class, it gets itself as the first argument (hence the name self) and method_two gives this error
>>> a.method_two()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: method_two() takes no arguments (1 given)
The second one won't work because when you call it like that python internally tries to call it with the a_test instance as the first argument, but your method_two doesn't accept any arguments, so it wont work, you'll get a runtime error.
If you want the equivalent of a static method you can use a class method.
There's much less need for class methods in Python than static methods in languages like Java or C#. Most often the best solution is to use a method in the module, outside a class definition, those work more efficiently than class methods.
The call to method_two will throw an exception for not accepting the self parameter the Python runtime will automatically pass it.
If you want to create a static method in a Python class, decorate it with the staticmethod decorator.
Class Test(Object):
#staticmethod
def method_two():
print "Called method_two"
Test.method_two()
Please read this docs from the Guido First Class everything Clearly explained how Unbound, Bound methods are born.
Bound method = instance method
Unbound method = static method.
The definition of method_two is invalid. When you call method_two, you'll get TypeError: method_two() takes 0 positional arguments but 1 was given from the interpreter.
An instance method is a bounded function when you call it like a_test.method_two(). It automatically accepts self, which points to an instance of Test, as its first parameter. Through the self parameter, an instance method can freely access attributes and modify them on the same object.
Unbound Methods
Unbound methods are methods that are not bound to any particular class instance yet.
Bound Methods
Bound methods are the ones which are bound to a specific instance of a class.
As its documented here, self can refer to different things depending on the function is bound, unbound or static.
Take a look at the following example:
class MyClass:
def some_method(self):
return self # For the sake of the example
>>> MyClass().some_method()
<__main__.MyClass object at 0x10e8e43a0># This can also be written as:>>> obj = MyClass()
>>> obj.some_method()
<__main__.MyClass object at 0x10ea12bb0>
# Bound method call:
>>> obj.some_method(10)
TypeError: some_method() takes 1 positional argument but 2 were given
# WHY IT DIDN'T WORK?
# obj.some_method(10) bound call translated as
# MyClass.some_method(obj, 10) unbound method and it takes 2
# arguments now instead of 1
# ----- USING THE UNBOUND METHOD ------
>>> MyClass.some_method(10)
10
Since we did not use the class instance — obj — on the last call, we can kinda say it looks like a static method.
If so, what is the difference between MyClass.some_method(10) call and a call to a static function decorated with a #staticmethod decorator?
By using the decorator, we explicitly make it clear that the method will be used without creating an instance for it first. Normally one would not expect the class member methods to be used without the instance and accesing them can cause possible errors depending on the structure of the method.
Also, by adding the #staticmethod decorator, we are making it possible to be reached through an object as well.
class MyClass:
def some_method(self):
return self
#staticmethod
def some_static_method(number):
return number
>>> MyClass.some_static_method(10) # without an instance
10
>>> MyClass().some_static_method(10) # Calling through an instance
10
You can’t do the above example with the instance methods. You may survive the first one (as we did before) but the second one will be translated into an unbound call MyClass.some_method(obj, 10) which will raise a TypeError since the instance method takes one argument and you unintentionally tried to pass two.
Then, you might say, “if I can call static methods through both an instance and a class, MyClass.some_static_method and MyClass().some_static_method should be the same methods.” Yes!
Coding exception classes, I came across this error:
TypeError: object.__new__(A) is not safe, use Exception.__new__()
There's a similar question posted here:
TypeError: object.__new__(int) is not safe, use int.__new__(). So __new__ was deprecated for the following reason:
[Python-Dev] __new__ deprecation
Guido van Rossum
"The message means just what it says. :-) There's no point in calling
object.__new__() with more than a class parameter, and any code that
did so was just dumping those args into a black hole."
But the warning in 3.3 that I get "is not safe" is scary. I try to understand the implication of using object.__new__, let's consider this example:
>>> class A(Exception):
... def __new__(cls, *args):
... return object.__new__(A)
...
>>> A()
TypeError: object.__new__(A) is not safe, use Exception.__new__()
Fails miserably. Another Example:
>>> class A(object):
... def __new__(cls, *args):
... return object.__new__(A)
...
>>>
>>> A()
<__main__.A object at 0x0000000002F2E278>
works fine. Although, object is a builtin class just like Exception with respect to their roles, they share the trait of being builtin-classes. Now with Exception, the first example raises TypeError, but with object, it does not?
(a) What are the downsides of using object.__new__ that made Python to raise the error (TypeError:...is not safe...) in the first Example?
(b) What sort of checking Python performs before to calling __new__? Or: What is the condition that makes Python raise the error in the first example?
There is no problem in calling object.__new__, but there is a problem in not calling Exception.__new__.
Exception class was designed in such way that it is crucial that its __new__ must be called, so it complains if that is not done.
There was a question why this happens only with built-in classes. Python in fact does it with every class which is programmed to do that.
Here is a simplified poor-mans implementation of the same mechanism in a custom class:
class A(object):
def __new__(cls):
rtn = object.__new__(cls)
rtn.new_called = True
return rtn
def __init__(self):
assert getattr(self,'new_called',False), \
"object.__new__ is unsafe, use A.__new__"
class B(A):
def __new__(cls):
return object.__new__(cls)
And now:
>>> A()
<__main__.A object at 0x00000000025CFF98>
>>> B()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __init__
AssertionError: object.__new__ is unsafe, use A.__new__
As a side note, this example from the question actually has two errors:
>>> class A(Exception):
... def __new__(cls, *args):
... return object.__new__(A)
The first is that __new__ is called on object, thus ignoring Exception.__new__.
The other, just as severe is that A is passed to __new__ instead of cls, which hinders all classes inherited from A.
See this example:
class A(object):
def __new__(cls):
return object.__new__(A) # The same erroneous line, without Exception
class B(A):
pass
And now B() does not create an instance of B:
>>> B()
<__main__.A object at 0x00000000025D30B8>
Calling object.__new__(A) returns an instance of A, but does so without calling Exception.__new__() if it is defined.
I know if you want to add a method to a class instance you can't do a simple assignment like this:
>>> def print_var(self): # method to be added
print(self.var)
>>> class MyClass:
var = 5
>>> c = MyClass()
>>> c.print_var = print_var
this indeed would cause print_var to behave like a normal function, so the self argument wouldn't have his typical meaning:
>>> c.print_var
<function print_var at 0x98e86ec>
>>> c.print_var()
Traceback (most recent call last):
File "<pyshell#149>", line 1, in <module>
c.print_var()
TypeError: print_var() takes exactly 1 argument (0 given)
In order to let the function be considered a method (i.e. to bind it to the instance), I used to use this code:
>>> import types
>>> c.print_var = types.MethodType(print_var, c)
>>> c.print_var
<bound method MyClass.print_var of <__main__.MyClass object at 0x98a1bac>>
>>> c.print_var()
5
but I found that .__get__ may also be used for this purpose:
>>> c.print_var = print_var.__get__(c)
>>> c.print_var
<bound method MyClass.print_var of <__main__.MyClass object at 0x98a1bac>>
>>> c.print_var()
5
The problem here is that it just works, but I can't understand how and why. The documentation about .__get__ doesn't seem to help very much.
I'd appreciate if someone could clarify this behaviour of python's interpreter.
The information you're looking for is in the Descriptor HowTo Guide:
To support method calls, functions include the __get__() method for binding methods during attribute access. This means that all functions are non-data descriptors which return bound or unbound methods depending whether they are invoked from an object or a class. In pure Python, it works like this:
class Function(object):
. . .
def __get__(self, obj, objtype=None):
"Simulate func_descr_get() in Objects/funcobject.c"
return types.MethodType(self, obj)
So there really isn't anything strange going on -- the __get__ method of a function object calls types.MethodType and returns the result.
I was wondering if you use #staticmethod decorator in your code.
Personally I don't use it, since it takes more letters to write #staticmethod then self.
The only benefit (which comes to me) from using it may be a better clarity of a code, but since I usually write a method description for sphinx, I always state whether a method is using object or not.
Or maybe I should start using #staticmethod decorator ?
Whether to use #staticmethod or not depends on what you want to achieve. Ignoring the decorator because there is more to type is a rather silly reason (no offense!) and indicates that you have not understood the concept of a static method in Python!
Static methods are independent of the class and any class instance. They only use the class scope as a namespace. If you omit the #staticmethod decorator, you are creating an instance method that cannot be used without constructing an instance.
Here is a very simple class Foo:
>>> class Foo(object):
... #staticmethod
... def foo():
... print 'foo'
...
... def bar(self):
... print 'bar'
Now, Foo.foo() is a static method that can be called directly:
>>> Foo.foo()
foo
Foo.bar() on the other hand is an instance method, that can only be called from instances (objects) of Foo:
>>> Foo.bar()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unbound method foo() must be called with Foo instance as first argument (got nothing instead)
>>> foo = Foo()
>>> foo.bar()
bar
To answer your question: If you want to define a static method, use #staticmethod. Otherwise, don't.
If you have a method that does not use self, and therefore could be written as a static method, ask yourself: Will you ever want to access this function from outside without having an instance? Most of the times, the answer will be: No.
Assume that we want to define abs method in class Math, then we have two choices:
class Math():
def abs(n):
if n>0:
return n
else:
return -n
class Math2():
#staticmethod
def abs(n):
if n>0:
return n
else:
return -n
In Python2:
>>> Math.abs(-2)
TypeError: unbound method abs() must be called with Math instance as
first argument (got int instance instead)
>>>Math().abs(-2)
TypeError: abs() takes exactly 1 argument (2 given)
>>> Math2.abs(-2)
2
>>> Math2().abs(-2)
2
python2 automatically takes Math().abs(-2) as Math().abs(self,-2),so that you have to use #staticmethod.
In Python3
>>>Math.abs(-3)
3
>>>Math().abs(-3)
TypeError: abs() takes 1 positional argument but 2 were given
>>>Math2.abs(-3)
3
>>>Math2().abs(-3)
3
In python3, you can use classname.method() without static method, but it will raise TypeError when someone tries to use instance.method().
#staticmethod decorator saves you typing and improves readability.
class Example:
#staticmethod
def some_method():
return
is the same as:
class Example:
def some_method():
return
some_method = staticmethod(some_method)
I think you may be confused about what a static method is in Python as the terminology differs from other languages. A regular method is "bound" to the instance (self), a class method is "bound" to the class (cls) while a static method is not "bound" at all (and can not access instance nor class attributes.
See:
http://docs.python.org/library/functions.html#classmethod
http://docs.python.org/library/functions.html#staticmethod
In addition to the previous answers,
from pythons doc #staticmethod :
It can be called either on the class (such as C.f()) or on an instance (such as C().f()). The instance is ignored except for its class.
class Test:
#staticmethod
def Foo():
print('static Foo')
def Bar():
print('static Bar')
Test.foo() # static Foo
Test.bar() # static Bar
obj = Test()
obj.foo() # static Foo ; note that you can call it from class instance
obj.bar() # ERROR