Develop Reference

How do python handles "self" attribute of a class internally, when the method is assigned to a variable

Recently, I was looking into the "getattr" method, and one interesting behaviour is that it could return a method of a class instance and assign it to a variable. Here's the example I played with:
class Example:
def some_method(self, param1):
print(param1)
example = Example()
method1 = getattr(example, "some_method")
print(method1)
method1("hi")
And here's the output:
<bound method Example.some_method of <__main__.Example object at 0x7f82cbcb2850>>
hi
I do understand for the first line of the output that the method1 is a "bound method" type related to the actual instance example. But for the function call involving the method method1("hi"), the "self" parameter is omitted, only the "param1" value is given. I wonder how this function call is processed successfully and where the "self" parameter is actually stored internally.

Methods rely on the mechanism called descriptors. Any type can be made a descriptor by implementing methods such as __get__. For example:
class D:
def __get__(self, obj, type=None):
return f'Some value from {obj.__class__.__name__}'
class A:
d = D()
print(A().d) # 'Some value from A'
Python attribute access mechanism will recognize the descriptor and delegate the access logic to its __get__(), __set__(), etc. Now, the plot twist is that Python functions also implement the descriptor protocol, that defines how they behave when accessed as methods. For example, if we have a method A.f():
class A:
def __init__(self, x):
self.x = x
def f(self, y):
return self.x + y
then this code:
a = A(2)
print(a.f(3)) # prints 5
in fact runs as:
a = A(2)
bound_f = A.__get__(a)
print(bound_f(3)) # also prints 5
So when a function is accessed as a non-static method, it intercepts the access via te descriptor protocol and returns a bound method object, with attribute __self__ set to the object on which it was accessed:
>>> print(bound_f)
<bound method A.f of <__main__.A object at 0x7fc13db99ac0>>
>>> print(bound_f.__self__)
<__main__.A at 0x7fc13db99ac0>
Static, class methods and properties are also implemented using descriptors.

Redefine Method of an Object

I've got a class, where a method should only run once. Of course, it could easily be done with artificial has_executed = True/False flag, but why use it, if you can just delete the method itself? python's a duck-typed language, everything is a reference, bla-bla-bla, what can go wrong?
At least it was the thought. I couldn't actually do it:
class A:
def b(self):
print("empty")
self.__delattr__('b')
a = A()
a.b()
raises AttributeError: b. However, executing self.__getattribute__('b') returns <bound method A.b of <__main__.A object at 0x000001CDC6742FD0>>, which sounds stupid to me: why is a method any different from an attribute, since everything in python is just a reference to an object? And why can I __getattribute__, but not __delattr__?
The same goes to redefinition. I can easily set any attribute, but methods are a no-no?
class A:
def b(self):
print("first")
self.__setattr__('b', lambda self: print(f"second"))
a = A()
a.b()
a.b()
results into TypeError: <lambda>() missing 1 required positional argument: 'self'. Which, of course, means, that now python isn't using dot-notation as intended. Of course, we could ditch the self attribute in the lambda altogether, considering we've got the reference to it already in b. But isn't it incorrect by design?
The further I'm trying to take python to the limit, the more frustrated I become. Some imposed limitations (or seemingly imposed?) seem so unnatural, considering the way the language is marketed. Shouldn't it allow this? Why doesn't it work?
UPD
Ok, consider this:
class A:
def __init__(self):
self.variable = 1
def b(self):
print("old")
self.variable += 1
def new_b():
print("new")
self.variable += 15
self.__setattr__('b', new_b)
It will work and do what we want: none of other objects will have their A.b method redefined once one object kind of overlays its b definition. (overlays, since everyone so far says that you cannot redefine a method for an object, but instead only kind of hide it from the caller behind another attribute with the same name, as far as I understand).
Is this good?

It doesn't work because b isn't an attribute belonging to the instance, it belongs to the class. So you can't delete it on the instance because it isn't there to be deleted.
>>> a = A()
>>> list(a.__dict__)
[]
>>> list(A.__dict__)
['__module__', 'b', '__dict__', '__weakref__', '__doc__']
When a.b is evaluated, Python will see that a has no instance attribute named b and fall back to the class. (It's a little more complicated because when falling back to the class, it will not simply return the method itself, but a version of the method which is bound to the instance a.)
Since you don't want to delete the method on the class, the way to go is to replace the method on the instance. I don't know why you tried to do this with __setattr__ - there is no need for that, simply assign self.b = ... as normal. The reason your attempt failed is because your lambda requires a positional parameter named self, but this parameter will not be automatically bound to the instance when you look it up, because it is an instance attribute, not a class attribute.
class A:
def b(self):
print('first')
self.b = lambda: print('second')
Usage:
>>> a = A()
>>> a.b()
first
>>> a.b()
second

Well in python you have 2 types of attributes
A class attribute is a variable that belongs to a certain class, and not a particular object. Every instance of this class shares the same variable. These attributes are usually defined outside the init constructor
An instance/object attribute is a variable that belongs to one (and only one) object. Every instance of a class points to its own attributes variables. These attributes are defined within the init constructor.
In case of a class attribute its part of the class descriptor, so you cannot delete it from the object attributes like self.__deleteattr__ or add new one with __setattr__ as it alters the class descriptor and reflects on all objects. Such an operation can have devastating effects.
Its very similar to a class variable as well. You can however change the behavior with overriding or reassigning like below
class A:
def b(self):
print("empty")
A.b = lambda self: print(f"second")
a = A()
a.b()
a.b()

How can one use a class method in a class attribute definition

All is in the title. I'd like to create a class method and a class attribute, both constructed only once, when the class is created, using the first in the second's definition.
With my best try, I just get a TypeError: 'classmethod' object is not callable.
Here is my code :
import numpy as np
class Foo( object ) :
#classmethod
def bar( cls, x ) :
return x+1
bar_vect = np.vectorize( bar )
Foo.bar_vect( np.array([ 1, 2, 3 ]) )
>> TypeError: 'classmethod' object is not callable
EDIT 1 :
'classmethod' object is not callable is a problem raising the same error, but with a lot of workarounds. My question is meant to go straight to the point and have a clear idea of how to use #classmethod without a scope giving access to cls.
Another try that I made was the following :
import numpy as np
class Foo( object ) :
#classmethod
def bar( cls, x ) :
return x+1
bar_vect = np.vectorize( bar )
>> NameError: name 'Foo' is not defined

#classmethods are implemented as a special object that gets processed using the descriptor protocol when looked up on the class; inside the definition, as a raw name (unqualified), it's a special classmethod object, not a normal function and it's not bound to the class properly. If you check the pure Python definition of classmethod, you'll note it's just a normal object that implements __init__ (for construction) and __get__ (for descriptor lookup), but not __call__, meaning that if you have the raw classmethod object, it's not actually a callable at all.
The trick is to qualify the reference so the "magic" happens to bind it to the class, and move the qualified reference outside the class definition (so Foo is a defined name and can be referenced for binding) changing:
class Foo(object):
... rest of class ...
bar_vect = np.vectorize(bar) # Indented and unqualified, BAD
to:
class Foo(object):
... rest of class ...
# Must qualify both bar_vect and bar, since no longer in class definition
Foo.bar_vect = np.vectorize(Foo.bar) # Dedented, so Foo is defined for referencing, GOOD
Note that since you're using a classmethod, I suspect you may eventually be interested in subclassing and overriding bar. As written, you'd need to explicitly redefine bar_vect after defining each subclass, or it would use the inherited bar_vect, based on Foo.bar, even if the subclass defines its own bar classmethod. Explicitly redefining bar_vect each time is an option, but the other approach is to use metaclasses to implicitly define bar_vect when a class redefines bar:
class BarVectorized(type):
def __new__(cls, name, bases, namespace, **kwargs):
newcls = type.__new__(cls, name, bases, dict(namespace))
# Make vectorized wrapper for this class (must use new wrapper
# even if bar unchanged, so cls in bar is correct for lookup of
# other class attributes/methods)
try:
newcls.bar_vect = np.vectorize(newcls.bar)
except AttributeError:
pass # Allow class w/o bar; remove try/except if class must have bar
return newcls
class Foo(object):
__metaclass__ = BarVectorized
#classmethod
def bar(cls, x): return x + 1
class Foo2(Foo):
ADD = 2 # Hardcoded 1 is dumb, use class attribute instead!
#classmethod
def bar(cls, x):
return x + cls.ADD
class Foo3(Foo2):
ADD = 3 # Provide new class attr to change Foo2.bar behavior when called via Foo3
>>> Foo.bar_vect([1,2,3])
array([2, 3, 4])
>>> Foo2.bar_vect([1,2,3])
array([3, 4, 5])
>>> Foo3.bar_vect([1,2,3])
array([4, 5, 6])
No need to define bar_vect explicitly at all, and bar_vect seamlessly uses the most local classes' definition of bar available at class definition time, so unless bar is redefined after class definition, it always works, and it works as efficiently as possible. To make it use bar live, you'd need to resort to more extreme measures that perform dynamic lookup and (barring a cache) reconstruction of the np.vectorize object on each use, which is suboptimal to say the least.
For completeness, a dynamic caching based solution (hat tip to Tadhg McDonald-Jensen's answer) that uses a dynamically populating cache that adds minimal overhead (and more importantly in my opinion, abstracts out the boilerplate code that's irrelevant to the work) for the case where the cache entry already exists by using a dict subclass defining __missing__:
import operator
import numpy as np
class ClassAttrRegistry(dict):
'''Dictionary keyed by classes which returns optionally wrapped cached attributes'''
__slots__ = '_wrapper', '_attrgetter'
def __init__(self, attr, wrapperfunc=lambda x: x):
self._wrapper = wrapperfunc
self._attrgetter = operator.attrgetter(attr)
def __missing__(self, cls):
self[cls] = wrapped = self._wrapper(self._attrgetter(cls))
return wrapped
class Foo(object):
#classmethod
def bar(cls, x):
return x + 1
# Dunder prefix makes cache private to Foo methods; if subclass overrides bar_vect,
# assumed it's more complex than "vectorized bar"; cache should not be used
__bar_vect_registry = ClassAttrRegistry('bar', np.vectorize)
#classmethod
def bar_vect(cls, x):
# Get cached vectorized bar (creating if needed) then call it
return cls.__bar_vect_registry[cls](x)
Subclasses don't need to (and should not) override bar_vect (and can't accidentally access __bar_vect_registry because it's name mangled such that only methods defined by Foo will see it; change name to _bar_vect_registry, one underscore, if it should be accessible to subclasses), they just override bar and Foo's bar_vect will create/cache vectorized accessors when bar_vect is first accessed on the subclass (or an instance thereof).

Your confusion to why this is not an easy work around is understandable, let me elaborate on to why using classmethod in this way isn't going to work...
The way classmethod works is that it creates a descriptor, an object that implements __get__ when it is retrieved as an attribute on an object.
So when you do Foo.bar it basically loads the bar classmethod and calls:
bar.__get__(None, Foo)
Where the None represents the instance (there is None because it is on the class itself) and the second argument represents the class, a classmethod is not callable because then it would not have a class to bind it too!
Not only this but the class object to bind it too doesn't exist until the class definition block has ended (and the metaclass type actually puts it together) so the bare minimum is to create bar_vect after the class is actually defined:
class Foo( object ):
a = 1 #lets use an example that actually uses the class
#classmethod
def bar( cls, x ):
return x+cls.a
Foo.bar_vect = np.vectorize( Foo.bar )
This will work sure, but then you break the functionality of subclasses, what if you wanted to change a?
class Subfoo(Foo):
a = 3 #this will have no effect on
assert Subfoo.bar_vect(np.array([ 1, 2, 3 ])) == np.array([ 4, 5, 6 ])
#this SHOULD work but doesn't because you bound bar_Vect to just Foo
#subclasses mean nothing to your class method
The only way to make it work in this case is to recreate the np.vectorize at least one for each subclass, the simplest version is to just do it every time you call bar_vect:
class Foo( object ):
a = 1
#classmethod
def bar( cls, x ):
return x+cls.a
#classmethod
def bar_vect(cls,arg):
return np.vectorize(cls.bar)(arg)
This is obviously undesirable because it calls np.vectorize every time x.bar_vect is used, however you could make a record of all the classes and only make it when a new class is used:
_bar_vect_registry = {}
#classmethod
def bar_vect(cls,arg):
try:
return cls._bar_vect_registry[cls](arg)
except KeyError:
cls._bar_vect_registry[cls] = np.vectorize(cls.bar)
return cls._bar_vect_registry[cls](arg)

You real problem is that you try to use bar before the class if fully constructed, so you do not get the expected object.
Here is a simplified example:
class Foo:
#classmethod
def bar(cls, x):
print ('bar called in', cls, 'with', x)
barv = str(bar)
print(str(Foo.bar))
print(Foo.barv)
gives:
<bound method Foo.bar of <class '__main__.Foo'>>
<classmethod object at 0x00000000035B0320>
That shows that until the class is fully constructed, the methods identifier are only bound to the method definitions and not to the real methods.
If you want achieve what you want, you must define the class variable outside of class definition (after last line), as explained by #ShadowRanger

Difference between #classmethod and a method in python [duplicate]

This question already has answers here:
What's an example use case for a Python classmethod?
(7 answers)
Closed 9 years ago.
What is the difference between #classmethod and a 'classic' method in python,
When should I use the #classmethod and when should I use a 'classic' method in python.
Is the classmethod must be an method who is referred to the class (I mean it's only a method who handle the class) ?
And I know what is the difference between a #staticmethod and classic method
Thx

Let's assume you have a class Car which represents the Car entity within your system.
A classmethod is a method that works for the class Car not on one of any of Car's instances. The first parameter to a function decorated with #classmethod, usually called cls, is therefore the class itself. Example:
class Car(object):
colour = 'red'
#classmethod
def blue_cars(cls):
# cls is the Car class
# return all blue cars by looping over cls instances
A function acts on a particular instance of the class; the first parameter usually called self is the instance itself:
def get_colour(self):
return self.colour
To sum up:
use classmethod to implement methods that work on a whole class (and not on particular class instances):
Car.blue_cars()
use instance methods to implement methods that work on a particular instance:
my_car = Car(colour='red')
my_car.get_colour() # should return 'red'

If you define a method inside a class, it is handled in a special way: access to it wraps it in a special object which modifies the calling arguments in order to include self, a reference to the referred object:
class A(object):
def f(self):
pass
a = A()
a.f()
This call to a.f actually asks f (via the descriptor protocol) for an object to really return. This object is then called without arguments and deflects the call to the real f, adding a in front.
So what a.f() really does is calling the original f function with (a) as arguments.
In order to prevent this, we can wrap the function
with a #staticmethod decorator,
with a #classmethod decorator,
with one of other, similiar working, self-made decorators.
#staticmethod turns it into an object which, when asked, changes the argument-passing behaviour so that it matches the intentions about calling the original f:
class A(object):
def method(self):
pass
#staticmethod
def stmethod():
pass
#classmethod
def clmethod(cls):
pass
a = A()
a.method() # the "function inside" gets told about a
A.method() # doesn't work because there is no reference to the needed object
a.clmethod() # the "function inside" gets told about a's class, A
A.clmethod() # works as well, because we only need the classgets told about a's class, A
a.stmethod() # the "function inside" gets told nothing about anything
A.stmethod() # works as well
So #classmethod and #staticmethod have in common that they "don't care about" the concrete object they were called with; the difference is that #staticmethod doesn't want to know anything at all about it, while #classmethod wants to know its class.
So the latter gets the class object the used object is an instance of. Just replace self with cls in this case.
Now, when to use what?
Well, that is easy to handle:
If you have an access to self, you clearly need an instance method.
If you don't access self, but want to know about its class, use #classmethod. This may for example be the case with factory methods. datetime.datetime.now() is such an example: you can call it via its class or via an instance, but it creates a new instance with completely different data. I even used them once for automatically generating subclasses of a given class.
If you need neither self nor cls, you use #staticmethod. This can as well be used for factory methods, if they don't need to care about subclassing.

#classmethod takes the class as first argument while function takes instance of the class
>>> class Test(object):
... def func(self):
... print self
... #classmethod
... def meth(self):
... print self
>>> t = Test()
>>> t.func()
<__main__.Test object at 0x00000000027238D0>
>>> t.meth()
<class '__main__.Test'>
I've used self argument in meth intentionally so it would be very close in syntax to the func. But usually you'd better use cls as argument:
... #classmethod
... def meth(cls):
... print cls

Difference between calling a method and accessing an attribute

I'm very new to Python, and I'm using Python 3.3.1.
class Parent: # define parent class
parentAttr = 100
age = 55
def __init__(self):
print ("Calling parent constructor")
def setAttr(self, attr):
Parent.parentAttr = attr
class Child(Parent):
def childMethod(self):
print ('Calling child method')
Now I'll create
c=child
c.[here every thing will appear methods and attr (age,setAttr)]
How can I distinguish between methods and atrributes? I mean, when do I use c.SetAtrr(Argument), and c.SetAtrr=value?

Methods are attributes too. They just happen to be callable objects.
You can detect if an object is callable by using the callable() function:
>>> def foo(): pass
...
>>> callable(foo)
True
>>> callable(1)
False
When you call a method, you look up the attribute (a getattr() operation) and then call the result:
c.setAttr(newvalue)
is two steps; finding the attribute (which in this case looks up the attribute on the class, and treats it as a descriptor), then calls the resulting object, a method.
When you assign to an attribute, you rebind that name to a new value:
c.setAttr = 'something else'
would be a setattr() operation.
If you wanted to intercept getting and setting attributes on instances of your class, you could provide the attribute access hooks, __getattr__, __setattr__ and __delattr__.
If you wanted to add a method to an instance, you would have to treat the function as a descriptor object, which produces a method object:
>>> class Foo: pass
...
>>> foo = Foo() # instance
>>> def bar(self): pass
...
>>> bar
<function bar at 0x10b85a320>
>>> bar.__get__(foo, Foo)
<bound method Foo.bar of <__main__.Foo instance at 0x10b85b830>>
The return value of function.__get__(), when given an instance and a class, is a bound method. Calling that method will call the underlying function with self bound to the instance.
And speaking of descriptors, the property() function returns a descriptor too, making it possible to have functions that behave like attributes; they can intercept the getattr(), setattr() and delattr() operations for just that attribute and turn it into a function call:
>>> class Foo:
... #property
... def bar(self):
... return "Hello World!"
...
>>> foo = Foo()
>>> foo.bar
"Hello World!"
Accessing .bar invoked the bar property get hook, which then calls the original bar method.
In almost all situations, you are not going to need the callable() function; you document your API, and provide methods and attributes and the user of your API will figure it out without testing each and every attribute to see if it is callable. With properties, you have the flexibility of providing attributes that are really callables in any case.