Develop Reference

How does Python turn a function into a method?

I know that functions are just descriptors, like this:
def func(self):
print(self.name)
class C:
def __init__(self, name):
self.name = name
C.func = func
c = C("foo")
c.func()
I thought at first that c.func equals C.func.__get__(c)，yes，C.func.__get__(c) return a bound method. But when I set the __get__ of func to None, c.func still returns a bound method.
def func(self):
print(self.name)
class C:
def __init__(self, name):
self.name = name
func.__get__ = None
C.func = func
c = C("foo")
c.func
output:
<bound method func of <__main__.C object at 0x0000027EB23BF088>>
So I'm confused. Moreover, I found that when calling a function from an instance, Python actually calls the class's ___getAttribute__ method, which returns a bound method.
def func(self):
print(self.name)
func.__get__ = None
class C:
def __getattribute__(self, name):
r = super().__getattribute__(name)
print(r) # r is a bound method already
return r
def __init__(self, name):
self.name = name
C.func = func
c = C("foo")
c.func
output:
<bound method func of <__main__.C object at 0x0000027EB243D1C8>>
func.__get__ doesn't seem to have any effect. So, What happended in __getattribute__? How does Python turn a function into a method? I've Googled and done some research, but I still can't find the answer.
Maybe I'm making things complicated, In my understanding, function is itself a descriptor, but just like the code below, I set the func to None, it works normally:
class C:
def func(self):
print('hello world')
func.__get__ = None
c = C()
c.func()
but if it's a descriptor, it will raise TypeError:
class C:
class D:
def __get__(self, inst, cls):
if inst is None:
return self
return 'hello world'
D.__get__ = None
func = D()
c = C()
c.func

Well, if I understand correctly from what I found. (Since I didn't know the descriptors, that's exactly why I like to help, still learning)
First, let's look at __getattr__ and __getattribute__.
Let's have an empty class A
class A:
pass
If I initialize an object and try to call a property, because there is none at the moment, we get AttributeError.
a = A()
a.some_property
The following occurs:
Simple check of flow:
class FlowDemo:
def __init__(self):
self.inited_property = True
def __getattribute__(self, item):
if item in ('__class__', '__len__') : # For less spam of getting this attribute, if you want, you can remove condition.
print('Get Attribute', item)
# Call default behavior
return super().__getattribute__(item)
def __getattr__(self, item):
print('Get Attr', item)
if item == 'some_magic_name':
return "It's magic!"
raise AttributeError
fd = FlowDemo()
fd.inited_property
# Get Attribute inited_property
# True
fd.some_magic_property
# Get Attribute some_magic_name
# Get Attr some_magic_name
# "It's magic!"
fd.some_property
# Get Attribute some_property
# Get Attr some_property
# Traceback (most recent call last):
# File "<input>", line 1, in <module>
# File "stack-class-property-and-descriptors.py", line 67, in # __getattr__
# raise AttributeError
# AttributeError
This is probably understandable, including the use. But to be sure, I'll give an example. This logic is used as a dynamic representation of the result from the databases (mapping of attributes to ordinary dict, list, etc.).
But it can also be just logic for accessing an attribute (property), such as an access counter or validation (but this applies to __setattr__ and __setattribute__)
And what about descriptors?
First let's look at data-descriptors, they are easier for me to understand.
This is a class or decoder that has __get__ and one or both of __set__ and __delete__.
Once this is defined, python, when used in the property definition with it and then does not return a class but the value it obtains through __get__, does not overwrite an already declared class when declaring a value, but uses its __set__.
Example:
class WeekDayDescriptor:
def __init__(self):
self.__week_day = 0
def __get__(self, instance, owner=None):
return self.__week_day
def __set__(self, instance, value):
if not isinstance(value, int):
raise TypeError('Value must be int')
if not (0 <= value < 6):
raise ValueError('Value must be in range 0 - 6')
self.__week_day = value
class Calendar:
week_day = WeekDayDescriptor()
def __init__(self, week_day):
self.week_day = week_day
Demo:
c = Calendar(9)
# ValueError: Value must be in range 0-6
c = Calendar('6')
# TypeError: Value must be int
c = Calendar(3)
c.week_day = 6
c.week_day = 10
# ValueError: Value must be in range 0-6
c.week_day = 'monday'
# TypeError: Value must be int
Decorator descriptor:
class Calendar:
#property
def week_day(self):
return self.__week_day
#week_day.setter
def week_day(self, week_day):
if not isinstance(week_day, int):
raise TypeError('Value must be int')
if not (0 <= week_day < 6):
raise ValueError('Value must be in range 0 - 6')
self.__week_day = week_day
def __init__(self, week_day):
self.week_day = week_day
pass
And now for non-data descriptors...
A non-data descriptor is one that has only __get__.
As I understand it, each method automatically has its own descriptor, thanks to which the functions get references to the object - self.
We can write our own descriptor for a function / method, but it's not that straightforward, we have to help ourselves and get around it a bit.
def function_as_method(self, value):
print(self, value)
class HelperDescriptor:
def __get__(self, instance, owner):
def wrapper(*args, **kwargs):
return function_as_method(instance, *args, **kwargs)
return wrapper
class Foo:
baz = HelperDescriptor()
>>> bar = Foo()
>>> bar.baz(1)
<__main__.Foo object at 0x7f64f7768b70> 1
Source of last code block, but in czech lang.
And finally, your mentioned problem, when we set __get__ to None and you still get a reference to the function.
It's simple, python doesn't directly distinguish between primitive data types and functions, it's all a variable (or attribute / property) that has a value. Whether it's just value or it's callable is a different matter.
def f(): return True
print(type(f), f())
# <class 'function'> True
f = 123
print(type(f), f)
# <class 'int'> 123
Therefore, when we ask for the obj.func method or call it obj.func() directly, the first two changed magic is called first - __getattribute__ and __getattr__.
And if we call a method, it is called only after we get a reference to a function in memory.
Again a simple example:
def func(self, value):
print('Printing:', value)
class PrintDescriptor:
def __init__(self, name):
self.name = name
def __get__(self, instance, owner):
def wrapper(*args, **kwargs):
print(f"Calling '{self.name}' method")
return func(instance, *args, **kwargs)
return wrapper
class B:
foo = PrintDescriptor('foo')
bar = PrintDescriptor('bar')
def __getattribute__(self, item):
if item not in ('__len__', '__class__', '__dict__'):
print('Get Attribute', item)
return super().__getattribute__(item)
Demo:
b = B()
b.foo
# Get Attribute foo
# <function PrintDescriptor.__get__.<locals>.wrapper at 0x7f774a782ee0>
b.foo(2)
# Get Attribute foo
# Calling 'foo' method
# Printing: 2
b.bar(4)
# Get Attribute bar
# Calling 'bar' method
# Printing: 4
Sources:
https://www.datacamp.com/community/tutorials/python-descriptors#above1
https://blog.milde.cz/post/319-pokrocile-techniky-v-pythonu-deskriptory/
Python Doc, __get__
Python Docs, __getattribute__
Python Docs, __getattr__

Prevent creating new attributes for class or module

I saw this question on SO Prevent creating new attributes outside init which shows how to prevent adding new attributes to objects of classes.
I wanted the same behaviour for the overall class or even the complete loaded module.
Example class:
class Klass:
a = 0
b = 1
Another module:
from Klass import Klass
Klass.c = 2 # this should raise an error
Is this possible?

If you're trying to prevent modifying the class itself, you can create a metaclass that defines the __setattr__ method for the class.
class FrozenMeta(type):
def __new__(cls, name, bases, dct):
inst = super().__new__(cls, name, bases, {"_FrozenMeta__frozen": False, **dct})
inst.__frozen = True
return inst
def __setattr__(self, key, value):
if self.__frozen and not hasattr(self, key):
raise TypeError("I am frozen")
super().__setattr__(key, value)
class A(metaclass=FrozenMeta):
a = 1
b = 2
A.a = 2
A.c = 1 # TypeError: I am frozen

The answer with slots would be the Pythonic way to do it.
class Klass:
__slots__ = ['a', 'b']
def __init__(self, a=0, b=1):
self.a = a
self.b = b
>>> k = klass.Klass()
>>> k.a
0
>>> k.b
1
>>> k.c = 2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Klass' object has no attribute 'c'
>>>

immutable objects in Python that can have weak references

I've been subclassing tuple or using namedtuple blissfully for a few years, but now I have a use case where I need a class that can be used as a weak referent. And today I learned tuples don't support weak references.
Is there another way to create an immutable object in Python with a fixed set of attributes? I don't need the numeric indexing or variable width of a tuple.
class SimpleThingWithMethods(object):
def __init__(self, n, x):
# I just need to store n and x as read-only attributes
... ??? ...
I guess this raises the obvious question of why immutable; "Pythonic" code usually just assumes we're all adults here and no one in their right mind would reach into a class and muck with its values if it risks ruining the class invariants. In my case I have a class in a library and I am worried about accidental modification of objects by end-users. The people I work with sometimes make incorrect assumptions about my code and start doing things I did not expect, so it's much cleaner if I can raise an error if they accidentally modify my code.
I'm not so worried about bulletproof immutability; if someone really nefarious wants to go and modify things, ok, fine, they're on their own. I just want to make it hard to accidentally modify my objects.

well, this isn't a great answer but it looks like I can modify the answer in https://stackoverflow.com/a/4828492/44330 --- essentially overriding __setattr__ and __delattr__ to meet my needs at least against accidental modification. (but not as nice as subclassing tuple)
class Point(object):
__slots__ = ('x','y','__weakref__')
def __init__(self, x, y):
object.__setattr__(self, "x", x)
object.__setattr__(self, "y", y)
def __setattr__(self, *args):
raise TypeError
def __delattr__(self, *args):
raise TypeError
def __eq__(self, other):
return self.x == other.x and self.y == other.y
def __hash__(self):
return self.x.__hash__() * 31 + self.y.__hash__()
Implementing #Elazar's idea:
class Point(object):
__slots__ = ('x','y','__weakref__')
def __new__(cls, x, y):
thing = object.__new__(cls)
object.__setattr__(thing, "x", x)
object.__setattr__(thing, "y", y)
return thing
def __setattr__(self, *args):
raise TypeError
def __delattr__(self, *args):
raise TypeError
def __eq__(self, other):
return self.x == other.x and self.y == other.y
def __hash__(self):
return self.x.__hash__() * 31 + self.y.__hash__()

If you don't worry about isinstance checks, you can strengthen you answer:
def Point(x, y):
class Point(object):
__slots__ = ('x','y','__weakref__')
def __setattr__(self, *args):
raise TypeError
def __delattr__(self, *args):
raise TypeError
def __eq__(self, other):
return x == other.x and y == other.y
def __hash__(self):
return x.__hash__() * 31 + y.__hash__()
p = Point()
object.__setattr__(p, "x", x)
object.__setattr__(p, "y", y)
return p
I don't really recommend it (every invocation creates a class!), just wanted to note the possibility.
It is also possible to go javascript all the way, and supply __getattr__ that will access the local variables. But that will also slow down access, in addition to creation. Now we don't need these slots at all:
class MetaImmutable:
def __setattr__(self, name, val):
raise TypeError
def Point(x, y):
class Point(object):
__metaclass__ = MetaImmutable
__slots__ = ('__weakref__',)
def __getattr__(self, name):
if name == 'x': return x
if name == 'y': return y
raise TypeError
#property
def x(self): return x
#property
def y(self): return y
def __eq__(self, other):
return x == other.x and y == other.y
def __hash__(self):
return x.__hash__() * 31 + y.__hash__()
return Point()
Test it:
>>> p = Point(1, 2)
>>> p.y
2
>>> p.z
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in __getattr__
TypeError
>>> p.z = 5
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Point' object has no attribute 'z'
>>> object.__setattr__(p, 'z', 5)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Point' object has no attribute 'z'
>>> from weakref import ref
>>> ref(p)().x
1
>>> type(p).x = property(lambda self: 3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in __setattr__
TypeError
And finally, you can still break it:
>>> type.__setattr__(type(p), 'x', property(lambda self: 5))
>>> p.x
5
Again, nothing here is recommended. Use #Jasons implementation.

What about using encapsulation and abstraction on the parameter (getter?):
class SimpleThingWithMethods(object):
def __init__(self, n, x):
self._n = n
self._x = x
def x(self):
return self._x
def n(self):
return self._n
SimpleThingWithMethods(2,3).x()
=> 3

Python __getattribute__ and __setattr__

I have the following code:
#-*-coding:utf-8-*-
class A(object):
def __init__(self, x):
self.x = x
def __getattr__(self, name): # `__getattr__` will be called undefined attribute
print "get: ", name
return self.__dict__.get(name)
def __setattr__(self, name, value):
print "set:", name, value
self.__dict__[name] = value
def __getattribute__(self, name): # `__getattribute__` will be called all attributes
print "attribute:", name
return object.__getattribute__(self, name)
if __name__ == "__main__":
a = A(10)
print '---------------'
a.x
print '---------------'
a.y = 20
print '---------------'
a.z
And the result is :
set: x 10
attribute: __dict__
---------------
attribute: x
---------------
set: y 20
attribute: __dict__
---------------
attribute: z
get: z
attribute: __dict__
When I called a=A(10), why __getattribute__ is called ? This is my thought: there is self.x = x in __init__ , and __setattr__ catch __init__, self.__dict__[name] = value catch __getattrbute__. So, __getattribute__ is called. Does my thought right ? What's wrong ?

The arrow is pointing to where __setattr__ invokes __getattribute__:
def __setattr__(self, name, value):
print "set:", name, value
self.__dict__[name] = value
# ^ attribute access!
__getattribute__ handles all explicit attribute lookup, including __dict__. I believe this is the conclusion you already came to; I couldn't quite understand what you were trying to say.

Why does #foo.setter in Python not work for me?

So, I'm playing with decorators in Python 2.6, and I'm having some trouble getting them to work. Here is my class file:
class testDec:
#property
def x(self):
print 'called getter'
return self._x
#x.setter
def x(self, value):
print 'called setter'
self._x = value
What I thought this meant is to treat x like a property, but call these functions on get and set. So, I fired up IDLE and checked it:
>>> from testDec import testDec
from testDec import testDec
>>> t = testDec()
t = testDec()
>>> t.x
t.x
called getter
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "testDec.py", line 18, in x
return self._x
AttributeError: testDec instance has no attribute '_x'
>>> t.x = 5
t.x = 5
>>> t.x
t.x
5
Clearly the first call works as expected, since I call the getter, and there is no default value, and it fails. OK, good, I understand. However, the call to assign t.x = 5 seems to create a new property x, and now the getter doesn't work!
What am I missing?

You seem to be using classic old-style classes in python 2. In order for properties to work correctly you need to use new-style classes instead (in python 2 you must inherit from object). Just declare your class as MyClass(object):
class testDec(object):
#property
def x(self):
print 'called getter'
return self._x
#x.setter
def x(self, value):
print 'called setter'
self._x = value
It works:
>>> k = testDec()
>>> k.x
called getter
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/devel/class_test.py", line 6, in x
return self._x
AttributeError: 'testDec' object has no attribute '_x'
>>> k.x = 5
called setter
>>> k.x
called getter
5
>>>
Another detail that might cause problems is that both methods need the same name for the property to work. If you define the setter with a different name like this it won't work:
#x.setter
def x_setter(self, value):
...
And one more thing that is not completely easy to spot at first, is the order: The getter must be defined first. If you define the setter first, you get name 'x' is not defined error.

Just a note for other people who stumble here looking for this exception: both functions need to have the same name. Naming the methods as follows will result in an exception:
#property
def x(self): pass
#x.setter
def x_setter(self, value): pass
Instead give both methods the same name
#property
def x(self): pass
#x.setter
def x(self, value): pass
It is also important to note that the order of the declaration matters. The getter must be defined before the setter in the file or else you will get a NameError: name 'x' is not defined

You need to use new-style classes which you do by deriving your class from object:
class testDec(object):
....
Then it should work.

In case anybody comes here from google, in addition to the above answers I would like to add that this needs careful attention when invoking the setter from the __init__ method of your class based on this answer
Specifically:
class testDec(object):
def __init__(self, value):
print 'We are in __init__'
self.x = value # Will call the setter. Note just x here
#self._x = value # Will not call the setter
#property
def x(self):
print 'called getter'
return self._x # Note the _x here
#x.setter
def x(self, value):
print 'called setter'
self._x = value # Note the _x here
t = testDec(17)
print t.x
Output:
We are in __init__
called setter
called getter
17