How to overload the constructor (not the initializer) in Python [duplicate] - python

I am working with Python code that calls into C wrappers, but the C code is very buggy (and I have no other alternative) and causes a segfault when a Python object managed in C goes out of scope, so I have to keep references to each object created.
Is there any good way to make an ergonomic "unique" wrapper where each class can only have one instance per set of constructor arguments, e.g.
#unique
class Test:
cls_val = 0
def __init__(self, val):
self.val = val
a = Test(1)
b = Test(1)
assert a is b
c = Test(2)
d = Test(2)
assert c is not b and c is not a
assert c is d
I've made this decorator, but it prevents any #unique-decorated class from being used as a base class (constructing an instance of the derived class calls the __new__ of the decorator).
def unique(unique_cls):
class Unique:
instances = {}
unique_class = unique_cls
def __new__(cls, *args, **kwargs):
if not Unique.instances.get(
(
cls.unique_class,
f_args := frozenset(args),
f_kwargs := frozenset(kwargs),
)
):
Unique.instances[
(Unique.unique_class, f_args, f_kwargs)
] = Unique.unique_class(*args, **kwargs)
return Unique.instances[(Unique.unique_class, f_args, f_kwargs)]
def __getattr__(self, name):
# Overloaded to get class attributes working for decorated classes
return object.__getattribute__(Unique.unique_class, name)
return Unique

You can use functools.lru_cache to store all instances in a cache, and set no limit on the cache size. When you call the constructor, you'll get a new instance, and it will be stored in the cache. Then whenever you call the constructor with the same arguments again, you'll get the cached instance. This also means that every object always has a reference from the cache.
from functools import lru_cache
#lru_cache(maxsize=None)
class Test:
def __init__(self, val):
self.val = val
Demonstration:
>>> a = Test(1)
>>> a2 = Test(1)
>>> a is a2
True
>>> b = Test(2)
>>> b2 = Test(2)
>>> b is b2
True
>>> a is b
False
If you need to be able to subclass Test (or really, do anything with Test itself except create instances), then you can override __new__ and apply the decorator there. This works because cls is an argument to __new__, so the cache will distinguish between different instances by their class as well as by their __init__ arguments.
class Test:
#lru_cache(maxsize=None)
def __new__(cls, *args, **kwargs):
return object.__new__(cls)
def __init__(self, val):
self.val = val
Demonstration:
>>> Test(1) is Test(1)
True
>>> Test(1) is Test(2)
False
>>> class SubTest(Test): pass
...
>>> Test(1) is SubTest(1)
False
>>> SubTest(1) is SubTest(1)
True

Related

Best way to make decorator for unique class instances Python

I am working with Python code that calls into C wrappers, but the C code is very buggy (and I have no other alternative) and causes a segfault when a Python object managed in C goes out of scope, so I have to keep references to each object created.
Is there any good way to make an ergonomic "unique" wrapper where each class can only have one instance per set of constructor arguments, e.g.
#unique
class Test:
cls_val = 0
def __init__(self, val):
self.val = val
a = Test(1)
b = Test(1)
assert a is b
c = Test(2)
d = Test(2)
assert c is not b and c is not a
assert c is d
I've made this decorator, but it prevents any #unique-decorated class from being used as a base class (constructing an instance of the derived class calls the __new__ of the decorator).
def unique(unique_cls):
class Unique:
instances = {}
unique_class = unique_cls
def __new__(cls, *args, **kwargs):
if not Unique.instances.get(
(
cls.unique_class,
f_args := frozenset(args),
f_kwargs := frozenset(kwargs),
)
):
Unique.instances[
(Unique.unique_class, f_args, f_kwargs)
] = Unique.unique_class(*args, **kwargs)
return Unique.instances[(Unique.unique_class, f_args, f_kwargs)]
def __getattr__(self, name):
# Overloaded to get class attributes working for decorated classes
return object.__getattribute__(Unique.unique_class, name)
return Unique
You can use functools.lru_cache to store all instances in a cache, and set no limit on the cache size. When you call the constructor, you'll get a new instance, and it will be stored in the cache. Then whenever you call the constructor with the same arguments again, you'll get the cached instance. This also means that every object always has a reference from the cache.
from functools import lru_cache
#lru_cache(maxsize=None)
class Test:
def __init__(self, val):
self.val = val
Demonstration:
>>> a = Test(1)
>>> a2 = Test(1)
>>> a is a2
True
>>> b = Test(2)
>>> b2 = Test(2)
>>> b is b2
True
>>> a is b
False
If you need to be able to subclass Test (or really, do anything with Test itself except create instances), then you can override __new__ and apply the decorator there. This works because cls is an argument to __new__, so the cache will distinguish between different instances by their class as well as by their __init__ arguments.
class Test:
#lru_cache(maxsize=None)
def __new__(cls, *args, **kwargs):
return object.__new__(cls)
def __init__(self, val):
self.val = val
Demonstration:
>>> Test(1) is Test(1)
True
>>> Test(1) is Test(2)
False
>>> class SubTest(Test): pass
...
>>> Test(1) is SubTest(1)
False
>>> SubTest(1) is SubTest(1)
True

How to avoid creating objects with same values?

I need to create a class whose instances can't have same values. If you create instance with value that have already been used you'll get old same instance.
I did it using special class method:
class A():
instances = []
def __init__(self, val):
self.val = val
#classmethod
def new(cls, val):
"""
Return instance with same value or create new.
"""
for ins in cls.instances:
if ins.val == val:
return ins
new_ins = A(val)
cls.instances.append(new_ins)
return new_ins
a1 = A.new("x")
a2 = A.new("x")
a3 = A.new("y")
print a1 # <__main__.A instance at 0x05B7FD00> S\ /M\
print a2 # <__main__.A instance at 0x05B7FD00> \A/ \E
print a3 # <__main__.A instance at 0x05B7FD28>
Is there a way to do it more elegant, without using .new method?
You could try functools.lru_cache.
For example:
from functools import lru_cache
class A:
#lru_cache()
def __new__(cls, arg):
return super().__new__(cls)
def __init__(self, arg):
self.n = arg
Sample usage:
>>> a1 = A('1')
>>> a2 = A('1')
>>> a1 is a2
True
>>> a1.n
'1'
>>> a2.n
'1'
Alternatively you could try building a custom caching class, as pointed out by Raymond Hettinger in this tweet: https://twitter.com/raymondh/status/977613745634471937.
This can be done by overriding the __new__ method, which is responsible for creating new instances of a class. Whenever you create a new instance you store it in a dict, and if the dict contains a matching instance then you return it instead of creating a new one:
class A:
instances = {}
def __new__(cls, val):
try:
return cls.instances[val]
except KeyError:
pass
obj = super().__new__(cls)
cls.instances[val] = obj
return obj
def __init__(self, val):
self.val = val
a = A(1)
b = A(2)
c = A(1)
print(a is b) # False
print(a is c) # True
One downside of this solution is that the __init__ method will be called regardless of whether the instance is a newly created one or one that's been stored in the dict. This can cause problems if your constructor has undesired side effects:
class A:
...
def __init__(self, val):
self.val = val
self.foo = 'foo'
a = A(1)
a.foo = 'bar'
b = A(1)
print(a.foo) # output: foo
Notice how a's foo attribute changed from "bar" to "foo" when b was created.
Another option is to use a metaclass and override its __call__ method:
class MemoMeta(type):
def __new__(mcs, name, bases, attrs):
cls = super().__new__(mcs, name, bases, attrs)
cls.instances = {}
return cls
def __call__(cls, val):
try:
return cls.instances[val]
except KeyError:
pass
obj = super().__call__(val)
cls.instances[val] = obj
return obj
class A(metaclass=MemoMeta):
def __init__(self, val):
self.val = val
self.foo = 'foo'
This bypasses the problem with __init__ being called on existing instances:
a = A(1)
a.foo = 'bar'
b = A(1)
print(a.foo) # output: bar
If you really want to make it more elegant, implement the duplicate check in __new__, so it will be performed when you call A(something).
Just do it in __new__:
def __new__(cls, val=None):
for i in cls.instances:
if val == i.val:
return i
return object.__new__(cls)

property decorator which functions on class (as opposed to instance) , setter [duplicate]

I have a class with two class methods (using the classmethod() function) for getting and setting what is essentially a static variable. I tried to use the property() function with these, but it results in an error. I was able to reproduce the error with the following in the interpreter:
class Foo(object):
_var = 5
#classmethod
def getvar(cls):
return cls._var
#classmethod
def setvar(cls, value):
cls._var = value
var = property(getvar, setvar)
I can demonstrate the class methods, but they don't work as properties:
>>> f = Foo()
>>> f.getvar()
5
>>> f.setvar(4)
>>> f.getvar()
4
>>> f.var
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: 'classmethod' object is not callable
>>> f.var=5
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: 'classmethod' object is not callable
Is it possible to use the property() function with #classmethod decorated functions?
3.8 < Python < 3.11
Can use both decorators together. See this answer.
Python < 3.9
A property is created on a class but affects an instance. So if you want a classmethod property, create the property on the metaclass.
>>> class foo(object):
... _var = 5
... class __metaclass__(type): # Python 2 syntax for metaclasses
... pass
... #classmethod
... def getvar(cls):
... return cls._var
... #classmethod
... def setvar(cls, value):
... cls._var = value
...
>>> foo.__metaclass__.var = property(foo.getvar.im_func, foo.setvar.im_func)
>>> foo.var
5
>>> foo.var = 3
>>> foo.var
3
But since you're using a metaclass anyway, it will read better if you just move the classmethods in there.
>>> class foo(object):
... _var = 5
... class __metaclass__(type): # Python 2 syntax for metaclasses
... #property
... def var(cls):
... return cls._var
... #var.setter
... def var(cls, value):
... cls._var = value
...
>>> foo.var
5
>>> foo.var = 3
>>> foo.var
3
or, using Python 3's metaclass=... syntax, and the metaclass defined outside of the foo class body, and the metaclass responsible for setting the initial value of _var:
>>> class foo_meta(type):
... def __init__(cls, *args, **kwargs):
... cls._var = 5
... #property
... def var(cls):
... return cls._var
... #var.setter
... def var(cls, value):
... cls._var = value
...
>>> class foo(metaclass=foo_meta):
... pass
...
>>> foo.var
5
>>> foo.var = 3
>>> foo.var
3
In Python 3.9 You could use them together, but (as noted in #xgt's comment) it was deprecated in Python 3.11, so it is not recommended to use it.
Check the version remarks here:
https://docs.python.org/3.11/library/functions.html#classmethod
However, it used to work like so:
class G:
#classmethod
#property
def __doc__(cls):
return f'A doc for {cls.__name__!r}'
Order matters - due to how the descriptors interact, #classmethod has to be on top.
I hope this dead-simple read-only #classproperty decorator would help somebody looking for classproperties.
class classproperty(property):
def __get__(self, owner_self, owner_cls):
return self.fget(owner_cls)
class C(object):
#classproperty
def x(cls):
return 1
assert C.x == 1
assert C().x == 1
Reading the Python 2.2 release notes, I find the following.
The get method [of a property] won't be called when
the property is accessed as a class
attribute (C.x) instead of as an
instance attribute (C().x). If you
want to override the __get__ operation
for properties when used as a class
attribute, you can subclass property -
it is a new-style type itself - to
extend its __get__ method, or you can
define a descriptor type from scratch
by creating a new-style class that
defines __get__, __set__ and
__delete__ methods.
NOTE: The below method doesn't actually work for setters, only getters.
Therefore, I believe the prescribed solution is to create a ClassProperty as a subclass of property.
class ClassProperty(property):
def __get__(self, cls, owner):
return self.fget.__get__(None, owner)()
class foo(object):
_var=5
def getvar(cls):
return cls._var
getvar=classmethod(getvar)
def setvar(cls,value):
cls._var=value
setvar=classmethod(setvar)
var=ClassProperty(getvar,setvar)
assert foo.getvar() == 5
foo.setvar(4)
assert foo.getvar() == 4
assert foo.var == 4
foo.var = 3
assert foo.var == 3
However, the setters don't actually work:
foo.var = 4
assert foo.var == foo._var # raises AssertionError
foo._var is unchanged, you've simply overwritten the property with a new value.
You can also use ClassProperty as a decorator:
class foo(object):
_var = 5
#ClassProperty
#classmethod
def var(cls):
return cls._var
#var.setter
#classmethod
def var(cls, value):
cls._var = value
assert foo.var == 5
Is it possible to use the property() function with classmethod decorated functions?
No.
However, a classmethod is simply a bound method (a partial function) on a class accessible from instances of that class.
Since the instance is a function of the class and you can derive the class from the instance, you can can get whatever desired behavior you might want from a class-property with property:
class Example(object):
_class_property = None
#property
def class_property(self):
return self._class_property
#class_property.setter
def class_property(self, value):
type(self)._class_property = value
#class_property.deleter
def class_property(self):
del type(self)._class_property
This code can be used to test - it should pass without raising any errors:
ex1 = Example()
ex2 = Example()
ex1.class_property = None
ex2.class_property = 'Example'
assert ex1.class_property is ex2.class_property
del ex2.class_property
assert not hasattr(ex1, 'class_property')
And note that we didn't need metaclasses at all - and you don't directly access a metaclass through its classes' instances anyways.
writing a #classproperty decorator
You can actually create a classproperty decorator in just a few lines of code by subclassing property (it's implemented in C, but you can see equivalent Python here):
class classproperty(property):
def __get__(self, obj, objtype=None):
return super(classproperty, self).__get__(objtype)
def __set__(self, obj, value):
super(classproperty, self).__set__(type(obj), value)
def __delete__(self, obj):
super(classproperty, self).__delete__(type(obj))
Then treat the decorator as if it were a classmethod combined with property:
class Foo(object):
_bar = 5
#classproperty
def bar(cls):
"""this is the bar attribute - each subclass of Foo gets its own.
Lookups should follow the method resolution order.
"""
return cls._bar
#bar.setter
def bar(cls, value):
cls._bar = value
#bar.deleter
def bar(cls):
del cls._bar
And this code should work without errors:
def main():
f = Foo()
print(f.bar)
f.bar = 4
print(f.bar)
del f.bar
try:
f.bar
except AttributeError:
pass
else:
raise RuntimeError('f.bar must have worked - inconceivable!')
help(f) # includes the Foo.bar help.
f.bar = 5
class Bar(Foo):
"a subclass of Foo, nothing more"
help(Bar) # includes the Foo.bar help!
b = Bar()
b.bar = 'baz'
print(b.bar) # prints baz
del b.bar
print(b.bar) # prints 5 - looked up from Foo!
if __name__ == '__main__':
main()
But I'm not sure how well-advised this would be. An old mailing list article suggests it shouldn't work.
Getting the property to work on the class:
The downside of the above is that the "class property" isn't accessible from the class, because it would simply overwrite the data descriptor from the class __dict__.
However, we can override this with a property defined in the metaclass __dict__. For example:
class MetaWithFooClassProperty(type):
#property
def foo(cls):
"""The foo property is a function of the class -
in this case, the trivial case of the identity function.
"""
return cls
And then a class instance of the metaclass could have a property that accesses the class's property using the principle already demonstrated in the prior sections:
class FooClassProperty(metaclass=MetaWithFooClassProperty):
#property
def foo(self):
"""access the class's property"""
return type(self).foo
And now we see both the instance
>>> FooClassProperty().foo
<class '__main__.FooClassProperty'>
and the class
>>> FooClassProperty.foo
<class '__main__.FooClassProperty'>
have access to the class property.
Python 3!
See #Amit Portnoy's answer for an even cleaner method in python >= 3.9
Old question, lots of views, sorely in need of a one-true Python 3 way.
Luckily, it's easy with the metaclass kwarg:
class FooProperties(type):
#property
def var(cls):
return cls._var
class Foo(object, metaclass=FooProperties):
_var = 'FOO!'
Then, >>> Foo.var
'FOO!'
There is no reasonable way to make this "class property" system to work in Python.
Here is one unreasonable way to make it work. You can certainly make it more seamless with increasing amounts of metaclass magic.
class ClassProperty(object):
def __init__(self, getter, setter):
self.getter = getter
self.setter = setter
def __get__(self, cls, owner):
return getattr(cls, self.getter)()
def __set__(self, cls, value):
getattr(cls, self.setter)(value)
class MetaFoo(type):
var = ClassProperty('getvar', 'setvar')
class Foo(object):
__metaclass__ = MetaFoo
_var = 5
#classmethod
def getvar(cls):
print "Getting var =", cls._var
return cls._var
#classmethod
def setvar(cls, value):
print "Setting var =", value
cls._var = value
x = Foo.var
print "Foo.var = ", x
Foo.var = 42
x = Foo.var
print "Foo.var = ", x
The knot of the issue is that properties are what Python calls "descriptors". There is no short and easy way to explain how this sort of metaprogramming works, so I must point you to the descriptor howto.
You only ever need to understand this sort of things if you are implementing a fairly advanced framework. Like a transparent object persistence or RPC system, or a kind of domain-specific language.
However, in a comment to a previous answer, you say that you
need to modify an attribute that in such a way that is seen by all instances of a class, and in the scope from which these class methods are called does not have references to all instances of the class.
It seems to me, what you really want is an Observer design pattern.
Setting it only on the meta class doesn't help if you want to access the class property via an instantiated object, in this case you need to install a normal property on the object as well (which dispatches to the class property). I think the following is a bit more clear:
#!/usr/bin/python
class classproperty(property):
def __get__(self, obj, type_):
return self.fget.__get__(None, type_)()
def __set__(self, obj, value):
cls = type(obj)
return self.fset.__get__(None, cls)(value)
class A (object):
_foo = 1
#classproperty
#classmethod
def foo(cls):
return cls._foo
#foo.setter
#classmethod
def foo(cls, value):
cls.foo = value
a = A()
print a.foo
b = A()
print b.foo
b.foo = 5
print a.foo
A.foo = 10
print b.foo
print A.foo
Half a solution, __set__ on the class does not work, still. The solution is a custom property class implementing both a property and a staticmethod
class ClassProperty(object):
def __init__(self, fget, fset):
self.fget = fget
self.fset = fset
def __get__(self, instance, owner):
return self.fget()
def __set__(self, instance, value):
self.fset(value)
class Foo(object):
_bar = 1
def get_bar():
print 'getting'
return Foo._bar
def set_bar(value):
print 'setting'
Foo._bar = value
bar = ClassProperty(get_bar, set_bar)
f = Foo()
#__get__ works
f.bar
Foo.bar
f.bar = 2
Foo.bar = 3 #__set__ does not
Because I need to modify an attribute that in such a way that is seen by all instances of a class, and in the scope from which these class methods are called does not have references to all instances of the class.
Do you have access to at least one instance of the class? I can think of a way to do it then:
class MyClass (object):
__var = None
def _set_var (self, value):
type (self).__var = value
def _get_var (self):
return self.__var
var = property (_get_var, _set_var)
a = MyClass ()
b = MyClass ()
a.var = "foo"
print b.var
Give this a try, it gets the job done without having to change/add a lot of existing code.
>>> class foo(object):
... _var = 5
... def getvar(cls):
... return cls._var
... getvar = classmethod(getvar)
... def setvar(cls, value):
... cls._var = value
... setvar = classmethod(setvar)
... var = property(lambda self: self.getvar(), lambda self, val: self.setvar(val))
...
>>> f = foo()
>>> f.var
5
>>> f.var = 3
>>> f.var
3
The property function needs two callable arguments. give them lambda wrappers (which it passes the instance as its first argument) and all is well.
Here's a solution which should work for both access via the class and access via an instance which uses a metaclass.
In [1]: class ClassPropertyMeta(type):
...: #property
...: def prop(cls):
...: return cls._prop
...: def __new__(cls, name, parents, dct):
...: # This makes overriding __getattr__ and __setattr__ in the class impossible, but should be fixable
...: dct['__getattr__'] = classmethod(lambda cls, attr: getattr(cls, attr))
...: dct['__setattr__'] = classmethod(lambda cls, attr, val: setattr(cls, attr, val))
...: return super(ClassPropertyMeta, cls).__new__(cls, name, parents, dct)
...:
In [2]: class ClassProperty(object):
...: __metaclass__ = ClassPropertyMeta
...: _prop = 42
...: def __getattr__(self, attr):
...: raise Exception('Never gets called')
...:
In [3]: ClassProperty.prop
Out[3]: 42
In [4]: ClassProperty.prop = 1
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-4-e2e8b423818a> in <module>()
----> 1 ClassProperty.prop = 1
AttributeError: can't set attribute
In [5]: cp = ClassProperty()
In [6]: cp.prop
Out[6]: 42
In [7]: cp.prop = 1
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-7-e8284a3ee950> in <module>()
----> 1 cp.prop = 1
<ipython-input-1-16b7c320d521> in <lambda>(cls, attr, val)
6 # This makes overriding __getattr__ and __setattr__ in the class impossible, but should be fixable
7 dct['__getattr__'] = classmethod(lambda cls, attr: getattr(cls, attr))
----> 8 dct['__setattr__'] = classmethod(lambda cls, attr, val: setattr(cls, attr, val))
9 return super(ClassPropertyMeta, cls).__new__(cls, name, parents, dct)
AttributeError: can't set attribute
This also works with a setter defined in the metaclass.
I found one clean solution to this problem. It's a package called classutilities (pip install classutilities), see the documentation here on PyPi.
Consider example:
import classutilities
class SomeClass(classutilities.ClassPropertiesMixin):
_some_variable = 8 # Some encapsulated class variable
#classutilities.classproperty
def some_variable(cls): # class property getter
return cls._some_variable
#some_variable.setter
def some_variable(cls, value): # class property setter
cls._some_variable = value
You can use it on both class level and instance level:
# Getter on class level:
value = SomeClass.some_variable
print(value) # >>> 8
# Getter on instance level
inst = SomeClass()
value = inst.some_variable
print(value) # >>> 8
# Setter on class level:
new_value = 9
SomeClass.some_variable = new_value
print(SomeClass.some_variable) # >>> 9
print(SomeClass._some_variable) # >>> 9
# Setter on instance level
inst = SomeClass()
inst.some_variable = new_value
print(SomeClass.some_variable) # >>> 9
print(SomeClass._some_variable) # >>> 9
print(inst.some_variable) # >>> 9
print(inst._some_variable) # >>> 9
As you can see, it works correctly under all circumstances.
Based on https://stackoverflow.com/a/1800999/2290820
class MetaProperty(type):
def __init__(cls, *args, **kwargs):
super()
#property
def praparty(cls):
return cls._var
#praparty.setter
def praparty(cls, val):
cls._var = val
class A(metaclass=MetaProperty):
_var = 5
print(A.praparty)
A.praparty = 6
print(A.praparty)
For a functional approach pre Python 3.9 you can use this:
def classproperty(fget):
return type(
'classproperty',
(),
{'__get__': lambda self, _, cls: fget(cls), '__module__': None}
)()
class Item:
a = 47
#classproperty
def x(cls):
return cls.a
Item.x
After searching different places, I found a method to define a classproperty
valid with Python 2 and 3.
from future.utils import with_metaclass
class BuilderMetaClass(type):
#property
def load_namespaces(self):
return (self.__sourcepath__)
class BuilderMixin(with_metaclass(BuilderMetaClass, object)):
__sourcepath__ = 'sp'
print(BuilderMixin.load_namespaces)
Hope this can help somebody :)
A code completion friendly solution for Python < 3.9
from typing import (
Callable,
Generic,
TypeVar,
)
T = TypeVar('T')
class classproperty(Generic[T]):
"""Converts a method to a class property.
"""
def __init__(self, f: Callable[..., T]):
self.fget = f
def __get__(self, instance, owner) -> T:
return self.fget(owner)
Here is my solution that also caches the class property
class class_property(object):
# this caches the result of the function call for fn with cls input
# use this as a decorator on function methods that you want converted
# into cached properties
def __init__(self, fn):
self._fn_name = fn.__name__
if not isinstance(fn, (classmethod, staticmethod)):
fn = classmethod(fn)
self._fn = fn
def __get__(self, obj, cls=None):
if cls is None:
cls = type(obj)
if (
self._fn_name in vars(cls) and
type(vars(cls)[self._fn_name]).__name__ != "class_property"
):
return vars(cls)[self._fn_name]
else:
value = self._fn.__get__(obj, cls)()
setattr(cls, self._fn_name, value)
return value
Here's my suggestion. Don't use class methods.
Seriously.
What's the reason for using class methods in this case? Why not have an ordinary object of an ordinary class?
If you simply want to change the value, a property isn't really very helpful is it? Just set the attribute value and be done with it.
A property should only be used if there's something to conceal -- something that might change in a future implementation.
Maybe your example is way stripped down, and there is some hellish calculation you've left off. But it doesn't look like the property adds significant value.
The Java-influenced "privacy" techniques (in Python, attribute names that begin with _) aren't really very helpful. Private from whom? The point of private is a little nebulous when you have the source (as you do in Python.)
The Java-influenced EJB-style getters and setters (often done as properties in Python) are there to facilitate Java's primitive introspection as well as to pass muster with the static language compiler. All those getters and setters aren't as helpful in Python.

Python : Set method attribute from within method

I am trying to make a python decorator that adds attributes to methods of a class so that I can access and modify those attributes from within the method itself. The decorator code is
from types import MethodType
class attribute(object):
def __init__(self, **attributes):
self.attributes = attributes
def __call__(self, function):
class override(object):
def __init__(self, function, attributes):
self.__function = function
for att in attributes:
setattr(self, att, attributes[att])
def __call__(self, *args, **kwargs):
return self.__function(*args, **kwargs)
def __get__(self, instance, owner):
return MethodType(self, instance, owner)
retval = override(function, self.attributes)
return retval
I tried this decorator on the toy example that follows.
class bar(object):
#attribute(a=2)
def foo(self):
print self.foo.a
self.foo.a = 1
Though I am able to access the value of attribute 'a' from within foo(), I can't set it to another value. Indeed, when I call bar().foo(), I get the following AttributeError.
AttributeError: 'instancemethod' object has no attribute 'a'
Why is this? More importantly how can I achieve my goal?
Edit
Just to be more specific, I am trying to find a simple way to implement static variable that are located within class methods. Continuing from the example above, I would like instantiate b = bar(), call both foo() and doo() methods and then access b.foo.a and b.doo.a later on.
class bar(object):
#attribute(a=2)
def foo(self):
self.foo.a = 1
#attribute(a=4)
def doo(self):
self.foo.a = 3
The best way to do this is to not do it at all.
First of all, there is no need for an attribute decorator; you can just assign it yourself:
class bar(object):
def foo(self):
print self.foo.a
self.foo.a = 1
foo.a = 2
However, this still encounters the same errors. You need to do:
self.foo.__dict__['a'] = 1
You can instead use a metaclass...but that gets messy quickly.
On the other hand, there are cleaner alternatives.
You can use defaults:
def foo(self, a):
print a[0]
a[0] = 2
foo.func_defaults = foo.func_defaults[:-1] + ([2],)
Of course, my preferred way is to avoid this altogether and use a callable class ("functor" in C++ words):
class bar(object):
def __init__(self):
self.foo = self.foo_method(self)
class foo_method(object):
def __init__(self, bar):
self.bar = bar
self.a = 2
def __call__(self):
print self.a
self.a = 1
Or just use classic class attributes:
class bar(object):
def __init__(self):
self.a = 1
def foo(self):
print self.a
self.a = 2
If it's that you want to hide a from derived classes, use whatever private attributes are called in Python terminology:
class bar(object):
def __init__(self):
self.__a = 1 # this will be implicitly mangled as __bar__a or similar
def foo(self):
print self.__a
self.__a = 2
EDIT: You want static attributes?
class bar(object):
a = 1
def foo(self):
print self.a
self.a = 2
EDIT 2: If you want static attributes visible to only the current function, you can use PyExt's modify_function:
import pyext
def wrap_mod(*args, **kw):
def inner(f):
return pyext.modify_function(f, *args, **kw)
return inner
class bar(object):
#wrap_mod(globals={'a': [1]})
def foo(self):
print a[0]
a[0] = 2
It's slightly ugly and hackish. But it works.
My recommendation would be just to use double underscores:
class bar(object):
__a = 1
def foo(self):
print self.__a
self.__a = 2
Although this is visible to the other functions, it's invisible to anything else (actually, it's there, but it's mangled).
FINAL EDIT: Use this:
import pyext
def wrap_mod(*args, **kw):
def inner(f):
return pyext.modify_function(f, *args, **kw)
return inner
class bar(object):
#wrap_mod(globals={'a': [1]})
def foo(self):
print a[0]
a[0] = 2
foo.a = foo.func_globals['a']
b = bar()
b.foo() # prints 1
b.foo() # prints 2
# external access
b.foo.a[0] = 77
b.foo() # prints 77
While You can accomplish Your goal by replacing self.foo.a = 1 with self.foo.__dict__['a'] = 1 it is generally not recommended.
If you are using Python2 - (and not Python3) - whenever you retrieve a method from an instance, a new instance method object is created which is a wrapper to the original function defined in the class body.
The instance method is a rather transparent proxy to the function - you can retrieve the function's attributes through it, but not set them - that is why setting an item in self.foo.__dict__ works.
Alternatively you can reach the function object itself using: self.foo.im_func - the im_func attribute of instance methods point the underlying function.
Based on other contributors's answers, I came up with the following workaround. First, wrap a dictionnary in a class resolving non-existant attributes to the wrapped dictionnary such as the following code.
class DictWrapper(object):
def __init__(self, d):
self.d = d
def __getattr__(self, key):
return self.d[key]
Credits to Lucas Jones for this code.
Then implement a addstatic decorator with a statics attribute that will store the static attributes.
class addstatic(object):
def __init__(self, **statics):
self.statics = statics
def __call__(self, function):
class override(object):
def __init__(self, function, statics):
self.__function = function
self.statics = DictWrapper(statics)
def __call__(self, *args, **kwargs):
return self.__function(*args, **kwargs)
def __get__(self, instance, objtype):
from types import MethodType
return MethodType(self, instance)
retval = override(function, self.statics)
return retval
The following code is an example of how the addstatic decorator can be used on methods.
class bar(object):
#attribute(a=2, b=3)
def foo(self):
self.foo.statics.a = 1
self.foo.statics.b = 2
Then, playing with an instance of the bar class yields :
>>> b = bar()
>>> b.foo.statics.a
2
>>> b.foo.statics.b
3
>>> b.foo()
>>> b.foo.statics.a
3
>>> b.foo.statics.b
5
The reason for using this statics dictionnary follows jsbueno's answer which suggest that what I want would require overloading the dot operator of and instance method wrapping the foo function, which I am not sure is possible. Of course, the method's attribute could be set in self.foo.__dict__, but since it not recommended (as suggested by brainovergrow), I came up with this workaround. I am not certain this would be recommended either and I guess it is up for comments.

Using property() on classmethods

I have a class with two class methods (using the classmethod() function) for getting and setting what is essentially a static variable. I tried to use the property() function with these, but it results in an error. I was able to reproduce the error with the following in the interpreter:
class Foo(object):
_var = 5
#classmethod
def getvar(cls):
return cls._var
#classmethod
def setvar(cls, value):
cls._var = value
var = property(getvar, setvar)
I can demonstrate the class methods, but they don't work as properties:
>>> f = Foo()
>>> f.getvar()
5
>>> f.setvar(4)
>>> f.getvar()
4
>>> f.var
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: 'classmethod' object is not callable
>>> f.var=5
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: 'classmethod' object is not callable
Is it possible to use the property() function with #classmethod decorated functions?
3.8 < Python < 3.11
Can use both decorators together. See this answer.
Python < 3.9
A property is created on a class but affects an instance. So if you want a classmethod property, create the property on the metaclass.
>>> class foo(object):
... _var = 5
... class __metaclass__(type): # Python 2 syntax for metaclasses
... pass
... #classmethod
... def getvar(cls):
... return cls._var
... #classmethod
... def setvar(cls, value):
... cls._var = value
...
>>> foo.__metaclass__.var = property(foo.getvar.im_func, foo.setvar.im_func)
>>> foo.var
5
>>> foo.var = 3
>>> foo.var
3
But since you're using a metaclass anyway, it will read better if you just move the classmethods in there.
>>> class foo(object):
... _var = 5
... class __metaclass__(type): # Python 2 syntax for metaclasses
... #property
... def var(cls):
... return cls._var
... #var.setter
... def var(cls, value):
... cls._var = value
...
>>> foo.var
5
>>> foo.var = 3
>>> foo.var
3
or, using Python 3's metaclass=... syntax, and the metaclass defined outside of the foo class body, and the metaclass responsible for setting the initial value of _var:
>>> class foo_meta(type):
... def __init__(cls, *args, **kwargs):
... cls._var = 5
... #property
... def var(cls):
... return cls._var
... #var.setter
... def var(cls, value):
... cls._var = value
...
>>> class foo(metaclass=foo_meta):
... pass
...
>>> foo.var
5
>>> foo.var = 3
>>> foo.var
3
In Python 3.9 You could use them together, but (as noted in #xgt's comment) it was deprecated in Python 3.11, so it is not recommended to use it.
Check the version remarks here:
https://docs.python.org/3.11/library/functions.html#classmethod
However, it used to work like so:
class G:
#classmethod
#property
def __doc__(cls):
return f'A doc for {cls.__name__!r}'
Order matters - due to how the descriptors interact, #classmethod has to be on top.
I hope this dead-simple read-only #classproperty decorator would help somebody looking for classproperties.
class classproperty(property):
def __get__(self, owner_self, owner_cls):
return self.fget(owner_cls)
class C(object):
#classproperty
def x(cls):
return 1
assert C.x == 1
assert C().x == 1
Reading the Python 2.2 release notes, I find the following.
The get method [of a property] won't be called when
the property is accessed as a class
attribute (C.x) instead of as an
instance attribute (C().x). If you
want to override the __get__ operation
for properties when used as a class
attribute, you can subclass property -
it is a new-style type itself - to
extend its __get__ method, or you can
define a descriptor type from scratch
by creating a new-style class that
defines __get__, __set__ and
__delete__ methods.
NOTE: The below method doesn't actually work for setters, only getters.
Therefore, I believe the prescribed solution is to create a ClassProperty as a subclass of property.
class ClassProperty(property):
def __get__(self, cls, owner):
return self.fget.__get__(None, owner)()
class foo(object):
_var=5
def getvar(cls):
return cls._var
getvar=classmethod(getvar)
def setvar(cls,value):
cls._var=value
setvar=classmethod(setvar)
var=ClassProperty(getvar,setvar)
assert foo.getvar() == 5
foo.setvar(4)
assert foo.getvar() == 4
assert foo.var == 4
foo.var = 3
assert foo.var == 3
However, the setters don't actually work:
foo.var = 4
assert foo.var == foo._var # raises AssertionError
foo._var is unchanged, you've simply overwritten the property with a new value.
You can also use ClassProperty as a decorator:
class foo(object):
_var = 5
#ClassProperty
#classmethod
def var(cls):
return cls._var
#var.setter
#classmethod
def var(cls, value):
cls._var = value
assert foo.var == 5
Is it possible to use the property() function with classmethod decorated functions?
No.
However, a classmethod is simply a bound method (a partial function) on a class accessible from instances of that class.
Since the instance is a function of the class and you can derive the class from the instance, you can can get whatever desired behavior you might want from a class-property with property:
class Example(object):
_class_property = None
#property
def class_property(self):
return self._class_property
#class_property.setter
def class_property(self, value):
type(self)._class_property = value
#class_property.deleter
def class_property(self):
del type(self)._class_property
This code can be used to test - it should pass without raising any errors:
ex1 = Example()
ex2 = Example()
ex1.class_property = None
ex2.class_property = 'Example'
assert ex1.class_property is ex2.class_property
del ex2.class_property
assert not hasattr(ex1, 'class_property')
And note that we didn't need metaclasses at all - and you don't directly access a metaclass through its classes' instances anyways.
writing a #classproperty decorator
You can actually create a classproperty decorator in just a few lines of code by subclassing property (it's implemented in C, but you can see equivalent Python here):
class classproperty(property):
def __get__(self, obj, objtype=None):
return super(classproperty, self).__get__(objtype)
def __set__(self, obj, value):
super(classproperty, self).__set__(type(obj), value)
def __delete__(self, obj):
super(classproperty, self).__delete__(type(obj))
Then treat the decorator as if it were a classmethod combined with property:
class Foo(object):
_bar = 5
#classproperty
def bar(cls):
"""this is the bar attribute - each subclass of Foo gets its own.
Lookups should follow the method resolution order.
"""
return cls._bar
#bar.setter
def bar(cls, value):
cls._bar = value
#bar.deleter
def bar(cls):
del cls._bar
And this code should work without errors:
def main():
f = Foo()
print(f.bar)
f.bar = 4
print(f.bar)
del f.bar
try:
f.bar
except AttributeError:
pass
else:
raise RuntimeError('f.bar must have worked - inconceivable!')
help(f) # includes the Foo.bar help.
f.bar = 5
class Bar(Foo):
"a subclass of Foo, nothing more"
help(Bar) # includes the Foo.bar help!
b = Bar()
b.bar = 'baz'
print(b.bar) # prints baz
del b.bar
print(b.bar) # prints 5 - looked up from Foo!
if __name__ == '__main__':
main()
But I'm not sure how well-advised this would be. An old mailing list article suggests it shouldn't work.
Getting the property to work on the class:
The downside of the above is that the "class property" isn't accessible from the class, because it would simply overwrite the data descriptor from the class __dict__.
However, we can override this with a property defined in the metaclass __dict__. For example:
class MetaWithFooClassProperty(type):
#property
def foo(cls):
"""The foo property is a function of the class -
in this case, the trivial case of the identity function.
"""
return cls
And then a class instance of the metaclass could have a property that accesses the class's property using the principle already demonstrated in the prior sections:
class FooClassProperty(metaclass=MetaWithFooClassProperty):
#property
def foo(self):
"""access the class's property"""
return type(self).foo
And now we see both the instance
>>> FooClassProperty().foo
<class '__main__.FooClassProperty'>
and the class
>>> FooClassProperty.foo
<class '__main__.FooClassProperty'>
have access to the class property.
Python 3!
See #Amit Portnoy's answer for an even cleaner method in python >= 3.9
Old question, lots of views, sorely in need of a one-true Python 3 way.
Luckily, it's easy with the metaclass kwarg:
class FooProperties(type):
#property
def var(cls):
return cls._var
class Foo(object, metaclass=FooProperties):
_var = 'FOO!'
Then, >>> Foo.var
'FOO!'
There is no reasonable way to make this "class property" system to work in Python.
Here is one unreasonable way to make it work. You can certainly make it more seamless with increasing amounts of metaclass magic.
class ClassProperty(object):
def __init__(self, getter, setter):
self.getter = getter
self.setter = setter
def __get__(self, cls, owner):
return getattr(cls, self.getter)()
def __set__(self, cls, value):
getattr(cls, self.setter)(value)
class MetaFoo(type):
var = ClassProperty('getvar', 'setvar')
class Foo(object):
__metaclass__ = MetaFoo
_var = 5
#classmethod
def getvar(cls):
print "Getting var =", cls._var
return cls._var
#classmethod
def setvar(cls, value):
print "Setting var =", value
cls._var = value
x = Foo.var
print "Foo.var = ", x
Foo.var = 42
x = Foo.var
print "Foo.var = ", x
The knot of the issue is that properties are what Python calls "descriptors". There is no short and easy way to explain how this sort of metaprogramming works, so I must point you to the descriptor howto.
You only ever need to understand this sort of things if you are implementing a fairly advanced framework. Like a transparent object persistence or RPC system, or a kind of domain-specific language.
However, in a comment to a previous answer, you say that you
need to modify an attribute that in such a way that is seen by all instances of a class, and in the scope from which these class methods are called does not have references to all instances of the class.
It seems to me, what you really want is an Observer design pattern.
Setting it only on the meta class doesn't help if you want to access the class property via an instantiated object, in this case you need to install a normal property on the object as well (which dispatches to the class property). I think the following is a bit more clear:
#!/usr/bin/python
class classproperty(property):
def __get__(self, obj, type_):
return self.fget.__get__(None, type_)()
def __set__(self, obj, value):
cls = type(obj)
return self.fset.__get__(None, cls)(value)
class A (object):
_foo = 1
#classproperty
#classmethod
def foo(cls):
return cls._foo
#foo.setter
#classmethod
def foo(cls, value):
cls.foo = value
a = A()
print a.foo
b = A()
print b.foo
b.foo = 5
print a.foo
A.foo = 10
print b.foo
print A.foo
Half a solution, __set__ on the class does not work, still. The solution is a custom property class implementing both a property and a staticmethod
class ClassProperty(object):
def __init__(self, fget, fset):
self.fget = fget
self.fset = fset
def __get__(self, instance, owner):
return self.fget()
def __set__(self, instance, value):
self.fset(value)
class Foo(object):
_bar = 1
def get_bar():
print 'getting'
return Foo._bar
def set_bar(value):
print 'setting'
Foo._bar = value
bar = ClassProperty(get_bar, set_bar)
f = Foo()
#__get__ works
f.bar
Foo.bar
f.bar = 2
Foo.bar = 3 #__set__ does not
Because I need to modify an attribute that in such a way that is seen by all instances of a class, and in the scope from which these class methods are called does not have references to all instances of the class.
Do you have access to at least one instance of the class? I can think of a way to do it then:
class MyClass (object):
__var = None
def _set_var (self, value):
type (self).__var = value
def _get_var (self):
return self.__var
var = property (_get_var, _set_var)
a = MyClass ()
b = MyClass ()
a.var = "foo"
print b.var
Give this a try, it gets the job done without having to change/add a lot of existing code.
>>> class foo(object):
... _var = 5
... def getvar(cls):
... return cls._var
... getvar = classmethod(getvar)
... def setvar(cls, value):
... cls._var = value
... setvar = classmethod(setvar)
... var = property(lambda self: self.getvar(), lambda self, val: self.setvar(val))
...
>>> f = foo()
>>> f.var
5
>>> f.var = 3
>>> f.var
3
The property function needs two callable arguments. give them lambda wrappers (which it passes the instance as its first argument) and all is well.
Here's a solution which should work for both access via the class and access via an instance which uses a metaclass.
In [1]: class ClassPropertyMeta(type):
...: #property
...: def prop(cls):
...: return cls._prop
...: def __new__(cls, name, parents, dct):
...: # This makes overriding __getattr__ and __setattr__ in the class impossible, but should be fixable
...: dct['__getattr__'] = classmethod(lambda cls, attr: getattr(cls, attr))
...: dct['__setattr__'] = classmethod(lambda cls, attr, val: setattr(cls, attr, val))
...: return super(ClassPropertyMeta, cls).__new__(cls, name, parents, dct)
...:
In [2]: class ClassProperty(object):
...: __metaclass__ = ClassPropertyMeta
...: _prop = 42
...: def __getattr__(self, attr):
...: raise Exception('Never gets called')
...:
In [3]: ClassProperty.prop
Out[3]: 42
In [4]: ClassProperty.prop = 1
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-4-e2e8b423818a> in <module>()
----> 1 ClassProperty.prop = 1
AttributeError: can't set attribute
In [5]: cp = ClassProperty()
In [6]: cp.prop
Out[6]: 42
In [7]: cp.prop = 1
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-7-e8284a3ee950> in <module>()
----> 1 cp.prop = 1
<ipython-input-1-16b7c320d521> in <lambda>(cls, attr, val)
6 # This makes overriding __getattr__ and __setattr__ in the class impossible, but should be fixable
7 dct['__getattr__'] = classmethod(lambda cls, attr: getattr(cls, attr))
----> 8 dct['__setattr__'] = classmethod(lambda cls, attr, val: setattr(cls, attr, val))
9 return super(ClassPropertyMeta, cls).__new__(cls, name, parents, dct)
AttributeError: can't set attribute
This also works with a setter defined in the metaclass.
I found one clean solution to this problem. It's a package called classutilities (pip install classutilities), see the documentation here on PyPi.
Consider example:
import classutilities
class SomeClass(classutilities.ClassPropertiesMixin):
_some_variable = 8 # Some encapsulated class variable
#classutilities.classproperty
def some_variable(cls): # class property getter
return cls._some_variable
#some_variable.setter
def some_variable(cls, value): # class property setter
cls._some_variable = value
You can use it on both class level and instance level:
# Getter on class level:
value = SomeClass.some_variable
print(value) # >>> 8
# Getter on instance level
inst = SomeClass()
value = inst.some_variable
print(value) # >>> 8
# Setter on class level:
new_value = 9
SomeClass.some_variable = new_value
print(SomeClass.some_variable) # >>> 9
print(SomeClass._some_variable) # >>> 9
# Setter on instance level
inst = SomeClass()
inst.some_variable = new_value
print(SomeClass.some_variable) # >>> 9
print(SomeClass._some_variable) # >>> 9
print(inst.some_variable) # >>> 9
print(inst._some_variable) # >>> 9
As you can see, it works correctly under all circumstances.
Based on https://stackoverflow.com/a/1800999/2290820
class MetaProperty(type):
def __init__(cls, *args, **kwargs):
super()
#property
def praparty(cls):
return cls._var
#praparty.setter
def praparty(cls, val):
cls._var = val
class A(metaclass=MetaProperty):
_var = 5
print(A.praparty)
A.praparty = 6
print(A.praparty)
For a functional approach pre Python 3.9 you can use this:
def classproperty(fget):
return type(
'classproperty',
(),
{'__get__': lambda self, _, cls: fget(cls), '__module__': None}
)()
class Item:
a = 47
#classproperty
def x(cls):
return cls.a
Item.x
After searching different places, I found a method to define a classproperty
valid with Python 2 and 3.
from future.utils import with_metaclass
class BuilderMetaClass(type):
#property
def load_namespaces(self):
return (self.__sourcepath__)
class BuilderMixin(with_metaclass(BuilderMetaClass, object)):
__sourcepath__ = 'sp'
print(BuilderMixin.load_namespaces)
Hope this can help somebody :)
A code completion friendly solution for Python < 3.9
from typing import (
Callable,
Generic,
TypeVar,
)
T = TypeVar('T')
class classproperty(Generic[T]):
"""Converts a method to a class property.
"""
def __init__(self, f: Callable[..., T]):
self.fget = f
def __get__(self, instance, owner) -> T:
return self.fget(owner)
Here is my solution that also caches the class property
class class_property(object):
# this caches the result of the function call for fn with cls input
# use this as a decorator on function methods that you want converted
# into cached properties
def __init__(self, fn):
self._fn_name = fn.__name__
if not isinstance(fn, (classmethod, staticmethod)):
fn = classmethod(fn)
self._fn = fn
def __get__(self, obj, cls=None):
if cls is None:
cls = type(obj)
if (
self._fn_name in vars(cls) and
type(vars(cls)[self._fn_name]).__name__ != "class_property"
):
return vars(cls)[self._fn_name]
else:
value = self._fn.__get__(obj, cls)()
setattr(cls, self._fn_name, value)
return value
Here's my suggestion. Don't use class methods.
Seriously.
What's the reason for using class methods in this case? Why not have an ordinary object of an ordinary class?
If you simply want to change the value, a property isn't really very helpful is it? Just set the attribute value and be done with it.
A property should only be used if there's something to conceal -- something that might change in a future implementation.
Maybe your example is way stripped down, and there is some hellish calculation you've left off. But it doesn't look like the property adds significant value.
The Java-influenced "privacy" techniques (in Python, attribute names that begin with _) aren't really very helpful. Private from whom? The point of private is a little nebulous when you have the source (as you do in Python.)
The Java-influenced EJB-style getters and setters (often done as properties in Python) are there to facilitate Java's primitive introspection as well as to pass muster with the static language compiler. All those getters and setters aren't as helpful in Python.

Categories

Resources