Having an instance c of a class C,
I would like to make c immutable, but other instances of C dont have to.
Is there an easy way to achieve this in python?
You can't make Python classes fully immutable. You can however imitate it:
class C:
_immutable = False
def __setattr__(self, name, value):
if self._immutable:
raise TypeError(f"Can't set attribute, {self!r} is immutable.")
super().__setattr__(name, value)
Example:
>>> c = C()
>>> c.hello = 123
>>> c.hello
123
>>> c._immutable = True
>>> c.hello = 456
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in __setattr__
TypeError: Can't set attribute, <__main__.C object at 0x000002087C679D20> is immutable.
If you wish to set it at initialization, you can add an __init__ like so:
class C:
_immutable = False
def __init__(self, immutable=False):
self._immutable = immutable
def __setattr__(self, name, value):
if self._immutable:
raise TypeError(f"Can't set attribute, {self!r} is immutable.")
super().__setattr__(name, value)
Keep in mind you can still bypass it by accessing and modifying the __dict__ of the instance directly:
>>> c = C(immutable=True)
>>> c.__dict__["hello"] = 123
>>> c.hello
123
You may attempt to block it like so:
class C:
_immutable = False
def __init__(self, immutable=False):
self._immutable = immutable
def __getattribute__(self, name):
if name == "__dict__":
raise TypeError("Can't access class dict.")
return super().__getattribute__(name)
def __setattr__(self, name, value):
if self._immutable:
raise TypeError(f"Can't set attribute, {self!r} is immutable.")
super().__setattr__(name, value)
But even then it's possible to bypass:
>>> c = C(immutable=True)
>>> object.__getattribute__(c, "__dict__")["hello"] = 123
>>> c.hello
123
I extended dict in a simple way to directly access it's values with the d.key notation instead of d['key']:
class ddict(dict):
def __getattr__(self, item):
return self[item]
def __setattr__(self, key, value):
self[key] = value
Now when I try to pickle it, it will call __getattr__ to find __getstate__, which is neither present nor necessary. The same will happen upon unpickling with __setstate__:
>>> import pickle
>>> class ddict(dict):
... def __getattr__(self, item):
... return self[item]
... def __setattr__(self, key, value):
... self[key] = value
...
>>> pickle.dumps(ddict())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __getattr__
KeyError: '__getstate__'
How do I have to modify the class ddict in order to be properly pickable?
The problem is not pickle but that your __getattr__ method breaks the expected contract by raising KeyError exceptions. You need to fix your __getattr__ method to raise AttributeError exceptions instead:
def __getattr__(self, item):
try:
return self[item]
except KeyError:
raise AttributeError(item)
Now pickle is given the expected signal for a missing __getstate__ customisation hook.
From the object.__getattr__ documentation:
This method should return the (computed) attribute value or raise an AttributeError exception.
(bold emphasis mine).
If you insist on keeping the KeyError, then at the very least you need to skip names that start and end with double underscores and raise an AttributeError just for those:
def __getattr__(self, item):
if isinstance(item, str) and item[:2] == item[-2:] == '__':
# skip non-existing dunder method lookups
raise AttributeError(item)
return self[item]
Note that you probably want to give your ddict() subclass an empty __slots__ tuple; you don't need the extra __dict__ attribute mapping on your instances, since you are diverting attributes to key-value pairs instead. That saves you a nice chunk of memory per instance.
Demo:
>>> import pickle
>>> class ddict(dict):
... __slots__ = ()
... def __getattr__(self, item):
... try:
... return self[item]
... except KeyError:
... raise AttributeError(item)
... def __setattr__(self, key, value):
... self[key] = value
...
>>> pickle.dumps(ddict())
b'\x80\x03c__main__\nddict\nq\x00)\x81q\x01.'
>>> type(pickle.loads(pickle.dumps(ddict())))
<class '__main__.ddict'>
>>> d = ddict()
>>> d.foo = 'bar'
>>> d.foo
'bar'
>>> pickle.loads(pickle.dumps(d))
{'foo': 'bar'}
That pickle tests for the __getstate__ method on the instance rather than on the class as is the norm for special methods, is a discussion for another day.
First of all, I think you may need to distinguish between instance attribute and class attribute.
In Python official document Chapter 11.1.4 about pickling, it says:
instances of such classes whose dict or the result of calling getstate() is picklable (see section The pickle protocol for details).
Therefore, the error message you're getting is when you try to pickle an instance of the class, but not the class itself - in fact, your class definition will just pickle fine.
Now for pickling an object of your class, the problem is that you need to call the parent class's serialization implementation first to properly set things up. The correct code is:
In [1]: import pickle
In [2]: class ddict(dict):
...:
...: def __getattr__(self, item):
...: super.__getattr__(self, item)
...: return self[item]
...:
...: def __setattr__(self, key, value):
...: super.__setattr__(self, key, value)
...: self[key] = value
...:
In [3]: d = ddict()
In [4]: d.name = "Sam"
In [5]: d
Out[5]: {'name': 'Sam'}
In [6]: pickle.dumps(d)
Out[6]: b'\x80\x03c__main__\nddict\nq\x00)\x81q\x01X\x04\x00\x00\x00nameq\x02X\x03\x00\x00\x00Samq\x03s}q\x04h\x02h\x03sb.'
In this example, I would like to avoid the # Oops eventuality.
def foo():
return "foo"
class MyClass(object):
def __init__(self):
setattr(self, 'foo', foo)
def bar(self):
return "bar"
-
>>> x = MyClass()
>>> x.foo()
>>> x.foo = 2 # Oops
>>> x.foo()
TypeError: 'int' object is not callable
How can I prevent my methods to be overwritten by mistake?
Make x.foo a property, without specifying a setter. However it is quite tricky to do it dynamically:
def add_property(inst, name, method):
'''
Adds a property to a class instance.
Property must be added to the CLASS.
'''
cls = type(inst)
if not hasattr(cls, '__perinstance'):
cls = type(cls.__name__, (cls,), {})
cls.__perinstance = True
inst.__class__ = cls
setattr(cls, name, property(method))
And then instead of just doing setattr do it like this:
class MyClass(object):
def __init__(self):
add_property(self, 'foo', lambda _ : 2)
(for more realistic use, replace the lambda function with the
function or method returning the value for foo)
Output:
>>> o=MyClass()
>>> o.foo
2
>>> o.foo=3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: can't set attribute
>>>
You can check if passed to setattr attribute name is already exists in class (and instance if needed) __dict__ and do not rewrite it in that case:
class MyClass(object):
def __setattr__(self, name, value):
if name not in self.__class__.__dict__ and name not in self.__dict__:
super(MyClass, self).__setattr__(name, value)
test:
>>> x = MyClass()
>>> x.foo = foo # equal to your __init__ setattr call
>>> x.foo()
'foo'
>>> x.foo = 2
>>> x.foo()
'foo'
class A():
def __init__(self, data=''):
self.data = data
def __str__(self):
return str(self.data)
d = {}
elem = A()
d[elem] = 'abc'
elem2 = A()
print d[elem2] # KeyError
# actually elem2! was used not elem
how can I implement this without error?
I tried to get d[elem2] (not elem) with another instance of A() BUT with the same content.
The answer is yes, you need to redefine __hash__() and __eq__():
>>> class A(object):
... def __init__(self, data=''):
... self.data = data
... def __eq__(self, another):
... return hasattr(another, 'data') and self.data == another.data
... def __hash__(self):
... return hash(self.data)
...
>>> a1, a2, a3 = A('foo'), A('foo'), A('bar')
>>> d = {a1: 'foo'}
>>> d[a1]
'foo'
>>> d[a2]
'foo'
>>> d[a3]
Traceback (most recent call last):
File "", line 1, in
KeyError: __main__.A object at 0x927d0>
As explained in another comment default implementation of __hash__ is just simple identity, so if you want to make it more sophisticated, you need to define it explicitly.
What you did should work, as long as you don't override the __hash__() and __eq__() methods. It will use object identity as equality. If you want a different notion of equality, you can override the __hash__() and __eq__() methods of your class.
I wish to create a class in Python that I can add and remove attributes and methods. How can I acomplish that?
Oh, and please don't ask why.
This example shows the differences between adding a method to a class and to an instance.
>>> class Dog():
... def __init__(self, name):
... self.name = name
...
>>> skip = Dog('Skip')
>>> spot = Dog('Spot')
>>> def talk(self):
... print 'Hi, my name is ' + self.name
...
>>> Dog.talk = talk # add method to class
>>> skip.talk()
Hi, my name is Skip
>>> spot.talk()
Hi, my name is Spot
>>> del Dog.talk # remove method from class
>>> skip.talk() # won't work anymore
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: Dog instance has no attribute 'talk'
>>> import types
>>> f = types.MethodType(talk, skip, Dog)
>>> skip.talk = f # add method to specific instance
>>> skip.talk()
Hi, my name is Skip
>>> spot.talk() # won't work, since we only modified skip
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: Dog instance has no attribute 'talk'
I wish to create a class in Python that I can add and remove attributes and methods.
import types
class SpecialClass(object):
#classmethod
def removeVariable(cls, name):
return delattr(cls, name)
#classmethod
def addMethod(cls, func):
return setattr(cls, func.__name__, types.MethodType(func, cls))
def hello(self, n):
print n
instance = SpecialClass()
SpecialClass.addMethod(hello)
>>> SpecialClass.hello(5)
5
>>> instance.hello(6)
6
>>> SpecialClass.removeVariable("hello")
>>> instance.hello(7)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'SpecialClass' object has no attribute 'hello'
>>> SpecialClass.hello(8)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'SpecialClass' has no attribute 'hello'
A possibly interesting alternative to using types.MethodType in:
>>> f = types.MethodType(talk, puppy, Dog)
>>> puppy.talk = f # add method to specific instance
would be to exploit the fact that functions are descriptors:
>>> puppy.talk = talk.__get__(puppy, Dog)
I wish to create a class in Python that I can add and remove attributes and methods. How can I acomplish that?
You can add and remove attributes and methods to any class, and they'll be available to all instances of the class:
>>> def method1(self):
pass
>>> def method1(self):
print "method1"
>>> def method2(self):
print "method2"
>>> class C():
pass
>>> c = C()
>>> c.method()
Traceback (most recent call last):
File "<pyshell#62>", line 1, in <module>
c.method()
AttributeError: C instance has no attribute 'method'
>>> C.method = method1
>>> c.method()
method1
>>> C.method = method2
>>> c.method()
method2
>>> del C.method
>>> c.method()
Traceback (most recent call last):
File "<pyshell#68>", line 1, in <module>
c.method()
AttributeError: C instance has no attribute 'method'
>>> C.attribute = "foo"
>>> c.attribute
'foo'
>>> c.attribute = "bar"
>>> c.attribute
'bar'
you can just assign directly to the class (either by accessing the original class name or via __class__ ):
class a : pass
ob=a()
ob.__class__.blah=lambda self,k: (3, self,k)
ob.blah(5)
ob2=a()
ob2.blah(7)
will print
(3, <__main__.a instance at 0x7f18e3c345f0>, 5)
(3, <__main__.a instance at 0x7f18e3c344d0>, 7)
Simply:
f1 = lambda:0 #method for instances
f2 = lambda _:0 #method for class
class C: pass #class
c1,c2 = C(),C() #instances
print dir(c1),dir(c2)
#add to the Instances
c1.func = f1
c1.any = 1.23
print dir(c1),dir(c2)
print c1.func(),c1.any
del c1.func,c1.any
#add to the Class
C.func = f2
C.any = 1.23
print dir(c1),dir(c2)
print c1.func(),c1.any
print c2.func(),c2.any
which results in:
['__doc__', '__module__'] ['__doc__', '__module__']
['__doc__', '__module__', 'any', 'func'] ['__doc__', '__module__']
0 1.23
['__doc__', '__module__', 'any', 'func'] ['__doc__', '__module__', 'any', 'func']
0 1.23
0 1.23
another alternative, if you need to replace the class wholesale is to modify the class attribute:
>>> class A(object):
... def foo(self):
... print 'A'
...
>>> class B(object):
... def foo(self):
... print 'Bar'
...
>>> a = A()
>>> a.foo()
A
>>> a.__class__ = B
>>> a.foo()
Bar
Does the class itself necessarily need to be modified? Or is the goal simply to replace what object.method() does at a particular point during runtime?
I ask because I sidestep the problem of actually modifying the class to monkey patch specific method calls in my framework with getattribute and a Runtime Decorator on my Base inheritance object.
Methods retrieved by a Base object in getattribute are wrapped in a Runtime_Decorator that parses the method calls keyword arguments for decorators/monkey patches to apply.
This enables you to utilize the syntax object.method(monkey_patch="mypatch"), object.method(decorator="mydecorator"), and even object.method(decorators=my_decorator_list).
This works for any individual method call (I leave out magic methods), does so without actually modifying any class/instance attributes, can utilize arbitrary, even foreign methods to patch, and will work transparently on sublcasses that inherit from Base (provided they don't override getattribute of course).
import trace
def monkey_patched(self, *args, **kwargs):
print self, "Tried to call a method, but it was monkey patched instead"
return "and now for something completely different"
class Base(object):
def __init__(self):
super(Base, self).__init__()
def testmethod(self):
print "%s test method" % self
def __getattribute__(self, attribute):
value = super(Base, self).__getattribute__(attribute)
if "__" not in attribute and callable(value):
value = Runtime_Decorator(value)
return value
class Runtime_Decorator(object):
def __init__(self, function):
self.function = function
def __call__(self, *args, **kwargs):
if kwargs.has_key("monkey_patch"):
module_name, patch_name = self._resolve_string(kwargs.pop("monkey_patch"))
module = self._get_module(module_name)
monkey_patch = getattr(module, patch_name)
return monkey_patch(self.function.im_self, *args, **kwargs)
if kwargs.has_key('decorator'):
decorator_type = str(kwargs['decorator'])
module_name, decorator_name = self._resolve_string(decorator_type)
decorator = self._get_decorator(decorator_name, module_name)
wrapped_function = decorator(self.function)
del kwargs['decorator']
return wrapped_function(*args, **kwargs)
elif kwargs.has_key('decorators'):
decorators = []
for item in kwargs['decorators']:
module_name, decorator_name = self._resolve_string(item)
decorator = self._get_decorator(decorator_name, module_name)
decorators.append(decorator)
wrapped_function = self.function
for item in reversed(decorators):
wrapped_function = item(wrapped_function)
del kwargs['decorators']
return wrapped_function(*args, **kwargs)
else:
return self.function(*args, **kwargs)
def _resolve_string(self, string):
try: # attempt to split the string into a module and attribute
module_name, decorator_name = string.split(".")
except ValueError: # there was no ".", it's just a single attribute
module_name = "__main__"
decorator_name = string
finally:
return module_name, decorator_name
def _get_module(self, module_name):
try: # attempt to load the module if it exists already
module = modules[module_name]
except KeyError: # import it if it doesn't
module = __import__(module_name)
finally:
return module
def _get_decorator(self, decorator_name, module_name):
module = self._get_module(module_name)
try: # attempt to procure the decorator class
decorator_wrap = getattr(module, decorator_name)
except AttributeError: # decorator not found in module
print("failed to locate decorators %s for function %s." %\
(kwargs["decorator"], self.function))
else:
return decorator_wrap # instantiate the class with self.function
class Tracer(object):
def __init__(self, function):
self.function = function
def __call__(self, *args, **kwargs):
tracer = trace.Trace(trace=1)
tracer.runfunc(self.function, *args, **kwargs)
b = Base()
b.testmethod(monkey_patch="monkey_patched")
b.testmethod(decorator="Tracer")
#b.testmethod(monkey_patch="external_module.my_patch")
The downside to this approach is getattribute hooks all access to attributes, so the checking of and potential wrapping of methods occurs even for attributes that are not methods + won't be utilizing the feature for the particular call in question. And using getattribute at all is inherently somewhat complicated.
The actual impact of this overhead in my experience/for my purposes has been negligible, and my machine runs a dual core Celeron. The previous implementation I used introspected methods upon object init and bound the Runtime_Decorator to methods then. Doing things that way eliminated the need to utilize getattribute and reduced the overhead mentioned previously... however, it also breaks pickle (maybe not dill) and is less dynamic then this approach.
The only use cases I have actually come across "in the wild" with this technique were with timing and tracing decorators. However, the possibilities it opens up are extremely wide ranging.
If you have a preexisting class that cannot be made to inherit from a different base (or utilize the technique it's own class definition or in it's base class'), then the whole thing simply does not apply to your issue at all unfortunately.
I don't think setting/removing non-callable attributes on a class at runtime is necessarily so challenging? unless you want classes that inherit from the modified class to automatically reflect the changes in themselves as well... That'd be a whole 'nother can o' worms by the sound of it though.