Develop Reference

Misspelled fields in a #dataclass Python class

How to make it raise an exception on setting a misspelled fields in a #dataclass-decorated Python class?
I want a practical way to do this. Do I need to write my own decorator instead?
#dataclass
class C(object):
x: int = 1
obj = C()
obj.y = 2 # should raise an exception

One straightforward way (which works with any class) is to define __slots__:
In [1]: from dataclasses import dataclass
In [2]: #dataclass
...: class Foo:
...: __slots__ = 'bar','baz'
...: bar: int
...: baz: int
...:
In [3]: foo = Foo(42, 88)
In [4]: foo.biz = 10
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-4-d52b60444257> in <module>()
----> 1 foo.biz = 10
AttributeError: 'Foo' object has no attribute 'biz'
The purpose of slots is to serve as a small optimization. It allows the instances of the class to use a symbol table instead of a dict as the namespace of the class. It increases the speed of attribute access slightly, and can significantly improve the per-instance memory usage (because the instance doesn't carry around a dict underneath the hood), however, it disallows dynamic attribute setting.
This is actually my favorite feature of __slots__.
Note, you must take care when using inheritance with slots, at least, if you want subclasses to retain the slots behavior.

One way to do so is to mark the dataclass as frozen:
>>> from dataclasses import dataclass
>>> #dataclass(frozen=True)
... class C:
... x: int = 1
...
>>> c = C()
>>> c.y = 1
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<string>", line 3, in __setattr__
dataclasses.FrozenInstanceError: cannot assign to field 'y'
But note that this makes all existing attributes, like x in this case, read-only as well.

You can provide your own __setattr__() method that only allows assignment to known fields.
Example:
#dataclass
class C:
x: int = 1
def __setattr__(self, k, v):
if k not in self.__annotations__:
raise AttributeError(f'{self.__class__.__name__} dataclass has no field {k}')
super().__setattr__(k, v)
In then fails/works like this:
[ins] In [3]: obj = C()
...: obj.y = 2
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-3-7a568eb098b1> in <module>
1 obj = C()
----> 2 obj.y = 2
<ipython-input-2-d30972f86fbb> in __setattr__(self, k, v)
5 def __setattr__(self, k, v):
6 if k not in self.__annotations__:
----> 7 raise AttributeError(f'{self.__class__.__name__} dataclass has no field {k}')
8 super().__setattr__(k, v)
9
AttributeError: C dataclass has no field y
[ins] In [4]: obj.x = 23
[ins] In [5]: obj.x
Out[5]: 23

Expose object properties dynamically

In [1]: class Foo():
...: pass
...:
In [2]: class Qux():
...: def __init__(self):
...: item = Foo()
...:
In [3]: a = Foo()
In [4]: setattr(a, 'superpower', 'strength')
In [5]: a.superpower
Out[5]: 'strength'
In [6]: b = Qux()
In [7]: b.item = a
In [8]: b.superpower
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-8-cf0e287006f1> in <module>()
----> 1 b.superpower
AttributeError: Qux instance has no attribute 'superpower'
What I would like is to define some way of calling any attribute on Qux and have it return getattr(Qux.item, <attributename>). In other words, to have b.superpower work without explicitly defining:
#property
def superpower(self):
return getattr(self.item, 'superpower')
I don't want to lose access to any properties defined on Qux itself as well, but rather to expose properties defined on Foo if they are not also on Qux.

Define a __getattr__:
class Qux(Foo):
def __init__(self):
self.item = Foo()
def __getattr__(self, attr):
return getattr(self.item, attr)
__getattr__ gets called whenever someone tries to look up an attribute of the object, but fails through normal means.
It has an evil twin called __getattribute__, which always gets called and must be used with extreme caution.

You do that by defining __getattr__, not with a property. For any attribute that cannot be found with the standard protocol, Python will call the __getattr__ method of a class.
Moreover, to store the item, you have to assign it to self.item, otherwise it is thrown at the end of Qux.__init__.
Finally, inheriting from Foo seems unecessary in that case.
class Foo:
def __init__(self, superpower):
self.superpower = superpower
class Qux:
def __init__(self, foo_item):
self.item = foo_item
def __getattr__(self, name):
return getattr(self.item, name)
Example
f = Foo('strenght')
q = Qux(f)
print(q.superpower) # 'strenght'
Inheritance
Although, it seems you half-tried to implement this with inheritance. If your intent was to extend Qux behaviour with Foo, then inheritance would be the way to go.
class Foo:
def __init__(self, superpower):
self.superpower = superpower
class Qux(Foo):
def __getattr__(self, name):
return getattr(self.item, name)
Example
q = Qux('strenght')
print(q.superpower) # 'strenght'

Is there any way to create a Python class method that does NOT pollute the attribute namespace of its instances?

I want to provide a method that can be used on a Python 2.7 class object, but does not pollute the attribute namespace of its instances. Is there any way to do this?
>>> class Foo(object):
... #classmethod
... def ugh(cls):
... return 33
...
>>> Foo.ugh()
33
>>> foo = Foo()
>>> foo.ugh()
33

You could subclass the classmethod descriptor:
class classonly(classmethod):
def __get__(self, obj, type):
if obj: raise AttributeError
return super(classonly, self).__get__(obj, type)
This is how it would behave:
class C(object):
#classonly
def foo(cls):
return 42
>>> C.foo()
42
>>> c=C()
>>> c.foo()
AttributeError
This desugars to the descriptor call (rather, it is invoked by the default implementation of __getattribute__):
>>> C.__dict__['foo'].__get__(None, C)
<bound method C.foo of <class '__main__.C'>>
>>> C.__dict__['foo'].__get__(c, type(c))
AttributeError
Required reading: Data Model — Implementing Descriptors and Descriptor HowTo Guide.

ugh is not in the namespace:
>>> foo.__dict__
{}
but the rules for attribute lookup fall back to the type of the instance for missing names. You can override Foo.__getattribute__ to prevent this.
class Foo(object):
#classmethod
def ugh(cls):
return 33
def __getattribute__(self, name):
if name == 'ugh':
raise AttributeError("Access to class method 'ugh' block from instance")
return super(Foo,self).__getattribute__(name)
This produces:
>>> foo = Foo()
>>> foo.ugh()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "tmp.py", line 8, in __getattribute__
raise AttributeError("Access to class method 'ugh' block from instance")
AttributeError: Access to class method 'ugh' block from instance
>>> Foo.ugh()
33
You must use __getattribute__, which is called unconditionally on any attribute access, rather than __getattr__, which is only called after the normal lookup (which includes checking the type's namespace) fails.

Python has quasi-private variables that use name-munging to reduce accidental access. Methods and object variables of the form __name are converted to _ClassName__name. Python automatically changes the name when compiling methods on the class but doesn't change the name for subclasses.
I can use the private method in a class
>>> class A(object):
... def __private(self):
... print('boo')
... def hello(self):
... self.__private()
...
>>>
>>> A().hello()
boo
But not outside the class
>>> A().__private()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'A' object has no attribute '__private'
>>>
Or in subclasses
>>> class B(A):
... def hello2(self):
... self.__private()
...
>>>
>>> B().hello()
boo
>>> B().hello2()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in hello2
AttributeError: 'B' object has no attribute '_B__private'

Yes, you can create the method in the metaclass.
class FooMeta(type):
# No #classmethod here
def ugh(cls):
return 33
class Foo(object):
__metaclass__ = FooMeta
Foo.ugh() # returns 33
Foo().ugh() # AttributeError
Note that metaclasses are a power feature, and their use is discouraged if unnecessary. In particular, multiple inheritance requires special care if the parent classes have different metaclasses.

Decide baseclass at runtime

I want to do something like this:
class A:
def methodA(self):
return 5
class B:
def methodB(self):
return 10
class X(...):
def __init__(self, baseclass):
if baseclass =='A' : derive X from A
elif baseclass == 'B' : derive X from B
else: raise Exception("Not supported baseclass %s!" % (baseclass))
def methodX(self):
return 42
X('A').methodA() # returns 5
X('A').methodX() # returns 42
X('A').methodB() # methodB not defined
X('B').methodB() # returns 10
X('B').methodX() # returns 42
X('A').methodA() # methodA not defined
How can I implement this?

If you want to add methodX to the existing classes, you could consider multiple inheritance:
class A:
def methodA(self):
return 5
class B:
def methodB(self):
return 10
class X():
#classmethod
def new(cls, baseclass):
if baseclass == A:
return AX()
elif baseclass == B:
return BX()
else: raise Exception("Not supported baseclass %s!" % str(baseclass))
def methodX(self):
return 42
class AX(A, X):
pass
class BX(B, X):
pass
You can add args and kwargs to X.new and pass them on to the specific constructors. Here are the outputs of your tests (I corrected the last on in your question):
>>> ax = X.new(A)
>>> ax.methodA() # returns 5
5
>>> ax.methodX() # returns 42
42
>>> ax.methodB() # methodB not defined
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: AX instance has no attribute 'methodB'
>>> bx = X.new(B)
>>> bx.methodB() # returns 10
10
>>> bx.new(B).methodX() # returns 42
42
>>> bx.new(B).methodA() # methodA not defined
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: BX instance has no attribute 'methodA'

You should define two classes, X and Y, and a factory-method to instantiate either X or Y, depending on a parameter.
In general, the behavior you try to implement is somewhat confusing. When you create an instance (that is what X(...) does) you should get an instance of X, and instances of a class should have same attributes. That is one of the main reasons why classes exist.
Example:
class A:
def methodA(self):
return 5
class B:
def methodB(self):
return 10
def x(class_name):
name2class = {"A":A, "B":B}
return name2class[class_name]()
for name in ["A","B","C"]:
instance = x(name)
print name, instance
will print
A <__main__.A instance at 0x022C8D50>
B <__main__.B instance at 0x022C8DF0>
Traceback (most recent call last):
File ".../14834949.py", line 21, in <module>
instance = x(name)
File ".../14834949.py", line 18, in x
return name2class[class_name]()
KeyError: 'C'

Python: changing methods and attributes at runtime

I wish to create a class in Python that I can add and remove attributes and methods. How can I acomplish that?
Oh, and please don't ask why.

This example shows the differences between adding a method to a class and to an instance.
>>> class Dog():
... def __init__(self, name):
... self.name = name
...
>>> skip = Dog('Skip')
>>> spot = Dog('Spot')
>>> def talk(self):
... print 'Hi, my name is ' + self.name
...
>>> Dog.talk = talk # add method to class
>>> skip.talk()
Hi, my name is Skip
>>> spot.talk()
Hi, my name is Spot
>>> del Dog.talk # remove method from class
>>> skip.talk() # won't work anymore
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: Dog instance has no attribute 'talk'
>>> import types
>>> f = types.MethodType(talk, skip, Dog)
>>> skip.talk = f # add method to specific instance
>>> skip.talk()
Hi, my name is Skip
>>> spot.talk() # won't work, since we only modified skip
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: Dog instance has no attribute 'talk'

I wish to create a class in Python that I can add and remove attributes and methods.
import types
class SpecialClass(object):
#classmethod
def removeVariable(cls, name):
return delattr(cls, name)
#classmethod
def addMethod(cls, func):
return setattr(cls, func.__name__, types.MethodType(func, cls))
def hello(self, n):
print n
instance = SpecialClass()
SpecialClass.addMethod(hello)
>>> SpecialClass.hello(5)
5
>>> instance.hello(6)
6
>>> SpecialClass.removeVariable("hello")
>>> instance.hello(7)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'SpecialClass' object has no attribute 'hello'
>>> SpecialClass.hello(8)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'SpecialClass' has no attribute 'hello'

A possibly interesting alternative to using types.MethodType in:
>>> f = types.MethodType(talk, puppy, Dog)
>>> puppy.talk = f # add method to specific instance
would be to exploit the fact that functions are descriptors:
>>> puppy.talk = talk.__get__(puppy, Dog)

I wish to create a class in Python that I can add and remove attributes and methods. How can I acomplish that?
You can add and remove attributes and methods to any class, and they'll be available to all instances of the class:
>>> def method1(self):
pass
>>> def method1(self):
print "method1"
>>> def method2(self):
print "method2"
>>> class C():
pass
>>> c = C()
>>> c.method()
Traceback (most recent call last):
File "<pyshell#62>", line 1, in <module>
c.method()
AttributeError: C instance has no attribute 'method'
>>> C.method = method1
>>> c.method()
method1
>>> C.method = method2
>>> c.method()
method2
>>> del C.method
>>> c.method()
Traceback (most recent call last):
File "<pyshell#68>", line 1, in <module>
c.method()
AttributeError: C instance has no attribute 'method'
>>> C.attribute = "foo"
>>> c.attribute
'foo'
>>> c.attribute = "bar"
>>> c.attribute
'bar'

you can just assign directly to the class (either by accessing the original class name or via __class__ ):
class a : pass
ob=a()
ob.__class__.blah=lambda self,k: (3, self,k)
ob.blah(5)
ob2=a()
ob2.blah(7)
will print
(3, <__main__.a instance at 0x7f18e3c345f0>, 5)
(3, <__main__.a instance at 0x7f18e3c344d0>, 7)

Simply:
f1 = lambda:0 #method for instances
f2 = lambda _:0 #method for class
class C: pass #class
c1,c2 = C(),C() #instances
print dir(c1),dir(c2)
#add to the Instances
c1.func = f1
c1.any = 1.23
print dir(c1),dir(c2)
print c1.func(),c1.any
del c1.func,c1.any
#add to the Class
C.func = f2
C.any = 1.23
print dir(c1),dir(c2)
print c1.func(),c1.any
print c2.func(),c2.any
which results in:
['__doc__', '__module__'] ['__doc__', '__module__']
['__doc__', '__module__', 'any', 'func'] ['__doc__', '__module__']
0 1.23
['__doc__', '__module__', 'any', 'func'] ['__doc__', '__module__', 'any', 'func']
0 1.23
0 1.23

another alternative, if you need to replace the class wholesale is to modify the class attribute:
>>> class A(object):
... def foo(self):
... print 'A'
...
>>> class B(object):
... def foo(self):
... print 'Bar'
...
>>> a = A()
>>> a.foo()
A
>>> a.__class__ = B
>>> a.foo()
Bar

Does the class itself necessarily need to be modified? Or is the goal simply to replace what object.method() does at a particular point during runtime?
I ask because I sidestep the problem of actually modifying the class to monkey patch specific method calls in my framework with getattribute and a Runtime Decorator on my Base inheritance object.
Methods retrieved by a Base object in getattribute are wrapped in a Runtime_Decorator that parses the method calls keyword arguments for decorators/monkey patches to apply.
This enables you to utilize the syntax object.method(monkey_patch="mypatch"), object.method(decorator="mydecorator"), and even object.method(decorators=my_decorator_list).
This works for any individual method call (I leave out magic methods), does so without actually modifying any class/instance attributes, can utilize arbitrary, even foreign methods to patch, and will work transparently on sublcasses that inherit from Base (provided they don't override getattribute of course).
import trace
def monkey_patched(self, *args, **kwargs):
print self, "Tried to call a method, but it was monkey patched instead"
return "and now for something completely different"
class Base(object):
def __init__(self):
super(Base, self).__init__()
def testmethod(self):
print "%s test method" % self
def __getattribute__(self, attribute):
value = super(Base, self).__getattribute__(attribute)
if "__" not in attribute and callable(value):
value = Runtime_Decorator(value)
return value
class Runtime_Decorator(object):
def __init__(self, function):
self.function = function
def __call__(self, *args, **kwargs):
if kwargs.has_key("monkey_patch"):
module_name, patch_name = self._resolve_string(kwargs.pop("monkey_patch"))
module = self._get_module(module_name)
monkey_patch = getattr(module, patch_name)
return monkey_patch(self.function.im_self, *args, **kwargs)
if kwargs.has_key('decorator'):
decorator_type = str(kwargs['decorator'])
module_name, decorator_name = self._resolve_string(decorator_type)
decorator = self._get_decorator(decorator_name, module_name)
wrapped_function = decorator(self.function)
del kwargs['decorator']
return wrapped_function(*args, **kwargs)
elif kwargs.has_key('decorators'):
decorators = []
for item in kwargs['decorators']:
module_name, decorator_name = self._resolve_string(item)
decorator = self._get_decorator(decorator_name, module_name)
decorators.append(decorator)
wrapped_function = self.function
for item in reversed(decorators):
wrapped_function = item(wrapped_function)
del kwargs['decorators']
return wrapped_function(*args, **kwargs)
else:
return self.function(*args, **kwargs)
def _resolve_string(self, string):
try: # attempt to split the string into a module and attribute
module_name, decorator_name = string.split(".")
except ValueError: # there was no ".", it's just a single attribute
module_name = "__main__"
decorator_name = string
finally:
return module_name, decorator_name
def _get_module(self, module_name):
try: # attempt to load the module if it exists already
module = modules[module_name]
except KeyError: # import it if it doesn't
module = __import__(module_name)
finally:
return module
def _get_decorator(self, decorator_name, module_name):
module = self._get_module(module_name)
try: # attempt to procure the decorator class
decorator_wrap = getattr(module, decorator_name)
except AttributeError: # decorator not found in module
print("failed to locate decorators %s for function %s." %\
(kwargs["decorator"], self.function))
else:
return decorator_wrap # instantiate the class with self.function
class Tracer(object):
def __init__(self, function):
self.function = function
def __call__(self, *args, **kwargs):
tracer = trace.Trace(trace=1)
tracer.runfunc(self.function, *args, **kwargs)
b = Base()
b.testmethod(monkey_patch="monkey_patched")
b.testmethod(decorator="Tracer")
#b.testmethod(monkey_patch="external_module.my_patch")
The downside to this approach is getattribute hooks all access to attributes, so the checking of and potential wrapping of methods occurs even for attributes that are not methods + won't be utilizing the feature for the particular call in question. And using getattribute at all is inherently somewhat complicated.
The actual impact of this overhead in my experience/for my purposes has been negligible, and my machine runs a dual core Celeron. The previous implementation I used introspected methods upon object init and bound the Runtime_Decorator to methods then. Doing things that way eliminated the need to utilize getattribute and reduced the overhead mentioned previously... however, it also breaks pickle (maybe not dill) and is less dynamic then this approach.
The only use cases I have actually come across "in the wild" with this technique were with timing and tracing decorators. However, the possibilities it opens up are extremely wide ranging.
If you have a preexisting class that cannot be made to inherit from a different base (or utilize the technique it's own class definition or in it's base class'), then the whole thing simply does not apply to your issue at all unfortunately.
I don't think setting/removing non-callable attributes on a class at runtime is necessarily so challenging? unless you want classes that inherit from the modified class to automatically reflect the changes in themselves as well... That'd be a whole 'nother can o' worms by the sound of it though.