Suppose I have a dataclass with a set method. How do I extend the repr method so that it also updates whenever the set method is called:
from dataclasses import dataclass
#dataclass
class State:
A: int = 1
B: int = 2
def set(self, var, val):
setattr(self, var, val)
Ex:
In [2]: x = State()
In [3]: x
Out[3]: State(A=1, B=2)
In [4]: x.set("C", 3)
In [5]: x
Out[5]: State(A=1, B=2)
In [6]: x.C
Out[6]: 3
The outcome I would like
In [7]: x
Out[7]: State(A=1, B=2, C=3)
The dataclass decorator lets you quickly and easily build classes that have specific fields that are predetermined when you define the class. The way you're intending to use your class, however, doesn't match up very well with what dataclasses are good for. You want to be able to dynamically add new fields after the class already exists, and have them work with various methods (like __init__, __repr__ and presumably __eq__). That removes almost all of the benefits of using dataclass. You should instead just write your own class that does what you want it to do.
Here's a quick and dirty version:
class State:
_defaults = {"A": 1, "B": 2}
def __init__(self, **kwargs):
self.__dict__.update(self._defaults)
self.__dict__.update(kwargs)
def __eq__(self, other):
return self.__dict__ == other.__dict__ # you might want to add some type checking here
def __repr__(self):
kws = [f"{key}={value!r}" for key, value in self.__dict__.items()]
return "{}({})".format(type(self).__name__, ", ".join(kws))
This is pretty similar to what you get from types.SimpleNamespace, so you might just be able to use that instead (it doesn't do default values though).
You could add your set method to this framework, though it seems to me like needless duplication of the builtin setattr function you're already using to implement it. If the caller needs to dynamically set an attribute, they can call setattr themselves. If the attribute name is constant, they can use normal attribute assignment syntax instead s.foo = "bar".
Related
I am trying to modify an already defined class by changing an attribute's value. Importantly, I want this change to propagate internally.
For example, consider this class:
class Base:
x = 1
y = 2 * x
# Other attributes and methods might follow
assert Base.x == 1
assert Base.y == 2
I would like to change x to 2, making it equivalent to this.
class Base:
x = 2
y = 2 * x
assert Base.x == 2
assert Base.y == 4
But I would like to make it in the following way:
Base = injector(Base, x=2)
Is there a way to achieve this WITHOUT recompile the original class source code?
The effect you want to achieve belongs to the realm of "reactive programing" - a programing paradigm (from were the now ubiquitous Javascript library got its name as an inspiration).
While Python has a lot of mechanisms to allow that, one needs to write his code to actually make use of these mechanisms.
By default, plain Python code as the one you put in your example, uses the Imperative paradigm, which is eager: whenever an expression is encoutered, it is executed, and the result of that expression is used (in this case, the result is stored in the class attribute).
Python's advantages also can make it so that once you write a codebase that will allow some reactive code to take place, users of your codebase don't have to be aware of that, and things work more or less "magically".
But, as stated above, that is not free. For the case of being able to redefine y when x changes in
class Base:
x = 1
y = 2 * x
There are a couple paths that can be followed - the most important is that, at the time the "*" operator is executed (and that happens when Python is parsing the class body), at least one side of the operation is not a plain number anymore, but a special object which implements a custom __mul__ method (or __rmul__) in this case. Then, instead of storing a resulting number in y, the expression is stored somewhere, and when y is retrieved either as a class attribute, other mechanisms force the expression to resolve.
If you want this at instance level, rather than at class level, it would be easier to implement. But keep in mind that you'd have to define each operator on your special "source" class for primitive values.
Also, both this and the easier, instance descriptor approach using property are "lazily evaluated": that means, the value for y is calcualted when it is to be used (it can be cached if it will be used more than once). If you want to evaluate it whenever x is assigned (and not when y is consumed), that will require other mechanisms. Although caching the lazy approach can mitigate the need for eager evaluation to the point it should not be needed.
1 - Before digging there
Python's easiest way to do code like this is simply to write the expressions to be calculated as functions - and use the property built-in as a descriptor to retrieve these values. The drawback is small:
you just have to wrap your expressions in a function (and then, that function
in something that will add the descriptor properties to it, such as property). The gain is huge: you are free to use any Python code inside your expression, including function calls, object instantiation, I/O, and the like. (Note that the other approach requires wiring up each desired operator, just to get started).
The plain "101" approach to have what you want working for instances of Base is:
class Base:
x = 1
#property
def y(self):
return self.x * 2
b = Base()
b.y
-> 2
Base.x = 3
b.y
-> 6
The work of property can be rewritten so that retrieving y from the class, instead of an instance, achieves the effect as well (this is still easier than the other approach).
If this will work for you somehow, I'd recommend doing it. If you need to cache y's value until x actually changes, that can be done with normal coding
2 - Exactly what you asked for, with a metaclass
as stated above, Python'd need to know about the special status of your y attribute when calculcating its expression 2 * x. At assignment time, it would be already too late.
Fortunately Python 3 allow class bodies to run in a custom namespace for the attribute assignment by implementing the __prepare__ method in a metaclass, and then recording all that takes place, and replacing primitive attributes of interest by special crafted objects implementing __mul__ and other special methods.
Going this way could even allow values to be eagerly calculated, so they can work as plain Python objects, but register information so that a special injector function could recreate the class redoing all the attributes that depend on expressions. It could also implement lazy evaluation, somewhat as described above.
from collections import UserDict
import operator
class Reactive:
def __init__(self, value):
self._initial_value = value
self.values = {}
def __set_name__(self, owner, name):
self.name = name
self.values[owner] = self._initial_value
def __get__(self, instance, owner):
return self.values[owner]
def __set__(self, instance, value):
raise AttributeError("value can't be set directly - call 'injector' to change this value")
def value(self, cls=None):
return self.values.get(cls, self._initial_value)
op1 = value
#property
def result(self):
return self.value
# dynamically populate magic methods for operation overloading:
for name in "mul add sub truediv pow contains".split():
op = getattr(operator, name)
locals()[f"__{name}__"] = (lambda operator: (lambda self, other: ReactiveExpr(self, other, operator)))(op)
locals()[f"__r{name}__"] = (lambda operator: (lambda self, other: ReactiveExpr(other, self, operator)))(op)
class ReactiveExpr(Reactive):
def __init__(self, value, op2, operator):
self.op2 = op2
self.operator = operator
super().__init__(value)
def result(self, cls):
op1, op2 = self.op1(cls), self.op2
if isinstance(op1, Reactive):
op1 = op1.result(cls)
if isinstance(op2, Reactive):
op2 = op2.result(cls)
return self.operator(op1, op2)
def __get__(self, instance, owner):
return self.result(owner)
class AuxDict(UserDict):
def __init__(self, *args, _parent, **kwargs):
self.parent = _parent
super().__init__(*args, **kwargs)
def __setitem__(self, item, value):
if isinstance(value, self.parent.reacttypes) and not item.startswith("_"):
value = Reactive(value)
super().__setitem__(item, value)
class MetaReact(type):
reacttypes = (int, float, str, bytes, list, tuple, dict)
def __prepare__(*args, **kwargs):
return AuxDict(_parent=__class__)
def __new__(mcls, name, bases, ns, **kwargs):
pre_registry = {}
cls = super().__new__(mcls, name, bases, ns.data, **kwargs)
#for name, obj in ns.items():
#if isinstance(obj, ReactiveExpr):
#pre_registry[name] = obj
#setattr(cls, name, obj.result()
for name, reactive in pre_registry.items():
_registry[cls, name] = reactive
return cls
def injector(cls, inplace=False, **kwargs):
original = cls
if not inplace:
cls = type(cls.__name__, (cls.__bases__), dict(cls.__dict__))
for name, attr in cls.__dict__.items():
if isinstance(attr, Reactive):
if isinstance(attr, ReactiveExpr) and name in kwargs:
raise AttributeError("Expression attributes can't be modified by injector")
attr.values[cls] = kwargs.get(name, attr.values[original])
return cls
class Base(metaclass=MetaReact):
x = 1
y = 2 * x
And, after pasting the snippet above in a REPL, here is the
result of using injector:
In [97]: Base2 = injector(Base, x=5)
In [98]: Base2.y
Out[98]: 10
The idea is complicated with that aspect that Base class is declared with dependent dynamically evaluated attributes. While we can inspect class's static attributes, I think there's no other way of getting dynamic expression except for parsing the class's sourcecode, find and replace the "injected" attribute name with its value and exec/eval the definition again. But that's the way you wanted to avoid. (moreover: if you expected injector to be unified for all classes).
If you want to proceed to rely on dynamically evaluated attributes define the dependent attribute as a lambda function.
class Base:
x = 1
y = lambda: 2 * Base.x
Base.x = 2
print(Base.y()) # 4
I have a problem with my base class. I started writing it after finding an answer on this site about more informative __repr__() methods. I added to it after finding a different answer on this site about using pprint() with my own classes. I tinkered with it a little more after finding a third answer on this site about making my classes unpackable with a ** operator.
I modified it again after seeing in yet another answer on this site that there was a distinction between merely giving it __getitem__(), __iter__(), and __len__() methods on the one hand, and actually making it a fully-qualified mapping by subclassing collections.abc.Mapping on the other. Further, I saw that doing so would remove the need for writing my own keys() method, as the Mapping would take care of that.
So I got rid of keys(), and a class method broke.
The problem
I have a method that iterates through my class' keys and values to produce one big string formatted as I'd like it. That class looks like this.
class MyObj():
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
def the_problem_method(self):
"""Method I'm getting divergent output for."""
longest = len(max((key for key in self.keys()), key=len))
key_width = longest + TAB_WIDTH - longest % TAB_WIDTH
return '\n'.join((f'{key:<{key_width}}{value}' for key, value in self))
Yes, that doesn't have the base class in it, but the MWE later on will account for that. The nut of it is that (key for key in self.keys()) part. When I have a keys() method written, I get the output I want.
def keys(self):
"""Get object attribute names."""
return self.__dict__.keys()
When I remove that to go with the keys() method supplied by collections.abc.Mapping, I get no space between key and value
The question
I can get the output I want by restoring the keys() method (and maybe adding values() and items() while I'm at it), but is that the best approach? Would it be better to go with the Mapping one and modify my class method to suit it? If so, how? Should I leave Mapping well enough alone until I know I need it?
This is my base class to be copied all aver creation and subclassed out the wazoo. I want to Get. It. Right.
There are already several considerations I can think of and many more of which I am wholly ignorant.
I use Python 3.9 and greater. I'll abandon 3.9 when conda does.
I want to keep my more-informative __repr__() methods.
I want pprint() to work, via the _dispatch table method with _format_dict_items().
I want to allow for duck typing my classes reliably.
I have not yet used type hinting, but I want to allow for using best practices there if I start.
Everything else I know nothing about.
The MWE
This has my problem class at the top and output stuff at the bottom. There are two series of classes building upon the previous ones.
The first are ever-more-inclusive base classes, and it is here that the difference between the instance with the keys() method and that without is shown. the class, BaseMap, subclasses the Mapping and has the __getitem__(), __iter__(), and __len__() methods. The next class up the chain, BaseMapKeys, subclasses that and adds the keys() method.
The second group, MapObj and MapKeysObj, are subclasses of the problem class that also subclass those different base classes respectively.
OK, maybe the WE isn't so M, but lots of things got me to this point and I don't want to neglect any.
import collections.abc
from pprint import pprint, PrettyPrinter
TAB_WIDTH = 3
class MyObj():
def __init__(self, foo, bar):
self.foo = foo
self.bar = bar
def the_problem_method(self):
"""Method I'm getting divergent output for."""
longest = len(max((key for key in self.keys()), key=len))
key_width = longest + TAB_WIDTH - longest % TAB_WIDTH
return '\n'.join((f'{key:<{key_width}}{value}' for key, value in self))
class Base(object):
"""Base class with more informative __repr__."""
def __repr__(self):
"""Object representation."""
params = (f'{key}={repr(value)}'
for key, value in self.__dict__.items())
return f'{repr(self.__class__)}({", ".join(params)})'
class BaseMap(Base, collections.abc.Mapping):
"""Enable class to be pprint-able, unpacked with **."""
def __getitem__(self, attr):
"""Get object attribute values."""
return getattr(self.__dict__, attr)
def __iter__(self):
"""Make object iterable."""
for attr in self.__dict__.keys():
yield attr, getattr(self, attr)
def __len__(self):
"""Get length of object."""
return len(self.__dict__)
class BaseMapKeys(BaseMap):
"""Overwrite KeysView output with what I thought it would be."""
def keys(self):
"""Get object attribute names."""
return self.__dict__.keys()
class MapObj(BaseMap, MyObj):
"""Problem class with collections.abc.Mapping."""
def __init__(self, foo, bar):
super().__init__(foo, bar)
class MapKeysObj(BaseMapKeys, MyObj):
"""Problem class with collections.abc.Mapping and keys method."""
def __init__(self, foo, bar):
super().__init__(foo, bar)
if isinstance(getattr(PrettyPrinter, '_dispatch'), dict):
# assume the dispatch table method still works
def pprint_basemap(printer, object, stream, indent, allowance, context,
level):
"""Implement pprint for subclasses of BaseMap class."""
write = stream.write
write(f'{object.__class__}(\n {indent * " "}')
printer._format_dict_items(object, stream, indent, allowance + 1,
context, level)
write(f'\n{indent * " "})')
map_classes = [MapObj, MapKeysObj]
for map_class in map_classes:
PrettyPrinter._dispatch[map_class.__repr__] = pprint_basemap
def print_stuff(map_obj):
print('pprint object:')
pprint(map_obj)
print()
print('print keys():')
print(map_obj.keys())
print()
print('print list(keys()):')
print(list(map_obj.keys()))
print()
print('print the problem method:')
print(map_obj.the_problem_method())
print('\n\n')
params = ['This is a really long line to force new line in pprint output', 2]
baz = MapObj(*params)
print_stuff(baz)
scoggs = MapKeysObj(*params)
print_stuff(scoggs)
I am currently working on a library.
I would like to write a wrapper that modifies a class (Mean in the example) in the inheritance tree by a new class (suppose WindowedMean) and I would like to initialize this class (for example k=10).
The Mean class can be anywhere in the heritage tree this is just one example.
This link shows the example
I know it's not advisable.
Do you have an elegant way to do this?
I imagine using the wrapper like this:
metric = Wrapper(MyClass, k=10)
update
Although the solution bellow will work exactly as you described, it came to my attention that with Multiple Inheritance, what you are asking can happen naturally.
Just inherit the class you want to modify, use the normal inheritance mechanisms to override the inherited class behavior. That means, setting a k=10 class attribute, and hardcoding super-calls to Mean's parent instead of using super, if needed.
Then, just create a new child of MyClass and add the overriden child of Mean to the inheritance Tree. Note that this sub-class of MyClass does not need a single statement in its body, and will behave exactly like MyClass, except for the modified Mean now being in the proper place in the mro.
Directly in the interpreter (I had to resort to exec to be able to type all the class hierarchy in a single line)
In [623]: exec("class GreatGrandParent1: pass\nclass GreatGrandParent2: pass\nclass GrandParent1(GreatGrandParent1, Gre
...: atGrandParent2): pass\nclass Mean: pass\nclass Parent1(GrandParent1, Mean): pass\nclass Parent2: pass\nclass
...: MyClass(Parent1, Parent2): pass")
In [624]: MyClass.__mro__
Out[624]:
(__main__.MyClass,
__main__.Parent1,
__main__.GrandParent1,
__main__.GreatGrandParent1,
__main__.GreatGrandParent2,
__main__.Mean,
__main__.Parent2,
object)
In [625]: class MeanMod(Mean):
...: k = 10
...:
In [626]: class MyClassMod(MyClass, MeanMod): pass
In [627]: MyClassMod.__mro__
Out[627]:
(__main__.MyClassMod,
__main__.MyClass,
__main__.Parent1,
__main__.GrandParent1,
__main__.GreatGrandParent1,
__main__.GreatGrandParent2,
__main__.MeanMod,
__main__.Mean,
__main__.Parent2,
object)
Note there is a case where that won't work straightforward: if a super call in Mean was supposed to call a method in Parent2 in your example. In this case, either resort to the original solution, or use some clever __mro__ manipulation to skip the method in Mean.
original answer
It looks like this would work with a class-decorator (which can also be used with this "wrapper" syntax you want).
there are sure cornercases, and things that might go wrong - but if your tree is somewhat well behaved, we have to recursively take all the bases on the class you want to wrap, and make the substitution in those bases - we can't simply take the final base, unroll all the ancestor classes in its __mro__ and just replace the desired class there: the heritage line would be broken bellow it.
That is, let's suppose you have classes A, B(A), C(B), D(C), and you want a clone D2 of D, replacing B by B2(A) - D.__bases__ is C. D.__mro__ is (D, C, B, A, object) If we try do create a new D2 forcing the __mro__ to be
(D, C, B2, A, object) , class C would break, as it would not see B anymore. (And the code in the previous version of this answer would leave both B and B2 in the inheritance line, leading to further brokerage).
The solution bellow recreates not only a new B class but also a new C.
Just beware that if B2 would not itself inherit from A, in this same example, A itself would be removed from the __mro__ for the new, replaced B2 does not need it. If D have any functionality that depends on A, it will break. This is not easily fixable, but for putting checking mechanisms to ensure the class being replaced contains the same ancestors the replaced class had, and raising an error otherwise - that would be easy to do. But figuring out how to pick and include the "now gone missing" ancestors is not easy, as it is impossible to know if they are needed at all, just to start.
As for the configuring part, without some example of how your RollingMean class is "configured" it is hard to give a proper concrete example - but let's make an subclass of it, updating its dict with the passed parameters - that should do for any configuration needed.
from types import new_class
def norm_name(config):
return "_" + "__".join(f"{key}_{value}" for key, value in config.items())
def ancestor_replace(oldclass: type, newclass: type, config: dict):
substitutions = {}
configured_newclass = type(newclass.__name__ + norm_name(config), (newclass,), config)
def replaced_factory(cls):
if cls in substitutions:
return substitutions[cls]
bases = cls.__bases__
new_bases = []
for base in bases:
if base is oldclass:
new_bases.append(configured_newclass)
else:
new_bases.append(replaced_factory(base))
if new_bases != bases:
new_cls = new_class(cls.__name__, tuple(new_bases), exec_body=lambda ns: ns.update(cls.__dict__.copy()))
substitutions[cls] = new_cls
return new_cls
return cls
return replaced_factory
# canbe used as:
#MyNewClass = ancestor_replace(Mean, RollingMean, {"ks": 10})(MyClass)
I take some care there to derive the proper metaclass - if no class in your inheritance tree is using a metaclass other than type, (usually ABCs or ORM models have differing metaclasses), you could use type instead of types.new_class in the call.
And finally, an example of using this in the interactive prompt:
In [152]: class A:
...: pass
...:
...: class B(A):
...: pass
...:
...: class B2(A):
...: pass
...:
...: class C(B):
...: pass
...:
...: class D(B):
...: pass
...:
...: class E(D):
...: pass
...:
In [153]: E2 = ancestor_replace(B, B2, {"k": 20})(E)
In [154]: E.__mro__
Out[154]: (__main__.E, __main__.D, __main__.B, __main__.A, object)
In [155]: E2.__mro__
Out[155]:
(__main__.E_modified,
__main__.D_modified,
__main__.B2_k_20,
__main__.B2,
__main__.A,
object)
In [156]: E2.k
Out[156]: 20
Maybe something like this?
class Parent1(object):
def __init__(self):
super(Parent1, self).__init__()
def speak(self):
print('Iam parent 1')
class Parent2(object):
def __init__(self):
super(Parent2, self).__init__()
def speak(self):
print('Iam parent 2')
class Child():
"""docstring for Child."""
def __init__(self):
pass
def child_method(self):
print('child method')
def make_child(inhert):
child = Child()
class ChildInhert(Child, inhert):
def __init__(self):
inhert.__init__(self)
return ChildInhert()
if __name__ == '__main__':
child = make_child(Parent1)
child.speak()
child.child_method()
In Python, how is it possible to reuse existing equal immutable objects (like is done for str)? Can this be done just by defining a __hash__ method, or does it require more complicated measures?
If you want to create via the class constructor and have it return a previously created object then you will need to provide a __new__ method (because by the time you get to __init__ the object has already been created).
Here is a simple example - if the value used to initialise has been seen before then a previously created object is returned rather than a new one created:
class Cached(object):
"""Simple example of immutable object reuse."""
def __init__(self, i):
self.i = i
def __new__(cls, i, _cache={}):
try:
return _cache[i]
except KeyError:
# you must call __new__ on the base class
x = super(Cached, cls).__new__(cls)
x.__init__(i)
_cache[i] = x
return x
Note that for this example you can use anything to initialise as long as it's hashable. And just to show that objects really are being reused:
>>> a = Cached(100)
>>> b = Cached(200)
>>> c = Cached(100)
>>> a is b
False
>>> a is c
True
There are two 'software engineering' solutions to this that don't require any low-level knowledge of Python. They apply in the following scenarios:
First Scenario: Objects of your class are 'equal' if they are constructed with the same constructor parameters, and equality won't change over time after construction. Solution: Use a factory that hashses the constructor parameters:
class MyClass:
def __init__(self, someint, someotherint):
self.a = someint
self.b = someotherint
cachedict = { }
def construct_myobject(someint, someotherint):
if (someint, someotherint) not in cachedict:
cachedict[(someint, someotherint)] = MyClass(someint, someotherint)
return cachedict[(someint, someotherint)]
This approach essentially limits the instances of your class to one unique object per distinct input pair. There are obvious drawbacks as well: not all types are easily hashable and so on.
Second Scenario: Objects of your class are mutable and their 'equality' may change over time. Solution: define a class-level registry of equal instances:
class MyClass:
registry = { }
def __init__(self, someint, someotherint, third):
MyClass.registry[id(self)] = (someint, someotherint)
self.someint = someint
self.someotherint = someotherint
self.third = third
def __eq__(self, other):
return MyClass.registry[id(self)] == MyClass.registry[id(other)]
def update(self, someint, someotherint):
MyClass.registry[id(self)] = (someint, someotherint)
In this example, objects with the same someint, someotherint pair are equal, while the third parameter does not factor in. The trick is to keep the parameters in registry in sync. As an alternative to update, you could override getattr and setattr for your class instead; this would ensure that any assignment foo.someint = y would be kept synced with your class-level dictionary. See an example here.
I believe you would have to keep a dict {args: object} of instances already created, then override the class' __new__ method to check in that dictionary, and return the relevant object if it already existed. Note that I haven't implemented or tested this idea. Of course, strings are handled at the C level.
I have a class that inherits from a dictionary in order to add some custom behavior - in this case it passes each key and value to a function for validation. In the example below, the 'validation' simply prints a message.
Assignment to the dictionary works as expected, printing messages whenever items are added to the dict. But when I try to use the custom dictionary type as the __dict__ attribute of a class, attribute assignments, which in turn puts keys/values into my custom dictionary class, somehow manages to insert values into the dictionary while completely bypassing __setitem__ (and the other methods I've defined that may add keys).
The custom dictionary:
from collections import MutableMapping
class ValidatedDict(dict):
"""A dictionary that passes each value it ends up storing through
a given validator function.
"""
def __init__(self, validator, *args, **kwargs):
self.__validator = validator
self.update(*args, **kwargs)
def __setitem__(self, key, value):
self.__validator(value)
self.__validator(key)
dict.__setitem__(self, key, value)
def copy(self): pass # snipped
def fromkeys(validator, seq, v = None): pass # snipped
setdefault = MutableMapping.setdefault
update = MutableMapping.update
def Validator(i): print "Validating:", i
Using it as the __dict__ attribute of a class yields behavior I don't understand.
>>> d = ValidatedDict(Validator)
>>> d["key"] = "value"
Validating: value
Validating: key
>>> class Foo(object): pass
...
>>> foo = Foo()
>>> foo.__dict__ = ValidatedDict(Validator)
>>> type(foo.__dict__)
<class '__main__.ValidatedDict'>
>>> foo.bar = 100 # Yields no message!
>>> foo.__dict__['odd'] = 99
Validating: 99
Validating: odd
>>> foo.__dict__
{'odd': 99, 'bar': 100}
Can someone explain why it doesn't behave the way I expect? Can it or can't it work the way I'm attempting?
This is an optimization. To support metamethods on __dict__, every single instance assignment would need to check the existance of the metamethod. This is a fundamental operation--every attribute lookup and assignment--so the extra couple branches needed to check this would become overhead for the whole language, for something that's more or less redundant with obj.__getattr__ and obj.__setattr__.