I have a Python program with multiple threads that manipulate a shared resource. I model this resource with an object and actions on this resource with methods in that object. A number of actions need to be performed as atomic actions, in a critical section. Since these functions follow a common pattern, I used a decorator:
def critical_section(inner):
def outer(self, *args, **kwargs):
self.enter()
inner(self, *args, **kwargs)
self.leave()
return outer
The real code is more complex than this, e.g. it handles exceptions, but this should be enough to get the gist of what I'm doing. Sample usage (I simplified all the thread-related stuff and just check for resource ownership):
class Base:
def __init__(self):
self.owner = None
def enter(self):
assert(self.owner == None)
self.owner = True
def leave(self):
assert(self.owner == True)
self.owner = None
#critical_section
def foo(self):
print('foo') # Dangerous stuff, must run atomically and not fail
Functions with the critical_section decorator have two properties:
The function's code is executed with the resource's lock held (enter and leave take care of that).
The function assumes that the resource is in a stable state on entry, and leaves it in a stable state on exit.
The second property implies that the lock is not a reentrant one. It is invalid to call a “critical section” function while already in a critical section, because the necessary invariants are not guaranteed.
The implementation of enter and leave check these properties: a thread can't call enter if the resource is currently owned (even by that thread itself), and enter gives ownership of the resource to the calling thread; leave symmetrically requires ownership and gives it up.
That architecture served me nicely until I wanted to have multiple resources with a similar structure, so I started to use inheritance.
class Derived(Base):
#critical_section
def foo(self):
print('fie') # Additional stuff to run first
Base.foo(self)
And now we come to the problem: that decorator is fundamentally inheritance-unfriendly.
base = Base()
base.foo()
derived = Derived()
derived.foo()
The call to derived.foo() fails because when Base.foo() is executed, the resource is already owned. At the point where the base class's method is called, the derived class has potentially already mucked up the object, violating the assumption that Base.foo starts with an object in known stable state.
The obvious solution would be to turn every critical-section function into a pair of functions: the outer function (meant to be called from outside, and never overridden in classes that modify the behavior of the resource), and the inner function (meant to be called only from derived methods).
Can Python decorators help me define such pairs of functions with a minimum of boilerplate?
Is there a better architecture that would maximize the clarity of where critical sections are entered and exited from, and minimize the risk of misuse? (Dynamic checks are good, but obvious correctness of source code is better.)
Edit: Here's a proper non-reentrant version.
You can have decorators take in arguments. so #dec(x); def f() ... will be called like dec(x)(f)(args). So, we have critical_section take in a string (#critical_section("Base")) and have one lock per string.
def critical_section(ident):
def _critical_section(inner):
def outer(self, *args, **kwargs):
self.enter(ident)
inner(self, *args, **kwargs)
self.leave(ident)
return outer
return _critical_section
class Base:
def __init__(self):
self.owner = {}
def enter(self, ident):
assert(ident not in self.owner)
self.owner[ident] = True
def leave(self, ident):
assert(ident in self.owner)
del self.owner[ident]
#critical_section("Base")
def foo(self):
print('foo') # Dangerous stuff, must run atomically and not fail
class Derived(Base):
#critical_section("Derived")
def foo(self):
print('fie') # Additional stuff to run first
Base.foo(self)
Hello, me from yesterday. Your fundamental problem is that you oversimplified the situation. You conflated two things: entering/leaving the critical section, and assuming/asserting the resource's invariants. But the resource's invariants can also be true in the middle of the critical section, and that's precisely what you're trying to convey when you say that Derived.foo method is allowed to call Base.foo (at some particular point during this execution).
You can model this in Python, but it does get a bit cumbersome.
def critical_section(inner):
def outer(self, *args, **kwargs):
self.enter()
inner(self, *args, **kwargs)
self.leave()
return outer
class Base:
def __init__(self):
self.owner = None
self.invariants = True
def enter(self):
assert(self.invariants)
self.invariants = False
assert(self.owner == None)
self.owner = True
def leave(self):
assert(self.owner == True)
self.owner = None
self.invariants = True
#critical_section
def foo(self):
print('foo') # Dangerous stuff, must run atomically and not fail
class Derived(Base):
#critical_section
def foo(self):
print('fie') # Additional stuff to run first
self.invariants = True
Base.foo(self)
The owner stuff should be a reentrant lock in the real world. Instead of having non-reentrancy to prevent modifying the resource while it's in an unstable state, there's this invariants check.
But all this is complicated and not really worth it when “checking invariants” amounts to “I state that the useful invariants are verified at this point in the code”. With invariants checked by static analysis, it would be another story, but then you wouldn't be using Python.
Coming back to your question, which was accessing the inner function when needed, Python does make it pretty easy. Store the inner function in an attribute of the decorated function.
def critical_section(inner):
def outer(self, *args, **kwargs):
self.enter()
inner(self, *args, **kwargs)
self.leave()
outer.inner_function = inner
return outer
…
class Derived(Base):
#critical_section
def foo(self):
print('fie') # Additional stuff to run first
Base.inner.foo(self)
Related
The case is such that I have an abstract class and a few child classes implementing it.
class Parent(metaclass=ABCMeta):
#abstract_method
def first_method(self, *args, **kwargs):
raise NotImplementedError()
#abstract_method
def second_method(self, *args, **kwargs):
raise NotImplementedError()
class Child(Parent):
def first_method(self, *args, **kwargs):
print('First method of the child class called!')
def second_method(self, *args, **kwargs):
print('Second method of the child class called!')
My goal is to make some kind of decorator, which will be used on methods of any child of the Parent class. I need this because every method make some kind of preparation before actually doing something, and this preparation is absolutely the same in all methods of all childs of the Parent class. Like:
class Child(Parent):
def first_method(self, *args, **kwargs):
print('Preparation!')
print('First method of the child class called!')
def second_method(self, *args, **kwargs):
print('Preparation!')
print('Second method of the child class called!')
The first thing came to my mind is to use Parent class method implementation: just remove "raise NotImplementedError()" and put some functionality, and then in child classes I would call, for example, super().first_method(self, *args, **kwargs) in the beginning of each method. It is good, but I also would want to return some data from the Parent method, and it would look weird when parent method and child method return something different in declaration. Not to mention that I would probably want to do some post-processing work after the method, so then I would need 2 different functions: for the beginning and after the performing the script.
The next thing I came up with is making MetaClass.
Just implement all the decoration of methods in the new MetaClass during creating a class, and pass the newly generated data which is used in child methods to them in kwargs.
This is the closest solution to my goal, but it feels wrong anyway. Because it is not explicit that some kwargs will be passed to child methods, and if you are new to this code, then you need to do some researches to understand how it works. I feel like I overengineering or so.
So the question: is there any pattern or something along these lines to implement this functionality?
Probably you can advise something better for my case?
Thank you a lot in advance!
So, existing patterns apart: I won't know if this has an specific name, what you need, that would be a "pattern" is the use of "slots": that is - you document special named methods that will be called as part of the execution of another method. This other method then performs its setup code, checks if the slotted method (usually identifiable by name) exists, call them, with a plain simple method call, which will run the most specialized version of it, even if the special method that calls the slots is in the base class, and you are on a big class-inheritance hierarchy.
One plain example of this pattern is the way Python instantiates objects: what one actually invokes calling the class with the same syntax that is used for function calls (MyClass()) is that class's class (its metaclass) __call__ method. (Usally type.__call__). In Python's code for type.__call__ the class' __new__ method is called, then the class' __init__ method is called and finally the value returned by the first call, to __new__ is returned. A custom metaclass can modify __call__ to run whatever code it wants before, between, or after these two calls.
So, if this was not Python, all you'd need is to spec down this, and document that these methods should not be called directly, but rather through an "entry point" method - which could simply feature an "ep_" prefix. These would have to be fixed and hardcoded on a baseclass, and you'd need one for each of the methods you want to prefix/postfix code to.
class Base(ABC):
def ep_first_method(self, *args, **kw);
# prefix code...
ret_val = self.first_method(*args, **kw)
# postfix code...
return ret_val
#abstractmethod
def first_method(self):
pass
class Child(Base):
def first_method(self, ...):
...
This being Python, it is easier to add some more magic to avoid code repetition and keep things concise.
One possible thing is to have a special class that, when detecting a method in a child class that should be called as a slot of a wrapper method, like above, to automatically rename that method: this way the entry point methods can feature the same name as the child methods - and better yet, a simple decorator can mark the methods that are meant to be "entrypoints", and inheritance would even work for them.
Basically, when building a new class we check all methods: if any of them has a correspondent part in the calling hierarchy which is marked as an entrypoint, the renaming takes place.
It is more practical if any entrypoint method will take as second parameter (the first being self), a reference for the slotted method to be called.
After some fiddling: the good news is that a custommetaclass is not needed - the __init_subclass__ special method in a baseclass is enough to enable the decorator.
The bad news: due to re-entry iterations in the entry-point triggered by potential calls to "super()" on the final methods, a somewhat intricate heuristic to call the original method in the intermediate classes is needed. I also took care to put some multi-threading protections - although this is not 100% bullet-proof.
import sys
import threading
from functools import wraps
def entrypoint(func):
name = func.__name__
slotted_name = f"_slotted_{name}"
recursion_control = threading.local()
recursion_control.depth = 0
lock = threading.Lock()
#wraps(func)
def wrapper(self, *args, **kw):
slotted_method = getattr(self, slotted_name, None)
if slotted_method is None:
# this check in place of abstractmethod errors. It is only raised when the method is called, though
raise TypeError("Child class {type(self).__name__} did not implement mandatory method {func.__name__}")
# recursion control logic: also handle when the slotted method calls "super",
# not just straightforward recursion
with lock:
recursion_control.depth += 1
if recursion_control.depth == 1:
normal_course = True
else:
normal_course = False
try:
if normal_course:
# runs through entrypoint
result = func(self, slotted_method, *args, **kw)
else:
# we are within a "super()" call - the only way to get the renamed method
# in the correct subclass is to recreate the callee's super, by fetching its
# implicit "__class__" variable.
try:
callee_super = super(sys._getframe(1).f_locals["__class__"], self)
except KeyError:
# callee did not make a "super" call, rather it likely is a recursive function "for real"
callee_super = type(self)
slotted_method = getattr(callee_super, slotted_name)
result = slotted_method(*args, **kw)
finally:
recursion_control.depth -= 1
return result
wrapper.__entrypoint__ = True
return wrapper
class SlottedBase:
def __init_subclass__(cls, *args, **kw):
super().__init_subclass__(*args, **kw)
for name, child_method in tuple(cls.__dict__.items()):
#breakpoint()
if not callable(child_method) or getattr(child_method, "__entrypoint__", None):
continue
for ancestor_cls in cls.__mro__[1:]:
parent_method = getattr(ancestor_cls, name, None)
if parent_method is None:
break
if not getattr(parent_method, "__entrypoint__", False):
continue
# if the code reaches here, this is a method that
# at some point up has been marked as having an entrypoint method: we rename it.
delattr (cls, name)
setattr(cls, f"_slotted_{name}", child_method)
break
# the chaeegs above are inplace, no need to return anything
class Parent(SlottedBase):
#entrypoint
def meth1(self, slotted, a, b):
print(f"at meth 1 entry, with {a=} and {b=}")
result = slotted(a, b)
print("exiting meth1\n")
return result
class Child(Parent):
def meth1(self, a, b):
print(f"at meth 1 on Child, with {a=} and {b=}")
class GrandChild(Child):
def meth1(self, a, b):
print(f"at meth 1 on grandchild, with {a=} and {b=}")
super().meth1(a,b)
class GrandGrandChild(GrandChild):
def meth1(self, a, b):
print(f"at meth 1 on grandgrandchild, with {a=} and {b=}")
super().meth1(a,b)
c = Child()
c.meth1(2, 3)
d = GrandChild()
d.meth1(2, 3)
e = GrandGrandChild()
e.meth1(2, 3)
Like the question posted here, I want to create a class that inherits from another class passed as an argument.
class A():
def __init__(self, args):
stuff
class B():
def __init__(self, args):
stuff
class C():
def __init__(self, cls, args):
self.inherit(cls, args)
args = #arguments to create instances of A and B
class_from_A = C(A, args) #instance of C inherited from A
class_from_B = C(B, args) #instance of C inherited from B
I want to do this so that I can keep track of calls I make to different web api's. The thought is that I am just adding my own functionality to any api-type object. The problem with the solution to the linked question is that I don't want to have to go through the additional "layer" to use the api-type object. I want to say obj.get_data() instead of obj.api.get_data().
I've tried looking into how super() works but haven't came across anything that would help (although I could've easily missed something). Any help would be nice, and I'm open to any other approaches for what I'm trying to do, however, just out of curiosity I'd like to know if this is possible.
I don't think it's possible because __init__ is called after __new__ which is where you would specify base classes, but I think you can achieve your goal of tracking api calls using a metaclass. Since you didn't give any examples of what tracking the calls means, I'll leave you with an example metaclass which counts method calls. You can adapt it to your needs.
Another alternative would be to subclass A and B with methods that track whatever you need, and just return super().whatever(). I think I'd prefer that method unless A and B contain too many methods worth managing like that.
Here's an implementation from python-course.eu, by Bernd Klein. Click the link for more detail.
class FuncCallCounter(type):
""" A Metaclass which decorates all the methods of the
subclass using call_counter as the decorator
"""
#staticmethod
def call_counter(func):
""" Decorator for counting the number of function
or method calls to the function or method func
"""
def helper(*args, **kwargs):
helper.calls += 1
return func(*args, **kwargs)
helper.calls = 0
helper.__name__= func.__name__
return helper
def __new__(cls, clsname, superclasses, attributedict):
""" Every method gets decorated with the decorator call_counter,
which will do the actual call counting
"""
for attr in attributedict:
if callable(attributedict[attr]) and not attr.startswith("__"):
attributedict[attr] = cls.call_counter(attributedict[attr])
return type.__new__(cls, clsname, superclasses, attributedict)
I have a base class whose method uses a with statement. In a child class, I override the same method, and would like to then access that same with statement (instead of having two with statements).
What are the standard ways of solving this problem?
For an example and possible solution, please see below.
Sample using threading.Lock
from threading import Lock
class BaseClass:
def __init__(self):
self.lock = Lock()
self._data = 0
def do_something_locked(self) -> None:
with self.lock:
self._data += 5
class ChildClass(BaseClass):
def do_something_locked(self) -> None:
super().do_something_locked()
# Obviously the parent class's self.lock's __exit__ method has
# already been called. What are accepted methods to add more
# functionality inside parent class's "with" statement?
with self.lock:
self._data += 1
Possible Solution
My first inclination is to define a private method in the BaseClass like so:
def do_something_locked(self) -> None:
with self.lock:
self._do_something()
def _do_something(self) -> None:
self._data += 5
And then the ChildClass can just override _do_something. This will work fine.
I am wondering, are there any other common patterns of solving this problem?
My first inclination is to define a private method in the BaseClass like so... And then the ChildClass can just override _do_something. This will work fine.
This is a good approach to the problem, even when you don't have a special requirement (like needing to remain within a with block context). I would not use a leading underscore for the "hook" method name, because anything that you are expecting to be overridden in derived classes, is logically part of the class interface. Also, if the self._data += 5 part always needs to happen, then leave it in do_something_locked.
are there any other common patterns of solving this problem?
Specific to the problem, you could use a re-entrant lock as shown in the other answer. You could also ignore the fact that the classes are related, and use dependency injection - make a general method in the base class that accepts a callable and executes it, using the lock:
# in base class
def do_locked(self, what, *args, **kwargs):
with self.lock:
what(*args, **kwargs)
# in derived class
def _implementation(self):
pass
def do_interesting_thing(self):
# pass in our own bound method, which takes no arguments
self._do_locked(self._implementation)
This way allows for client code to make use of the lock in custom ways. It's probably not a great idea if you don't need or want that functionality.
Use a re-entrant Lock. This will automatically "connect" the nested with statements, releasing the lock only after the outer-most with.
from threading import RLock
class BaseClass:
def __init__(self):
self.lock = RLock()
self._data = 0
def do_something_locked(self) -> None:
with self.lock:
self._data += 5
class ChildClass(BaseClass):
def do_something_locked(self) -> None:
with self.lock:
super().do_something_locked()
self._data += 1
In general, the pattern of reentrant context managers exists explicitly to allow possibly-nested contexts.
These context managers can not only be used in multiple with statements, but may also be used inside a with statement that is already using the same context manager.
In a Python class, I would like to automatically assign member variables to be the same as the __init__ function arguments, like this:
class Foo(object):
def __init__(self, arg1, arg2 = 1):
self.arg1 = arg1
self.arg2 = arg2
I would like to explicitly have argument names in the init
function for the sake of code clarity.
I don't want to use decorators for the same reason.
Is it possible to achieve this using a custom metaclass?
First, a disclaimer. Python object creation and initialization can be complicated and highly dynamic. This means that it can be difficult to come up with a solution that works for the corner cases. Moreover, the solutions tend to use some darker magic, and so when they inevitably do go wrong they can be hard to debug.
Second, the fact that your class has so many initialization parameters might be a hint that it has too many parameters. Some of them are probably related and can fit together in a smaller class. For example, if I'm building a car, it's better to have:
class Car:
def __init__(self, tires, engine):
self.tires = tires
self.engine = engine
class Tire:
def __init__(self, radius, winter=False):
self.radius = radius
self.winter = winter
class Engine:
def __init__(self, big=True, loud=True):
self.big = big
self.loud = loud
as opposed to
class Car:
def __init__(self, tire_radius, winter_tires=False,
engine_big=True, engine_loud=True):
self.tire_radius = tire_radius
self.winter_tires winter_tires
self.engine_big = engine_big
self.engine_loud = engine_loud
All of that said, here is a solution. I haven't used this in my own code, so it isn't "battle-tested". But it at least appears to work in the simple case. Use at your own risk.
First, metaclasses aren't necessary here, and we can use a simple decorator on the __init__ method. I think this is more readable, anyways, since it is clear that we are only modifying the behavior of __init__, and not something deeper about class creation.
import inspect
import functools
def unpack(__init__):
sig = inspect.signature(__init__)
#functools.wraps(__init__)
def __init_wrapped__(self, *args, **kwargs):
bound = sig.bind(self, *args, **kwargs)
bound.apply_defaults()
# first entry is the instance, should not be set
# discard it, taking only the rest
attrs = list(bound.arguments.items())[1:]
for attr, value in attrs:
setattr(self, attr, value)
return __init__(self, *args, **kwargs)
return __init_wrapped__
This decorator uses the inspect module to retrieve the signature of the __init__ method. Then we simply loop through the attributes and use setattr to assign them.
In use, it looks like:
class Foo(object):
#unpack
def __init__(self, a, b=88):
print('This still runs!')
so that
>>> foo = Foo(42)
This still runs!
>>> foo.a
42
>>> foo.b
88
I am not certain that every introspection tool will see the right signature of the decorated __init__. In particular, I'm not sure if Sphinx will do the "right thing". But at least the inspect module's signature function will return the signature of the wrapped function, as can be tested.
If you really want a metaclass solution, it's simple enough (but less readable and more magic, IMO). You need only write a class factory that applies the unpack decorator:
def unpackmeta(clsname, bases, dct):
dct['__init__'] = unpack(dct['__init__'])
return type(clsname, bases, dct)
class Foo(metaclass=unpackmeta):
def __init__(self, a, b=88):
print('This still runs!')
The output will be the same as the above example.
My issue is that I am using a metaclass to wrap certain class methods in a timer for logging purposes.
For example:
class MyMeta(type):
#staticmethod
def time_method(method):
def __wrapper(self, *args, **kwargs):
start = time.time()
result = method(self, *args, **kwargs)
finish = time.time()
sys.stdout.write('instancemethod %s took %0.3f s.\n' %(
method.__name__, (finish - start)))
return result
return __wrapper
def __new__(cls, name, bases, attrs):
for attr in ['__init__', 'run']:
if not attr in attrs:
continue
attrs[attr] = cls.time_method(attrs[attr])
return super(MetaBuilderModule, cls).__new__(cls, name, bases, attrs)
The problem I'm having is that my wrapper runs for every '__init__' even though I really only want it for the current module I am instantiating. The same goes for any method want to time. I dont want the timing to run on any inherited methods UNLESS they aren't being overridden.
class MyClass0(object):
__metaclass__ = MyMeta
def __init__(self):
pass
def run(self):
sys.stdout.write('running')
return True
class MyClass1(MyClass0):
def __init__(self): # I want this timed
MyClass0.__init__(self) # But not this.
pass
''' I need the inherited 'run' to be timed. '''
I've tried a few things but so far I've had no success.
Guard the timing code with an attribute. That way, only the outermost decorated method on an object will actually get timed.
#staticmethod
def time_method(method):
def __wrapper(self, *args, **kwargs):
if hasattr(self, '_being_timed'):
# We're being timed already; just run the method
return method(self, *args, **kwargs)
else:
# Not timed yet; run the timing code
self._being_timed = True # remember we're being timed
try:
start = time.time()
result = method(self, *args, **kwargs)
finish = time.time()
sys.stdout.write('instancemethod %s took %0.3f s.\n' %(
method.__name__, (finish - start)))
return result
finally:
# Done timing, reset to original state
del self._being_timed
return __wrapper
Timing only the outermost method is slightly different than “not timing inherited methods unless they aren't being overridden”, but I believe it solves your problem.
I'm not sure this has anything to do with multiple inheritance.
The trouble is that any subclass of MyClass0 has to be an instance of the same metaclass, which means MyClass1 gets created with MyMeta.__new__, so its methods get processed and wrapped in the timing code.
Effectively, what you need is that MyClass0.__init__ somehow returns something different in the two following circumstances:
When called directly (instantiating MyClass0 directly, or when MyClass1 doesn't override it), it needs to return the timed method
When called within a subclass definition, it needs to return the original untimed method
This is impossible, since MyClass0.__init__ doesn't know why it's being called.
I see three options:
Make the metaclass more complex. It can check through the base classes to see if they're already instances of the metaclass; if so it can make a new copy of them that removes the timed wrapper from the methods that are present in the class being constructed. You don't want to mutate the base classes directly, as that will affect all uses of them (including when they're instantiated directly, or when they're subclassed by other classes that override different methods). A downside of this is it really screws up the instanceof relationships; unless you construct the slight variations on the base classes by creating new subclasses of them (ugh!) and caching all the variations so you never construct duplicates (ugh!), you completely void natural assumptions that two classes share a base class (they may only share a template from which two completely independent base classes were generated).
Make the timing code more complex. Have a start_timing and stop_timing method, and if start_timing is called when the method is already being timed you just increment a counter, and stop_timing just decrements a counter and only stops timing when the counter hits zero. Be careful of timed methods that call other timed methods; you'll need to have separate counters per method name.
Give up on metaclasses and just use a decorator on the methods you want timed explicitly, with some way of getting at the undecorated method so that overriding definitions can call it. This will involve a couple of lines of boiler plate per use; that will quite possibly add up to less lines of code than either of the other two options.