I have a base class whose method uses a with statement. In a child class, I override the same method, and would like to then access that same with statement (instead of having two with statements).
What are the standard ways of solving this problem?
For an example and possible solution, please see below.
Sample using threading.Lock
from threading import Lock
class BaseClass:
def __init__(self):
self.lock = Lock()
self._data = 0
def do_something_locked(self) -> None:
with self.lock:
self._data += 5
class ChildClass(BaseClass):
def do_something_locked(self) -> None:
super().do_something_locked()
# Obviously the parent class's self.lock's __exit__ method has
# already been called. What are accepted methods to add more
# functionality inside parent class's "with" statement?
with self.lock:
self._data += 1
Possible Solution
My first inclination is to define a private method in the BaseClass like so:
def do_something_locked(self) -> None:
with self.lock:
self._do_something()
def _do_something(self) -> None:
self._data += 5
And then the ChildClass can just override _do_something. This will work fine.
I am wondering, are there any other common patterns of solving this problem?
My first inclination is to define a private method in the BaseClass like so... And then the ChildClass can just override _do_something. This will work fine.
This is a good approach to the problem, even when you don't have a special requirement (like needing to remain within a with block context). I would not use a leading underscore for the "hook" method name, because anything that you are expecting to be overridden in derived classes, is logically part of the class interface. Also, if the self._data += 5 part always needs to happen, then leave it in do_something_locked.
are there any other common patterns of solving this problem?
Specific to the problem, you could use a re-entrant lock as shown in the other answer. You could also ignore the fact that the classes are related, and use dependency injection - make a general method in the base class that accepts a callable and executes it, using the lock:
# in base class
def do_locked(self, what, *args, **kwargs):
with self.lock:
what(*args, **kwargs)
# in derived class
def _implementation(self):
pass
def do_interesting_thing(self):
# pass in our own bound method, which takes no arguments
self._do_locked(self._implementation)
This way allows for client code to make use of the lock in custom ways. It's probably not a great idea if you don't need or want that functionality.
Use a re-entrant Lock. This will automatically "connect" the nested with statements, releasing the lock only after the outer-most with.
from threading import RLock
class BaseClass:
def __init__(self):
self.lock = RLock()
self._data = 0
def do_something_locked(self) -> None:
with self.lock:
self._data += 5
class ChildClass(BaseClass):
def do_something_locked(self) -> None:
with self.lock:
super().do_something_locked()
self._data += 1
In general, the pattern of reentrant context managers exists explicitly to allow possibly-nested contexts.
These context managers can not only be used in multiple with statements, but may also be used inside a with statement that is already using the same context manager.
Related
I have a need to run some code at class creation time, invoking a function (in this case it happens to be a method) that I need to pass the cls class object and a few other things (mostly defined in the parent).
My solution so far is this:
#PostConstruct()
class Child(Parent):
X = 1
Y = Parent.A
Z = 2
#classmethod
def __post_construct__(cls):
cls.add_thing(cls.X, as_key=True, before=cls.Y)
cls.add_thing(cls.Z, as_key=False, before=cls.Y)
Supporting code:
class PostConstruct:
"""
Runs a class's ``__post_construct__`` class method immediately after
the body code of the class is run. Allows an author to make small
modifications to the class (e.g. modifying class-level variables) at
class creation time.
"""
def __call__(self, cls):
cls.__post_construct__()
return cls
class Parent:
A = 0
#classmethod
def add_thing(cls, thing, as_key, before):
print("Adding thing...")
Is there some built-in post-class-construction hook method I've overlooked, so I wouldn't need to write this decorator myself? I've looked at https://docs.python.org/3/reference/datamodel.html#customizing-class-creation but I haven't seen anything that seems relevant. But this wouldn't be the first time I've implemented some "clever" thing and then learned later that I could have done it simpler.
Or any other suggestion to achieve a similar result?
Thanks, all. Putting together the suggested tweaks from the comments, here's how I'm moving forward:
#post_construct()
class Child(Parent):
X = 1
Y = Parent.A
Z = 2
#classmethod
def _post_construct(cls):
cls.add_thing(cls.X, as_key=True, before=cls.Y)
cls.add_thing(cls.Z, as_key=False, before=cls.Y)
Supporting code:
def post_construct(cls):
"""
Runs a class's ``_post_construct`` class method immediately after the body code of the class is run. Allows an author to make
small modifications to the class (e.g. modifying class-level variables) at class creation time.
"""
cls._post_construct()
return cls
class Parent:
A = 0
#classmethod
def add_thing(cls, thing, as_key, before):
print("Adding thing...")
The case is such that I have an abstract class and a few child classes implementing it.
class Parent(metaclass=ABCMeta):
#abstract_method
def first_method(self, *args, **kwargs):
raise NotImplementedError()
#abstract_method
def second_method(self, *args, **kwargs):
raise NotImplementedError()
class Child(Parent):
def first_method(self, *args, **kwargs):
print('First method of the child class called!')
def second_method(self, *args, **kwargs):
print('Second method of the child class called!')
My goal is to make some kind of decorator, which will be used on methods of any child of the Parent class. I need this because every method make some kind of preparation before actually doing something, and this preparation is absolutely the same in all methods of all childs of the Parent class. Like:
class Child(Parent):
def first_method(self, *args, **kwargs):
print('Preparation!')
print('First method of the child class called!')
def second_method(self, *args, **kwargs):
print('Preparation!')
print('Second method of the child class called!')
The first thing came to my mind is to use Parent class method implementation: just remove "raise NotImplementedError()" and put some functionality, and then in child classes I would call, for example, super().first_method(self, *args, **kwargs) in the beginning of each method. It is good, but I also would want to return some data from the Parent method, and it would look weird when parent method and child method return something different in declaration. Not to mention that I would probably want to do some post-processing work after the method, so then I would need 2 different functions: for the beginning and after the performing the script.
The next thing I came up with is making MetaClass.
Just implement all the decoration of methods in the new MetaClass during creating a class, and pass the newly generated data which is used in child methods to them in kwargs.
This is the closest solution to my goal, but it feels wrong anyway. Because it is not explicit that some kwargs will be passed to child methods, and if you are new to this code, then you need to do some researches to understand how it works. I feel like I overengineering or so.
So the question: is there any pattern or something along these lines to implement this functionality?
Probably you can advise something better for my case?
Thank you a lot in advance!
So, existing patterns apart: I won't know if this has an specific name, what you need, that would be a "pattern" is the use of "slots": that is - you document special named methods that will be called as part of the execution of another method. This other method then performs its setup code, checks if the slotted method (usually identifiable by name) exists, call them, with a plain simple method call, which will run the most specialized version of it, even if the special method that calls the slots is in the base class, and you are on a big class-inheritance hierarchy.
One plain example of this pattern is the way Python instantiates objects: what one actually invokes calling the class with the same syntax that is used for function calls (MyClass()) is that class's class (its metaclass) __call__ method. (Usally type.__call__). In Python's code for type.__call__ the class' __new__ method is called, then the class' __init__ method is called and finally the value returned by the first call, to __new__ is returned. A custom metaclass can modify __call__ to run whatever code it wants before, between, or after these two calls.
So, if this was not Python, all you'd need is to spec down this, and document that these methods should not be called directly, but rather through an "entry point" method - which could simply feature an "ep_" prefix. These would have to be fixed and hardcoded on a baseclass, and you'd need one for each of the methods you want to prefix/postfix code to.
class Base(ABC):
def ep_first_method(self, *args, **kw);
# prefix code...
ret_val = self.first_method(*args, **kw)
# postfix code...
return ret_val
#abstractmethod
def first_method(self):
pass
class Child(Base):
def first_method(self, ...):
...
This being Python, it is easier to add some more magic to avoid code repetition and keep things concise.
One possible thing is to have a special class that, when detecting a method in a child class that should be called as a slot of a wrapper method, like above, to automatically rename that method: this way the entry point methods can feature the same name as the child methods - and better yet, a simple decorator can mark the methods that are meant to be "entrypoints", and inheritance would even work for them.
Basically, when building a new class we check all methods: if any of them has a correspondent part in the calling hierarchy which is marked as an entrypoint, the renaming takes place.
It is more practical if any entrypoint method will take as second parameter (the first being self), a reference for the slotted method to be called.
After some fiddling: the good news is that a custommetaclass is not needed - the __init_subclass__ special method in a baseclass is enough to enable the decorator.
The bad news: due to re-entry iterations in the entry-point triggered by potential calls to "super()" on the final methods, a somewhat intricate heuristic to call the original method in the intermediate classes is needed. I also took care to put some multi-threading protections - although this is not 100% bullet-proof.
import sys
import threading
from functools import wraps
def entrypoint(func):
name = func.__name__
slotted_name = f"_slotted_{name}"
recursion_control = threading.local()
recursion_control.depth = 0
lock = threading.Lock()
#wraps(func)
def wrapper(self, *args, **kw):
slotted_method = getattr(self, slotted_name, None)
if slotted_method is None:
# this check in place of abstractmethod errors. It is only raised when the method is called, though
raise TypeError("Child class {type(self).__name__} did not implement mandatory method {func.__name__}")
# recursion control logic: also handle when the slotted method calls "super",
# not just straightforward recursion
with lock:
recursion_control.depth += 1
if recursion_control.depth == 1:
normal_course = True
else:
normal_course = False
try:
if normal_course:
# runs through entrypoint
result = func(self, slotted_method, *args, **kw)
else:
# we are within a "super()" call - the only way to get the renamed method
# in the correct subclass is to recreate the callee's super, by fetching its
# implicit "__class__" variable.
try:
callee_super = super(sys._getframe(1).f_locals["__class__"], self)
except KeyError:
# callee did not make a "super" call, rather it likely is a recursive function "for real"
callee_super = type(self)
slotted_method = getattr(callee_super, slotted_name)
result = slotted_method(*args, **kw)
finally:
recursion_control.depth -= 1
return result
wrapper.__entrypoint__ = True
return wrapper
class SlottedBase:
def __init_subclass__(cls, *args, **kw):
super().__init_subclass__(*args, **kw)
for name, child_method in tuple(cls.__dict__.items()):
#breakpoint()
if not callable(child_method) or getattr(child_method, "__entrypoint__", None):
continue
for ancestor_cls in cls.__mro__[1:]:
parent_method = getattr(ancestor_cls, name, None)
if parent_method is None:
break
if not getattr(parent_method, "__entrypoint__", False):
continue
# if the code reaches here, this is a method that
# at some point up has been marked as having an entrypoint method: we rename it.
delattr (cls, name)
setattr(cls, f"_slotted_{name}", child_method)
break
# the chaeegs above are inplace, no need to return anything
class Parent(SlottedBase):
#entrypoint
def meth1(self, slotted, a, b):
print(f"at meth 1 entry, with {a=} and {b=}")
result = slotted(a, b)
print("exiting meth1\n")
return result
class Child(Parent):
def meth1(self, a, b):
print(f"at meth 1 on Child, with {a=} and {b=}")
class GrandChild(Child):
def meth1(self, a, b):
print(f"at meth 1 on grandchild, with {a=} and {b=}")
super().meth1(a,b)
class GrandGrandChild(GrandChild):
def meth1(self, a, b):
print(f"at meth 1 on grandgrandchild, with {a=} and {b=}")
super().meth1(a,b)
c = Child()
c.meth1(2, 3)
d = GrandChild()
d.meth1(2, 3)
e = GrandGrandChild()
e.meth1(2, 3)
I have a Python program with multiple threads that manipulate a shared resource. I model this resource with an object and actions on this resource with methods in that object. A number of actions need to be performed as atomic actions, in a critical section. Since these functions follow a common pattern, I used a decorator:
def critical_section(inner):
def outer(self, *args, **kwargs):
self.enter()
inner(self, *args, **kwargs)
self.leave()
return outer
The real code is more complex than this, e.g. it handles exceptions, but this should be enough to get the gist of what I'm doing. Sample usage (I simplified all the thread-related stuff and just check for resource ownership):
class Base:
def __init__(self):
self.owner = None
def enter(self):
assert(self.owner == None)
self.owner = True
def leave(self):
assert(self.owner == True)
self.owner = None
#critical_section
def foo(self):
print('foo') # Dangerous stuff, must run atomically and not fail
Functions with the critical_section decorator have two properties:
The function's code is executed with the resource's lock held (enter and leave take care of that).
The function assumes that the resource is in a stable state on entry, and leaves it in a stable state on exit.
The second property implies that the lock is not a reentrant one. It is invalid to call a “critical section” function while already in a critical section, because the necessary invariants are not guaranteed.
The implementation of enter and leave check these properties: a thread can't call enter if the resource is currently owned (even by that thread itself), and enter gives ownership of the resource to the calling thread; leave symmetrically requires ownership and gives it up.
That architecture served me nicely until I wanted to have multiple resources with a similar structure, so I started to use inheritance.
class Derived(Base):
#critical_section
def foo(self):
print('fie') # Additional stuff to run first
Base.foo(self)
And now we come to the problem: that decorator is fundamentally inheritance-unfriendly.
base = Base()
base.foo()
derived = Derived()
derived.foo()
The call to derived.foo() fails because when Base.foo() is executed, the resource is already owned. At the point where the base class's method is called, the derived class has potentially already mucked up the object, violating the assumption that Base.foo starts with an object in known stable state.
The obvious solution would be to turn every critical-section function into a pair of functions: the outer function (meant to be called from outside, and never overridden in classes that modify the behavior of the resource), and the inner function (meant to be called only from derived methods).
Can Python decorators help me define such pairs of functions with a minimum of boilerplate?
Is there a better architecture that would maximize the clarity of where critical sections are entered and exited from, and minimize the risk of misuse? (Dynamic checks are good, but obvious correctness of source code is better.)
Edit: Here's a proper non-reentrant version.
You can have decorators take in arguments. so #dec(x); def f() ... will be called like dec(x)(f)(args). So, we have critical_section take in a string (#critical_section("Base")) and have one lock per string.
def critical_section(ident):
def _critical_section(inner):
def outer(self, *args, **kwargs):
self.enter(ident)
inner(self, *args, **kwargs)
self.leave(ident)
return outer
return _critical_section
class Base:
def __init__(self):
self.owner = {}
def enter(self, ident):
assert(ident not in self.owner)
self.owner[ident] = True
def leave(self, ident):
assert(ident in self.owner)
del self.owner[ident]
#critical_section("Base")
def foo(self):
print('foo') # Dangerous stuff, must run atomically and not fail
class Derived(Base):
#critical_section("Derived")
def foo(self):
print('fie') # Additional stuff to run first
Base.foo(self)
Hello, me from yesterday. Your fundamental problem is that you oversimplified the situation. You conflated two things: entering/leaving the critical section, and assuming/asserting the resource's invariants. But the resource's invariants can also be true in the middle of the critical section, and that's precisely what you're trying to convey when you say that Derived.foo method is allowed to call Base.foo (at some particular point during this execution).
You can model this in Python, but it does get a bit cumbersome.
def critical_section(inner):
def outer(self, *args, **kwargs):
self.enter()
inner(self, *args, **kwargs)
self.leave()
return outer
class Base:
def __init__(self):
self.owner = None
self.invariants = True
def enter(self):
assert(self.invariants)
self.invariants = False
assert(self.owner == None)
self.owner = True
def leave(self):
assert(self.owner == True)
self.owner = None
self.invariants = True
#critical_section
def foo(self):
print('foo') # Dangerous stuff, must run atomically and not fail
class Derived(Base):
#critical_section
def foo(self):
print('fie') # Additional stuff to run first
self.invariants = True
Base.foo(self)
The owner stuff should be a reentrant lock in the real world. Instead of having non-reentrancy to prevent modifying the resource while it's in an unstable state, there's this invariants check.
But all this is complicated and not really worth it when “checking invariants” amounts to “I state that the useful invariants are verified at this point in the code”. With invariants checked by static analysis, it would be another story, but then you wouldn't be using Python.
Coming back to your question, which was accessing the inner function when needed, Python does make it pretty easy. Store the inner function in an attribute of the decorated function.
def critical_section(inner):
def outer(self, *args, **kwargs):
self.enter()
inner(self, *args, **kwargs)
self.leave()
outer.inner_function = inner
return outer
…
class Derived(Base):
#critical_section
def foo(self):
print('fie') # Additional stuff to run first
Base.inner.foo(self)
I am trying to implement thread safe code but encounter some simple problem. I searched and not found solution.
Let me show abstract code to describe problem:
import threading
class A(object):
sharedLock = threading.Lock()
shared = 0
#classmethod
def getIncremented(cls):
with cls.sharedLock:
cls.shared += 1
return cls.shared
class B(A):
pass
class C(A):
#classmethod
def getIncremented(cls):
with cls.sharedLock:
cls.shared += B.getIncremented()
return cls.shared
I want to define class A to inherit many child classes for example for enumerations or lazy variables - specific use no matter. I am already done single thread version now want update multi thread.
This code will give such results as should do:
id(A.sharedLock) 11694384
id(B.sharedLock) 11694384
id(C.sharedLock) 11694384
I means that lock in class A is lock in class B so it is bad since first entry into class B will lock also class A and class C. If C will use B it will lead to dedlock.
I can use RLock but it is invalid programming pattern and not sure if it not produce more serious deadlock.
How can I change sharedLock value during initialization of class to new lock to make id(A.sharedLock) != id(B.sharedLock) and same for A and C and B and C?
How can I hook class initialization in python in generic to change some class variables?
That question is not too complex but I do not know what to do with it.
I want inherit parent share variables except shared parent locks
You must not do this. It makes access to "share variables" not thread-safe.
sharedLock protects shared variable. If the same shared variable can be modified in a recursive call then you need RLock(). Here shared means shared among all subclasses.
It looks like you want a standalone function (or a static method) instead of the classmethod:
def getIncremented(_lock=Lock(), _shared=[0]):
with _lock:
_shared[0] += 1
return _shared[0]
Thus all classes use the same shared variable (and the corresponding lock).
If you want each class to have its own shared variable (here shared means shared among instances of this particular class) then don't use cls.shared that may traverse ancestors to get it.
To hint that subclasses shouldn't use a variable directly, you could use the syntax for a private variable:
class A:
__shared = 0
__lock = Lock()
If a subclass overrides a method that uses __shared then it won't use A.__shared by accident in the code directly.
As you noticed, if you expose shared locks as class attributes, the locks are shared by subclasses.
You could hack around this by redefining the lock on each subclass:
class B(A):
sharedLock = threading.Lock()
You could even use metaclasses to achieve this (please don't). It seems to me that you're approaching the program from the wrong angle.
This task is easier if you assign locks explicitly to instances (not classes).
class A(object):
def __init__(self, lock):
this.sharedLock= lock
my_lock= threading.Lock()
a= A(my_lock)
Of course, you run into the "problem" of having to explicitly pass the lock for each instance. This is traditionally solved using a factory pattern, but in python you can simply use functions properly:
from functools import partial
A_with_mylock= partial(A, my_lock)
a2= A_with_mylock()
Here is solution - this allow separate lock per each class since it done on class constructor level (metaclass). Thank you for all hints and help to achieve this code it looks very nice.
I can be also mangled variable but need to use hardcode '_A__lock' what can be problematic and not tested by me.
import threading
class MetaA(type):
def __new__(self, name, bases, clsDict):
# change <type> behavior
clsDict['_lock'] = threading.Lock()
return super(MetaA, self).__new__(self, name, bases, clsDict)
class A(object):
__metaclass__ = MetaA
#classmethod
def getLock(cls):
return cls._lock
class B(A):
pass
print 'id(A.getLock())', id(A.getLock())
print 'id(B.getLock())', id(B.getLock())
print A.getLock() == B.getLock()
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Is there a simple, elegant way to define Singletons in Python?
I have the following example code, in which I derive a class from a Singleton (hope it is one):
class Singleton(object):
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = object.__new__(cls, *args, **kwargs)
return cls._instance
class Tracer(Singleton):
def __init__(self):
print "Init"
a = Tracer()
b = Tracer()
When you try it, you will see the __init__ method of Tracer is called again. Isn't the sense of having a singleton to make another instance refer to the original one? I do not want to run the __init__ method again, as it probably overwrites previous information. Maybe the singleton is wrong or it's use?
My previous answer didn't work and I've deleted it. However I've found a highly rated SO answer that does. The primary differences are that it uses a Singleton metaclass instead of a baseclass and overloads the __call__() method of its instance classes instead of their __new__() method. This gives it the control required over the creation process of instances of its singleton class instances. It would be possible to define an additional method for deleting one or more of these — say for testing purposes.
Another notable implementation detail is that the metaclass maintains a dictionary of _instances rather than something that can only hold a single value. This allows it keep track of an indefinite number of singleton instances (since it might be the metaclass of more than one since it's reusable).
Applying it to your sample code would be done something like this:
class Singleton(type):
"""Metaclass."""
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
return cls._instances[cls]
class Tracer(object):
__metaclass__ = Singleton
def __init__(self):
print("Init")
a = Tracer()
b = Tracer()
print('a is b: {}'.format(a is b)) # same object? -> True
Output:
Init
a is b: True
Update
The syntax for specifying a metaclass varies between Python 2 and 3. For the latter you'd need to change the Tracer class definition to this:
#!/usr/bin/env python3
class Tracer(object, metaclass=Singleton):
def __init__(self):
print("Init")
Writing a something that would work in both version 2 and 3 of Python is possible, but is a little more complicated since you can't simply conditionally define it like this:
## Won't work ##
if sys.version_info[0] < 3: # Python 2?
class Tracer(object):
__metaclass__ = Singleton
def __init__(self):
print("Init")
else: # Python 3
class Tracer(object, metaclass=Singleton): # causes SyntaxError in Python 2
def __init__(self):
print("Init")
because the definition in the else clause causes a SyntaxError in Python 2 (even though the code in the block will never actually be executed). A workaround similar to what Benjamin Peterson's six module's with_metaclass() function does and would look like this:
class Tracer(Singleton("SingletonBaseClass", (object,), {})):
def __init__(self):
print("Init")
This dynamically creates a baseclass that inherits the desired metaclass—thereby avoiding any errors due to metaclass syntax differences between the two Python versions. (Which it does by explicitly using the defined metaclass to create the temporary baseclass.)
Your __init__ is called twice, but on the same object. You have created a singleton, but Python doesn't know it is, so it initializes each object that gets created.
If you want to pursue the singleton pattern, you'll have to move your initializing code into the __new__, or into another method that your __new__ calls.
Keep in mind:
Singletons are the norm in Java, but are frowned upon in Python.
Singletons make your code harder to test, because they are global state carried from one test to the next.