I have a problem in Python, for which I cannot find any clean solution ...
When calling some methods, I want to execute some code before the method execution and after. In order (among many other things) to automatically set and clean a context variable.
In order to achieve this, I have declared the following metaclass :
class MyType(type):
def __new__(cls, name, bases, attrs):
#wraps the 'test' method to automate context management and other stuff
attrs['test'] = cls.other_wrapper(attrs['test'])
attrs['test'] = cls.context_wrapper(attrs['test'])
return super(MyType, cls).__new__(cls, name, bases, attrs)
#classmethod
def context_wrapper(cls, operation):
def _manage_context(self, *args, **kwargs):
#Sets the context to 'blabla' before the execution
self.context = 'blabla'
returned = operation(self, *args, **kwargs)
#Cleans the context after execution
self.context = None
return returned
return _manage_context
#classmethod
def other_wrapper(cls, operation):
def _wrapped(self, *args, **kwargs):
#DO something with self and *args and **kwargs
return operation(self, *args, **kwargs)
return _wrapped
This works like a charm :
class Parent(object):
__metaclass__ = MyType
def test(self):
#Here the context is set:
print self.context #prints blabla
But as soon as I want to subclass Parent, problems appear, when I call the parent method with super :
class Child(Parent):
def test(self):
#Here the context is set too
print self.context #prints blabla
super(Child, self).test()
#But now the context is unset, because Parent.test is also wrapped by _manage_context
#so this prints 'None', which is not what was expected
print self.context
I have thought of saving the context before setting it to a new value, but that only solves partially the problem...
Indeed, (hang on, this is hard to explain), the parent method is called, the wrappers are executed, but they receive *args and **kwargs addressed to Parent.test, while self is a Child instance, so self attributes have irrelevant values if I want to challenge them with *args and **kwargs (for example for automated validation purpose), example :
#classmethod
def validation_wrapper(cls, operation):
def _wrapped(self, *args, **kwargs):
#Validate the value of a kwarg
#But if this is executed because we called super(Child, self).test(...
#`self.some_minimum` will be `Child.some_minimum`, which is irrelevant
#considering that we called `Parent.test`
if not kwarg['some_arg'] > self.some_minimum:
raise ValueError('Validation failed')
return operation(self, *args, **kwargs)
return _wrapped
So basically, to solve this problem I see two solutions :
preventing the wrappers to be executed when the method was called with super(Child, self)
having a self that is always of the "right" type
Both solutions seem impossible to me ... Do somebody has an idea on how to solve this ? A suggestion ?
Well, can't you just check if the context is already set in _manage_context? Like this:
def _manage_context(self, *args, **kwargs):
#Sets the context to 'blabla' before the execution
if self.context is None:
self.context = 'blabla'
returned = operation(self, *args, **kwargs)
#Cleans the context after execution
self.context = None
return returned
else:
return operation(self, *args, **kwargs)
Also, this should probably be wrapped in a try-catch block, to ensure resetting of the context in case of exceptions.
Actually I have found out a way to prevent the wrappers to be executed when the method was called with super(Child, self) :
class MyType(type):
def __new__(cls, name, bases, attrs):
#wraps the 'test' method to automate context management and other stuff
new_class = super(MyType, cls).__new__(cls, name, bases, attrs)
new_class.test = cls.other_wrapper(new_class.test, new_class)
#classmethod
def other_wrapper(cls, operation, new_class):
def _wrapped(self, *args, **kwargs):
#DO something with self and *args and **kwargs ...
#ONLY if self is of type *new_class* !!!
if type(self) == new_class:
pass #do things
return operation(self, *args, **kwargs)
return _wrapped
That way, when calling :
super(Child, self).a_wrapped_method
The wrapping code is by-passed !!! That's quite hackish, but it works ...
Ok, first, your "solution" is really ugly, but I suppose you know that. :-) So let's try to answer your questions.
First is an implicit "question": why don't you use Python's context managers? They give you much nicer syntax and error management practically for free. See contextlib module, it can help you greatly. Especially see section about reentrancy.
Then you'll see that people usually have problems when trying to stack context managers. That's not surprising, since to properly support recursion you need a stack of values, not a single value. [You could see the source for some reentrant cm, for example redirect_stdout, to see how it's handled.] So your context_wrapper should either:
(cleaner) keep a list of self.contexts, append to it when entering context, and pop from it when exiting. That way you always get your context.
(more like what you want) keep a single self.context, but also a global value DEPTH, increased by one on entering, decreased by one on exiting, and self.context being reset to None when DEPTH is 0.
As for your second question, I must say I don't quite understand you. self is of the right type. If A is subclass of B, and self is instance of A, then it is also instance of B. If self.some_minimum is "wrong" whether you consider self an instance of A or of B, that means that some_minimum is not really an instance attribute of self, but a class attribute of A or B. Right? They can be freely different on A and on B, because A and B are different objects (of their metaclass).
Related
So I am trying to override __new__ and let it exist as a factory to create
derived instances. After reading a bit on SO, I am under the impression that I should be calling __new__ on the derived instance as well.
BaseThing
class BaseThing:
def __init(self, name, **kwargs):
self.name = name
# methods to be derived
ThingFactory
class Thing(BaseThing):
def __new__(cls, name, **kwargs):
if name == 'A':
return A.__new__(name, **kwargs)
if name == 'B':
return B.__new__(name, **kwargs)
def __init__(self, *args, **kwargs):
super().__init__(name, **kwargs)
# methods to be implemented by concrete class (same as those in base)
A
class A(BaseThing):
def __init__(self, name, **kwargs):
super().__init__(name, **kwargs)
B
class B(BaseThing):
def __init__(self, name, **kwargs):
super().__init__(name, **kwargs)
what I am expecting was that it'd just work.
>>> a = Thing('A')
gives me TypeError: object.__new__(X): X is not a type object (str)
I am bit confused by this; when I just return a concrete instance of derived classes, it just worked. i.e.
def __new__(cls, name, **kwargs):
if name == 'A':
return A(name)
if name == 'B':
return B(name)
I don't think this is the correct way to return in __new__; it may duplicate the calls to __init__.
when I am checking signatures of __new__ in object it seems be this one:
#staticmethod # known case of __new__
def __new__(cls, *more): # known special case of object.__new__
""" Create and return a new object. See help(type) for accurate signature. """
pass
I didn't expect this was the one; I'd expect it came with args and kwargs as well. I must have done something wrong here.
it seems to me that I need to inherit object directly in my base but could anyone explain the correct way of doing it?
You're calling __new__ wrong. If you want your __new__ to create an instance of a subclass, you don't call the subclass's __new__; you call the superclass's __new__ as usual, but pass it the subclass as the first argument:
instance = super().__new__(A)
I can't guarantee that this will be enough to fix your problems, since the code you've posted wouldn't reproduce the error you claim; it has other problems that would have caused a different error first (infinite recursion). Particularly, if A and B don't really descend from Thing, that needs different handling.
I have a class "Wrapper". Wrapper inherits from object, and it's primary responsibility is to hook method calls to selenium during a test. It's __init__ method signature is
__init__(self, object_to_wrap, actions={}, **kwargs)
Part of it's functionality is to take a dictionary as an argument in the init method, and to expose the keys as attributes on the object_to_wrap object by defining the __getattr__ magic method
(signature: __getattr__(self, item):)
and checking self.actions's keys for item. If it is found, the method is invoked.
In the test code, the initialization would look similar to:
def navigate(scoped_self, to=''):
self.driver.switch_to_default_content()
self.driver.switch_to.frame(to)
scoped_self.navigations.append(to)
# the navigate method is scoped in an
# instance method of the test class, so it has access to self
So, my question is, how can I make the scope, or self in the above method, be the scope of my Wrapper class, and not the test class?
For clarity, if the solution I am looking for in this question was found, the navigate implementation would change to be:
def navigate(scoped_self, to=''):
self.object_to_wrap.switch_to_default_content()
self.object_to_wrap.switch_to.frame(to)
Also for clarity, I'm pretty sure what I'm looking for is exactly what Javascripts Function.prototype.bind accomplishes.
UPDATE: Defining the methods, such as navigate, inside the Wrapper class is not an option, as the Wrapper class cannot have test-specific logic. The test suite is done on n > 1 DOM, that are completely unrelated. For example, one of the tests requires the method "alert_handlers" (to overwrite the window.alert function and return the alerts presented as a string), another might require the navigate method, and a third might require both.
UPDATE #2: Thanks to the answer below from c17r, I realized that without the getattribute code included in my example, that it would appear as if I was asking for something we had already accomplished. What I am looking for is the ability, in the above navigate method, is for scoped_self to be the instance of Wrapper.
Furthermore, I am specifically looking for a way to "dynamically" pass scoped_self to the functions (the way that Function.prototype.bind"dynamically" sets this where this is myEventListener.bind(window); if you're unfamiliar with javascript, make an event listener on the body and console.log this without binding, and then with binding, to see the difference)
Furthermore, I figured it might help to give an example implementation without the solution I am looking for. This is currently working as expected:
class Wrapper(object):
def __init__(self, wrapped, actions={}):
self.wrapped = wrapped
self.actions = actions
self.navigations = [] # EXAMPLE, SEE THE TEST CLASS CODE
def __getattr__(self, item):
if item in self.actions:
return self.actions[item]
# do other fancy stuff here
# UPDATE #2: added for clarity. this is the current implementation
orig_attr = self.wrapped.__getattribute__(item)
if callable(orig_attr):
def hooked(*args, **kwargs):
self.pre(item, *args, **kwargs)
self.err = False
try:
result = orig_attr(*args, **kwargs)
except Exception as e:
#logs
self.post(*args, **kwargs)
raise
if type(self.wrapped) == type(result):
return self
return result
return hooked
else:
return orig_attr
class SomeTest():
#blah blah init stuff, set self.driver = selenium.webdriver.Phantomjs
def spawn_actions(self):
def navigate(scoped_self, to=''):
self.driver.switch_to_default_content()
self.driver.switch_to.frame(to)
scoped_self.navigations.append(to) # <--- appended to wrapper.navigations
return {'navigate': navigate}
def run(self):
driver = Wrapper(self.driver, self.spawn_actions())
driver.get(url)
driver.navigate('New Request')
# fun tests!
If I understand you correctly, 3 things:
Wrapper would need to pass any unknown functions down to the wrapped item, otherwise driver.get() won't work properly.
The navigate function needs self as the first parameter, like it would if it was actually defined on Wrapper
Wrapper needs to pass self into the dict-based function. This is a little tricky since __getattr__ doesn't actually call the function, so you need to return a function that calls the underlying function properly, similar to how decorators work.
Like this:
class Driver(object):
def get(self, url):
print('get')
print(repr(self))
print(repr(url))
print('--')
class Wrapper(object):
def __init__(self, wrapped, actions={}):
self.wrapped = wrapped
self.actions = actions
def __getattr__(self, item):
if item in self.actions:
def unwrap(*args, **kwargs):
return self.actions[item](self, *args, **kwargs)
return unwrap
else:
return getattr(self.wrapped, item)
class Test(object):
def __init__(self):
self.driver = Driver();
def spawn_actions(self):
def navigate(self, to=''):
print('navigate')
print(repr(self))
print(repr(to))
print(repr(self.wrapped))
print('--')
return {
'navigate': navigate
}
def run(self):
driver = Wrapper(self.driver, self.spawn_actions())
driver.get('url')
driver.navigate('thing')
Now calling:
t = Test()
t.run()
Outputs:
get
<__main__.Driver object at 0x104008630>
'url'
--
navigate
<__main__.Wrapper object at 0x104008ba8>
'thing'
<__main__.Driver object at 0x104008630>
--
EDIT
You can also dynamically bind the methods to the instance instead of __getattr__ returning the unwrap function:
import types
class Wrapper(object):
def __init__(self, wrapped, actions={}):
self.wrapped = wrapped
for name, func in actions.items():
setattr(self, name, types.MethodType(func, self))
def __getattr__(self, item):
return getattr(self.wrapped, item)
I'm writing a class that parses HTML in order to provide an interface to a profile on a webpage. It looks something like this:
class Profile(BeautifulSoup):
def __init__(self, page_source):
super().__init__(page_source)
def username(self):
return self.title.split(':')[0]
Except more complex and time consuming. Since I know that the underlying profiles aren't going to be changing during the lifetime of a Profile object, I thought this would be a good place to cache results in order to avoid recalculating values that are already known. I implemented this with a decorator, and the result looks like this:
def cached_resource(method_to_cache):
def decorator(self, *args, **kwargs):
method_name = method_to_cache.__name__
try:
return self._cache[method_name]
except KeyError:
self._cache[method_name] = method_to_cache(self, *args, **kwargs)
return self._cache[method_name]
return decorator
class Profile(BeautifulSoup):
def __init__(self, page_source):
super().__init__(page_source)
self._cache = {}
#cached_resource
def username(self):
return self.title.split(':')[0]
When I give this code to pylint, it complains about cached_resource having access to a protected variable of a client class.
I realize that the distinction between public and private isn't a huge deal in Python, but I'm still curious -- have I done something bad here? Is it poor style to have decorators rely on implementation details of the classes they're associated with?
EDIT: I'm unclear about how the closure in Duncan's answer works, so maybe this is a little bit kludge-y, but would this be a simpler solution?
def cached_resource(method_to_cache):
def decorator(self, *args, **kwargs):
method_name = method_to_cache.__name__
try:
return self._cache[method_name]
except KeyError:
self._cache[method_name] = method_to_cache(self, *args, **kwargs)
except AttributeError:
self._cache = {}
self._cache[method_name] = method_to_cache(self, *args, **kwargs)
finally:
return self._cache[method_name]
return decorator
There is a bit of a code smell about that, I think I would agree with pylint on this one though it is quite subjective.
Your decorator looks like it is a general-purpose decorator, but it is tied into the internal implementation detail of the class. If you tried to use it from another class it won't work without the initialisation of _cache in __init__. The linkage I don't like is that the knowledge of an attribute called '_cache' is shared between both the class and the decorator.
You could move the initialisation of _cache out of __init__ and into the decorator. I don't know if that would help pacify pylint and it still requires the class to know about and avoid using the attribute. A cleaner solution here (I think) would be to pass the name of the cache attribute into the decorator. That should break the linkage cleanly:
def cached_resource(cache_attribute):
def decorator_factory(method_to_cache):
def decorator(self, *args, **kwargs):
method_name = method_to_cache.__name__
cache = getattr(self, cache_attribute)
try:
return cache[method_name]
except KeyError:
result = cache[method_name] = method_to_cache(self, *args, **kwargs)
return result
return decorator
return decorator_factory
class Profile(BeautifulSoup):
def __init__(self, page_source):
super().__init__(page_source)
self._cache = {}
#cached_resource('_cache')
def username(self):
return self.title.split(':')[0]
And if you don't like a lot of decorator calls repeating the name of the attribute then:
class Profile(BeautifulSoup):
def __init__(self, page_source):
super().__init__(page_source)
self._cache = {}
with_cache = cached_resource('_cache')
#with_cache
def username(self):
return self.title.split(':')[0]
Edit:
Martineau suggests this may be overkill. It could be if you don't actually need separate access to the _cache attribute inside the class (e.g. to have a cache reset method). In that case you could manage the cache entirely within the decorator, but if you are going to do that you don't need a cache dictionary on the instance at all, as you can store the cache in the decorator and key on the Profile instance:
from weakref import WeakKeyDictionary
def cached_resource(method_to_cache):
cache = WeakKeyDictionary()
def decorator(self, *args, **kwargs):
try:
return cache[self]
except KeyError:
result = cache[self] = method_to_cache(self, *args, **kwargs)
return result
return decorator
class Profile(BeautifulSoup):
def __init__(self, page_source):
super().__init__(page_source)
self._cache = {}
#cached_resource
def username(self):
return self.title.split(':')[0]
What you did looks fine to me. The error is presumably because pylint can't figure out that cached_resource is only "accessing" self._cache via its inner function, which ultimately is a method of the class (assigned by the decorator).
It might be worth raising an issue on the pylint tracker for this. It could be tough to handle with static analysis but the current behavior doesn't seem right.
I'm trying to find the best way to create a class decorator that does the following:
Injects a few functions into the decorated class
Forces a call to one of these functions AFTER the decorated class' __init__ is called
Currently, I'm just saving off a reference to the 'original' __init__ method and replacing it with my __init__ that calls the original and my additional function. It looks similar to this:
orig_init = cls.__init__
def new_init(self, *args, **kwargs):
"""
'Extend' wrapped class' __init__ so we can attach to all signals
automatically
"""
orig_init(self, *args, **kwargs)
self._debugSignals()
cls.__init__ = new_init
Is there a better way to 'augment' the original __init__ or inject my call somewhere else? All I really need is for my self._debugSignals() to be called sometime after the object is created. I also want it happen automatically, which is why I thought after __init__ was a good place.
Extra misc. decorator notes
It might be worth mentioning some background on this decorator. You can find the full code here. The point of the decorator is to automatically attach to any PyQt signals and print when they are emitted. The decorator works fine when I decorate my own subclasses of QtCore.QObject, however I've been recently trying to automatically decorate all QObject children.
I'd like to have a 'debug' mode in the application where I can automatically print ALL signals just to make sure things are doing what I expect. I'm sure this will result in TONS of debug, but I'd still like to see what's happening.
The problem is my current version of the decorator is causing a segfault when replacing QtCore.QObject.__init__. I've tried to debug this, but the code is all SIP generated, which I don't have much experience with.
So, I was wondering if there was a safer, more pythonic way to inject a function call AFTER the __init__ and hopefully avoid the segfault.
Based on this post and this answer, an alternative way to do this is through a custom metaclass. This would work as follows (tested in Python 2.7):
# define a new metaclass which overrides the "__call__" function
class NewInitCaller(type):
def __call__(cls, *args, **kwargs):
"""Called when you call MyNewClass() """
obj = type.__call__(cls, *args, **kwargs)
obj.new_init()
return obj
# then create a new class with the __metaclass__ set as our custom metaclass
class MyNewClass(object):
__metaclass__ = NewInitCaller
def __init__(self):
print "Init class"
def new_init(self):
print "New init!!"
# when you create an instance
a = MyNewClass()
>>> Init class
>>> New init!!
The basic idea is that:
when you call MyNewClass() it searches for the metaclass, finds that you have defined NewInitCaller
The metaclass __call__ function is called.
This function creates the MyNewClass instance using type,
The instance runs its own __init__ (printing "Init class").
The meta class then calls the new_init function of the instance.
Here is the solution for Python 3.x, based on this post's accepted answer. Also see PEP 3115 for reference, I think the rationale is an interesting read.
Changes in the example above are shown with comments; the only real change is the way the metaclass is defined, all other are trivial 2to3 modifications.
# define a new metaclass which overrides the "__call__" function
class NewInitCaller(type):
def __call__(cls, *args, **kwargs):
"""Called when you call MyNewClass() """
obj = type.__call__(cls, *args, **kwargs)
obj.new_init()
return obj
# then create a new class with the metaclass passed as an argument
class MyNewClass(object, metaclass=NewInitCaller): # added argument
# __metaclass__ = NewInitCaller this line is removed; would not have effect
def __init__(self):
print("Init class") # function, not command
def new_init(self):
print("New init!!") # function, not command
# when you create an instance
a = MyNewClass()
>>> Init class
>>> New init!!
Here's a generalized form of jake77's example which implements __post_init__ on a non-dataclass. This enables a subclass's configure() to be automatically invoked in correct sequence after the base & subclass __init__s have completed.
# define a new metaclass which overrides the "__call__" function
class PostInitCaller(type):
def __call__(cls, *args, **kwargs):
"""Called when you call BaseClass() """
print(f"{__class__.__name__}.__call__({args}, {kwargs})")
obj = type.__call__(cls, *args, **kwargs)
obj.__post_init__(*args, **kwargs)
return obj
# then create a new class with the metaclass passed as an argument
class BaseClass(object, metaclass=PostInitCaller):
def __init__(self, *args, **kwargs):
print(f"{__class__.__name__}.__init__({args}, {kwargs})")
super().__init__()
def __post_init__(self, *args, **kwargs):
print(f"{__class__.__name__}.__post_init__({args}, {kwargs})")
self.configure(*args, **kwargs)
def configure(self, *args, **kwargs):
print(f"{__class__.__name__}.configure({args}, {kwargs})")
class SubClass(BaseClass):
def __init__(self, *args, **kwargs):
print(f"{__class__.__name__}.__init__({args}, {kwargs})")
super().__init__(*args, **kwargs)
def configure(self, *args, **kwargs):
print(f"{__class__.__name__}.configure({args}, {kwargs})")
super().configure(*args, **kwargs)
# when you create an instance
a = SubClass('a', b='b')
running gives:
PostInitCaller.__call__(('a',), {'b': 'b'})
SubClass.__init__(('a',), {'b': 'b'})
BaseClass.__init__(('a',), {'b': 'b'})
BaseClass.__post_init__(('a',), {'b': 'b'})
SubClass.configure(('a',), {'b': 'b'})
BaseClass.configure(('a',), {'b': 'b'})
I know that the metaclass approach is the Pro way, but I've a more readable and easy proposal using #staticmethod:
class Invites(TimestampModel, db.Model):
id = db.Column(db.Integer, primary_key=True, autoincrement=True)
invitee_email = db.Column(db.String(128), nullable=False)
def __init__(self, invitee_email):
invitee_email = invitee_email
#staticmethod
def create_invitation(invitee_email):
"""
Create an invitation
saves it and fetches it because the id
is being generated in the DB
"""
invitation = Invites(invitee_email)
db.session.save(invitation)
db.session.commit()
return Invites.query.filter(
PartnerInvites.invitee_email == invitee_email
).one_or_none()
So I could use it this way:
invitation = Invites.create_invitation("jim#mail.com")
print(invitation.id, invitation.invitee_email)
>>>> 1 jim#mail.com
I would like to provide a decorator that allows for an optional configuration when applied to a function.
A simple implementation follows:
import functools
class Deco(object):
config = {'message': 'hello'}
def __init__(self, func):
self.func = func
functools.wraps(func)(self)
def __call__(self, *args, **kwargs):
print self.config['message']
return self.func(*args, **kwargs)
#classmethod
def customize(cls, **kwargs):
"""Return a customized instance of this class. """
return type(cls.__name__, (Deco, ), {'config': kwargs})
#Deco
def add(a, b):
return a + b
#Deco.customize(message='bye')
def sub(a, b):
return a - b
>>> add(1, 2)
'hello'
>>> sub(2, 1)
'bye'
I would like to use it to provide user-friendly decorators for Django views.
This approach works without errors, but is there something bad about allowing a class to have a static factory method instantiating customized instances of it self, as a decorator?
You could work without creating an extra sub-class for each time the decorator is used there, but your code is fine. The way without extra subclass could be something along:
class Deco(object):
config = {'message': 'hello'}
def __init__(self, func=None, **kwargs):
if kwargs:
self.config = kwargs
if func is not None:
self._decorate(func)
def _decorate(self, func):
self.func = func
functools.wraps(func)(self)
def __call__(self, *args, **kwargs):
if not hasattr(self, "func"):
self._decorate(func)
return self
print self.config['message']
return self.func(*args, **kwargs)
So, while performance wise there would be no difference to your code (unless you would be decorating at least hundreds of thousands of functions - your code create an extra object - a class - for each time the decorator is used, besides the instance of that class) - there is an impact on people would review your code (either to use your modules, or to maintain it after you are done). I mean "a decorator that dynamically generates subclasses of itself" may sound too advanced and scare people away. Although it is as simple as my suggestion above once one understands the mechanisms of class generation in Python as you had.