I am working with htcondor python bindings (https://htcondor.readthedocs.io/en/latest/apis/python-bindings/index.html)
You don't need to know htcondor, but for reference I am working with the
htcondor.JobEvent class to get the data that I want.
From the description of that object it follows that it behaves like a dictionary,
but it has no __dict__ property.
Basically you can't tell what this object is, because it's translated from C++ into python, hence I want to wrap it with all it's functionalities to add more functionalities.
The way I am solving this atm is:
class HTCJobEventWrapper:
"""
Wrapper for HTCondor JobEvent.
Extracts event number and time_stamp of an event.
The wrapped event can be printed to the terminal for dev purpose.
:param job_event: HTCJobEvent
"""
def __init__(self, job_event: HTCJobEvent):
self.wrapped_class = job_event
self.event_number = job_event.get('EventTypeNumber')
self.time_stamp = date_time.strptime(
job_event.get('EventTime'),
STRP_FORMAT
)
def __getattr__(self, attr):
return getattr(self.wrapped_class, attr)
def get(self, *args, **kwargs):
"""Wraps wrapped_class get function."""
return self.wrapped_class.get(*args, **kwargs)
def items(self):
"""Wraps wrapped_class items method."""
return self.wrapped_class.items()
def keys(self):
"""Wraps wrapped_class keys method."""
return self.wrapped_class.keys()
def values(self):
"""Wraps wrapped_class values method."""
return self.wrapped_class.values()
def to_dict(self):
"""Turns wrapped_class items into a dictionary."""
return dict(self.items())
def __repr__(self):
return json.dumps(
self.to_dict(),
indent=2
)
With this it's possible to get any attribute and to use the methods described in the documentation.
However as you can see HTCJobEventWrapper is not of type htcondor.JobEvent
and is not inheriting from it.
If you try to instatiate a htcondor.JobEvent class it results in the following error: RuntimeError: This class cannot be instantiated from Python.
What I want:
I would like it to be a child class which copies a given htcondor.JobEvent object completely
and adds the functionalities I want and returns a HTCJobEventWrapper object
This kind of relates to this question: completely wrap an object in python
Is there a pythonic way to dynamically call every attribute, function or method on self.wrapped_class ? Just like with getattr ?
But in this case I've tried getattr but it works only for attributes.
Related
I have two classes, Manager and DataManager, simplified in the example below:
import numpy as np
class Manager:
def __init__(self, value, delay_init=True):
self.value = value
self.is_init = False
self.data = None
if not delay_init:
self._initialize()
#staticmethod
def delayed_init(fn):
def wrapped_delayed_init(obj, *args, **kwargs):
if not obj.is_init:
obj.data = np.random.randn(obj.value, obj.value)
obj.is_init = True
return fn(obj, *args, **kwargs)
return wrapped_delayed_init
#delayed_init.__get__(object)
def _initialize(self):
pass
class DataManager(Manager):
def __init__(self, value):
super().__init__(value)
#Manager.delayed_init
def calculate_mean(self):
return np.mean(self.data)
data_manager = DataManager(100)
assert data_manager.data is None
mean = data_manager.calculate_mean()
What my code needs to do is pass the method calculate as an argument to some other function as part of a test suite. In order to do this I need to create an instance of DataManager. What I must avoid is the time incurred by the full instance creation (since it involved downloading data), and so I delegate this task to some function in the parent class called delayed_init. There are a subset of methods belonging to DataManager that require this delayed_init to have been run, and so I choose to decorate them with delayed_init to ensure it is run whenever 1) another method requires it and 2) it has not already been run.
Now my problem: Currently it appears I need to explicitly define the decorator as #Manager.delayed_init, but this can be re-written as #<parent>.delayed_init. I would like to write it this way if possible given that in my opinion it is cleaner to not have to explicitly write out a given type if the type is always the parent. However, I cannot find a way to properly reference the parent class before an instance/object is created. Is it possible to access the parent class without the creation of any instances?
Thank you for the assistance.
I have an external API that I cannot modify. For each call to this API, I need to be able to perform an operation before and after. This API is used like this:
def get_api():
"""
Return an initiated ClassAPI object
"""
token = Token.objects.last()
api = ClassAPI(
settings.CLASS_API_ID,
settings.CLASS_API_SECRET,
last_token)
return api
get_api() is called everywhere in the code and the result is then used to perform request (like: api.book_store.get(id=book_id)).
My goal is to return a virtual object that will perform the same operations than the ClassAPI adding print "Before" and print "After".
The ClassAPI looks like this:
class ClassAPI
class BookStore
def get(...)
def list(...)
class PenStore
def get(...)
def list(...)
I tried to create a class inheriting from (ClassApi, object) [as ClassAPI doesn't inherit from object] and add to this class a metaclass that decorates all the methods, but I cannot impact the methods from BookStore (get and list)
Any idea about how to perform this modifying only get_api() and adding additional classes? I try to avoid copying the whole structure of the API to add my operations.
I am using python 2.7.
You could do this with a Proxy:
class Proxy:
def __init__(self, other):
self.other = other
self.calls = []
def __getattr__(self, name):
self.calls.append(name)
return self
def __call__(self, *args, **kwargs):
self.before()
ret = self.call_proxied(*args, **kwargs)
self.after()
return ret
def call_proxied(self, *args, **kwargs):
other = self.other
calls = self.calls
self.calls = []
for item in calls:
other = getattr(other, item)
return other(*args, **kwargs)
This class intercepts unknown members in the __getattr__() method, saving the names that are used and returning itself.
When a method is called (eg. api.book_store.get(id=book_id) ), it calls a before() and after() method on itself and in between it fetches the members of other and forwards the arguments in a call.
You use this class like this:
def get_api():
...
return Proxy(api)
Update: corrected the call to self.call_proxied(*args, **kwargs). Also allow any return value to be returned.
I have a class "Wrapper". Wrapper inherits from object, and it's primary responsibility is to hook method calls to selenium during a test. It's __init__ method signature is
__init__(self, object_to_wrap, actions={}, **kwargs)
Part of it's functionality is to take a dictionary as an argument in the init method, and to expose the keys as attributes on the object_to_wrap object by defining the __getattr__ magic method
(signature: __getattr__(self, item):)
and checking self.actions's keys for item. If it is found, the method is invoked.
In the test code, the initialization would look similar to:
def navigate(scoped_self, to=''):
self.driver.switch_to_default_content()
self.driver.switch_to.frame(to)
scoped_self.navigations.append(to)
# the navigate method is scoped in an
# instance method of the test class, so it has access to self
So, my question is, how can I make the scope, or self in the above method, be the scope of my Wrapper class, and not the test class?
For clarity, if the solution I am looking for in this question was found, the navigate implementation would change to be:
def navigate(scoped_self, to=''):
self.object_to_wrap.switch_to_default_content()
self.object_to_wrap.switch_to.frame(to)
Also for clarity, I'm pretty sure what I'm looking for is exactly what Javascripts Function.prototype.bind accomplishes.
UPDATE: Defining the methods, such as navigate, inside the Wrapper class is not an option, as the Wrapper class cannot have test-specific logic. The test suite is done on n > 1 DOM, that are completely unrelated. For example, one of the tests requires the method "alert_handlers" (to overwrite the window.alert function and return the alerts presented as a string), another might require the navigate method, and a third might require both.
UPDATE #2: Thanks to the answer below from c17r, I realized that without the getattribute code included in my example, that it would appear as if I was asking for something we had already accomplished. What I am looking for is the ability, in the above navigate method, is for scoped_self to be the instance of Wrapper.
Furthermore, I am specifically looking for a way to "dynamically" pass scoped_self to the functions (the way that Function.prototype.bind"dynamically" sets this where this is myEventListener.bind(window); if you're unfamiliar with javascript, make an event listener on the body and console.log this without binding, and then with binding, to see the difference)
Furthermore, I figured it might help to give an example implementation without the solution I am looking for. This is currently working as expected:
class Wrapper(object):
def __init__(self, wrapped, actions={}):
self.wrapped = wrapped
self.actions = actions
self.navigations = [] # EXAMPLE, SEE THE TEST CLASS CODE
def __getattr__(self, item):
if item in self.actions:
return self.actions[item]
# do other fancy stuff here
# UPDATE #2: added for clarity. this is the current implementation
orig_attr = self.wrapped.__getattribute__(item)
if callable(orig_attr):
def hooked(*args, **kwargs):
self.pre(item, *args, **kwargs)
self.err = False
try:
result = orig_attr(*args, **kwargs)
except Exception as e:
#logs
self.post(*args, **kwargs)
raise
if type(self.wrapped) == type(result):
return self
return result
return hooked
else:
return orig_attr
class SomeTest():
#blah blah init stuff, set self.driver = selenium.webdriver.Phantomjs
def spawn_actions(self):
def navigate(scoped_self, to=''):
self.driver.switch_to_default_content()
self.driver.switch_to.frame(to)
scoped_self.navigations.append(to) # <--- appended to wrapper.navigations
return {'navigate': navigate}
def run(self):
driver = Wrapper(self.driver, self.spawn_actions())
driver.get(url)
driver.navigate('New Request')
# fun tests!
If I understand you correctly, 3 things:
Wrapper would need to pass any unknown functions down to the wrapped item, otherwise driver.get() won't work properly.
The navigate function needs self as the first parameter, like it would if it was actually defined on Wrapper
Wrapper needs to pass self into the dict-based function. This is a little tricky since __getattr__ doesn't actually call the function, so you need to return a function that calls the underlying function properly, similar to how decorators work.
Like this:
class Driver(object):
def get(self, url):
print('get')
print(repr(self))
print(repr(url))
print('--')
class Wrapper(object):
def __init__(self, wrapped, actions={}):
self.wrapped = wrapped
self.actions = actions
def __getattr__(self, item):
if item in self.actions:
def unwrap(*args, **kwargs):
return self.actions[item](self, *args, **kwargs)
return unwrap
else:
return getattr(self.wrapped, item)
class Test(object):
def __init__(self):
self.driver = Driver();
def spawn_actions(self):
def navigate(self, to=''):
print('navigate')
print(repr(self))
print(repr(to))
print(repr(self.wrapped))
print('--')
return {
'navigate': navigate
}
def run(self):
driver = Wrapper(self.driver, self.spawn_actions())
driver.get('url')
driver.navigate('thing')
Now calling:
t = Test()
t.run()
Outputs:
get
<__main__.Driver object at 0x104008630>
'url'
--
navigate
<__main__.Wrapper object at 0x104008ba8>
'thing'
<__main__.Driver object at 0x104008630>
--
EDIT
You can also dynamically bind the methods to the instance instead of __getattr__ returning the unwrap function:
import types
class Wrapper(object):
def __init__(self, wrapped, actions={}):
self.wrapped = wrapped
for name, func in actions.items():
setattr(self, name, types.MethodType(func, self))
def __getattr__(self, item):
return getattr(self.wrapped, item)
Suppose I have defined:
def to_class(cls):
""" returns a decorator
aimed to force the result to be of class cls. """
def decorating_func(func):
def wrapper(*args, **kwargs):
return cls(func(*args, **kwargs))
return wrapper
return decorator(decorating_func)
I wish to use it to create decorators which turn function results to objects of a given class. However, this will not work:
class TestClass(object):
def __init__(self, value):
self._value = (value, value)
def __str__(self):
return str(self._value)
#staticmethod
#to_test_class
def test_func(value):
return value
to_test_class = to_class(TestClass)
as test_func will look for to_test_class and will not find it. On the other hand, putting the assignment to to_test_class before the class definition will fail as well, as TestClass will not be defined yet.
Trying to put #to_class(TestClass) above the definition of test_func will also fail, as the method is constructed before the class (if I am not wrong).
The only workaround I have found is to define to_test_class manually as a decorator, and not as one returned from the general "to_class" def.
It might be important to mention that this is only a basic example, but I wish to use to_class for many applications, such as modifying the returned value before 'plugging' it into the class' constructor; and I wish to use it as a decorator for other class' methods as well.
I am sure some think a "to_class" decorator is pointless; manipulations can be done within the decorated method, instead. Though, I find it convenient, and it helps me with readability.
Finally I wish to add that this interests me 20% for practical reasons and 80% for studying reasons, as I find this is something I do not fully understand about decorators in Python in general.
Indeed, at class construction time, the class object itself has not yet been constructed, thus you cannot use it as the basis of a decorator.
One work-around I can think of, is to not use the staticmethod decorator. Instead, internally in your own decorator, re-use the classmethod decorator. That way you ensure that Python at the very least passes in the associated class for you:
def to_class(func):
""" returns a decorator
aimed to force the result to be of class cls. """
def wrapper(cls, *args, **kwargs):
return cls(func(*args, **kwargs))
return classmethod(wrapper)
Then use it like this:
class TestClass(object):
def __init__(self, value):
self._value = (value, value)
def __str__(self):
return str(self._value)
#to_class
def test_func(value):
return value
Demonstration:
>>> def to_class(func):
... """ returns a decorator
... aimed to force the result to be of class cls. """
... def wrapper(cls, *args, **kwargs):
... return cls(func(*args, **kwargs))
... return classmethod(wrapper)
...
>>> class TestClass(object):
... def __init__(self, value):
... self._value = (value, value)
... def __str__(self):
... return str(self._value)
... #to_class
... def test_func(value):
... return value
...
>>> TestClass.test_func('foo')
<__main__.TestClass object at 0x102a77210>
>>> print TestClass.test_func('foo')
('foo', 'foo')
A generic version of your decorator is not easy; the only other workaround to your conundrum is to use a metaclass hack; see another answer of mine where I describe the method in more detail.
You basically need to reach into the class-under-construction namespace, set a temporary metaclass, and then rely on there being at least one instance of the class before your decorator will work; the temporary metaclass approach hooks into the class creation mechanisms to retrieve the constructed class at a later time.
Seeing as you are using this decorator as an alternative class factory however, that is probably not going to be ideal; if someone used your decorated functions to create class instances exclusively the metaclass would be called too late.
Well, you forgot that class is the first parameter passed to method decorated with classmethod, so you can write it like this:
def to_this_class(func):
def wrapped(cls, value):
res = func(cls, value)
return cls(res)
return wrapped
class TestClass(object):
def __init__(self, value):
self._value = (value, value)
def __str__(self):
return str(self._value)
#classmethod
#to_this_class
def test_func(cls, value):
return value
x = TestClass('a')
print x.test_func('b')
The problem is that a decorator gets evaluated upon defining the thing it decorates, so when defining the method test_func(), the decorator to_test_class gets called, and even if it already exists, the thing it shall work on (the class TestClass) does not exist yet (as this is created after all methods are created).
Maybe you can use a placeholder at the point where the class is used and later (after the class is created) fill in that value (the class) at the point of the placeholder.
Example:
lazyClasses = {}
def to_lazy_class(className):
""" returns a decorator
aimed to force the result to be of class cls. """
def decorating_func(func):
def wrapper(*args, **kwargs):
return lazyClasses[className](func(*args, **kwargs))
return wrapper
return decorating_func
class TestClass(object):
def __init__(self, value):
self._value = (value, value)
def __str__(self):
return str(self._value)
#staticmethod
#to_lazy_class('TestClass')
def test_func(value):
return value
lazyClasses['TestClass'] = TestClass
>>> TestClass.test_func('hallo')
<__main__.TestClass object at 0x7f76d8cba190>
I've been trying to pickle an object which contains references to static class methods.
Pickle fails (for example on module.MyClass.foo) stating it cannot be pickled, as module.foo does not exist.
I have come up with the following solution, using a wrapper object to locate the function upon invocation, saving the container class and function name:
class PicklableStaticMethod(object):
"""Picklable version of a static method.
Typical usage:
class MyClass:
#staticmethod
def doit():
print "done"
# This cannot be pickled:
non_picklable = MyClass.doit
# This can be pickled:
picklable = PicklableStaticMethod(MyClass.doit, MyClass)
"""
def __init__(self, func, parent_class):
self.func_name = func.func_name
self.parent_class = parent_class
def __call__(self, *args, **kwargs):
func = getattr(self.parent_class, self.func_name)
return func(*args, **kwargs)
I am wondering though, is there a better - more standard way - to pickle such an object?
I do not want to make changes to the global pickle process (using copy_reg for example), but the following pattern would be great:
class MyClass(object):
#picklable_staticmethod
def foo():
print "done."
My attempts at this were unsuccessful, specifically because I could not extract the owner class from the foo function. I was even willing to settle for explicit specification (such as #picklable_staticmethod(MyClass)) but I don't know of any way to refer to the MyClass class right where it's being defined.
Any ideas would be great!
Yonatan
This seems to work.
class PickleableStaticMethod(object):
def __init__(self, fn, cls=None):
self.cls = cls
self.fn = fn
def __call__(self, *args, **kwargs):
return self.fn(*args, **kwargs)
def __get__(self, obj, cls):
return PickleableStaticMethod(self.fn, cls)
def __getstate__(self):
return (self.cls, self.fn.__name__)
def __setstate__(self, state):
self.cls, name = state
self.fn = getattr(self.cls, name).fn
The trick is to snag the class when the static method is gotten from it.
Alternatives: You could use metaclassing to give all your static methods a .__parentclass__ attribute. Then you could subclass Pickler and give each subclass instance its own .dispatch table which you can then modify without affecting the global dispatch table (Pickler.dispatch). Pickling, unpickling, and calling the method might then be a little faster.
EDIT: modified after Jason comment.
I think python is correct in not letting pickling a staticmethod object - as it is impossible to pickle instance or class methods! Such an object would make very little sense outside of its context:
Check this: Descriptor Tutorial
import pickle
def dosomething(a, b):
print a, b
class MyClass(object):
dosomething = staticmethod(dosomething)
o = MyClass()
pickled = pickle.dumps(dosomething)
This works, and that's what should be done - define a function, pickle it, and use such function as a staticmethod in a certain class.
If you've got an use case for your need, please write it down and I'll be glad to discuss it.