Develop Reference

A good practice to implement with python multiple inheritance class?

The Scenario:
class A:
def __init__(self, key, secret):
self.key = key
self.secret = secret
def same_name_method(self):
do_some_staff
def method_a(self):
pass
class B:
def __init__(self, key, secret):
self.key = key
self.secret = secret
def same_name_method(self):
do_another_staff
def method_b(self):
pass
class C(A,B):
def __init__(self, *args, **kwargs):
# I want to init both class A and B's key and secret
## I want to rename class A and B's same method
any_ideas()
...
What I Want:
I want the instance of class C initialize both class A and B, because they are different api key.
And I want rename class A and B's same_name_method, so I will not confused at which same_name_method.
What I Have Done:
For problem one, I have done this:
class C(A,B):
def __init__(self, *args, **kwargs):
A.__init__(self, a_api_key,a_api_secret)
B.__init__(self, b_api_key,b_api_secret)
Comment: I know about super(), but for this situation I do not know how to use it.
For problem two, I add a __new__ for class C
def __new__(cls, *args, **kwargs):
cls.platforms = []
cls.rename_method = []
for platform in cls.__bases__:
# fetch platform module name
module_name = platform.__module__.split('.')[0]
cls.platforms.append(module_name)
# rename attr
for k, v in platform.__dict__.items():
if not k.startswith('__'):
setattr(cls, module_name+'_'+k, v)
cls.rename_method.append(k)
for i in cls.rename_method:
delattr(cls, i) ## this line will raise AttributeError!!
return super().__new__(cls)
Comment: because I rename the new method names and add it to cls attr. I need to delete the old method attr, but do not know how to delattr. Now I just leave them alone, did not delete the old methods.
Question:
Any Suggestions?

So, you want some pretty advanced things, some complicated things, and you don't understand well how classes behave in Python.
So, for your first thing: initializing both classes, and every other method that should run in all classes: the correct solution is to make use of cooperative calls to super() methods.
A call to super() in Python returns you a very special proxy objects that reflects all methods available in the next class, obeying the proper method Resolution Order.
So, if A.__init__ and B.__init__ have to be called, both methods should include a super().__init__ call - and one will call the other's __init__ in the appropriate order, regardless of how they are used as bases in subclasses. As object also have __init__, the last super().__init__ will just call it that is a no-op. If you have more methods in your classes that should be run in all base classes, you'd rather build a proper base class so that the top-most super() call don't try to propagate to a non-existing method.
Otherwise, it is just:
class A:
def __init__(self, akey, asecret, **kwargs):
self.key = akey
self.secret = asecret
super().__init__(**kwargs)
class B:
def __init__(self, bkey, bsecret, **kwargs):
self.key = bkey
self.secret = bsecret
super().__init__(**kwargs)
class C(A,B):
# does not even need an explicit `__init__`.
I think you can get the idea. Of course, the parameter names have to differ - ideally, when writing C you don't have to worry about parameter order - but when calling C you have to worry about suplying all mandatory parameters for C and its bases. If you can't rename the parameters in A or B to be distinct, you could try to use the parameter order for the call, though, with each __init__ consuming two position-parameters - but that will require some extra care in inheritance order.
So - up to this point, it is basic Python multiple-inheritance "howto", and should be pretty straightforward. Now comes your strange stuff.
As for the auto-renaming of methods: first things first -
are you quite sure you need inheritance? Maybe having your granular classes for each external service, and a registry and dispatch class that call the methods on the others by composition would be more sane. (I may come back to this later)
Are you aware that __new__ is called for each instantiation of the class, and all class-attribute mangling you are performing there happens at each new instance of your classes?
So, if the needed method-renaming + shadowing needs to take place at class creation time, you can do that using the special method __init_subclass__ that exists from Python 3.6. It is a special class method that is called once for each derived class of the class it is defined on. So, just create a base class, from which A and B themselves will inherit, and move a properly modified version the thing you are putting in __new__ there. If you are not using Python 3.6, this should be done on the __new__ or __init__ of a metaclass, not on the __new__ of the class itself.
Another approach would be to have a custom __getattribute__ method - this could be crafted to provide namespaces for the base classes. It would owrk ony on instances, not on the classes themselves (but could be made to, again, using a metaclass). __getattribute__ can even hide the same-name-methods.
class Base:
#classmethod
def _get_base_modules(cls):
result = {}
for base in cls.__bases__:
module_name = cls.__module__.split(".")[0]
result[module_name] = base
return result
#classmethod
def _proxy(self, module_name):
class base:
def __dir__(base_self):
return dir(self._base_modules[module_name])
def __getattr__(base_self, attr):
original_value = self._base_modules[module_name].__dict__[attr]
if hasattr(original_value, "__get__"):
original_value = original_value.__get__(self, self.__class__)
return original_value
base.__name__ = module_name
return base()
def __init_subclass__(cls):
cls._base_modules = cls._get_base_modules()
cls._shadowed = {name for module_class in cls._base_modules.values() for name in module_class.__dict__ if not name.startswith("_")}
def __getattribute__(self, attr):
if attr.startswith("_"):
return super().__getattribute__(attr)
cls = self.__class__
if attr in cls._shadowed:
raise AttributeError(attr)
if attr in cls._base_modules:
return cls._proxy(attr)
return super().__getattribute__(attr)
def __dir__(self):
return super().dir() + list(self._base_modules)
class A(Base):
...
class B(Base):
...
class C(A, B):
...
As you can see - this is some fun, but starts getting really complicated - and all the hoola-boops that are needed to retrieve the actual attributes from the superclasses after ading an artificial namespace seem to indicate your problem is not calling for using inheritance after all, as I suggested above.
Since you have your small, functional, atomic classes for each "service" , you could use a plain, simple, non-meta-at-all class that would work as a registry for the various services - and you can even enhance it to call the equivalent method in several of the services it is handling with a single call:
class Services:
def __init__(self):
self.registry = {}
def register(self, cls, key, secret):
name = cls.__module__.split(".")[0]
service= cls(key, secret)
self.registry[name] = service
def __getattr__(self, attr):
if attr in self.registry:
return self.registry[attr]

cls behaviour in inherited classmethod of a decorated class

I'm trying to make some validations for the class methods of a class using one of the parameters used when calling them.
To do this, I'm using a decorator for the class that will apply a decorator to the required methods, which will perform a validation function using one of the parameters in the function.
This all works well for the base class (for this example I will call it Parent).
However, if I make another class which inherits Parent, (for this example I will call it Child), the inherited decorated classmethod no longer behaves normally.
The cls parameter inside the classmethod for the Child class is not Child as expected, but is Parent instead.
Taking the following example
import inspect
def is_number(word):
if word.isdigit():
print('Validation passed')
else:
raise Exception('Validation failed')
class ClassDecorator(object):
def __init__(self, *args):
self.validators = args
def __decorateMethod(self):
def wrapped(method):
def wrapper(cls, word, *args, **kwargs):
for validator in self.validators:
validator(word)
return method(word, *args, **kwargs)
return wrapper
return wrapped
def __call__(self, cls):
for name, method in inspect.getmembers(cls):
if name == 'shout':
decoratedMethod = self.__decorateMethod()(method)
setattr(cls, name, classmethod(decoratedMethod))
return cls
#ClassDecorator(is_number)
class Parent(object):
#classmethod
def shout(cls, word):
print('{} is shouting {}'.format(cls, word))
#classmethod
def say(cls):
print('{} is talking'.format(cls))
class Child(Parent):
pass
Parent.shout('123')
Child.shout('321')
Will result in the following output:
Validation passed
<class '__main__.Parent'> is shouting 123
Validation passed
<class '__main__.Parent'> is shouting 321
My questions are:
Why does the classmethod for Child get called with Parent as cls
Is it possible using this design to get the wanted behaviour?
P.S.: I've tried this on both Python 2.7.10 and Python 3.5.2 and have gotten the same behaviour

You are decorating the bound class method; it is this object that holds on to Parent and passes it into the original shout function when called; whatever cls is bound to in your wrapper() method is not passed in and ignored.
Unwrap classmethods first, you can get to the underlying function object with the __func__ attribute:
def __call__(self, cls):
for name, method in inspect.getmembers(cls):
if name == 'shout':
decoratedMethod = self.__decorateMethod()(method.__func__)
setattr(cls, name, classmethod(decoratedMethod))
return cls
You now have to take into account that your wrapper is handling an unbound function too, so pass on the cls argument or manually bind:
# pass in cls explicitly:
return method(cls, word, *args, **kwargs)
# or bind the descriptor manually:
return method.__get__(cls)(word, *args, **kwargs)

Trying to program a "to_class" general decorator

Suppose I have defined:
def to_class(cls):
""" returns a decorator
aimed to force the result to be of class cls. """
def decorating_func(func):
def wrapper(*args, **kwargs):
return cls(func(*args, **kwargs))
return wrapper
return decorator(decorating_func)
I wish to use it to create decorators which turn function results to objects of a given class. However, this will not work:
class TestClass(object):
def __init__(self, value):
self._value = (value, value)
def __str__(self):
return str(self._value)
#staticmethod
#to_test_class
def test_func(value):
return value
to_test_class = to_class(TestClass)
as test_func will look for to_test_class and will not find it. On the other hand, putting the assignment to to_test_class before the class definition will fail as well, as TestClass will not be defined yet.
Trying to put #to_class(TestClass) above the definition of test_func will also fail, as the method is constructed before the class (if I am not wrong).
The only workaround I have found is to define to_test_class manually as a decorator, and not as one returned from the general "to_class" def.
It might be important to mention that this is only a basic example, but I wish to use to_class for many applications, such as modifying the returned value before 'plugging' it into the class' constructor; and I wish to use it as a decorator for other class' methods as well.
I am sure some think a "to_class" decorator is pointless; manipulations can be done within the decorated method, instead. Though, I find it convenient, and it helps me with readability.
Finally I wish to add that this interests me 20% for practical reasons and 80% for studying reasons, as I find this is something I do not fully understand about decorators in Python in general.

Indeed, at class construction time, the class object itself has not yet been constructed, thus you cannot use it as the basis of a decorator.
One work-around I can think of, is to not use the staticmethod decorator. Instead, internally in your own decorator, re-use the classmethod decorator. That way you ensure that Python at the very least passes in the associated class for you:
def to_class(func):
""" returns a decorator
aimed to force the result to be of class cls. """
def wrapper(cls, *args, **kwargs):
return cls(func(*args, **kwargs))
return classmethod(wrapper)
Then use it like this:
class TestClass(object):
def __init__(self, value):
self._value = (value, value)
def __str__(self):
return str(self._value)
#to_class
def test_func(value):
return value
Demonstration:
>>> def to_class(func):
... """ returns a decorator
... aimed to force the result to be of class cls. """
... def wrapper(cls, *args, **kwargs):
... return cls(func(*args, **kwargs))
... return classmethod(wrapper)
...
>>> class TestClass(object):
... def __init__(self, value):
... self._value = (value, value)
... def __str__(self):
... return str(self._value)
... #to_class
... def test_func(value):
... return value
...
>>> TestClass.test_func('foo')
<__main__.TestClass object at 0x102a77210>
>>> print TestClass.test_func('foo')
('foo', 'foo')
A generic version of your decorator is not easy; the only other workaround to your conundrum is to use a metaclass hack; see another answer of mine where I describe the method in more detail.
You basically need to reach into the class-under-construction namespace, set a temporary metaclass, and then rely on there being at least one instance of the class before your decorator will work; the temporary metaclass approach hooks into the class creation mechanisms to retrieve the constructed class at a later time.
Seeing as you are using this decorator as an alternative class factory however, that is probably not going to be ideal; if someone used your decorated functions to create class instances exclusively the metaclass would be called too late.

Well, you forgot that class is the first parameter passed to method decorated with classmethod, so you can write it like this:
def to_this_class(func):
def wrapped(cls, value):
res = func(cls, value)
return cls(res)
return wrapped
class TestClass(object):
def __init__(self, value):
self._value = (value, value)
def __str__(self):
return str(self._value)
#classmethod
#to_this_class
def test_func(cls, value):
return value
x = TestClass('a')
print x.test_func('b')

The problem is that a decorator gets evaluated upon defining the thing it decorates, so when defining the method test_func(), the decorator to_test_class gets called, and even if it already exists, the thing it shall work on (the class TestClass) does not exist yet (as this is created after all methods are created).
Maybe you can use a placeholder at the point where the class is used and later (after the class is created) fill in that value (the class) at the point of the placeholder.
Example:
lazyClasses = {}
def to_lazy_class(className):
""" returns a decorator
aimed to force the result to be of class cls. """
def decorating_func(func):
def wrapper(*args, **kwargs):
return lazyClasses[className](func(*args, **kwargs))
return wrapper
return decorating_func
class TestClass(object):
def __init__(self, value):
self._value = (value, value)
def __str__(self):
return str(self._value)
#staticmethod
#to_lazy_class('TestClass')
def test_func(value):
return value
lazyClasses['TestClass'] = TestClass
>>> TestClass.test_func('hallo')
<__main__.TestClass object at 0x7f76d8cba190>

Pickling a staticmethod in Python

I've been trying to pickle an object which contains references to static class methods.
Pickle fails (for example on module.MyClass.foo) stating it cannot be pickled, as module.foo does not exist.
I have come up with the following solution, using a wrapper object to locate the function upon invocation, saving the container class and function name:
class PicklableStaticMethod(object):
"""Picklable version of a static method.
Typical usage:
class MyClass:
#staticmethod
def doit():
print "done"
# This cannot be pickled:
non_picklable = MyClass.doit
# This can be pickled:
picklable = PicklableStaticMethod(MyClass.doit, MyClass)
"""
def __init__(self, func, parent_class):
self.func_name = func.func_name
self.parent_class = parent_class
def __call__(self, *args, **kwargs):
func = getattr(self.parent_class, self.func_name)
return func(*args, **kwargs)
I am wondering though, is there a better - more standard way - to pickle such an object?
I do not want to make changes to the global pickle process (using copy_reg for example), but the following pattern would be great:
class MyClass(object):
#picklable_staticmethod
def foo():
print "done."
My attempts at this were unsuccessful, specifically because I could not extract the owner class from the foo function. I was even willing to settle for explicit specification (such as #picklable_staticmethod(MyClass)) but I don't know of any way to refer to the MyClass class right where it's being defined.
Any ideas would be great!
Yonatan

This seems to work.
class PickleableStaticMethod(object):
def __init__(self, fn, cls=None):
self.cls = cls
self.fn = fn
def __call__(self, *args, **kwargs):
return self.fn(*args, **kwargs)
def __get__(self, obj, cls):
return PickleableStaticMethod(self.fn, cls)
def __getstate__(self):
return (self.cls, self.fn.__name__)
def __setstate__(self, state):
self.cls, name = state
self.fn = getattr(self.cls, name).fn
The trick is to snag the class when the static method is gotten from it.
Alternatives: You could use metaclassing to give all your static methods a .__parentclass__ attribute. Then you could subclass Pickler and give each subclass instance its own .dispatch table which you can then modify without affecting the global dispatch table (Pickler.dispatch). Pickling, unpickling, and calling the method might then be a little faster.

EDIT: modified after Jason comment.
I think python is correct in not letting pickling a staticmethod object - as it is impossible to pickle instance or class methods! Such an object would make very little sense outside of its context:
Check this: Descriptor Tutorial
import pickle
def dosomething(a, b):
print a, b
class MyClass(object):
dosomething = staticmethod(dosomething)
o = MyClass()
pickled = pickle.dumps(dosomething)
This works, and that's what should be done - define a function, pickle it, and use such function as a staticmethod in a certain class.
If you've got an use case for your need, please write it down and I'll be glad to discuss it.

Accessing the class that owns a decorated method from the decorator

I'm writing a decorator for methods that must inspect the parent methods (the methods of the same name in the parents of the class in which I'm decorating).
Example (from the fourth example of PEP 318):
def returns(rtype):
def check_returns(f):
def new_f(*args, **kwds):
result = f(*args, **kwds)
assert isinstance(result, rtype), \
"return value %r does not match %s" % (result,rtype)
return result
new_f.func_name = f.func_name
# here I want to reach the class owning the decorated method f,
# it should give me the class A
return new_f
return check_returns
class A(object):
#returns(int)
def compute(self, value):
return value * 3
So I'm looking for the code to type in place of # here I want...
Thanks.

As bobince said it, you can't access the surrounding class, because at the time the decorator is invoked, the class does not exist yet. If you need access to the full dictionary of the class and the bases, you should consider a metaclass:
__metaclass__
This variable can be any callable accepting arguments for name, bases, and dict. Upon class creation, the callable is used instead of the built-in type().
Basically, we convert the returns decorator into something that just tells the metaclass to do some magic on class construction:
class CheckedReturnType(object):
def __init__(self, meth, rtype):
self.meth = meth
self.rtype = rtype
def returns(rtype):
def _inner(f):
return CheckedReturnType(f, rtype)
return _inner
class BaseInspector(type):
def __new__(mcs, name, bases, dct):
for obj_name, obj in dct.iteritems():
if isinstance(obj, CheckedReturnType):
# do your wrapping & checking here, base classes are in bases
# reassign to dct
return type.__new__(mcs, name, bases, dct)
class A(object):
__metaclass__ = BaseInspector
#returns(int)
def compute(self, value):
return value * 3
Mind that I have not tested this code, please leave comments if I should update this.
There are some articles on metaclasses by the highly recommendable David Mertz, which you might find interesting in this context.

here I want to reach the class owning the decorated method f
You can't because at the point of decoration, no class owns the method f.
class A(object):
#returns(int)
def compute(self, value):
return value * 3
Is the same as saying:
class A(object):
pass
#returns(int)
def compute(self, value):
return value*3
A.compute= compute
Clearly, the returns() decorator is built before the function is assigned to an owner class.
Now when you write a function to a class (either inline, or explicitly like this) it becomes an unbound method object. Now it has a reference to its owner class, which you can get by saying:
>>> A.compute.im_class
<class '__main__.A'>
So you can read f.im_class inside ‘new_f’, which is executed after the assignment, but not in the decorator itself.
(And even then it's a bit ugly relying on a CPython implementation detail if you don't need to. I'm not quite sure what you're trying to do, but things involving “get the owner class” are often doable using metaclasses.)