Develop Reference

A good practice to implement with python multiple inheritance class?

The Scenario:
class A:
def __init__(self, key, secret):
self.key = key
self.secret = secret
def same_name_method(self):
do_some_staff
def method_a(self):
pass
class B:
def __init__(self, key, secret):
self.key = key
self.secret = secret
def same_name_method(self):
do_another_staff
def method_b(self):
pass
class C(A,B):
def __init__(self, *args, **kwargs):
# I want to init both class A and B's key and secret
## I want to rename class A and B's same method
any_ideas()
...
What I Want:
I want the instance of class C initialize both class A and B, because they are different api key.
And I want rename class A and B's same_name_method, so I will not confused at which same_name_method.
What I Have Done:
For problem one, I have done this:
class C(A,B):
def __init__(self, *args, **kwargs):
A.__init__(self, a_api_key,a_api_secret)
B.__init__(self, b_api_key,b_api_secret)
Comment: I know about super(), but for this situation I do not know how to use it.
For problem two, I add a __new__ for class C
def __new__(cls, *args, **kwargs):
cls.platforms = []
cls.rename_method = []
for platform in cls.__bases__:
# fetch platform module name
module_name = platform.__module__.split('.')[0]
cls.platforms.append(module_name)
# rename attr
for k, v in platform.__dict__.items():
if not k.startswith('__'):
setattr(cls, module_name+'_'+k, v)
cls.rename_method.append(k)
for i in cls.rename_method:
delattr(cls, i) ## this line will raise AttributeError!!
return super().__new__(cls)
Comment: because I rename the new method names and add it to cls attr. I need to delete the old method attr, but do not know how to delattr. Now I just leave them alone, did not delete the old methods.
Question:
Any Suggestions?

So, you want some pretty advanced things, some complicated things, and you don't understand well how classes behave in Python.
So, for your first thing: initializing both classes, and every other method that should run in all classes: the correct solution is to make use of cooperative calls to super() methods.
A call to super() in Python returns you a very special proxy objects that reflects all methods available in the next class, obeying the proper method Resolution Order.
So, if A.__init__ and B.__init__ have to be called, both methods should include a super().__init__ call - and one will call the other's __init__ in the appropriate order, regardless of how they are used as bases in subclasses. As object also have __init__, the last super().__init__ will just call it that is a no-op. If you have more methods in your classes that should be run in all base classes, you'd rather build a proper base class so that the top-most super() call don't try to propagate to a non-existing method.
Otherwise, it is just:
class A:
def __init__(self, akey, asecret, **kwargs):
self.key = akey
self.secret = asecret
super().__init__(**kwargs)
class B:
def __init__(self, bkey, bsecret, **kwargs):
self.key = bkey
self.secret = bsecret
super().__init__(**kwargs)
class C(A,B):
# does not even need an explicit `__init__`.
I think you can get the idea. Of course, the parameter names have to differ - ideally, when writing C you don't have to worry about parameter order - but when calling C you have to worry about suplying all mandatory parameters for C and its bases. If you can't rename the parameters in A or B to be distinct, you could try to use the parameter order for the call, though, with each __init__ consuming two position-parameters - but that will require some extra care in inheritance order.
So - up to this point, it is basic Python multiple-inheritance "howto", and should be pretty straightforward. Now comes your strange stuff.
As for the auto-renaming of methods: first things first -
are you quite sure you need inheritance? Maybe having your granular classes for each external service, and a registry and dispatch class that call the methods on the others by composition would be more sane. (I may come back to this later)
Are you aware that __new__ is called for each instantiation of the class, and all class-attribute mangling you are performing there happens at each new instance of your classes?
So, if the needed method-renaming + shadowing needs to take place at class creation time, you can do that using the special method __init_subclass__ that exists from Python 3.6. It is a special class method that is called once for each derived class of the class it is defined on. So, just create a base class, from which A and B themselves will inherit, and move a properly modified version the thing you are putting in __new__ there. If you are not using Python 3.6, this should be done on the __new__ or __init__ of a metaclass, not on the __new__ of the class itself.
Another approach would be to have a custom __getattribute__ method - this could be crafted to provide namespaces for the base classes. It would owrk ony on instances, not on the classes themselves (but could be made to, again, using a metaclass). __getattribute__ can even hide the same-name-methods.
class Base:
#classmethod
def _get_base_modules(cls):
result = {}
for base in cls.__bases__:
module_name = cls.__module__.split(".")[0]
result[module_name] = base
return result
#classmethod
def _proxy(self, module_name):
class base:
def __dir__(base_self):
return dir(self._base_modules[module_name])
def __getattr__(base_self, attr):
original_value = self._base_modules[module_name].__dict__[attr]
if hasattr(original_value, "__get__"):
original_value = original_value.__get__(self, self.__class__)
return original_value
base.__name__ = module_name
return base()
def __init_subclass__(cls):
cls._base_modules = cls._get_base_modules()
cls._shadowed = {name for module_class in cls._base_modules.values() for name in module_class.__dict__ if not name.startswith("_")}
def __getattribute__(self, attr):
if attr.startswith("_"):
return super().__getattribute__(attr)
cls = self.__class__
if attr in cls._shadowed:
raise AttributeError(attr)
if attr in cls._base_modules:
return cls._proxy(attr)
return super().__getattribute__(attr)
def __dir__(self):
return super().dir() + list(self._base_modules)
class A(Base):
...
class B(Base):
...
class C(A, B):
...
As you can see - this is some fun, but starts getting really complicated - and all the hoola-boops that are needed to retrieve the actual attributes from the superclasses after ading an artificial namespace seem to indicate your problem is not calling for using inheritance after all, as I suggested above.
Since you have your small, functional, atomic classes for each "service" , you could use a plain, simple, non-meta-at-all class that would work as a registry for the various services - and you can even enhance it to call the equivalent method in several of the services it is handling with a single call:
class Services:
def __init__(self):
self.registry = {}
def register(self, cls, key, secret):
name = cls.__module__.split(".")[0]
service= cls(key, secret)
self.registry[name] = service
def __getattr__(self, attr):
if attr in self.registry:
return self.registry[attr]

Decorator for a class method that caches return value after first access

My problem, and why
I'm trying to write a decorator for a class method, #cachedproperty. I want it to behave so that when the method is first called, the method is replaced with its return value. I also want it to behave like #property so that it doesn't need to be explicitly called. Basically, it should be indistinguishable from #property except that it's faster, because it only calculates the value once and then stores it. My idea is that this would not slow down instantiation like defining it in __init__ would. That's why I want to do this.
What I tried
First, I tried to override the fget method of the property, but it's read-only.
Next, I figured I'd try to implement a decorator that does needs to be called the first time but then caches the values. This isn't my final goal of a property-type decorator that never needs to be called, but I thought this would be a simpler problem to tackle first. In other words, this is a not-working solution to a slightly simpler problem.
I tried:
def cachedproperty(func):
""" Used on methods to convert them to methods that replace themselves
with their return value once they are called. """
def cache(*args):
self = args[0] # Reference to the class who owns the method
funcname = inspect.stack()[0][3] # Name of the function, so that it can be overridden.
setattr(self, funcname, func()) # Replace the function with its value
return func() # Return the result of the function
return cache
However, this doesn't seem work. I tested this with:
>>> class Test:
... #cachedproperty
... def test(self):
... print "Execute"
... return "Return"
...
>>> Test.test
<unbound method Test.cache>
>>> Test.test()
but I get an error about how the class didn't pass itself to the method:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unbound method cache() must be called with Test instance as first argument (got nothing instead)
At this point, me and my limited knowledge of deep Python methods are very confused, and I have no idea where my code went wrong or how to fix it. (I've never tried to write a decorator before)
The question
How can I write a decorator that will return the result of calling a class method the first time it's accessed (like #property does), and be replaced with a cached value for all subsequent queries?
I hope this question isn't too confusing, I tried to explain it as well as I could.

If you don't mind alternative solutions, I'd recommend lru_cache
for example
from functools import lru_cache
class Test:
#property
#lru_cache(maxsize=None)
def calc(self):
print("Calculating")
return 1
Expected output
In [2]: t = Test()
In [3]: t.calc
Calculating
Out[3]: 1
In [4]: t.calc
Out[4]: 1

First of all Test should be instantiated
test = Test()
Second, there is no need for inspect cause we can get the property name from func.__name__
And third, we return property(cache) to make python to do all the magic.
def cachedproperty(func):
" Used on methods to convert them to methods that replace themselves\
with their return value once they are called. "
def cache(*args):
self = args[0] # Reference to the class who owns the method
funcname = func.__name__
ret_value = func(self)
setattr(self, funcname, ret_value) # Replace the function with its value
return ret_value # Return the result of the function
return property(cache)
class Test:
#cachedproperty
def test(self):
print "Execute"
return "Return"
>>> test = Test()
>>> test.test
Execute
'Return'
>>> test.test
'Return'
>>>
"""

With Python 3.8 or later you can use functools.cached_property().
It works similar as the previously proposed lru_cache solution.
Example usage:
import functools
class Test:
#functools.cached_property
def calc(self):
print("Calculating")
return 1
Test output:
In [2]: t = Test()
In [3]: t.calc
Calculating
Out[3]: 1
In [4]: t.calc
Out[4]: 1

I think you're better off with a custom descriptor, since this is exactly the kind of thing descriptors are for. Like so:
class CachedProperty:
def __init__(self, name, get_the_value):
self.name = name
self.get_the_value = get_the_value
def __get__(self, obj, typ):
name = self.name
while True:
try:
return getattr(obj, name)
except AttributeError:
get_the_value = self.get_the_value
try:
# get_the_value can be a string which is the name of an obj method
value = getattr(obj, get_the_value)()
except AttributeError:
# or it can be another external function
value = get_the_value()
setattr(obj, name, value)
continue
break
class Mine:
cached_property = CachedProperty("_cached_property ", get_cached_property_value)
# OR:
class Mine:
cached_property = CachedProperty("_cached_property", "get_cached_property_value")
def get_cached_property_value(self):
return "the_value"
EDIT: By the way, you don't even actually need a custom descriptor. You could just cache the value inside of your property function. E.g.:
#property
def test(self):
while True:
try:
return self._test
except AttributeError:
self._test = get_initial_value()
That's all there is to it.
However, many would consider this a bit of an abuse of property, and to be an unexpected way of using it. And unexpected usually means you should do it another, more explicit way. A custom CachedProperty descriptor is very explicit, so for that reason I would prefer it to the property approach, though it requires more code.

Django's version of this decorator does exactly what you describe and is simple, so besides my comment I'll just copy it here:
class cached_property(object):
"""
Decorator that converts a method with a single self argument into a
property cached on the instance.
Optional ``name`` argument allows you to make cached properties of other
methods. (e.g. url = cached_property(get_absolute_url, name='url') )
"""
def __init__(self, func, name=None):
self.func = func
self.__doc__ = getattr(func, '__doc__')
self.name = name or func.__name__
def __get__(self, instance, type=None):
if instance is None:
return self
res = instance.__dict__[self.name] = self.func(instance)
return res
(source).
As you can see, it uses func.name to determine the name of the function (no need to fiddle with inspect.stack) and it replaces the method with its result by mutating instance.__dict__. So subsequent "calls" are just an attribute lookup and there is no need for any caches, et cetera.

You can use something like this:
def cached(timeout=None):
def decorator(func):
def wrapper(self, *args, **kwargs):
value = None
key = '_'.join([type(self).__name__, str(self.id) if hasattr(self, 'id') else '', func.__name__])
if settings.CACHING_ENABLED:
value = cache.get(key)
if value is None:
value = func(self, *args, **kwargs)
if settings.CACHING_ENABLED:
# if timeout=None Django cache reads a global value from settings
cache.set(key, value, timeout=timeout)
return value
return wrapper
return decorator
When adding to the cache dictionary it generates keys based on the convention class_id_function in case you are caching entities and the property could possibly return a different value for each one.
It also checks a settings key CACHING_ENABLED in case you want to turn it off temporarily when doing benchmarks.
But it does not encapsulate the standard property decorator so you should still call it like a function, or you can use it like this (why restrict it to properties only):
#cached
#property
def total_sales(self):
# Some calculations here...
pass
Also it may be worth noting that in case you are caching a result from lazy foreign key relationships, there are times depending on your data where it would be faster to simply run an aggregate function when doing your select query and fetching everything at once, than visiting the cache for every record in your result-set. So use some tool like django-debug-toolbar for your framework to compare what performs best in your scenario.

#functools.lru_cache()
def func(....):
....
Reference: #functools.lru_cache() | Python

Have u tried djangos built in:
from django.utils.functional import cached_property
please don't use lru_cache as suggested by multiple people as it opens up a host of possible memory leak issues

Trying to program a "to_class" general decorator

Suppose I have defined:
def to_class(cls):
""" returns a decorator
aimed to force the result to be of class cls. """
def decorating_func(func):
def wrapper(*args, **kwargs):
return cls(func(*args, **kwargs))
return wrapper
return decorator(decorating_func)
I wish to use it to create decorators which turn function results to objects of a given class. However, this will not work:
class TestClass(object):
def __init__(self, value):
self._value = (value, value)
def __str__(self):
return str(self._value)
#staticmethod
#to_test_class
def test_func(value):
return value
to_test_class = to_class(TestClass)
as test_func will look for to_test_class and will not find it. On the other hand, putting the assignment to to_test_class before the class definition will fail as well, as TestClass will not be defined yet.
Trying to put #to_class(TestClass) above the definition of test_func will also fail, as the method is constructed before the class (if I am not wrong).
The only workaround I have found is to define to_test_class manually as a decorator, and not as one returned from the general "to_class" def.
It might be important to mention that this is only a basic example, but I wish to use to_class for many applications, such as modifying the returned value before 'plugging' it into the class' constructor; and I wish to use it as a decorator for other class' methods as well.
I am sure some think a "to_class" decorator is pointless; manipulations can be done within the decorated method, instead. Though, I find it convenient, and it helps me with readability.
Finally I wish to add that this interests me 20% for practical reasons and 80% for studying reasons, as I find this is something I do not fully understand about decorators in Python in general.

Indeed, at class construction time, the class object itself has not yet been constructed, thus you cannot use it as the basis of a decorator.
One work-around I can think of, is to not use the staticmethod decorator. Instead, internally in your own decorator, re-use the classmethod decorator. That way you ensure that Python at the very least passes in the associated class for you:
def to_class(func):
""" returns a decorator
aimed to force the result to be of class cls. """
def wrapper(cls, *args, **kwargs):
return cls(func(*args, **kwargs))
return classmethod(wrapper)
Then use it like this:
class TestClass(object):
def __init__(self, value):
self._value = (value, value)
def __str__(self):
return str(self._value)
#to_class
def test_func(value):
return value
Demonstration:
>>> def to_class(func):
... """ returns a decorator
... aimed to force the result to be of class cls. """
... def wrapper(cls, *args, **kwargs):
... return cls(func(*args, **kwargs))
... return classmethod(wrapper)
...
>>> class TestClass(object):
... def __init__(self, value):
... self._value = (value, value)
... def __str__(self):
... return str(self._value)
... #to_class
... def test_func(value):
... return value
...
>>> TestClass.test_func('foo')
<__main__.TestClass object at 0x102a77210>
>>> print TestClass.test_func('foo')
('foo', 'foo')
A generic version of your decorator is not easy; the only other workaround to your conundrum is to use a metaclass hack; see another answer of mine where I describe the method in more detail.
You basically need to reach into the class-under-construction namespace, set a temporary metaclass, and then rely on there being at least one instance of the class before your decorator will work; the temporary metaclass approach hooks into the class creation mechanisms to retrieve the constructed class at a later time.
Seeing as you are using this decorator as an alternative class factory however, that is probably not going to be ideal; if someone used your decorated functions to create class instances exclusively the metaclass would be called too late.

Well, you forgot that class is the first parameter passed to method decorated with classmethod, so you can write it like this:
def to_this_class(func):
def wrapped(cls, value):
res = func(cls, value)
return cls(res)
return wrapped
class TestClass(object):
def __init__(self, value):
self._value = (value, value)
def __str__(self):
return str(self._value)
#classmethod
#to_this_class
def test_func(cls, value):
return value
x = TestClass('a')
print x.test_func('b')

The problem is that a decorator gets evaluated upon defining the thing it decorates, so when defining the method test_func(), the decorator to_test_class gets called, and even if it already exists, the thing it shall work on (the class TestClass) does not exist yet (as this is created after all methods are created).
Maybe you can use a placeholder at the point where the class is used and later (after the class is created) fill in that value (the class) at the point of the placeholder.
Example:
lazyClasses = {}
def to_lazy_class(className):
""" returns a decorator
aimed to force the result to be of class cls. """
def decorating_func(func):
def wrapper(*args, **kwargs):
return lazyClasses[className](func(*args, **kwargs))
return wrapper
return decorating_func
class TestClass(object):
def __init__(self, value):
self._value = (value, value)
def __str__(self):
return str(self._value)
#staticmethod
#to_lazy_class('TestClass')
def test_func(value):
return value
lazyClasses['TestClass'] = TestClass
>>> TestClass.test_func('hallo')
<__main__.TestClass object at 0x7f76d8cba190>

Python name of class in class body

Is it possible to get the class name within the body of a class definition?
For example,
class Foo():
x = magic() # x should now be 'Foo'
I know that I can do this statically outside of the class body using a class method:
class Bar():
#classmethod
def magic(cls):
print cls.__name__
Bar.magic()
However this isn't what I want, I want the class name in the class body

Ok - got one more solution - this one is actually not that complex!
import traceback
def magic():
return traceback.extract_stack()[-2][2]
class Something(object):
print magic()
It will print out "Something". I'm not sure if extracted stack format is standardised in any way, but it works for python 2.6 (and 2.7 and 3.1)

AFAIK, the class object is not available until the class definition has been "executed", so it's not possible to get it during class definition.
If you need the class name for later use but don't use it during class definition (e.g. to compute other field names, or some such thing), then you can still automate the process using a class decorator.
def classname ( field ):
def decorator ( klass ):
setattr(klass, field, klass.__name__)
return klass
return decorator
(Caveat: not tested.)
With this definition, you can get something like:
#classname(field='x')
class Foo:
pass
and you would get field x with the class name in it, as in:
print Foo.x

Here you have a working solution for your specific case, but beware (I wrote it mainly to demonstrate that it IS indeed possible to do something like this):
You shouldn't use it
It is very specific
It has many limitations
I was just having fun with this
It is black magic
It may not work for your use case
It is not threadsafe
Do I have already said that you shouldn't use it?
Anyway, here you have the code:
import inspect
def NameAwareClassType():
frameInfo = inspect.getouterframes(inspect.currentframe())[1]
codeContext = frameInfo[4][0]
className = codeContext.split(' ', 1)[1].split('(', 1)[0]
class ClassNameGlobalRemoverType(type):
def __new__(mcs, name, bases, dict):
if name == className:
del globals()['__clsname__']
return type.__new__(mcs, name, bases, dict)
class NameAwareClass(object):
__metaclass__ = ClassNameGlobalRemoverType
globals()['__clsname__'] = className
return NameAwareClass
class A(NameAwareClassType()):
print __clsname__
def __init__(self):
pass
print __clsname__
Edit: https://gist.github.com/1085475 — there you have a version which allows to use __clsname__ during method execution; makes not much sense, as self.__class__.__name__ is a better approach and the __clsname__ variable does not hold a string anymore (I'm having fun experimenting with this)

I don't know of an elegant way to do this in Python 2.x -- but it's an interpreted language which means that something relatively simple along the following lines will do what you want and would be safe if you're sure of the code being executed:
classdef = """\
class %(classname)s(object):
x = '%(classname)s'
print x
"""
exec classdef % {'classname': 'Foo'}
foo = Foo()
print foo

class Bar():
#classmethod
def magic(cls):
return cls.__name__
#property
def x(self):
return self.magic()
def y(self):
return self.x
>>> a = Bar()
>>> a.x
'Bar'
>>> a.y()
'Bar'
This way you can use x as an attribute, at least within any instance and static methods. In class methods, you can just get the class name from the cls attribute anyway.

Accessing the class that owns a decorated method from the decorator

I'm writing a decorator for methods that must inspect the parent methods (the methods of the same name in the parents of the class in which I'm decorating).
Example (from the fourth example of PEP 318):
def returns(rtype):
def check_returns(f):
def new_f(*args, **kwds):
result = f(*args, **kwds)
assert isinstance(result, rtype), \
"return value %r does not match %s" % (result,rtype)
return result
new_f.func_name = f.func_name
# here I want to reach the class owning the decorated method f,
# it should give me the class A
return new_f
return check_returns
class A(object):
#returns(int)
def compute(self, value):
return value * 3
So I'm looking for the code to type in place of # here I want...
Thanks.

As bobince said it, you can't access the surrounding class, because at the time the decorator is invoked, the class does not exist yet. If you need access to the full dictionary of the class and the bases, you should consider a metaclass:
__metaclass__
This variable can be any callable accepting arguments for name, bases, and dict. Upon class creation, the callable is used instead of the built-in type().
Basically, we convert the returns decorator into something that just tells the metaclass to do some magic on class construction:
class CheckedReturnType(object):
def __init__(self, meth, rtype):
self.meth = meth
self.rtype = rtype
def returns(rtype):
def _inner(f):
return CheckedReturnType(f, rtype)
return _inner
class BaseInspector(type):
def __new__(mcs, name, bases, dct):
for obj_name, obj in dct.iteritems():
if isinstance(obj, CheckedReturnType):
# do your wrapping & checking here, base classes are in bases
# reassign to dct
return type.__new__(mcs, name, bases, dct)
class A(object):
__metaclass__ = BaseInspector
#returns(int)
def compute(self, value):
return value * 3
Mind that I have not tested this code, please leave comments if I should update this.
There are some articles on metaclasses by the highly recommendable David Mertz, which you might find interesting in this context.

here I want to reach the class owning the decorated method f
You can't because at the point of decoration, no class owns the method f.
class A(object):
#returns(int)
def compute(self, value):
return value * 3
Is the same as saying:
class A(object):
pass
#returns(int)
def compute(self, value):
return value*3
A.compute= compute
Clearly, the returns() decorator is built before the function is assigned to an owner class.
Now when you write a function to a class (either inline, or explicitly like this) it becomes an unbound method object. Now it has a reference to its owner class, which you can get by saying:
>>> A.compute.im_class
<class '__main__.A'>
So you can read f.im_class inside ‘new_f’, which is executed after the assignment, but not in the decorator itself.
(And even then it's a bit ugly relying on a CPython implementation detail if you don't need to. I'm not quite sure what you're trying to do, but things involving “get the owner class” are often doable using metaclasses.)