Develop Reference

A good practice to implement with python multiple inheritance class?

The Scenario:
class A:
def __init__(self, key, secret):
self.key = key
self.secret = secret
def same_name_method(self):
do_some_staff
def method_a(self):
pass
class B:
def __init__(self, key, secret):
self.key = key
self.secret = secret
def same_name_method(self):
do_another_staff
def method_b(self):
pass
class C(A,B):
def __init__(self, *args, **kwargs):
# I want to init both class A and B's key and secret
## I want to rename class A and B's same method
any_ideas()
...
What I Want:
I want the instance of class C initialize both class A and B, because they are different api key.
And I want rename class A and B's same_name_method, so I will not confused at which same_name_method.
What I Have Done:
For problem one, I have done this:
class C(A,B):
def __init__(self, *args, **kwargs):
A.__init__(self, a_api_key,a_api_secret)
B.__init__(self, b_api_key,b_api_secret)
Comment: I know about super(), but for this situation I do not know how to use it.
For problem two, I add a __new__ for class C
def __new__(cls, *args, **kwargs):
cls.platforms = []
cls.rename_method = []
for platform in cls.__bases__:
# fetch platform module name
module_name = platform.__module__.split('.')[0]
cls.platforms.append(module_name)
# rename attr
for k, v in platform.__dict__.items():
if not k.startswith('__'):
setattr(cls, module_name+'_'+k, v)
cls.rename_method.append(k)
for i in cls.rename_method:
delattr(cls, i) ## this line will raise AttributeError!!
return super().__new__(cls)
Comment: because I rename the new method names and add it to cls attr. I need to delete the old method attr, but do not know how to delattr. Now I just leave them alone, did not delete the old methods.
Question:
Any Suggestions?

So, you want some pretty advanced things, some complicated things, and you don't understand well how classes behave in Python.
So, for your first thing: initializing both classes, and every other method that should run in all classes: the correct solution is to make use of cooperative calls to super() methods.
A call to super() in Python returns you a very special proxy objects that reflects all methods available in the next class, obeying the proper method Resolution Order.
So, if A.__init__ and B.__init__ have to be called, both methods should include a super().__init__ call - and one will call the other's __init__ in the appropriate order, regardless of how they are used as bases in subclasses. As object also have __init__, the last super().__init__ will just call it that is a no-op. If you have more methods in your classes that should be run in all base classes, you'd rather build a proper base class so that the top-most super() call don't try to propagate to a non-existing method.
Otherwise, it is just:
class A:
def __init__(self, akey, asecret, **kwargs):
self.key = akey
self.secret = asecret
super().__init__(**kwargs)
class B:
def __init__(self, bkey, bsecret, **kwargs):
self.key = bkey
self.secret = bsecret
super().__init__(**kwargs)
class C(A,B):
# does not even need an explicit `__init__`.
I think you can get the idea. Of course, the parameter names have to differ - ideally, when writing C you don't have to worry about parameter order - but when calling C you have to worry about suplying all mandatory parameters for C and its bases. If you can't rename the parameters in A or B to be distinct, you could try to use the parameter order for the call, though, with each __init__ consuming two position-parameters - but that will require some extra care in inheritance order.
So - up to this point, it is basic Python multiple-inheritance "howto", and should be pretty straightforward. Now comes your strange stuff.
As for the auto-renaming of methods: first things first -
are you quite sure you need inheritance? Maybe having your granular classes for each external service, and a registry and dispatch class that call the methods on the others by composition would be more sane. (I may come back to this later)
Are you aware that __new__ is called for each instantiation of the class, and all class-attribute mangling you are performing there happens at each new instance of your classes?
So, if the needed method-renaming + shadowing needs to take place at class creation time, you can do that using the special method __init_subclass__ that exists from Python 3.6. It is a special class method that is called once for each derived class of the class it is defined on. So, just create a base class, from which A and B themselves will inherit, and move a properly modified version the thing you are putting in __new__ there. If you are not using Python 3.6, this should be done on the __new__ or __init__ of a metaclass, not on the __new__ of the class itself.
Another approach would be to have a custom __getattribute__ method - this could be crafted to provide namespaces for the base classes. It would owrk ony on instances, not on the classes themselves (but could be made to, again, using a metaclass). __getattribute__ can even hide the same-name-methods.
class Base:
#classmethod
def _get_base_modules(cls):
result = {}
for base in cls.__bases__:
module_name = cls.__module__.split(".")[0]
result[module_name] = base
return result
#classmethod
def _proxy(self, module_name):
class base:
def __dir__(base_self):
return dir(self._base_modules[module_name])
def __getattr__(base_self, attr):
original_value = self._base_modules[module_name].__dict__[attr]
if hasattr(original_value, "__get__"):
original_value = original_value.__get__(self, self.__class__)
return original_value
base.__name__ = module_name
return base()
def __init_subclass__(cls):
cls._base_modules = cls._get_base_modules()
cls._shadowed = {name for module_class in cls._base_modules.values() for name in module_class.__dict__ if not name.startswith("_")}
def __getattribute__(self, attr):
if attr.startswith("_"):
return super().__getattribute__(attr)
cls = self.__class__
if attr in cls._shadowed:
raise AttributeError(attr)
if attr in cls._base_modules:
return cls._proxy(attr)
return super().__getattribute__(attr)
def __dir__(self):
return super().dir() + list(self._base_modules)
class A(Base):
...
class B(Base):
...
class C(A, B):
...
As you can see - this is some fun, but starts getting really complicated - and all the hoola-boops that are needed to retrieve the actual attributes from the superclasses after ading an artificial namespace seem to indicate your problem is not calling for using inheritance after all, as I suggested above.
Since you have your small, functional, atomic classes for each "service" , you could use a plain, simple, non-meta-at-all class that would work as a registry for the various services - and you can even enhance it to call the equivalent method in several of the services it is handling with a single call:
class Services:
def __init__(self):
self.registry = {}
def register(self, cls, key, secret):
name = cls.__module__.split(".")[0]
service= cls(key, secret)
self.registry[name] = service
def __getattr__(self, attr):
if attr in self.registry:
return self.registry[attr]

Decorator for a class method that caches return value after first access

My problem, and why
I'm trying to write a decorator for a class method, #cachedproperty. I want it to behave so that when the method is first called, the method is replaced with its return value. I also want it to behave like #property so that it doesn't need to be explicitly called. Basically, it should be indistinguishable from #property except that it's faster, because it only calculates the value once and then stores it. My idea is that this would not slow down instantiation like defining it in __init__ would. That's why I want to do this.
What I tried
First, I tried to override the fget method of the property, but it's read-only.
Next, I figured I'd try to implement a decorator that does needs to be called the first time but then caches the values. This isn't my final goal of a property-type decorator that never needs to be called, but I thought this would be a simpler problem to tackle first. In other words, this is a not-working solution to a slightly simpler problem.
I tried:
def cachedproperty(func):
""" Used on methods to convert them to methods that replace themselves
with their return value once they are called. """
def cache(*args):
self = args[0] # Reference to the class who owns the method
funcname = inspect.stack()[0][3] # Name of the function, so that it can be overridden.
setattr(self, funcname, func()) # Replace the function with its value
return func() # Return the result of the function
return cache
However, this doesn't seem work. I tested this with:
>>> class Test:
... #cachedproperty
... def test(self):
... print "Execute"
... return "Return"
...
>>> Test.test
<unbound method Test.cache>
>>> Test.test()
but I get an error about how the class didn't pass itself to the method:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unbound method cache() must be called with Test instance as first argument (got nothing instead)
At this point, me and my limited knowledge of deep Python methods are very confused, and I have no idea where my code went wrong or how to fix it. (I've never tried to write a decorator before)
The question
How can I write a decorator that will return the result of calling a class method the first time it's accessed (like #property does), and be replaced with a cached value for all subsequent queries?
I hope this question isn't too confusing, I tried to explain it as well as I could.

If you don't mind alternative solutions, I'd recommend lru_cache
for example
from functools import lru_cache
class Test:
#property
#lru_cache(maxsize=None)
def calc(self):
print("Calculating")
return 1
Expected output
In [2]: t = Test()
In [3]: t.calc
Calculating
Out[3]: 1
In [4]: t.calc
Out[4]: 1

First of all Test should be instantiated
test = Test()
Second, there is no need for inspect cause we can get the property name from func.__name__
And third, we return property(cache) to make python to do all the magic.
def cachedproperty(func):
" Used on methods to convert them to methods that replace themselves\
with their return value once they are called. "
def cache(*args):
self = args[0] # Reference to the class who owns the method
funcname = func.__name__
ret_value = func(self)
setattr(self, funcname, ret_value) # Replace the function with its value
return ret_value # Return the result of the function
return property(cache)
class Test:
#cachedproperty
def test(self):
print "Execute"
return "Return"
>>> test = Test()
>>> test.test
Execute
'Return'
>>> test.test
'Return'
>>>
"""

With Python 3.8 or later you can use functools.cached_property().
It works similar as the previously proposed lru_cache solution.
Example usage:
import functools
class Test:
#functools.cached_property
def calc(self):
print("Calculating")
return 1
Test output:
In [2]: t = Test()
In [3]: t.calc
Calculating
Out[3]: 1
In [4]: t.calc
Out[4]: 1

I think you're better off with a custom descriptor, since this is exactly the kind of thing descriptors are for. Like so:
class CachedProperty:
def __init__(self, name, get_the_value):
self.name = name
self.get_the_value = get_the_value
def __get__(self, obj, typ):
name = self.name
while True:
try:
return getattr(obj, name)
except AttributeError:
get_the_value = self.get_the_value
try:
# get_the_value can be a string which is the name of an obj method
value = getattr(obj, get_the_value)()
except AttributeError:
# or it can be another external function
value = get_the_value()
setattr(obj, name, value)
continue
break
class Mine:
cached_property = CachedProperty("_cached_property ", get_cached_property_value)
# OR:
class Mine:
cached_property = CachedProperty("_cached_property", "get_cached_property_value")
def get_cached_property_value(self):
return "the_value"
EDIT: By the way, you don't even actually need a custom descriptor. You could just cache the value inside of your property function. E.g.:
#property
def test(self):
while True:
try:
return self._test
except AttributeError:
self._test = get_initial_value()
That's all there is to it.
However, many would consider this a bit of an abuse of property, and to be an unexpected way of using it. And unexpected usually means you should do it another, more explicit way. A custom CachedProperty descriptor is very explicit, so for that reason I would prefer it to the property approach, though it requires more code.

Django's version of this decorator does exactly what you describe and is simple, so besides my comment I'll just copy it here:
class cached_property(object):
"""
Decorator that converts a method with a single self argument into a
property cached on the instance.
Optional ``name`` argument allows you to make cached properties of other
methods. (e.g. url = cached_property(get_absolute_url, name='url') )
"""
def __init__(self, func, name=None):
self.func = func
self.__doc__ = getattr(func, '__doc__')
self.name = name or func.__name__
def __get__(self, instance, type=None):
if instance is None:
return self
res = instance.__dict__[self.name] = self.func(instance)
return res
(source).
As you can see, it uses func.name to determine the name of the function (no need to fiddle with inspect.stack) and it replaces the method with its result by mutating instance.__dict__. So subsequent "calls" are just an attribute lookup and there is no need for any caches, et cetera.

You can use something like this:
def cached(timeout=None):
def decorator(func):
def wrapper(self, *args, **kwargs):
value = None
key = '_'.join([type(self).__name__, str(self.id) if hasattr(self, 'id') else '', func.__name__])
if settings.CACHING_ENABLED:
value = cache.get(key)
if value is None:
value = func(self, *args, **kwargs)
if settings.CACHING_ENABLED:
# if timeout=None Django cache reads a global value from settings
cache.set(key, value, timeout=timeout)
return value
return wrapper
return decorator
When adding to the cache dictionary it generates keys based on the convention class_id_function in case you are caching entities and the property could possibly return a different value for each one.
It also checks a settings key CACHING_ENABLED in case you want to turn it off temporarily when doing benchmarks.
But it does not encapsulate the standard property decorator so you should still call it like a function, or you can use it like this (why restrict it to properties only):
#cached
#property
def total_sales(self):
# Some calculations here...
pass
Also it may be worth noting that in case you are caching a result from lazy foreign key relationships, there are times depending on your data where it would be faster to simply run an aggregate function when doing your select query and fetching everything at once, than visiting the cache for every record in your result-set. So use some tool like django-debug-toolbar for your framework to compare what performs best in your scenario.

#functools.lru_cache()
def func(....):
....
Reference: #functools.lru_cache() | Python

Have u tried djangos built in:
from django.utils.functional import cached_property
please don't use lru_cache as suggested by multiple people as it opens up a host of possible memory leak issues

Access a class attribute inside a python class decorator

EDIT: i found this method decorator and was able to use it to individually wrap the methods (omitting __init__) of ClassA and ClassB. however, instead of manually wrapping individual methods, i'd like to just wrap the class.
i've created my own logging class, MyLogger, which inherits logging.Logger. in this class, (among other things) i have a FileHandler which prints the logger name in its output:
import logging
class MyLogger(logging.Logger):
def __init__(self, name, path="output.log"):
logging.Logger.__init__(self, name, logging.DEBUG)
logpath = path
fh = logging.FileHandler(logpath)
fh.setLevel(logging.DEBUG)
fh.setFormatter(logging.Formatter("%(name)s - %(message)s"))
# stream handler omitted
self.addHandler(fh)
i also have ClassA and ClassB, which both get the same instance of MyLogger:
class ClassA(object):
def __init__(self, mylogger):
self.log = mylogger
def fn1(self):
self.log.debug("message1 from ClassA fn1")
self.fn2()
b = ClassB(self.log)
b.fn1()
self.log.debug("message2 from ClassA fn1")
def fn2(self):
self.log.debug("message1 from ClassA fn2")
# many more functions
class ClassB(object):
def __init__(self, mylogger):
self.log = mylogger
def fn1(self):
self.log.debug("message1 from ClassB fn1")
# many more functions
here's a simple "main" function:
print "inside main"
log = MyLogger("main")
a = ClassA(log)
a.fn1()
because the MyLogger instance is being passed around, i'd like to ensure the log name (i'm just using the class name) is printed correctly by each function. so i'm attempting to decorate all methods of each class so that the the previous log name is remembered, then the log name is set to the name of the class, the method is run, and finally the log name is set back to what it previously was. i'm using the decorator/descriptor from here. for the sake of brevity, i will only post my changes to it. i renamed the decorator setlogger, have added print statements inside each method in the descript class, and have altered make_bound as follows:
def make_bound(self, instance):
print "in __BOUND__"
#functools.wraps(self.f)
def wrapper(*args, **kwargs):
'''This documentation will disapear :)'''
prev = instance.log.name
print "about to wrap %s.%s, prev = %s" % (instance.__class__.__name__, self.f.__name__, prev)
ret = self.f(instance, *args, **kwargs)
instance.log.name = prev
print "done wrapping %s.%s, now = %s" % (instance.__class__.__name__, self.f.__name__, prev)
return ret
# This instance does not need the descriptor anymore,
# let it find the wrapper directly next time:
setattr(instance, self.f.__name__, wrapper)
return wrapper
if i use the setlogger decorator/descriptor to wrap individual methods in ClassA and ClassB, it works fine. however, i'd like to just wrap the two classes. so here's my class decorator:
def setloggerforallmethods(cls):
def decorate(*args, **kwargs):
for name, m in inspect.getmembers(cls, inspect.ismethod):
if name != "__init__":
print "calling setattr on %s.%s" % (cls.__name__, name)
setattr(cls, name, setlogger(m))
return cls
return decorate
if i wrap ClassA and ClassB with #setloggerforallmethods, and run the main function, heres the output:
inside main
calling setattr on ClassA.fn1
in __INIT__: f = fn1
calling setattr on ClassA.fn2
in __INIT__: f = fn2
in __GET__
in __UNBOUND__
Traceback (most recent call last):
File "/ws/maleva-rcd/yacht/classa.py", line 23, in <module>
a.fn1()
File "/ws/maleva-rcd/yacht/yachtlogger.py", line 34, in wrapper
self.f.__name__)
ValueError: zero length field name in format
i dont understand why fn1 is unbound at this time. isnt it bound to a as in a.fn1()?

I think you're trying to solve the wrong problem in the wrong way. But I can explain why your code isn't doing what you're trying to make it do.
First, in your decorator, you do this:
for name, fn in inspect.getmembers(cls, inspect.ismethod):
if name != "__init__":
print "calling setlogger on %s" % cls.__name__ + "." + name
fn = setlogger(fn)
That has no effect. For each bound method fn, you create a wrapper function, then rebind the local variable fn to that function. That has no more effect than doing this:
def foo(a):
a = 3
i = 0
foo(i)
If you want to set an attribute on the class, you have to set an attribute on the class, like this:
setattr(cls, name, setlogger(fn))
Now your wrapper will get called.
Next, cls.log is a class attribute named log—that is, an attribute on the class itself, which is shared by all instances of that class. But all of the code within the classes uses instance attributes, where each instance has its own copy. That's what you get when you assign self.log in your __init__. So, there is no class attribute named log, meaning you'll just get this:
AttributeError: type object 'ClassA' has no attribute 'log'
You could of course create a class attribute… but that won't do any good. The instance attribute of the same name will just shadow it.
You need to access the instance attribute inside inner, which means you need a self to access it off. And you obviously don't have self inside setlogger. But think about what you're doing: you're wrapping a method with another method. Methods get self as their first argument. In fact, if you modify inner to print out its args, you'll see that the first one is always something like <__main__.ClassA object at 0x12345678>. So:
def inner(self, *args, **kwargs):
prevname = self.log.name
self.log.name = cls.__name__
ret = func(self, *args, **kwargs) # don't forget to forward self
self.log.name = prevname
return ret
But if any of these wrapped methods ever raises an exception, they'll leave the name in the wrong state. So really, you need to either create a context manager for stashing and restoring the value, or just a try/finally. Which also happens to make the wrapper a little easier to write:
def inner(self, *args, **kwargs):
prevname = self.log.name
self.log.name = cls.__name__
try:
return func(self, *args, **kwargs)
finally:
self.log.name = prevname
Finally, you need to remove the self.log.name = in each __init__ method. Otherwise, when you construct a B instance in the middle of A.fn1, you're changing the logger's name without going through the wrapper that restores the previous name.
Again, I don't think this is a good solution. But it will do what you're trying to do.

I still don't completely understand the problem you're trying to solve, but I think it's this:
Constructing a MyLogger takes two pieces of information: a name, and a path. You don't want every class to have to know that path. So, you figured you needed to share the MyLogger instance, because there's no other way around that. And then, because the MyLogger stores its name as an attribute, you had to hack up that attribute in wrappers around every method.
But there is a much simpler way around that: Make your classes take a "logger factory"—that is, a callable which constructs an appropriate logger for them—instead of a logger. The MyLogger class itself already is such a callable, since it takes a default value for path and you just use it. But let's pretend that weren't true, and you wanted to use some non-default path. Still easy; you just need to wrap it up:
class ClassA(object):
def __init__(self, log_factory):
self.log_factory = log_factory
self.log = log_factory("ClassA")
def fn1(self):
# ...
b = ClassB(self.log_factory)
# ...
class ClassB(object):
def __init__(self, log_factory):
self.log_factory = log_factory
self.log = log_factory("ClassB")
# ...
# or just log_factory = functools.partial(MyLogger, log="output.log")
def log_factory(name):
return MyLogger(name, "output.log")
a = ClassA(log_factory)
a.fn1()
You may notice that the __init__ method in both classes does the same thing. So, why not extract it into a mixin base class?
class LogUserMixin(object):
def __init__(self, log_factory):
self.log_factory = log_factory
self.log = log_factory(self.__class__.__name__)
Now:
class ClassA(LogUserMixin):
def fn1(self):
# ...
When it's a ClassA being initialized, self.__class__ will be "ClassA", not "LogUserMixin", so this does exactly what you want. It works even if your real classes already have base classes, or a hierarchy of subclasses, or if they do additional stuff in __init__, or take additional arguments; you just need to do a tiny bit more work in some of those cases.

Is it best-practice to place __init__ in the beginning or end of a class?

Consider the following code:
class AClass():
def defaultMethod(self):
return 1
def __init__(self, methodToUse = defaultMethod):
print (methodToUse(self))
if __name__== "__main__":
AClass()
In this case one cannot move the defaultMethod below the __init__ method, if I do, it causes "NameError: name 'defaultMethod' is not defined"
This means that I need to define this method before the __init__ or else Python does not know about it. This again, means that I no longer have __init__ as the first method, which leaves me to wonder whether it is usual to place the __init__ method at the end of a class or in the beginning.

What do you mean, "I need to define this method before the init or else Python does not know about it" ?
>>> class A(object):
... def __init__(self):
... self.foo()
... def foo(self):
... print '42'
...
>>> A()
42
I usually place __ init__() before other instance methods, but after class methods/property/attributes.

I think you're doing things a little peculiarly. You should still put __init__ high up if not the first method. Readability is key and __init__ exposes what you expect the main instance fields to be.
Here are three alternatives. My preference is for the first as it documents the default method and will require the least modification to your code. The last works, but could be confusing for anyone having to maintain your code.
class A(object):
def __init__(self, method="foo"):
if callable(method):
method(self)
else:
getattr(self, method)()
def foo(self):
print "something"
class B(object):
def __init__(self, method = None):
if method is None:
self.defaultMethod()
else:
method(self)
def defaultMethod(self):
print "foo"
def _defaultMethod(self):
print self.x
class C(object):
def __init__(self, method = _defaultMethod):
self.x = "bleh"
method(self)
def anotherMethod(self):
print "doing something else"
def defaultMethodProxy(self):
_defaultMethod(self)

__init__ is most commonly placed at the beginning of a class since they are the first thing run when the class is instantiated. Since your situation requires it to exist further down in the class, it would be nice to other devs to leave a note in the comments for the class.

I prefer init at the beginning and I would actually not write the class that way, but rather something like this:
class AClass():
def __init__(self, methodToUse = 'defaultMethod'):
print getattr(self, methodToUse)()
def defaultMethod(self):
return 1
if __name__== "__main__":
AClass()
The problem is that at compile time (when the default arguments are created), there is no function defaultMethod, but if you use it inside __init__, then the method is there.

How to access the parent class during initialisation in python?

How do I find out which class I am initialising a decorator in? It makes sense that I wouldn't be able to find this out as the decorator is not yet bound to the class, but is there a way of getting round this?
class A(object):
def dec(f):
# I am in class 'A'
def func(cls):
f(cls)
return func
#dec
def test(self):
pass
I need to know which class I am (indicated by the commented line).

I don't think this is possible. At the very moment when you define test, the class doesn't exist yet.
When Python encounters
class A(object):
it creates a new namespace in which it runs all code that it finds in the class definition (including the definition of test() and the call to the decorator), and when it's done, it creates a new class object and puts everything into this class that was left in the namespace after the code was executed.
So when the decorator is called, it doesn't know anything yet. At this moment, test is just a function.

I don't get the question.
>>> class A(object):
def dec(f):
def func(cls):
print cls
return func
#dec
def test(self):
pass
>>> a=A()
>>> a.test()
<__main__.A object at 0x00C56330>
>>>
The argument (cls) is the class, A.

As Nadia pointed out you will need to be more specific. Python does not allow this kind of things, which means that what you are trying to do is probably something wrong.
In the meantime, here is my contribution: a little story about a sailor and a frog. (use a constructor after the class initialization)
class Cruise(object):
def arewelostyet(self):
print 'Young sailor: I think I am lost, help me :s'
instance = Cruise()
instance.arewelostyet()
def whereami(lostfunc):
"""
decorator
"""
def decorated(*args, **kwargs):
lostfunc(*args, **kwargs)
print 'Frog: Crôak! thou art sailing in class', lostfunc.im_class.__name__
# don't forget to write name and doc
decorated.func_name = lostfunc.func_name
decorated.func_doc = lostfunc.func_name
return decorated
print '[i]A frog pops out of nowhere[/i]'
# decorate the method:
Cruise.arewelostyet = whereami(Cruise.arewelostyet)
instance.arewelostyet()