If you have multiple layers of inheritance and know that a particular variable exists, is there a way to trace back to where the variable originated? Without having to navigate backwards by looking through each file and classes. Possibly calling some sort of function that will do it?
Example:
parent.py
class parent(object):
def __init__(self):
findMe = "Here I am!"
child.py
from parent import parent
class child(parent):
pass
grandson.py
from child import child
class grandson(child):
def printVar(self):
print self.findMe
Try to locate where the findMe variable came from with a function call.
If the "variable" is an instance variable - , so , if at any point in chain of __init__ methods you do:
def __init__(self):
self.findMe = "Here I am!"
It is an instance variable from that point on, and cannot, for all effects, be made distinct of any other instance variable. (Unless you put in place a mechanism, like a class with a special __setattr__ method, that will keep track of attributes changing, and introspect back which part of the code set the attribute - see last example on this answer)
Please also note that on your example,
class parent(object):
def __init__(self):
findMe = "Here I am!"
findMe is defined as a local variable to that method and does not even exist after __init__ is finished.
Now, if your variable is set as a class attribute somewhere on the inheritance chain:
class parent(object):
findMe = False
class childone(parent):
...
It is possible to find the class where findMe is defined by introspecting each class' __dict__ in the MRO (method resolution order) chain . Of course, there is no way, and no sense, in doing that without introspecting all classes in the MRO chain - except if one keeps track of attributes as defined, like in the example bellow this - but introspecting the MRO itself is a oneliner in Python:
def __init__(self):
super().__init__()
...
findme_definer = [cls for cls in self.__class__.__mro__ if "findMe" in cls.__dict__][0]
Again - it would be possible to have a metaclass to your inheritance chain which would keep track of all defined attributes in the inheritance tree, and use a dictionary to retrieve where each attribute is defined. The same metaclass could also auto-decorate all __init__ (or all methods), and set a special __setitem__ so that it could track instance attributes as they are created, as listed above.
That can be done, is a bit complicated, would be hard to maintain, and probably is a signal you are taking the wrong approach to your problem.
So, the metaclass to record just class attributes could simply be (python3 syntax - define a __metaclass__ attribute on the class body if you are still using Python 2.7):
class MetaBase(type):
definitions = {}
def __init__(cls, name, bases, dct):
for attr in dct.keys():
cls.__class__.definitions[attr] = cls
class parent(metaclass=MetaBase):
findMe = 5
def __init__(self):
print(self.__class__.definitions["findMe"])
Now, if one wants to find which of the superclasses defined an attribute of the currentclass, just a "live" tracking mechanism, wrapping each method in each class can work - it is a lot trickier.
I've made it - even if you won't need this much, this combines both methods - keeping track of class attributes in the class'class definitions and on an instance _definitions dictionary - since in each created instance an arbitrary method might have been the last to set a particular instance attribute: (This is pure Python3, and maybe not that straighforward porting to Python2 due to the "unbound method" that Python2 uses, and is a simple function in Python3)
from threading import current_thread
from functools import wraps
from types import MethodType
from collections import defaultdict
def method_decorator(func, cls):
#wraps(func)
def wrapper(self, *args, **kw):
self.__class__.__class__.current_running_class[current_thread()].append(cls)
result = MethodType(func, self)(*args, **kw)
self.__class__.__class__.current_running_class[current_thread()].pop()
return result
return wrapper
class MetaBase(type):
definitions = {}
current_running_class = defaultdict(list)
def __init__(cls, name, bases, dct):
for attrname, attr in dct.items():
cls.__class__.definitions[attr] = cls
if callable(attr) and attrname != "__setattr__":
setattr(cls, attrname, method_decorator(attr, cls))
class Base(object, metaclass=MetaBase):
def __setattr__(self, attr, value):
if not hasattr(self, "_definitions"):
super().__setattr__("_definitions", {})
self._definitions[attr] = self.__class__.current_running_class[current_thread()][-1]
return super().__setattr__(attr,value)
Example Classes for the code above:
class Parent(Base):
def __init__(self):
super().__init__()
self.findMe = 10
class Child1(Parent):
def __init__(self):
super().__init__()
self.findMe1 = 20
class Child2(Parent):
def __init__(self):
super().__init__()
self.findMe2 = 30
class GrandChild(Child1, Child2):
def __init__(self):
super().__init__()
def findall(self):
for attr in "findMe findMe1 findMe2".split():
print("Attr '{}' defined in class '{}' ".format(attr, self._definitions[attr].__name__))
And on the console one will get this result:
In [87]: g = GrandChild()
In [88]: g.findall()
Attr 'findMe' defined in class 'Parent'
Attr 'findMe1' defined in class 'Child1'
Attr 'findMe2' defined in class 'Child2'
Related
The Scenario:
class A:
def __init__(self, key, secret):
self.key = key
self.secret = secret
def same_name_method(self):
do_some_staff
def method_a(self):
pass
class B:
def __init__(self, key, secret):
self.key = key
self.secret = secret
def same_name_method(self):
do_another_staff
def method_b(self):
pass
class C(A,B):
def __init__(self, *args, **kwargs):
# I want to init both class A and B's key and secret
## I want to rename class A and B's same method
any_ideas()
...
What I Want:
I want the instance of class C initialize both class A and B, because they are different api key.
And I want rename class A and B's same_name_method, so I will not confused at which same_name_method.
What I Have Done:
For problem one, I have done this:
class C(A,B):
def __init__(self, *args, **kwargs):
A.__init__(self, a_api_key,a_api_secret)
B.__init__(self, b_api_key,b_api_secret)
Comment: I know about super(), but for this situation I do not know how to use it.
For problem two, I add a __new__ for class C
def __new__(cls, *args, **kwargs):
cls.platforms = []
cls.rename_method = []
for platform in cls.__bases__:
# fetch platform module name
module_name = platform.__module__.split('.')[0]
cls.platforms.append(module_name)
# rename attr
for k, v in platform.__dict__.items():
if not k.startswith('__'):
setattr(cls, module_name+'_'+k, v)
cls.rename_method.append(k)
for i in cls.rename_method:
delattr(cls, i) ## this line will raise AttributeError!!
return super().__new__(cls)
Comment: because I rename the new method names and add it to cls attr. I need to delete the old method attr, but do not know how to delattr. Now I just leave them alone, did not delete the old methods.
Question:
Any Suggestions?
So, you want some pretty advanced things, some complicated things, and you don't understand well how classes behave in Python.
So, for your first thing: initializing both classes, and every other method that should run in all classes: the correct solution is to make use of cooperative calls to super() methods.
A call to super() in Python returns you a very special proxy objects that reflects all methods available in the next class, obeying the proper method Resolution Order.
So, if A.__init__ and B.__init__ have to be called, both methods should include a super().__init__ call - and one will call the other's __init__ in the appropriate order, regardless of how they are used as bases in subclasses. As object also have __init__, the last super().__init__ will just call it that is a no-op. If you have more methods in your classes that should be run in all base classes, you'd rather build a proper base class so that the top-most super() call don't try to propagate to a non-existing method.
Otherwise, it is just:
class A:
def __init__(self, akey, asecret, **kwargs):
self.key = akey
self.secret = asecret
super().__init__(**kwargs)
class B:
def __init__(self, bkey, bsecret, **kwargs):
self.key = bkey
self.secret = bsecret
super().__init__(**kwargs)
class C(A,B):
# does not even need an explicit `__init__`.
I think you can get the idea. Of course, the parameter names have to differ - ideally, when writing C you don't have to worry about parameter order - but when calling C you have to worry about suplying all mandatory parameters for C and its bases. If you can't rename the parameters in A or B to be distinct, you could try to use the parameter order for the call, though, with each __init__ consuming two position-parameters - but that will require some extra care in inheritance order.
So - up to this point, it is basic Python multiple-inheritance "howto", and should be pretty straightforward. Now comes your strange stuff.
As for the auto-renaming of methods: first things first -
are you quite sure you need inheritance? Maybe having your granular classes for each external service, and a registry and dispatch class that call the methods on the others by composition would be more sane. (I may come back to this later)
Are you aware that __new__ is called for each instantiation of the class, and all class-attribute mangling you are performing there happens at each new instance of your classes?
So, if the needed method-renaming + shadowing needs to take place at class creation time, you can do that using the special method __init_subclass__ that exists from Python 3.6. It is a special class method that is called once for each derived class of the class it is defined on. So, just create a base class, from which A and B themselves will inherit, and move a properly modified version the thing you are putting in __new__ there. If you are not using Python 3.6, this should be done on the __new__ or __init__ of a metaclass, not on the __new__ of the class itself.
Another approach would be to have a custom __getattribute__ method - this could be crafted to provide namespaces for the base classes. It would owrk ony on instances, not on the classes themselves (but could be made to, again, using a metaclass). __getattribute__ can even hide the same-name-methods.
class Base:
#classmethod
def _get_base_modules(cls):
result = {}
for base in cls.__bases__:
module_name = cls.__module__.split(".")[0]
result[module_name] = base
return result
#classmethod
def _proxy(self, module_name):
class base:
def __dir__(base_self):
return dir(self._base_modules[module_name])
def __getattr__(base_self, attr):
original_value = self._base_modules[module_name].__dict__[attr]
if hasattr(original_value, "__get__"):
original_value = original_value.__get__(self, self.__class__)
return original_value
base.__name__ = module_name
return base()
def __init_subclass__(cls):
cls._base_modules = cls._get_base_modules()
cls._shadowed = {name for module_class in cls._base_modules.values() for name in module_class.__dict__ if not name.startswith("_")}
def __getattribute__(self, attr):
if attr.startswith("_"):
return super().__getattribute__(attr)
cls = self.__class__
if attr in cls._shadowed:
raise AttributeError(attr)
if attr in cls._base_modules:
return cls._proxy(attr)
return super().__getattribute__(attr)
def __dir__(self):
return super().dir() + list(self._base_modules)
class A(Base):
...
class B(Base):
...
class C(A, B):
...
As you can see - this is some fun, but starts getting really complicated - and all the hoola-boops that are needed to retrieve the actual attributes from the superclasses after ading an artificial namespace seem to indicate your problem is not calling for using inheritance after all, as I suggested above.
Since you have your small, functional, atomic classes for each "service" , you could use a plain, simple, non-meta-at-all class that would work as a registry for the various services - and you can even enhance it to call the equivalent method in several of the services it is handling with a single call:
class Services:
def __init__(self):
self.registry = {}
def register(self, cls, key, secret):
name = cls.__module__.split(".")[0]
service= cls(key, secret)
self.registry[name] = service
def __getattr__(self, attr):
if attr in self.registry:
return self.registry[attr]
Lets define simple class decorator function, which creates subclass and adds 'Dec' to original class name only:
def decorate_class(klass):
new_class = type(klass.__name__ + 'Dec', (klass,), {})
return new_class
Now apply it on a simple subclass definition:
class Base(object):
def __init__(self):
print 'Base init'
#decorate_class
class MyClass(Base):
def __init__(self):
print 'MyClass init'
super(MyClass, self).__init__()
Now, if you try instantiate decorated MyClass, it will end up in an infinite loop:
c = MyClass()
# ...
# File "test.py", line 40, in __init__
# super(MyClass, self).__init__()
# RuntimeError: maximum recursion depth exceeded while calling a Python object
It seems, super can't handle this case and does not skip current class from inheritance chain.
The question, how correctly use class decorator on classes using super ?
Bonus question, how get final class from proxy-object created by super ? Ie. get object class from super(Base, self).__init__ expression, as determined parent class defining called __init__.
If you just want to change the class's .__name__ attribute, make a decorator that does that.
from __future__ import print_function
def decorate_class(klass):
klass.__name__ += 'Dec'
return klass
class Base(object):
def __init__(self):
print('Base init')
#decorate_class
class MyClass(Base):
def __init__(self):
print('MyClass init')
super(MyClass, self).__init__()
c = MyClass()
cls = c.__class__
print(cls, cls.__name__)
Python 2 output
MyClass init
Base init
<class '__main__.MyClassDec'> MyClassDec
Python 3 output
MyClass init
Base init
<class '__main__.MyClass'> MyClassDec
Note the difference in the repr of cls. (I'm not sure why you'd want to change a class's name though, it sounds like a recipe for confusion, but I guess it's ok for this simple example).
As others have said, an #decorator isn't intended to create a subclass. You can do it in Python 3 by using the arg-less form of super (i.e., super().__init__()). And you can make it work in both Python 3 and Python 2 by explicitly supplying the parent class rather than using super.
from __future__ import print_function
def decorate_class(klass):
name = klass.__name__
return type(name + 'Dec', (klass,), {})
class Base(object):
def __init__(self):
print('Base init')
#decorate_class
class MyClass(Base):
def __init__(self):
print('MyClass init')
Base.__init__(self)
c = MyClass()
cls = c.__class__
print(cls, cls.__name__)
Python 2 & 3 output
MyClass init
Base init
<class '__main__.MyClassDec'> MyClassDec
Finally, if we just call decorate_class using normal function syntax rather than as an #decorator we can use super.
from __future__ import print_function
def decorate_class(klass):
name = klass.__name__
return type(name + 'Dec', (klass,), {})
class Base(object):
def __init__(self):
print('Base init')
class MyClass(Base):
def __init__(self):
print('MyClass init')
super(MyClass, self).__init__()
MyClassDec = decorate_class(MyClass)
c = MyClassDec()
cls = c.__class__
print(cls, cls.__name__)
The output is the same as in the last version.
Since your decorator returns an entirely new class with different name, for that class MyClass object doesn't even exist. This is not the case class decorators are intended for. They are intended to add additional functionality to an existing class, not outright replacing it with some other class.
Still if you are using Python3, solution is simple -
#decorate_class
class MyClass(Base):
def __init__(self):
print 'MyClass init'
super().__init__()
Otherwise, I doubt there is any straight-forward solution, you just need to change your implementation. When you are renaming the class, you need to rewrite overwrite __init__ as well with newer name.
The problem is that your decorator creates a subclass of the original one. That means that super(Myclass) now point to... the original class itself!
I cannot even explain how the 0 arg form of super manages to do the job in Python 3, I could not find anything explicit in the reference manual. I assume it must use the class in which it is used at the time of declaration. But I cannot imagine a way to get that result in Python2.
If you want to be able to use super in the decorated class in Python 2, you should not create a derived class, but directly modify the original class in place.
For example, here is a decorator that prints a line before and after calling any method:
def decorate_class(klass):
for name, method in klass.__dict__.iteritems(): # iterate the class attributes
if isinstance(method, types.FunctionType): # identify the methods
def meth(*args, **kwargs): # define a wrapper
print "Before", name
method(*args, **kwargs)
print "After", name
setattr(klass, name, meth) # tell the class to use the wrapper
return klass
With your example it gives as expected:
>>> c = MyClass()
Before __init__
MyClass init
Base init
After __init__
I have a python abstract base class as follows:
class Node(object):
"""
All concrete node classes should inherit from this
"""
__metaclass__ = ABCMeta
def __init__(self, name):
self.name = name
self.inputs = dict()
def add_input(self, key, value=None, d=None):
self.inputs[key] = (d, value)
def bind_input(self):
print "Binding inputs"
#abstractmethod
def run(self):
pass
Now, various derived classes will inherit from this node class and override the run method. It is always the case that bind_input() must be the first thing that should be called in the run method. Currently, for all derived classes the developer has to make sure to first call self.bind_input(). This is not a huge problem per se but out of curiosity is it possible to ensure this somehow from the base class itself that bind_input is called before executing the child object's run?
The usual object-oriented approach is this:
def run(self):
self.bind_input()
return self.do_run()
#abstractmethod
def do_run(self):
pass # override this method
Have your subclasses override the inner method, instead of the outer one.
I have the following python code:
class FooMeta(type):
def __setattr__(self, name, value):
print name, value
return super(FooMeta, self).__setattr__(name, value)
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
I would have expected __setattr__ of the meta class being called for both FOO and a. However, it is not called at all. When I assign something to Foo.whatever after the class has been defined the method is called.
What's the reason for this behaviour and is there a way to intercept the assignments that happen during the creation of the class? Using attrs in __new__ won't work since I'd like to check if a method is being redefined.
A class block is roughly syntactic sugar for building a dictionary, and then invoking a metaclass to build the class object.
This:
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
Comes out pretty much as if you'd written:
d = {}
d['__metaclass__'] = FooMeta
d['FOO'] = 123
def a(self):
pass
d['a'] = a
Foo = d.get('__metaclass__', type)('Foo', (object,), d)
Only without the namespace pollution (and in reality there's also a search through all the bases to determine the metaclass, or whether there's a metaclass conflict, but I'm ignoring that here).
The metaclass' __setattr__ can control what happens when you try to set an attribute on one of its instances (the class object), but inside the class block you're not doing that, you're inserting into a dictionary object, so the dict class controls what's going on, not your metaclass. So you're out of luck.
Unless you're using Python 3.x! In Python 3.x you can define a __prepare__ classmethod (or staticmethod) on a metaclass, which controls what object is used to accumulate attributes set within a class block before they're passed to the metaclass constructor. The default __prepare__ simply returns a normal dictionary, but you could build a custom dict-like class that doesn't allow keys to be redefined, and use that to accumulate your attributes:
from collections import MutableMapping
class SingleAssignDict(MutableMapping):
def __init__(self, *args, **kwargs):
self._d = dict(*args, **kwargs)
def __getitem__(self, key):
return self._d[key]
def __setitem__(self, key, value):
if key in self._d:
raise ValueError(
'Key {!r} already exists in SingleAssignDict'.format(key)
)
else:
self._d[key] = value
def __delitem__(self, key):
del self._d[key]
def __iter__(self):
return iter(self._d)
def __len__(self):
return len(self._d)
def __contains__(self, key):
return key in self._d
def __repr__(self):
return '{}({!r})'.format(type(self).__name__, self._d)
class RedefBlocker(type):
#classmethod
def __prepare__(metacls, name, bases, **kwargs):
return SingleAssignDict()
def __new__(metacls, name, bases, sad):
return super().__new__(metacls, name, bases, dict(sad))
class Okay(metaclass=RedefBlocker):
a = 1
b = 2
class Boom(metaclass=RedefBlocker):
a = 1
b = 2
a = 3
Running this gives me:
Traceback (most recent call last):
File "/tmp/redef.py", line 50, in <module>
class Boom(metaclass=RedefBlocker):
File "/tmp/redef.py", line 53, in Boom
a = 3
File "/tmp/redef.py", line 15, in __setitem__
'Key {!r} already exists in SingleAssignDict'.format(key)
ValueError: Key 'a' already exists in SingleAssignDict
Some notes:
__prepare__ has to be a classmethod or staticmethod, because it's being called before the metaclass' instance (your class) exists.
type still needs its third parameter to be a real dict, so you have to have a __new__ method that converts the SingleAssignDict to a normal one
I could have subclassed dict, which would probably have avoided (2), but I really dislike doing that because of how the non-basic methods like update don't respect your overrides of the basic methods like __setitem__. So I prefer to subclass collections.MutableMapping and wrap a dictionary.
The actual Okay.__dict__ object is a normal dictionary, because it was set by type and type is finicky about the kind of dictionary it wants. This means that overwriting class attributes after class creation does not raise an exception. You can overwrite the __dict__ attribute after the superclass call in __new__ if you want to maintain the no-overwriting forced by the class object's dictionary.
Sadly this technique is unavailable in Python 2.x (I checked). The __prepare__ method isn't invoked, which makes sense as in Python 2.x the metaclass is determined by the __metaclass__ magic attribute rather than a special keyword in the classblock; which means the dict object used to accumulate attributes for the class block already exists by the time the metaclass is known.
Compare Python 2:
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
Being roughly equivalent to:
d = {}
d['__metaclass__'] = FooMeta
d['FOO'] = 123
def a(self):
pass
d['a'] = a
Foo = d.get('__metaclass__', type)('Foo', (object,), d)
Where the metaclass to invoke is determined from the dictionary, versus Python 3:
class Foo(metaclass=FooMeta):
FOO = 123
def a(self):
pass
Being roughly equivalent to:
d = FooMeta.__prepare__('Foo', ())
d['Foo'] = 123
def a(self):
pass
d['a'] = a
Foo = FooMeta('Foo', (), d)
Where the dictionary to use is determined from the metaclass.
There are no assignments happening during the creation of the class. Or: they are happening, but not in the context you think they are. All class attributes are collected from class body scope and passed to metaclass' __new__, as the last argument:
class FooMeta(type):
def __new__(self, name, bases, attrs):
print attrs
return type.__new__(self, name, bases, attrs)
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
Reason: when the code in the class body executes, there's no class yet. Which means there's no opportunity for metaclass to intercept anything yet.
Class attributes are passed to the metaclass as a single dictionary and my hypothesis is that this is used to update the __dict__ attribute of the class all at once, e.g. something like cls.__dict__.update(dct) rather than doing setattr() on each item. More to the point, it's all handled in C-land and simply wasn't written to call a custom __setattr__().
It's easy enough to do whatever you want to the attributes of the class in your metaclass's __init__() method, since you're passed the class namespace as a dict, so just do that.
During the class creation, your namespace is evaluated to a dict and passed as an argument to the metaclass, together with the class name and base classes. Because of that, assigning a class attribute inside the class definition wouldn't work the way you expect. It doesn't create an empty class and assign everything. You also can't have duplicated keys in a dict, so during class creation attributes are already deduplicated. Only by setting an attribute after the class definition you can trigger your custom __setattr__.
Because the namespace is a dict, there's no way for you to check duplicated methods, as suggested by your other question. The only practical way to do that is parsing the source code.
Here's a very simple example of what I'm trying to get around:
class Test(object):
some_dict = {Test: True}
The problem is that I cannot refer to Test while it's still being defined
Normally, I'd just do this:
class Test(object):
some_dict = {}
def __init__(self):
if self.__class__.some_dict == {}:
self.__class__.some_dict = {Test: True}
But I never create an instance of this class. It's really just a container to hold a group of related functions and data (I have several of these classes, and I pass around references to them, so it is necessary for Test to be it's own class)
So my question is, how could I refer to Test while it's being defined, or is there something similar to __init__ that get's called as soon as the class is defined? If possible, I want self.some_dict = {Test: True} to remain inside the class definition. This is the only way I know how to do this so far:
class Test(object):
#classmethod
def class_init(cls):
cls.some_dict = {Test: True}
Test.class_init()
The class does in fact not exist while it is being defined. The way the class statement works is that the body of the statement is executed, as a block of code, in a separate namespace. At the end of the execution, that namespace is passed to the metaclass (such as type) and the metaclass creates the class using the namespace as the attributespace.
From your description, it does not sound necessary for Test to be a class. It sounds like it should be a module instead. some_dict is a global -- even if it's a class attribute, there's only one such attribute in your program, so it's not any better than having a global -- and any classmethods you have in the class can just be functions.
If you really want it to be a class, you have three options: set the dict after defining the class:
class Test:
some_dict = {}
Test.some_dict[Test] = True
Use a class decorator (in Python 2.6 or later):
def set_some_dict(cls):
cls.some_dict[cls] = True
#set_some_dict
class Test:
some_dict = {}
Or use a metaclass:
class SomeDictSetterType(type):
def __init__(self, name, bases, attrs):
self.some_dict[self] = True
super(SomeDictSetterType, self).__init__(name, bases, attrs)
class Test(object):
__metaclass__ = SomeDictSetterType
some_dict = {}
You could add the some_dict attribute after the main class definition.
class Test(object):
pass
Test.some_dict = {Test: True}
I've tried to use classes in this way in the past, and it gets ugly pretty quickly (for example, all the methods will need to be class methods or static methods, and you will probably realise eventually that you want to define certain special methods, for which you will have to start using metaclasses). It could make things a lot easier if you just use class instances instead - there aren't really any downsides.
A (weird-looking) alternative to what others have suggested: you could use __new__:
class Test(object):
def __new__(cls):
cls.some_dict = {cls: True}
Test()
You could even have __new__ return a reference to the class and use a decorator to call it:
def instantiate(cls):
return cls()
#instantiate
class Test(object):
def __new__(cls):
cls.some_dict = {cls: True}
return cls
You can also use a metaclass (a function here but there are other ways):
def Meta(name, bases, ns):
klass = type(name, bases, ns)
setattr(klass, 'some_dict', { klass: True })
return klass
class Test(object):
__metaclass__ = Meta
print Test.some_dict
Thomas's first example is very good, but here's a more Pythonic way of doing the same thing.
class Test:
x = {}
#classmethod
def init(cls):
# do whatever setup you need here
cls.x[cls] = True
Test.init()