Suppose we have a base class, A, that contains some class variables. This class also has a class method foo that does something with those variables. Since this behavior shouldn't be hard-coded (e.g. we don't want to have to modify foo when adding new class variables), foo reads cls.__dict__ instead of directly referencing the variables.
Now we introduce a derived class: B extends A with some more class variables, as well as inherits foo. Code example:
class A:
x = 0
y = 1
#classmethod
def foo(cls):
print([name for name, prop in cls.__dict__.items() if type(prop) is int])
class B(A):
z = 3
print(B.x) # prints "0"
A.foo() # prints "['x', 'y']"
B.foo() # prints "['z']" -- why not "['x', 'y', 'z']"?
Therefore, my question is: why B.__dict__ does not contain the variables inherited from A, and if not there, then where are they?
This is not a duplicate of Accessing class attributes from parents in instance methods, because I don't just want to query specific variables that happen to be in the base class - I want to list them without knowing their names. The answers related to MRO given in this question might happen to also apply here, but the original problem is in my view different.
I solved this with a metaclass, using it to redefine the way derived classes are created. In this approach, all class variables that we are interested in get copied from the base classes to the derived one:
class M(type):
def __new__(cls, name, bases, dct):
for base in bases:
for name, prop in base.__dict__.items():
if type(prop) is int:
dct[name] = prop
return super(M, cls).__new__(cls, name, bases, dct)
class A(metaclass=M):
x = 0
y = 1
#classmethod
def foo(cls):
print([name for name, prop in cls.__dict__.items() if type(prop) is int])
class B(A):
z = 3
A.foo() # prints "['y', 'x']"
B.foo() # prints "['y', 'x', 'z']"
Besides, the metaclass can be defined in such a way that foo will not even have to query the __dict__ - variables of interest could be put into a list instead of modifying __dict__, should this for some reason had to be avoided.
Related
Let B inherit from A. Suppose that some of B's behavior depends on the class attribute cls_x and we want to set up this dependency during construction of B objects. Since it is not a simple operation, we want to wrap it in a class method, which the constructor will call. Example:
class B(A):
cls_x = 'B'
#classmethod
def cm(cls):
return cls.cls_x
def __init__(self):
self.attr = B.cm()
Problem: cm as well as __init__ will always be doing the same things and their behavior must stay the same in each derived class. Thus, we would like to put them both in the base class and not define it in any of the derived classes. The only difference will be the caller of cm - either A or B (or any of B1, B2, each inheriting from A), whatever is being constructed. So what we'd like to have is something like this:
class A:
cls_x = 'A'
#classmethod
def cm(cls):
return cls.cls_x
def __init__(self):
self.attr = ClassOfWhateverIsInstantiated.cm() #how to do this?
class B(A):
cls_x = 'B'
I feel like it's either something very simple I'm missing about Python's inheritance mechanics or the whole issue should be handled entirely differently.
This is different than this question as I do not want to override the class method, but move its implementation to the base class entirely.
Look at it this way: Your question is essentially "How do I get the class of an instance?". The answer to that question is to use the type function:
ClassOfWhateverIsInstantiated = type(self)
But you don't even need to do that, because classmethods can be called directly through an instance:
def __init__(self):
self.attr = self.cm() # just use `self`
This works because classmethods automatically look up the class of the instance for you. From the docs:
[A classmethod] can be called either on the class (such as C.f()) or on an instance
(such as C().f()). The instance is ignored except for its class.
For ClassOfWhateverIsInstantiated you can just use self:
class A:
cls_x = 'A'
#classmethod
def cm(cls):
return cls.cls_x
def __init__(self):
self.attr = self.cm() # 'self' refers to B, if called from B
class B(A):
cls_x = 'B'
a = A()
print(a.cls_x) # = 'A'
print(A.cls_x) # = 'A'
b = B()
print(b.cls_x) # = 'B'
print(B.cls_x) # = 'B'
To understand this, just remember that class B is inheriting the methods of class A. So when __init__() is called during B's instantiation, it's called in the context of class B, to which self refers.
Class Foo is defined with a metaclass Meta. The metaclass loops over the class attributes and prints them to screen.
Class Bar subclasses Foo. However, the metaclass does not print the inherited attributes from Bar.
Why doesn't the metaclass have access to Foo's attributes inherited in Bar? What am I not understanding about python's metaclass system?
Here is the sample code in 2.7:
class Meta(type):
def __init__(cls, name, bases, attrs):
print "bases = {}".format(bases)
items = {k:v for k,v in attrs.iteritems() if not k.startswith('__')}
for k,v in items.iteritems():
print k, v
class Foo(object):
__metaclass__ = Meta
hi = 1
# This prints:
# bases = (<type 'object'>,)
# hi 1
class Bar(Foo):
pass
# This prints:
# bases = (<class '__main__.Foo'>,)
Foo.hi
#prints 1
Bar.hi
#prints 1
The attrs parameter to __init__ only contains the attributes for that class, not for its bases.
A Bar object does not have an attribute hi. Instead, when you ask for Bar.hi the lookup will start at Bar, find out that it doesn't have hi, then look in base Foo to find it.
As #orlp says, attrs contains only the class dictionary for the class being created. You still have access to hi, however, because it's in the __dict__ attribute of one of Foo's bases. That is, you could do something similar to what you have, but recurse through the base classes and print out the entries in each base class dictionary.
Another approach is to use dir(), which should roughly return a list of all attributes a class has. I say roughly because a class can implement __getattr__ or __getattribute__ to return attributes "on the fly", meaning that the class may not have a well-defined set of attributes for dir() to return -- see the full disclaimer here. But in many common cases, something like the following will work:
class Meta(type):
def __init__(cls, name, bases, attrs):
print "bases = {}".format(bases)
for attr in dir(cls):
if not attr.startswith('_'):
print attr, getattr(cls, attr)
class Foo(object):
__metaclass__ = Meta
hi = 1
class Bar(Foo):
pass
Which prints:
bases = (<type 'object'>,)
hi 1
bases = (<class '__main__.Foo'>,)
hi 1
I have a parent class and two child class. The parent class is an abstract base class that has method combine that gets inherited by the child classes. But each child implements combine differently from a parameter perspective therefore each of their own methods take different number of parameters. In Python, when a child inherits a method and requires re-implementing it, that newly re-implemented method must match parameter by parameter. Is there a way around this? I.e. the inherited method can have dynamic parameter composition?
This code demonstrates that signature of overridden method can easily change.
class Parent(object):
def foo(self, number):
for _ in range(number):
print "Hello from parent"
class Child(Parent):
def foo(self, number, greeting):
for _ in range(number):
print greeting
class GrandChild(Child):
def foo(self):
super(GrandChild,self).foo(1, "hey")
p = Parent()
p.foo(3)
c = Child()
c.foo(2, "Hi")
g = GrandChild()
g.foo()
As the other answer demonstrates for plain classes, the signature of an overridden inherited method can be different in the child than in the parent.
The same is true even if the parent is an abstract base class:
import abc
class Foo:
__metaclass__ = abc.ABCMeta
#abc.abstractmethod
def bar(self, x, y):
return x + y
class ChildFoo(Foo):
def bar(self, x):
return super(self.__class__, self).bar(x, 3)
class DumbFoo(Foo):
def bar(self):
return "derp derp derp"
cf = ChildFoo()
print cf.bar(5)
df = DumbFoo()
print df.bar()
Inappropriately complicated detour
It is an interesting exercise in Python metaclasses to try to restrict the ability to override methods, such that their argument signature must match that of the base class. Here is an attempt.
Note: I'm not endorsing this as a good engineering idea, and I did not spend time tying up loose ends so there are likely little caveats about the code below that could make it more efficient or something.
import types
import inspect
def strict(func):
"""Add some info for functions having strict signature.
"""
arg_sig = inspect.getargspec(func)
func.is_strict = True
func.arg_signature = arg_sig
return func
class StrictSignature(type):
def __new__(cls, name, bases, attrs):
func_types = (types.MethodType,) # include types.FunctionType?
# Check each attribute in the class being created.
for attr_name, attr_value in attrs.iteritems():
if isinstance(attr_value, func_types):
# Check every base for #strict functions.
for base in bases:
base_attr = base.__dict__.get(attr_name)
base_attr_is_function = isinstance(base_attr, func_types)
base_attr_is_strict = hasattr(base_attr, "is_strict")
# Assert that inspected signatures match.
if base_attr_is_function and base_attr_is_strict:
assert (inspect.getargspec(attr_value) ==
base_attr.arg_signature)
# If everything passed, create the class.
return super(StrictSignature, cls).__new__(cls, name, bases, attrs)
# Make a base class to try out strictness
class Base:
__metaclass__ = StrictSignature
#strict
def foo(self, a, b, c="blah"):
return a + b + len(c)
def bar(self, x, y, z):
return x
#####
# Now try to make some classes inheriting from Base.
#####
class GoodChild(Base):
# Was declared strict, better match the signature.
def foo(self, a, b, c="blah"):
return c
# Was never declared as strict, so no rules!
def bar(im_a_little, teapot):
return teapot/2
# These below can't even be created. Uncomment and try to run the file
# and see. It's not just that you can't instantiate them, you can't
# even get the *class object* defined at class creation time.
#
#class WrongChild(Base):
# def foo(self, a):
# return super(self.__class__, self).foo(a, 5)
#
#class BadChild(Base):
# def foo(self, a, b, c="halb"):
# return super(self.__class__, self).foo(a, b, c)
Note, like with most "strict" or "private" type ideas in Python, that you are still free to monkey-patch functions onto even a "good class" and those monkey-patched functions don't have to satisfy the signature constraint.
# Instance level
gc = GoodChild()
gc.foo = lambda self=gc: "Haha, I changed the signature!"
# Class level
GoodChild.foo = lambda self: "Haha, I changed the signature!"
and even if you add more complexity to the meta class that checks whenever any method type attributes are updated in the class's __dict__ and keeps making the assert statement when the class is modified, you can still use type.__setattr__ to bypass customized behavior and set an attribute anyway.
In these cases, I imagine Jeff Goldblum as Ian Malcolm from Jurassic Park, looking at you blankly and saying "Consenting adults, uhh, find a way.."
Toward the end of a program I'm looking to load a specific variable from all the instances of a class into a dictionary.
For example:
class Foo():
def __init__(self):
self.x = {}
foo1 = Foo()
foo2 = Foo()
...
Let's say the number of instances will vary and I want the x dict from each instance of Foo() loaded into a new dict. How would I do that?
The examples I've seen in SO assume one already has the list of instances.
One way to keep track of instances is with a class variable:
class A(object):
instances = []
def __init__(self, foo):
self.foo = foo
A.instances.append(self)
At the end of the program, you can create your dict like this:
foo_vars = {id(instance): instance.foo for instance in A.instances}
There is only one list:
>>> a = A(1)
>>> b = A(2)
>>> A.instances
[<__main__.A object at 0x1004d44d0>, <__main__.A object at 0x1004d4510>]
>>> id(A.instances)
4299683456
>>> id(a.instances)
4299683456
>>> id(b.instances)
4299683456
#JoelCornett's answer covers the basics perfectly. This is a slightly more complicated version, which might help with a few subtle issues.
If you want to be able to access all the "live" instances of a given class, subclass the following (or include equivalent code in your own base class):
from weakref import WeakSet
class base(object):
def __new__(cls, *args, **kwargs):
instance = object.__new__(cls, *args, **kwargs)
if "instances" not in cls.__dict__:
cls.instances = WeakSet()
cls.instances.add(instance)
return instance
This addresses two possible issues with the simpler implementation that #JoelCornett presented:
Each subclass of base will keep track of its own instances separately. You won't get subclass instances in a parent class's instance list, and one subclass will never stumble over instances of a sibling subclass. This might be undesirable, depending on your use case, but it's probably easier to merge the sets back together than it is to split them apart.
The instances set uses weak references to the class's instances, so if you del or reassign all the other references to an instance elsewhere in your code, the bookkeeping code will not prevent it from being garbage collected. Again, this might not be desirable for some use cases, but it is easy enough to use regular sets (or lists) instead of a weakset if you really want every instance to last forever.
Some handy-dandy test output (with the instances sets always being passed to list only because they don't print out nicely):
>>> b = base()
>>> list(base.instances)
[<__main__.base object at 0x00000000026067F0>]
>>> class foo(base):
... pass
...
>>> f = foo()
>>> list(foo.instances)
[<__main__.foo object at 0x0000000002606898>]
>>> list(base.instances)
[<__main__.base object at 0x00000000026067F0>]
>>> del f
>>> list(foo.instances)
[]
You would probably want to use weak references to your instances. Otherwise the class could likely end up keeping track of instances that were meant to have been deleted. A weakref.WeakSet will automatically remove any dead instances from its set.
One way to keep track of instances is with a class variable:
import weakref
class A(object):
instances = weakref.WeakSet()
def __init__(self, foo):
self.foo = foo
A.instances.add(self)
#classmethod
def get_instances(cls):
return list(A.instances) #Returns list of all current instances
At the end of the program, you can create your dict like this:
foo_vars = {id(instance): instance.foo for instance in A.instances}
There is only one list:
>>> a = A(1)
>>> b = A(2)
>>> A.get_instances()
[<inst.A object at 0x100587290>, <inst.A object at 0x100587250>]
>>> id(A.instances)
4299861712
>>> id(a.instances)
4299861712
>>> id(b.instances)
4299861712
>>> a = A(3) #original a will be dereferenced and replaced with new instance
>>> A.get_instances()
[<inst.A object at 0x100587290>, <inst.A object at 0x1005872d0>]
You can also solve this problem using a metaclass:
When a class is created (__init__ method of metaclass), add a new instance registry
When a new instance of this class is created (__call__ method of metaclass), add it to the instance registry.
The advantage of this approach is that each class has a registry - even if no instance exists. In contrast, when overriding __new__ (as in Blckknght's answer), the registry is added when the first instance is created.
class MetaInstanceRegistry(type):
"""Metaclass providing an instance registry"""
def __init__(cls, name, bases, attrs):
# Create class
super(MetaInstanceRegistry, cls).__init__(name, bases, attrs)
# Initialize fresh instance storage
cls._instances = weakref.WeakSet()
def __call__(cls, *args, **kwargs):
# Create instance (calls __init__ and __new__ methods)
inst = super(MetaInstanceRegistry, cls).__call__(*args, **kwargs)
# Store weak reference to instance. WeakSet will automatically remove
# references to objects that have been garbage collected
cls._instances.add(inst)
return inst
def _get_instances(cls, recursive=False):
"""Get all instances of this class in the registry. If recursive=True
search subclasses recursively"""
instances = list(cls._instances)
if recursive:
for Child in cls.__subclasses__():
instances += Child._get_instances(recursive=recursive)
# Remove duplicates from multiple inheritance.
return list(set(instances))
Usage: Create a registry and subclass it.
class Registry(object):
__metaclass__ = MetaInstanceRegistry
class Base(Registry):
def __init__(self, x):
self.x = x
class A(Base):
pass
class B(Base):
pass
class C(B):
pass
a = A(x=1)
a2 = A(2)
b = B(x=3)
c = C(4)
for cls in [Base, A, B, C]:
print cls.__name__
print cls._get_instances()
print cls._get_instances(recursive=True)
print
del c
print C._get_instances()
If using abstract base classes from the abc module, just subclass abc.ABCMeta to avoid metaclass conflicts:
from abc import ABCMeta, abstractmethod
class ABCMetaInstanceRegistry(MetaInstanceRegistry, ABCMeta):
pass
class ABCRegistry(object):
__metaclass__ = ABCMetaInstanceRegistry
class ABCBase(ABCRegistry):
__metaclass__ = ABCMeta
#abstractmethod
def f(self):
pass
class E(ABCBase):
def __init__(self, x):
self.x = x
def f(self):
return self.x
e = E(x=5)
print E._get_instances()
Another option for quick low-level hacks and debugging is to filter the list of objects returned by gc.get_objects() and generate the dictionary on the fly that way. In CPython that function will return you a (generally huge) list of everything the garbage collector knows about, so it will definitely contain all of the instances of any particular user-defined class.
Note that this is digging a bit into the internals of the interpreter, so it may or may not work (or work well) with the likes of Jython, PyPy, IronPython, etc. I haven't checked. It's also likely to be really slow regardless. Use with caution/YMMV/etc.
However, I imagine that some people running into this question might eventually want to do this sort of thing as a one-off to figure out what's going on with the runtime state of some slice of code that's behaving strangely. This method has the benefit of not affecting the instances or their construction at all, which might be useful if the code in question is coming out of a third-party library or something.
Here's a similar approach to Blckknght's, which works with subclasses as well. Thought this might be of interest, if someone ends up here. One difference, if B is a subclass of A, and b is an instance of B, b will appear in both A.instances and B.instances. As stated by Blckknght, this depends on the use case.
from weakref import WeakSet
class RegisterInstancesMixin:
instances = WeakSet()
def __new__(cls, *args, **kargs):
o = object.__new__(cls, *args, **kargs)
cls._register_instance(o)
return o
#classmethod
def print_instances(cls):
for instance in cls.instances:
print(instance)
#classmethod
def _register_instance(cls, instance):
cls.instances.add(instance)
for b in cls.__bases__:
if issubclass(b, RegisterInstancesMixin):
b._register_instance(instance)
def __init_subclass__(cls):
cls.instances = WeakSet()
class Animal(RegisterInstancesMixin):
pass
class Mammal(Animal):
pass
class Human(Mammal):
pass
class Dog(Mammal):
pass
alice = Human()
bob = Human()
cannelle = Dog()
Animal.print_instances()
Mammal.print_instances()
Human.print_instances()
Animal.print_instances() will print three objects, whereas Human.print_instances() will print two.
Using the answer from #Joel Cornett I've come up with the following, which seems to work. i.e. i'm able to total up object variables.
import os
os.system("clear")
class Foo():
instances = []
def __init__(self):
Foo.instances.append(self)
self.x = 5
class Bar():
def __init__(self):
pass
def testy(self):
self.foo1 = Foo()
self.foo2 = Foo()
self.foo3 = Foo()
foo = Foo()
print Foo.instances
bar = Bar()
bar.testy()
print Foo.instances
x_tot = 0
for inst in Foo.instances:
x_tot += inst.x
print x_tot
output:
[<__main__.Foo instance at 0x108e334d0>]
[<__main__.Foo instance at 0x108e334d0>, <__main__.Foo instance at 0x108e33560>, <__main__.Foo instance at 0x108e335a8>, <__main__.Foo instance at 0x108e335f0>]
5
10
15
20
(For Python)
I have found a way to record the class instances via the "dataclass" decorator while defining a class. Define a class attribute 'instances' (or any other name) as a list of the instances you want to record. Append that list with the 'dict' form of created objects via the dunder method __dict__. Thus, the class attribute 'instances' will record instances in the dict form, which you want.
For example,
from dataclasses import dataclass
#dataclass
class player:
instances=[]
def __init__(self,name,rank):
self.name=name
self.rank=rank
self.instances.append(self.__dict__)
I'd like to set a class property that can be shared by all instances of the class or its subclasses. It should be possible for any instance to also set the property.
I tried the following:
class A:
x = 1
#classmethod
def setX(cls, val):
if cls.__bases__:
cls = cls.__bases__[-1]
cls.x = val
This seems to work fine in case of single inheritance. But if I use multiple inheritance, depending on the order of inheritance, it either works or doesn't (i.e., class A is not always the last of bases).
Any ideas for a robust implementation?
Use lexical scoping:
class A(object):
x = 1
#classmethod
def setX(cls, val):
A.x = val
The right way, IMO, is to go with a class property. Since properties are tied to an object's classes __dict__, as opposed to the object's very own __dict__, and since your object happens to be a class, you must attach it to the classes class, its metaclass:
class A(object):
_x = None
class __metaclass__(type):
#property
def x(cls):
return A._x
#x.setter
def x(cls, val):
A._x = val
class B(A):
pass
A.x = 'jim'
B.x = 'joe'
print A.x, B.x
Result:
joe joe
Also, your classes must be new style classes, i.e. they must inherit from object in Python 2.x. And metaclasses are defined differently in Python 3.x:
class MetaA(type):
""" like ___metaclass__ above """
class A(metaclass=MetaA):
_x = None
...