Develop Reference

Python - using metaclass & class attributes in object instances

I have a number of classes that inherit a common one. I need the parent class to keep track of a bunch of dependencies/relationships that are defined at the class level. Something like:
class Meta(type):
ALLDEPENDENCIES = {}
def __new__(meta, name, bases, attrs):
if "DEPENDENCIES" in attrs.keys():
for key, value in attrs.items():
if key == "DEPENDENCIES":
meta.ALLDEPENDENCIES.update(attrs["DEPENDENCIES"])
return type.__new__(meta, name, bases, attrs)
class DataTable(DataFrameWrapper, metaclass=Meta):
pass
class Foo(DataTable):
DEPENDENCIES = {"a":1}
class Bar(DataTable):
DEPENDENCIES = {"b":2}
So essentially, as I create new classes (Foo, Bar, Baz...) each of them has a dictionary. I need to merge the info from each dictionary. So I'm using the metaclass, as shown above. Each class as an DEPENDENCIES attribute, and I'm gathering all of those into the ALLDEPENDENCIES attribute defined in the metaclass.
If I do this, it seems to work alright:
import Foo, Bar
print(Foo.ALLDEPENDENCIES)
>> {"a":1, "b":2}
print(Bar.ALLDEPENDENCIES)
>> {"a":1, "b":2}
However, when working if obj instances, the ALLDEPENDENCIES attributes is missing:
f = Foo()
b = Bar()
print(f.ALLDEPENDENCIES)
print(b.ALLDEPENDENCIES)
Attribute error - there is no ALLDEPENDENCIES.
I thought that the class attribute defined in the metaclass would be accessible from self.myattribute in the instances, just like DEPENDENCIES is. What am I doing wrong?

Meta describes how to create class but not what class that will be.
Meta != Parent with inherited attributes
So you have to pass proper attributes into new class:
class Meta(type):
_a = {}
def __new__(meta, name, bases, attrs):
if "d" in attrs:
meta._a.update(attrs["d"])
attrs["a"] = meta._a
return type.__new__(meta, name, bases, attrs)
class Data:
pass
class DataTable(Data, metaclass=Meta):
pass
class Foo(DataTable):
d = {"a":1}
class Bar(DataTable):
d = {"b":2}
f = Foo()
print(Foo.a)
print(f.a)
{'a': 1, 'b': 2}
{'a': 1, 'b': 2}

Instance class attribute search does not go into the metaclass - just to the class. The metaclass could set ALLDEPENDANCIES in each new class, with a single line in its __new__, but if you want cleaner code, in the sense the dictionary is not aliased everywhere, you can just access the attribute through the class.
Using your code, as is:
Foo().__class__.ALLDEPENDANCIES
will work from anywhere (just as `type(Foo()).ALLDEPENDANCIES).
In order to set the attribute in the new classes, so that it will be visible in the newly created classes, an option is:
from types import MappingProxyType
class Meta(type):
ALLDEPENDANCIES = {}
ALLDEPSVIEW = MappingProxyType(ALLDEPENDANCIES)
def __new__(meta, name, bases, attrs):
if "DEPENDANCIES" in attrs.keys():
for key, value in attrs.items():
if key == "DEPENDANCIES":
meta.ALLDEPENDANCIES.update(attrs["DEPENDANCIES"])
new_cls = super().__new__(meta, name, bases, attrs)
new_cls.ALLDEPENDANCIES = meta.ALLDEPSVIEW
return new_cls
(Inserting the new attr in attrs before calling type.__new__ will also work)
Here I do two other extras: (1) call super().__new__ instead of hardcoding a call to type.__new__: this allows your metaclass to be composable with other metaclasses, which might be needed if one of your classes will cross with other metaclass (for example, if you are using abstract base classes from abc or collections.abc). And (2) using a MappingProxyType which is a "read only" dictionary view, and will stop acidental direct updates of the dict through classes or instances.

Python: Class instances are classes

Is it possible to create a class whose instances are classes?
At first it seemed possible, but I wondered if this would actually be possible without any external interference.
Eg:
NewClass = ClassClass(someArgs)
instance = NewClass(someOtherArgs)
Is this possible? If so, how would it structurally work? Is this even useful?

If you check the type of 1 you get type(1) == int. If you check the type of int you get type(int) == type.
A class whose instances are classes is called a metaclass and in Python you create metaclasses by subclassing type.
class ClassClass(type):
pass
Technically collections.namedtuple is a metaclass, but it's actually cheating.
There's a great PyCon talk by David Beazly about metaprogramming in Python. It's long, but very interesting. He starts talking about metaclasses around the 27 minute mark.

You're looking for metaclasses:
class Foo(type):
# Foo is a subclass of type and just like instances of type are
# classes, instances of Foo are classes.
pass
class Bar(object):
# The Bar class is an instance of Foo
__metaclass__ = Foo
# You can also create instances of Foo dynamically.
Bar = Foo("Bar", (object,), {})
More on metaclasses.

From the Python help:
type(name, bases, dict) -> a new type
So lets create a simple class:
Dog = type('Dog', (object,), {})
And now you can create a dog:
fido = Dog()
If you literally want a class that creates classes, you can do it, by extending type... Honestly I have no idea what you want it for, but here it is:
class ClassMaker(type):
def __new__(self, name):
return type(name, (object,), {})
NewClass = ClassMaker('NewClass')
print NewClass

In (modern) python, classes are first order objects. This is easy to verify by accessing a class' __class__ property:
class A:
pass
print(A.__class__)
-> <class 'type'>
Note that type's class is also type:
print(type.__class__)
-> <class 'type'>
It is possible to actually derive from type:
class A:
def do(self):
print("DOING")
class mytype(type):
def __new__(cls, *args, **kwargs):
return type.__new__(cls, 'mytype', cls.__bases__, {})
def __init__(self):
return type.__init__(self, 'mytype', self.__class__.__bases__, {})
def __call__(self, *args, **kwargs):
return A(*args, **kwargs)
aClass = mytype()
-> <class '__main__.mytype'>
aObj = aClass()
print(aObj)
-> <__main__.A object at 0xdeadbeaf>
aObj.do()
-> DOING
However, with proper metaclasses I do not see a good reason jumping through such hoops. Even metaclasses are only needed in special cases. Most often, a factory method that returns a class object is sufficient:
def ClassClass():
return A
aClass = ClassClass()
aObj = aClass()
Syntax is the same and if the only operation on the factory is instantiation, there is no difference.

Why does a metaclass not have access to the attributes inhereited from a subclass of a class defined by the metaclass?

Class Foo is defined with a metaclass Meta. The metaclass loops over the class attributes and prints them to screen.
Class Bar subclasses Foo. However, the metaclass does not print the inherited attributes from Bar.
Why doesn't the metaclass have access to Foo's attributes inherited in Bar? What am I not understanding about python's metaclass system?
Here is the sample code in 2.7:
class Meta(type):
def __init__(cls, name, bases, attrs):
print "bases = {}".format(bases)
items = {k:v for k,v in attrs.iteritems() if not k.startswith('__')}
for k,v in items.iteritems():
print k, v
class Foo(object):
__metaclass__ = Meta
hi = 1
# This prints:
# bases = (<type 'object'>,)
# hi 1
class Bar(Foo):
pass
# This prints:
# bases = (<class '__main__.Foo'>,)
Foo.hi
#prints 1
Bar.hi
#prints 1

The attrs parameter to __init__ only contains the attributes for that class, not for its bases.
A Bar object does not have an attribute hi. Instead, when you ask for Bar.hi the lookup will start at Bar, find out that it doesn't have hi, then look in base Foo to find it.

As #orlp says, attrs contains only the class dictionary for the class being created. You still have access to hi, however, because it's in the __dict__ attribute of one of Foo's bases. That is, you could do something similar to what you have, but recurse through the base classes and print out the entries in each base class dictionary.
Another approach is to use dir(), which should roughly return a list of all attributes a class has. I say roughly because a class can implement __getattr__ or __getattribute__ to return attributes "on the fly", meaning that the class may not have a well-defined set of attributes for dir() to return -- see the full disclaimer here. But in many common cases, something like the following will work:
class Meta(type):
def __init__(cls, name, bases, attrs):
print "bases = {}".format(bases)
for attr in dir(cls):
if not attr.startswith('_'):
print attr, getattr(cls, attr)
class Foo(object):
__metaclass__ = Meta
hi = 1
class Bar(Foo):
pass
Which prints:
bases = (<type 'object'>,)
hi 1
bases = (<class '__main__.Foo'>,)
hi 1

Using __delitem__ with a class object rather than an instance in Python

I'd like to be able to use __delitem__ with a class-level variable.
My use case can be found here (the answer that uses _reg_funcs) but it basically involves a decorator class keeping a list of all the functions it has decorated. Is there a way I can get the class object to support __delitem__? I know I could keep an instance around specially for this purpose but I'd rather not have to do that.
class Foo(object):
_instances = {}
def __init__(self, my_str):
n = len(self._instances) + 1
self._instances[my_str] = n
print "Now up to {} instances".format(n)
#classmethod
def __delitem__(cls, my_str):
del cls._instances[my_str]
abcd = Foo('abcd')
defg = Foo('defg')
print "Deleting via instance..."
del abcd['abcd']
print "Done!\n"
print "Deleting via class object..."
del Foo['defg']
print "You'll never get here because of a TypeError: 'type' object does not support item deletion"

When you write del obj[key], Python calls the __delitem__ method of the class of obj, not of obj. So del obj[key] results in type(obj).__delitem__(obj, key).
In your case, that means type(Foo).__delitem__(Foo, 'abcd'). type(Foo) is type, and type.__delitem__ is not defined. You can't modify type itself, you'll need to change the type of Foo itself to something that does.
You do that by defining a new metaclass, which is simply a subclass of type, then instructing Python to use your new metaclass to create the Foo class (not instances of Foo, but Foo itself).
class ClassMapping(type):
def __new__(cls, name, bases, dct):
t = type.__new__(cls, name, bases, dct)
t._instances = {}
return t
def __delitem__(cls, my_str):
del cls._instances[my_str]
class Foo(object):
__metaclass__ = ClassMapping
def __init__(self, my_str):
n = len(Foo._instances) + 1
Foo._instances[my_str] = n
print "Now up to {} instances".format(n)
Changing the metaclass of Foo from type to ClassMapping provides Foo with
a class variable _instances that refers to a dictionary
a __delitem__ method that removes arguments from _instances.

How to keep track of class instances?

Toward the end of a program I'm looking to load a specific variable from all the instances of a class into a dictionary.
For example:
class Foo():
def __init__(self):
self.x = {}
foo1 = Foo()
foo2 = Foo()
...
Let's say the number of instances will vary and I want the x dict from each instance of Foo() loaded into a new dict. How would I do that?
The examples I've seen in SO assume one already has the list of instances.

One way to keep track of instances is with a class variable:
class A(object):
instances = []
def __init__(self, foo):
self.foo = foo
A.instances.append(self)
At the end of the program, you can create your dict like this:
foo_vars = {id(instance): instance.foo for instance in A.instances}
There is only one list:
>>> a = A(1)
>>> b = A(2)
>>> A.instances
[<__main__.A object at 0x1004d44d0>, <__main__.A object at 0x1004d4510>]
>>> id(A.instances)
4299683456
>>> id(a.instances)
4299683456
>>> id(b.instances)
4299683456

#JoelCornett's answer covers the basics perfectly. This is a slightly more complicated version, which might help with a few subtle issues.
If you want to be able to access all the "live" instances of a given class, subclass the following (or include equivalent code in your own base class):
from weakref import WeakSet
class base(object):
def __new__(cls, *args, **kwargs):
instance = object.__new__(cls, *args, **kwargs)
if "instances" not in cls.__dict__:
cls.instances = WeakSet()
cls.instances.add(instance)
return instance
This addresses two possible issues with the simpler implementation that #JoelCornett presented:
Each subclass of base will keep track of its own instances separately. You won't get subclass instances in a parent class's instance list, and one subclass will never stumble over instances of a sibling subclass. This might be undesirable, depending on your use case, but it's probably easier to merge the sets back together than it is to split them apart.
The instances set uses weak references to the class's instances, so if you del or reassign all the other references to an instance elsewhere in your code, the bookkeeping code will not prevent it from being garbage collected. Again, this might not be desirable for some use cases, but it is easy enough to use regular sets (or lists) instead of a weakset if you really want every instance to last forever.
Some handy-dandy test output (with the instances sets always being passed to list only because they don't print out nicely):
>>> b = base()
>>> list(base.instances)
[<__main__.base object at 0x00000000026067F0>]
>>> class foo(base):
... pass
...
>>> f = foo()
>>> list(foo.instances)
[<__main__.foo object at 0x0000000002606898>]
>>> list(base.instances)
[<__main__.base object at 0x00000000026067F0>]
>>> del f
>>> list(foo.instances)
[]

You would probably want to use weak references to your instances. Otherwise the class could likely end up keeping track of instances that were meant to have been deleted. A weakref.WeakSet will automatically remove any dead instances from its set.
One way to keep track of instances is with a class variable:
import weakref
class A(object):
instances = weakref.WeakSet()
def __init__(self, foo):
self.foo = foo
A.instances.add(self)
#classmethod
def get_instances(cls):
return list(A.instances) #Returns list of all current instances
At the end of the program, you can create your dict like this:
foo_vars = {id(instance): instance.foo for instance in A.instances}
There is only one list:
>>> a = A(1)
>>> b = A(2)
>>> A.get_instances()
[<inst.A object at 0x100587290>, <inst.A object at 0x100587250>]
>>> id(A.instances)
4299861712
>>> id(a.instances)
4299861712
>>> id(b.instances)
4299861712
>>> a = A(3) #original a will be dereferenced and replaced with new instance
>>> A.get_instances()
[<inst.A object at 0x100587290>, <inst.A object at 0x1005872d0>]

You can also solve this problem using a metaclass:
When a class is created (__init__ method of metaclass), add a new instance registry
When a new instance of this class is created (__call__ method of metaclass), add it to the instance registry.
The advantage of this approach is that each class has a registry - even if no instance exists. In contrast, when overriding __new__ (as in Blckknght's answer), the registry is added when the first instance is created.
class MetaInstanceRegistry(type):
"""Metaclass providing an instance registry"""
def __init__(cls, name, bases, attrs):
# Create class
super(MetaInstanceRegistry, cls).__init__(name, bases, attrs)
# Initialize fresh instance storage
cls._instances = weakref.WeakSet()
def __call__(cls, *args, **kwargs):
# Create instance (calls __init__ and __new__ methods)
inst = super(MetaInstanceRegistry, cls).__call__(*args, **kwargs)
# Store weak reference to instance. WeakSet will automatically remove
# references to objects that have been garbage collected
cls._instances.add(inst)
return inst
def _get_instances(cls, recursive=False):
"""Get all instances of this class in the registry. If recursive=True
search subclasses recursively"""
instances = list(cls._instances)
if recursive:
for Child in cls.__subclasses__():
instances += Child._get_instances(recursive=recursive)
# Remove duplicates from multiple inheritance.
return list(set(instances))
Usage: Create a registry and subclass it.
class Registry(object):
__metaclass__ = MetaInstanceRegistry
class Base(Registry):
def __init__(self, x):
self.x = x
class A(Base):
pass
class B(Base):
pass
class C(B):
pass
a = A(x=1)
a2 = A(2)
b = B(x=3)
c = C(4)
for cls in [Base, A, B, C]:
print cls.__name__
print cls._get_instances()
print cls._get_instances(recursive=True)
print
del c
print C._get_instances()
If using abstract base classes from the abc module, just subclass abc.ABCMeta to avoid metaclass conflicts:
from abc import ABCMeta, abstractmethod
class ABCMetaInstanceRegistry(MetaInstanceRegistry, ABCMeta):
pass
class ABCRegistry(object):
__metaclass__ = ABCMetaInstanceRegistry
class ABCBase(ABCRegistry):
__metaclass__ = ABCMeta
#abstractmethod
def f(self):
pass
class E(ABCBase):
def __init__(self, x):
self.x = x
def f(self):
return self.x
e = E(x=5)
print E._get_instances()

Another option for quick low-level hacks and debugging is to filter the list of objects returned by gc.get_objects() and generate the dictionary on the fly that way. In CPython that function will return you a (generally huge) list of everything the garbage collector knows about, so it will definitely contain all of the instances of any particular user-defined class.
Note that this is digging a bit into the internals of the interpreter, so it may or may not work (or work well) with the likes of Jython, PyPy, IronPython, etc. I haven't checked. It's also likely to be really slow regardless. Use with caution/YMMV/etc.
However, I imagine that some people running into this question might eventually want to do this sort of thing as a one-off to figure out what's going on with the runtime state of some slice of code that's behaving strangely. This method has the benefit of not affecting the instances or their construction at all, which might be useful if the code in question is coming out of a third-party library or something.

Here's a similar approach to Blckknght's, which works with subclasses as well. Thought this might be of interest, if someone ends up here. One difference, if B is a subclass of A, and b is an instance of B, b will appear in both A.instances and B.instances. As stated by Blckknght, this depends on the use case.
from weakref import WeakSet
class RegisterInstancesMixin:
instances = WeakSet()
def __new__(cls, *args, **kargs):
o = object.__new__(cls, *args, **kargs)
cls._register_instance(o)
return o
#classmethod
def print_instances(cls):
for instance in cls.instances:
print(instance)
#classmethod
def _register_instance(cls, instance):
cls.instances.add(instance)
for b in cls.__bases__:
if issubclass(b, RegisterInstancesMixin):
b._register_instance(instance)
def __init_subclass__(cls):
cls.instances = WeakSet()
class Animal(RegisterInstancesMixin):
pass
class Mammal(Animal):
pass
class Human(Mammal):
pass
class Dog(Mammal):
pass
alice = Human()
bob = Human()
cannelle = Dog()
Animal.print_instances()
Mammal.print_instances()
Human.print_instances()
Animal.print_instances() will print three objects, whereas Human.print_instances() will print two.

Using the answer from #Joel Cornett I've come up with the following, which seems to work. i.e. i'm able to total up object variables.
import os
os.system("clear")
class Foo():
instances = []
def __init__(self):
Foo.instances.append(self)
self.x = 5
class Bar():
def __init__(self):
pass
def testy(self):
self.foo1 = Foo()
self.foo2 = Foo()
self.foo3 = Foo()
foo = Foo()
print Foo.instances
bar = Bar()
bar.testy()
print Foo.instances
x_tot = 0
for inst in Foo.instances:
x_tot += inst.x
print x_tot
output:
[<__main__.Foo instance at 0x108e334d0>]
[<__main__.Foo instance at 0x108e334d0>, <__main__.Foo instance at 0x108e33560>, <__main__.Foo instance at 0x108e335a8>, <__main__.Foo instance at 0x108e335f0>]
5
10
15
20

(For Python)
I have found a way to record the class instances via the "dataclass" decorator while defining a class. Define a class attribute 'instances' (or any other name) as a list of the instances you want to record. Append that list with the 'dict' form of created objects via the dunder method __dict__. Thus, the class attribute 'instances' will record instances in the dict form, which you want.
For example,
from dataclasses import dataclass
#dataclass
class player:
instances=[]
def __init__(self,name,rank):
self.name=name
self.rank=rank
self.instances.append(self.__dict__)