TL;DR -
I have a class that uses a metaclass.
I would like to access the parameters of the object's constructor from the metaclass, just before the initialization process, but I couldn't find a way to access those parameters.
How can I access the constructor's parameters from the metaclass function __new__?
In order to practice the use of metaclasses in python, I would like to create a class that would be used as the supercomputer "Deep Thought" from the book "The Hitchhiker's Guide to the Galaxy".
The purpose of my class would be to store the various queries the supercomputer gets from users.
At the bottom line, it would just get some arguments and store them.
If one of the given arguments is number 42 or the string "The answer to life, the universe, and everything", I don't want to create a new object but rather return a pointer to an existing object.
The idea behind this is that those objects would be the exact same so when using the is operator to compare those two, the result would be true.
In order to be able to use the is operator and get True as an answer, I would need to make sure those variables point to the same object. So, in order to return a pointer to an existing object, I need to intervene in the middle of the initialization process of the object. I cannot check the given arguments at the constructor itself and modify the object's inner-variables accordingly because it would be too late: If I check the given parameters only as part of the __init__ function, those two objects would be allocated on different portions of the memory (they might be equal but won't return True when using the is operator).
I thought of doing something like that:
class SuperComputer(type):
answer = 42
def __new__(meta, name, bases, attributes):
# Check if args contains the number "42"
# or has the string "The answer to life, the universe, and everything"
# If so, just return a pointer to an existing object:
return SuperComputer.answer
# Else, just create the object as it is:
return super(SuperComputer, meta).__new__(meta, name, bases, attributes)
class Query(object):
__metaclass__ = SuperComputer
def __init__(self, *args, **kwargs):
self.args = args
for key, value in kwargs.items():
setattr(self, key, value)
def main():
number = Query(42)
string = Query("The answer to life, the universe, and everything")
other = Query("Sunny", "Sunday", 123)
num2 = Query(45)
print number is string # Should print True
print other is string # Should print False
print number is num2 # Should print False
if __name__ == '__main__':
main()
But I'm stuck on getting the parameters from the constructor.
I saw that the __new__ method gets only four arguments:
The metaclass instance itself, the name of the class, its bases, and its attributes.
How can I send the parameters from the constructor to the metaclass?
What can I do in order to achieve my goal?
You don't need a metaclass for that.
The fact is __init__ is not the "constructor" of an object in Python, rather, it is commonly called an "initializator" . The __new__ is closer to the role of a "constructor" in other languages, and it is not available only for the metaclass - all classes have a __new__ method. If it is not explicitly implemented, the object.__new__ is called directly.
And actually, it is object.__new__ which creates a new object in Python. From pure Python code, there is no other possible way to create an object: it will always go through there. That means that if you implement the __new__ method on your own class, you have the option of not creating a new instance, and instead return another pre-existing instance of the same class (or any other object).
You only have to keep in mind that: if __new__ returns an instance of the same class, then the default behavior is that __init__ is called on the same instance. Otherwise, __init__ is not called.
It is also worth noting that in recent years some recipe for creating "singletons" in Python using metaclasses became popular - it is actually an overkill approach,a s overriding __new__ is also preferable for creating singletons.
In your case, you just need to have a dictionary with the parameters you want to track as your keys, and check if you create a new instance or "recycle" one whenever __new__ runs. The dictionary may be a class attribute, or a global variable at module level - that is your pick:
class Recycler:
_instances = {}
def __new__(cls, parameter1, ...):
if parameter1 in cls._instances:
return cls._instances[parameter1]
self = super().__new__(cls) # don't pass remaining parameters to object.__new__
_instances[parameter1] = self
return self
If you'd have any code in __init__ besides that, move it to __new__ as well.
You can have a baseclass with this behavior and have a class hierarchy without needing to re-implement __new__ for every class.
As for a metaclass, none of its methods are called when actually creating a new instance of the classes created with that metaclass. It would only be of use to automatically insert this behavior, by decorating or creating a fresh __new__ method, on classes created with that metaclass. Since this behavior is easier to track, maintain, and overall to combine with other classes just using ordinary inheritance, no need for a metaclass at all.
Related
I'm in the process of migrating from 2.7 to 3.x and I'm trying to understand the __prepare__ method for metaclasses introduced in PEP3115.
In most of the examples I've seen, implementations of this method ignore the parameters (name, bases, and **kwargs) simply returns a custom dictionary that does something interesting to the namespace provided to the __new__ and __init__ methods for the metaclass. Even the example in PEP3115 does nothing with the parameters.
I don't doubt that there is some good reason for the signature of __prepare__ but I haven't seen the use case.
What are some good examples that demonstrate the rational for making the signature of __prepare__ take these parameters?
__prepare__ will create a namespace for the class like you said, so we can do some logic inside of it like this:
class MyMeta(type):
#classmethod
def __prepare__(metacls, klass_name, bases):
namespace = {'s': 'my string',
'description': 'N/A'}
if klass_name.endswith('Base'):
namespace.update({'description': 'base class'})
return namespace
class KlassBase(metaclass=MyMeta):
def __init__(self, value):
self.value = value
class SubKlass(KlassBase):
def __init__(self, value):
super().__init__(value)
print(KlassBase(5).s)
print(KlassBase(5).description)
print(SubKlass(5).s)
print(SubKlass(5).description)
And you got:
my string
base class
my string
N/A
The reason why we don't do it, because same things could be done in other part of the meta class like : __new__, __init__, or be overrided by the latter. So most of time, we won't do it in __prepare__
The following is an image of the class creation work-flow, which is much more clearer:
[sorry I cannot find the original source of this pic]
When you look at __prepare__, and the state of Python class creation mechanism at the time, it is pretty much clear that what was really needed was a mechanism to enable attribute order preservation.
This way, one would be able to create classes that would describe data records with ordered fields, which is pretty much what humans would expect when describing a record. (Try to imagine a form that each time it is rendered, it shuffles the field ordes, so half the time you will be filling in the country you live in before the country).
Instead of a fixed mechanism to just enable the class body namespace to be an collections.OrderedDict, they came up with __prepare__ which enables this easily, with a single line returning a new OrderedDict instance.
But __prepare__ can have so many uses and abuses, that I think no one really thought of all the possibilities. The parameters ou mentioned are avaliable at the time it is called, and since it exists, there is no reason whatsoever for they not to be passed to the function. Why to cripple one of knowing the class' name inside the __prepare__ function?
So, it is just a powerful and flexible mechanism put in place, and not necessarily all possible use cases were thought of when making it. The "Ordered Attributes" thing n the other hand is so important that it became the default for Python 3.6, with PEP 520, even without any custom metaclass declaration.
__prepare__ receivog bases, for example, would allow one to pre-populate the namespace with certain objects that would be found in the superclass namespaces, overriding the inheritance attribute access, for example.
Or it could simply check the class name against a pre-existing registry, and either raise or pre-populate stuff from there.
One super-easy usage would be to pre-populate __name__ for example, so that class attributes could make use of it:
import collections
class M(type):
#classmethod
def __prepare__(metacls, name, bases):
ns = collections.OrderedDict()
ns["__name__"] = name
class T(metaclass=M):
__table__ = f"_{__name__.lower()}"
trivia
Due to the way functions work as methods in Python 3, one interesting related thing that is not documented anywhere is that if __prepare__ is not explicitly decorated to be a classmethod, it works as a staticmethod, and the target class name is passed directly in the first parameter to it.
I want to implement a singleton pattern in python, and I liked the pattern described in the http://www.python-course.eu/python3_metaclasses.php.
class Singleton(type):
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
return cls._instances[cls]
class SingletonClass(metaclass=Singleton):
pass
class RegularClass():
pass
x = SingletonClass()
y = SingletonClass()
print(x == y)
x = RegularClass()
y = RegularClass()
print(x == y)
And the code works perfect. But, the __call__() does not have the self, and it also does not have #classmethod or #staticmethod declaration.
But, in the Python data model https://docs.python.org/3/reference/datamodel.html#object.__call__ the __call__() method has a self in the arguments.
The code does not work if I pass self, or declare as #staticmethod or #classmethod.
Can someone please explain the logic of the syntax behind the __call__() method.
Naming the first argument of a method cls or self are just a convention. The __call__ method does have a self argument, only it is named cls here. That's because for a metaclass, the method is bound to a class object, and the name reflects this.
The same convention is applied to #classmethod methods; the first argument is a class, always, due to the nature of how a classmethod object is bound, so it makes sense to name that first argument cls.
But you are free to name that first argument anything else. It is not the name that makes a classmethod or a regular method or a method on a metatype work. All that using self or cls does is document what type of object this is, making it easier for other developers to mentally track what is going on.
So no, this is not an implicit class method. That first argument is not bound to the Singleton metaclass object, it is bound to the class that was called. That makes sense, because that class object is an instance of the Singleton metatype.
If you want to dive into how binding works (the process that causes that first argument to be passed in, whatever the name), you can read up on the Descriptor HOWTO. TLDR: functions, property, classmethod and staticmethod objects are all descriptors, and whenever you access them as an attribute on a supporting object such as an instance or a class, they are bound, often causing a different object to be returned as a result, which when called passes in the bound object to the actual function.
Martin's answer says it all. I am adding this so maybe a different wording can throw in more light for different people:
Python call() is this an implicit classmethod?
No. But all "metaclass" methods are implicit "class methods" for the classes that use that metaclass.
That is implicit when we take in account the fact that classes are simply instances of the metaclass. From the language point of view, a class behave almost exactly like any other instance - and any interactions with a class that would trigger dunder (__magic__) methods in an object - like using the "+, -, *, /" operators, or index retrieval with [ ], or calling them with ( ) trigger the corresponding methods on its class. That is ordinarily type.
And, as put in the other answer, Python does not care what name you ut on the first argument of a method. For metaclasses it makes sense to use cls there, since the "instances" the methods are dealing with are metaclasses. As it makes sense that the first argument to a metaclass' __new__ method be named metacls (like in the example bellow). Because the new "cls" is the object we get after calling type.__new__ - the only call possible in pure Python that will actually create a class object.
class Meta(type):
def __new__(metacls, name, bases, namespace):
...
cls = super().__new__(metacls, name, bases, namespace)
...
return cls
(Now, still on the topic of the question: __new__ is a special case
of an implicit static method (even for ordinary classes that are not intended to be metaclasses) - to which Python specially add the first argument, using a different mechanism than what is done to regular classmethods. Thats is why the super().__new__ call above needs to include the metacls as the first parameter)
I'm a little confused by the following example from the python documentation here.
>>> class inch(float):
... "Convert from inch to meter"
... def __new__(cls, arg=0.0):
... return float.__new__(cls, arg*0.0254)
...
>>> print inch(12)
0.3048
>>>
Presumably, float is here is the actual float class defined somewhere deep inside Python. When we call float.__new__(cls, argument) , we're sneakily calling the function that returns instances of float for a given argument, but we're passing it the inch class instead of the float class. Since the inch class doesn't really do anything, why does this work?
Because inch is a subclass of float, it satisfies all the requirements that the float.__new__() instance factory has. It is the job of the __new__(cls) static method to create instances of the first argument, not of it's 'own' class.
Note the word 'static method' there. The __new__ factory is really just a specialist function tied to a class only for inheritance reasons. In other words, it is a function that plays well in a object-oriented hierarchy. You are supposed to find it via super() or perhaps call it directly (as done here). The following would actually be a little more pythonic:
def __new__(cls, arg=0.0):
return super(inch, cls).__new__(cls, arg*0.0254)
because that would call the 'correct' __new__ function if inch were to be used in a multiple-inheritance hierarchy; in this simple example it'll end up calling float.__new__ just the same.
So, __new__(cls, ...) is expected to create an instance of type cls. Why then tie it to a class at all and not make it a more generic function then? Because in the case of float.__new__(cls, value) it not only creates a new instance of type cls, it also sets it's initial value to value. And in order for that to work, float.__new__(...) needs to have intimate knowledge of what the float class looks like. Because inch() is a subclass of float(), it has the exact same necessary bits to be a float() too, and thus when the float.__new__() factory creates a new inch instance, all those bits are there to make it a inch() instance instead.
A little background is needed to answer this question:
Only object.__new__() can create a new instances type of objects, this
kind of objects cannot be subclassed.
An instance has a type, which can be assigned when passing the type name
cls to __new__(cls) as the first argument. class keyword creats
another kind of objects: classes (a.k.a types), and these kinds of objects
can be subclassed.
Now, go back to your example, what
float.__new__(cls, argument)
essentially does is using object.__new__(cls) to create a new instance (float.__base__ is object),
assign the type cls (inch in this case) to it, and also does something with argument defined in float.__new__.
So it is not surprising that it'd work when cls isn't float but inch: the class/type is already created by class inch(float), you are just assigning this type to a new instance.
I am learning Python and so far I can tell the things below about __new__ and __init__:
__new__ is for object creation
__init__ is for object initialization
__new__ is invoked before __init__ as __new__ returns a new instance and __init__ invoked afterwards to initialize inner state.
__new__ is good for immutable object as they cannot be changed once they are assigned. So we can return new instance which has new state.
We can use __new__ and __init__ for both mutable object as its inner state can be changed.
But I have another questions now.
When I create a new instance such as a = MyClass("hello","world"), how these arguments are passed? I mean how I should structure the class using __init__ and __new__ as they are different and both accepts arbitrary arguments besides default first argument.
self keyword is in terms of name can be changed to something else? But I am wondering cls is in terms of name is subject to change to something else as it is just a parameter name?
I made a little experiments as such below:
>>> class MyClass(tuple):
def __new__(tuple):
return [1,2,3]
and I did below:
>>> a = MyClass()
>>> a
[1, 2, 3]
Albeit I said I want to return tuple, this code works fine and returned me [1,2,3]. I knew we were passing the first parameters as the type we wanted to receive once the __new__ function is invoked. We are talking about New function right? I don't know other languages return type other than bound type?
And I did anther things as well:
>>> issubclass(MyClass,list)
False
>>> issubclass(MyClass,tuple)
True
>>> isinstance(a,MyClass)
False
>>> isinstance(a,tuple)
False
>>> isinstance(a,list)
True
I didn't do more experiment because the further wasn't bright and I decided to stop there and decided to ask StackOverflow.
The SO posts I read:
Python object creation
Python's use of __new__ and __init__?
how I should structure the class using __init__ and __new__ as they are different and both accepts arbitrary arguments besides default first argument.
Only rarely will you have to worry about __new__. Usually, you'll just define __init__ and let the default __new__ pass the constructor arguments to it.
self keyword is in terms of name can be changed to something else? But I am wondering cls is in terms of name is subject to change to something else as it is just a parameter name?
Both are just parameter names with no special meaning in the language. But their use is a very strong convention in the Python community; most Pythonistas will never change the names self and cls in these contexts and will be confused when someone else does.
Note that your use of def __new__(tuple) re-binds the name tuple inside the constructor function. When actually implementing __new__, you'll want to do it as
def __new__(cls, *args, **kwargs):
# do allocation to get an object, say, obj
return obj
Albeit I said I want to return tuple, this code works fine and returned me [1,2,3].
MyClass() will have the value that __new__ returns. There's no implicit type checking in Python; it's the responsibility of the programmer to return the correct type ("we're all consenting adults here"). Being able to return a different type than requested can be useful for implementing factories: you can return a subclass of the type requested.
This also explains the issubclass/isinstance behavior you observe: the subclass relationship follows from your use of class MyClass(tuple), the isinstance reflects that you return the "wrong" type from __new__.
For reference, check out the requirements for __new__ in the Python Language Reference.
Edit: ok, here's an example of potentially useful use of __new__. The class Eel keeps track of how many eels are alive in the process and refuses to allocate if this exceeds some maximum.
class Eel(object):
MAX_EELS = 20
n_eels = 0
def __new__(cls, *args, **kwargs):
if cls.n_eels == cls.MAX_EELS:
raise HovercraftFull()
obj = super(Eel, cls).__new__(cls)
cls.n_eels += 1
return obj
def __init__(self, voltage):
self.voltage = voltage
def __del__(self):
type(self).n_eels -= 1
def electric(self):
"""Is this an electric eel?"""
return self.voltage > 0
Mind you, there are smarter ways to accomplish this behavior.
I know that I can dynamically add an instance method to an object by doing something like:
import types
def my_method(self):
# logic of method
# ...
# instance is some instance of some class
instance.my_method = types.MethodType(my_method, instance)
Later on I can call instance.my_method() and self will be bound correctly and everything works.
Now, my question: how to do the exact same thing to obtain the behavior that decorating the new method with #property would give?
I would guess something like:
instance.my_method = types.MethodType(my_method, instance)
instance.my_method = property(instance.my_method)
But, doing that instance.my_method returns a property object.
The property descriptor objects needs to live in the class, not in the instance, to have the effect you desire. If you don't want to alter the existing class in order to avoid altering the behavior of other instances, you'll need to make a "per-instance class", e.g.:
def addprop(inst, name, method):
cls = type(inst)
if not hasattr(cls, '__perinstance'):
cls = type(cls.__name__, (cls,), {})
cls.__perinstance = True
inst.__class__ = cls
setattr(cls, name, property(method))
I'm marking these special "per-instance" classes with an attribute to avoid needlessly making multiple ones if you're doing several addprop calls on the same instance.
Note that, like for other uses of property, you need the class in play to be new-style (typically obtained by inheriting directly or indirectly from object), not the ancient legacy style (dropped in Python 3) that's assigned by default to a class without bases.
Since this question isn't asking about only adding to a spesific instance,
the following method can be used to add a property to the class, this will expose the properties to all instances of the class YMMV.
cls = type(my_instance)
cls.my_prop = property(lambda self: "hello world")
print(my_instance.my_prop)
# >>> hello world
Note: Adding another answer because I think #Alex Martelli, while correct, is achieving the desired result by creating a new class that holds the property, this answer is intended to be more direct/straightforward without abstracting whats going on into its own method.