I experiment with metaclasses to generate the class with the custom special method - particularly, __call__. The generation of the class depends on the parameters the constructor was called with. I've faced a strange effect, simplified example is below:
def trick(self, *args, **kwargs):
print "Works!"
class Test1Factory(type):
def __new__(mcls, name, bases, namespace):
namespace['__call__'] = trick
return type.__new__(mcls, name, bases, namespace)
class Test1(object):
__metaclass__ = Test1Factory
def __init__(self, value):
self._value = value
t1 = Test1(1)
t1() # "Works!"
It works, but it is not really useful, because there is no access to constructor arguments within __new__. type.__call__ should do the trick:
import types
class Test2Factory(type):
def __call__(self, *args, **kwargs):
obj = type.__call__(self, *args, **kwargs)
setattr(obj, '__call__', types.MethodType(trick, obj, Test2))
return obj
class Test2(object):
__metaclass__ = Test2Factory
def __init__(self, value):
self._value = value
t2 = Test2(2)
t2.__call__() # "Works!"
t2() # TypeError: 'Test2' object is not callable
As far as I understand, instance() is similar to instance.__call__(), but it is not the case here. Using __new__ static method of the class does the same. I have a workaround that does not use metaclasses at all, but just want to understand the phenomena. Python version is 2.7.5
The wrong assumption may be in “instance() is similar to instance.__call__()”, as __call__ is not looked up for in instance, but in instance's type. That is, the __call__ used is not that of instance, but that of instance.__class__ or type(instance).
Any __call__ attribute defined on the instance solely, may be accessed regularly as any other attribute, but will not be used when instance is called as in instance(). That's part of Python's semantic.
Try to define a __call__ both on an instance and on its type, and see what you get.
If I understand the question correctly, the question has the same background as another I had, and which gets an answer (summarized, with demonstrations by experiments, in the question's post) : “How do Python tell “this is called as a function”?”.
Related
In my undertstanding, decorator class should contain __call__ or __new__ method. But cached_property in cpython repo doesn't follow the rules. Can anyone explain it for me?
class cached_property:
def __init__(self, func):
xxx
def __set_name__(self, owner, name):
xxx
def __get__(self, instance, owner=None):
xxx
__class_getitem__ = classmethod(GenericAlias)
Do all decorator classes need __call__?
decorator class should contain __call__ or __new__ method
Not all decorator classes need to implement __call__.
It's only required when we want to call the decorated object with ().
A decorator class that takes a callable to produce a callable has to implement __call__.
In this example, __call__ is implemented because we want to do data.calculate().
# Decorator to call and cache the function immediately
class PreCompute:
def __init__(self, func):
self.value = func()
def __call__(self, *args, **kwds):
return self.value
class Data:
#PreCompute
def calculate():
print("Data.calculate called")
return 42
data = Data()
# This actually calls PreCompute's __call__
print(data.calculate())
The definition of class Data here is roughly desugared to something like this,
so when calling data.calculate() we're actually calling the __call__ function from class PreCompute.
class Data:
def calculate():
print("Data.calculate called")
return 42
calculate = PreCompute(calculate)
A decorator class that takes a callable but does not produce a callable does not have to implement __call__.
For example, we can modify the class Precompute decorator to the following code, which allows us to access data.calculate as if it's an attribute.
For more information about what __get__ does, see Descriptor HowTo Guide from Python docs.
class PreCompute:
def __init__(self, func):
self.value = func()
def __get__(self, instance, owner):
return self.value
class Data:
#PreCompute
def calculate():
print("Data.calculate called")
return 42
data = Data()
# Access .calculate like an attribute
print(data.calculate)
What about __new__?
I'm not sure how OP got the impression that decorator classes must define either __call__ or __new__. I've seen __new__ being defined for use cases like #singleton decorator for classes, but as discussed in the previous section about __call__, this is also not strictly required. The only function we must define is an __init__ that receives the object to be decorated.
How does #functools.cached_property work, then?
Now going back to the question, notice from the documentation of #functools.cached_property that
it "transform a method of a class into a property", which is to be accessed without the parentheses ().
Therefore, class cached_property implements __get__ but not __call__, which is similar to the second example above.
So I am trying to override __new__ and let it exist as a factory to create
derived instances. After reading a bit on SO, I am under the impression that I should be calling __new__ on the derived instance as well.
BaseThing
class BaseThing:
def __init(self, name, **kwargs):
self.name = name
# methods to be derived
ThingFactory
class Thing(BaseThing):
def __new__(cls, name, **kwargs):
if name == 'A':
return A.__new__(name, **kwargs)
if name == 'B':
return B.__new__(name, **kwargs)
def __init__(self, *args, **kwargs):
super().__init__(name, **kwargs)
# methods to be implemented by concrete class (same as those in base)
A
class A(BaseThing):
def __init__(self, name, **kwargs):
super().__init__(name, **kwargs)
B
class B(BaseThing):
def __init__(self, name, **kwargs):
super().__init__(name, **kwargs)
what I am expecting was that it'd just work.
>>> a = Thing('A')
gives me TypeError: object.__new__(X): X is not a type object (str)
I am bit confused by this; when I just return a concrete instance of derived classes, it just worked. i.e.
def __new__(cls, name, **kwargs):
if name == 'A':
return A(name)
if name == 'B':
return B(name)
I don't think this is the correct way to return in __new__; it may duplicate the calls to __init__.
when I am checking signatures of __new__ in object it seems be this one:
#staticmethod # known case of __new__
def __new__(cls, *more): # known special case of object.__new__
""" Create and return a new object. See help(type) for accurate signature. """
pass
I didn't expect this was the one; I'd expect it came with args and kwargs as well. I must have done something wrong here.
it seems to me that I need to inherit object directly in my base but could anyone explain the correct way of doing it?
You're calling __new__ wrong. If you want your __new__ to create an instance of a subclass, you don't call the subclass's __new__; you call the superclass's __new__ as usual, but pass it the subclass as the first argument:
instance = super().__new__(A)
I can't guarantee that this will be enough to fix your problems, since the code you've posted wouldn't reproduce the error you claim; it has other problems that would have caused a different error first (infinite recursion). Particularly, if A and B don't really descend from Thing, that needs different handling.
I'm interested in using the __new__ functionality to inject code into the __init__ function of subclasses. My understanding from the documentation is that python will call __init__ on the instance returned by __new__. However, my efforts to change the value of __init__ in the instance before returning it from __new__ don't seem to work.
class Parent(object):
def __new__(cls, *args, **kwargs):
new_object = super(Parent, cls).__new__(cls)
user_init = new_object.__init__
def __init__(self, *args, **kwargs):
print("New __init__ called")
user_init(self, *args, **kwargs)
self.extra()
print("Replacing __init__")
setattr(new_object, '__init__', __init__)
return new_object
def extra(self):
print("Extra called")
class Child(Parent):
def __init__(self):
print("Original __init__ called")
super(Child, self).__init__()
c = Child()
The above code prints:
Replacing __init__
Original __init__ called
but I would expect it to print
Replacing __init__
New __init__ called
Original __init__ called
Extra called
Why not?
I feel like Python is calling the original value of __init__, regardless of what I set it to in __new__. Running introspection on c.__init__ shows that the new version is in place, but it hasn't been called as part of the object creation.
Well, the new object is expected to be empty before the __init__ is called. So probably python, as optimization, does not bother to query the object and goes to fetch __init__ straight from the class.
Therefore you'll have to modify __init__ of the subclasses themselves. Fortunately Python has a tool for that, metaclasses.
In Python 2, you set metaclass by setting special member:
class Parent(object):
__metaclass__ = Meta
...
See Python2 documentation
In Python 3, you set metaclass via keyword attribute in the parent list, so
class Parent(metaclass=Meta):
...
See Python3 documentation
The metaclass is a base class for the class instance. It has to be derived from type and in it's __new__ it can modify the class being created (I believe the __init__ should be called too, but the examples override __new__, so I'll go with it). The __new__ will be similar to what you have:
class Meta(type):
def __new__(mcs, name, bases, namespace, **kwargs):
new_cls = super(Meta, mcs).__new__(mcs, name, bases, namespace, **kwargs)
user_init = new_cls.__init__
def __init__(self, *args, **kwargs):
print("New __init__ called")
user_init(self, *args, **kwargs)
self.extra()
print("Replacing __init__")
setattr(new_cls, '__init__', __init__)
return new_cls
(using the Python 3 example, but the signature in Python 2 seems to be the same except there are no **kwargs, but adding them shouldn't hurt; I didn't test it).
I suspect the answer is that __init__ is a special function, internally it is defined as a class method, and as a result cannot be replaced by reassigning it in an instance of the object.
In Python, all objects are represented by the PyObject in C, which has a pointer to a PyTypeObject. This contains a member called tp_init that I believe contains a pointer to the __init__ function.
The other solution works, because we are modifying the class, not an instance of the object.
I'm editing the original question because we're all focusing on SHOULD you ever want to do this. My question is simply CAN I do this and HOW (understanding that there may be several solutions). So I'm just going to leave the actual question and cut out the background.
Suppose I have a base class and a child class. Is there anything I can do in the base class to prevent __init__ from being called on the child class - or at least throw an exception or even log if __init__ exists or is called on the child class? I do want the __init__ method to be called on the parent class.
Edit/Conclusion - After exploring the options presented in the answers, I decided that doing this would be bad style. I will solve my problem a different way. Nonetheless, hopefully the answers below are helpful in case someone else wants to do this.
" Whether or not I should or need to do it is a separate discussion :)"
Please, keep that in mind.
But it can be done - when a class is instantiated, not only the syntax is just like
a method call - with the class object name followed y a parenthesis - the class itself (which is a Python object), is called - as a callable object.
Calling an object in Python invokes the __call__ magic method in its class. Therefore, instantiating a class, invokes the __call__ method on its metaclass.
What is inside this __call__ method in the standard metaclass (which is "type") is roughly equivalent to:
def __call__(cls, *args, **kw):
self = cls.__new__(cls, *args, **kw)
cls.__init__(self, *args, **kw)
return self
So, if you write a metaclass, overriding __call__ and suppress the call to __init__ in these, it won't be called at all:
class Meta(type):
def __call__(cls, *args, **kw):
return cls.__new__(cls, *args, **kw)
class NoInit(object):
__metaclass__ = Meta
def __init__(self):
print "Hello!"
NoInit()
If you want just to avoid that sublcasses have __init__ instead of not calling it, you can do a much simpler metaclass that would just raise an exception at class instantiation time:
class Meta(type):
def __new__(metacls, name, bases, dct):
if "__init__" in dct:
raise NameError("Classes in this hierarchy should not have an __init__ method")
return type.__new__(metacls, name, bases, dct)
That's quite doable, but I don't think you should.
Tell the users how to use your class and they should obey. Also, if someone is subclassing he should know how to call the parent's initialization method.
As a proof of concept, here's how it can be done with metaclasses (Python 2.x syntax):
>>> class WhoMovedMyInit(object):
class __metaclass__(type):
def __init__(self, *args, **kw):
super(type,self).__init__(*args, **kw)
if self.__init__ is not WhoMovedMyInit.__init__:
raise Exception('Dude, I told not to override my __init__')
>>> class IAmOk(WhoMovedMyInit):
pass
>>> class Lol(WhoMovedMyInit):
def __init__(self):
pass
Traceback (most recent call last):
File "<pyshell#35>", line 1, in <module>
class Lol(WhoMovedMyInit):
File "<pyshell#31>", line 6, in __init__
raise Exception('Dude, I told not to override my __init__')
Exception: Dude, I told not to override my __init__
You can also replace the subclass __init__ method to one which warns the user or raises an error on "runtime".
Most of these answers are outdated.
This can be easily done since python 3.6. It was defined in PEP487.
class Base:
def __init_subclass__(cls):
if Base.__init__ is not cls.__init__:
raise Exception(f'Do not override {cls}.__init__')
class Good(Base):
pass
class Bad(Base):
def __init__(self):
pass
Good() # No problem
Bad() # Raises Exception
I'm writing a decorator for methods that must inspect the parent methods (the methods of the same name in the parents of the class in which I'm decorating).
Example (from the fourth example of PEP 318):
def returns(rtype):
def check_returns(f):
def new_f(*args, **kwds):
result = f(*args, **kwds)
assert isinstance(result, rtype), \
"return value %r does not match %s" % (result,rtype)
return result
new_f.func_name = f.func_name
# here I want to reach the class owning the decorated method f,
# it should give me the class A
return new_f
return check_returns
class A(object):
#returns(int)
def compute(self, value):
return value * 3
So I'm looking for the code to type in place of # here I want...
Thanks.
As bobince said it, you can't access the surrounding class, because at the time the decorator is invoked, the class does not exist yet. If you need access to the full dictionary of the class and the bases, you should consider a metaclass:
__metaclass__
This variable can be any callable accepting arguments for name, bases, and dict. Upon class creation, the callable is used instead of the built-in type().
Basically, we convert the returns decorator into something that just tells the metaclass to do some magic on class construction:
class CheckedReturnType(object):
def __init__(self, meth, rtype):
self.meth = meth
self.rtype = rtype
def returns(rtype):
def _inner(f):
return CheckedReturnType(f, rtype)
return _inner
class BaseInspector(type):
def __new__(mcs, name, bases, dct):
for obj_name, obj in dct.iteritems():
if isinstance(obj, CheckedReturnType):
# do your wrapping & checking here, base classes are in bases
# reassign to dct
return type.__new__(mcs, name, bases, dct)
class A(object):
__metaclass__ = BaseInspector
#returns(int)
def compute(self, value):
return value * 3
Mind that I have not tested this code, please leave comments if I should update this.
There are some articles on metaclasses by the highly recommendable David Mertz, which you might find interesting in this context.
here I want to reach the class owning the decorated method f
You can't because at the point of decoration, no class owns the method f.
class A(object):
#returns(int)
def compute(self, value):
return value * 3
Is the same as saying:
class A(object):
pass
#returns(int)
def compute(self, value):
return value*3
A.compute= compute
Clearly, the returns() decorator is built before the function is assigned to an owner class.
Now when you write a function to a class (either inline, or explicitly like this) it becomes an unbound method object. Now it has a reference to its owner class, which you can get by saying:
>>> A.compute.im_class
<class '__main__.A'>
So you can read f.im_class inside ‘new_f’, which is executed after the assignment, but not in the decorator itself.
(And even then it's a bit ugly relying on a CPython implementation detail if you don't need to. I'm not quite sure what you're trying to do, but things involving “get the owner class” are often doable using metaclasses.)