I'm writing a custom type Foo and I'd like to achieve the following: when writing
foo = Foo(bar)
then if bar is already an instance of Foo, then Foo(foo) returns that object unmodified, so foo and bar refer to the same object. Otherwise a new object of type Foo should be created, using information from bar in whatever way Foo.__init__ deems appropriate.
How can I do this?
I would assume that Foo.__new__ is the key to this. It should be fairly easy to have Foo.__new__ return the existing object if the instanceof check succeeds, and to call super().__new__ otherwise. But the documentation for __new__ writes this:
If __new__() returns an instance of cls, then the new instance’s __init__() method will be invoked like __init__(self[, ...]), where self is the new instance and the remaining arguments are the same as were passed to __new__().
In this case I would be returning an instance of the requested class, albeit not a new one. Can I somehow prevent the call to __init__? Or do I have to add a check inside __init__ to detect whether it is being invoked for a newly created instance or for an already existing one? The latter sounds like code duplication which should be avoidable.
IMHO, you should directly use __new__ and __init__. The test in init to see whether you should init a new object or already have an existing one is so simple that there is no code duplication and the added complexity is IMHO acceptable
class Foo:
def __new__(cls, obj):
if isinstance(obj, cls):
print("New: same object")
return obj
else:
print("New: create object")
return super(Foo, cls).__new__(cls)
def __init__(self, obj):
if self is obj:
print("init: same object")
else:
print("init: brand new object from ", obj)
# do the actual initialization
It gives as expected:
>>> foo = Foo("x")
New: create object
init: brand new object from x
>>> bar = Foo(foo)
New: same object
init: same object
>>> bar is foo
True
One way to achieve this is by moving the required code into a metaclass like this:
class IdempotentCast(type):
"""Metaclass to ensure that Foo(x) is x if isinstance(x, Foo)"""
def __new__(cls, name, bases, namespace, **kwds):
res = None
defineNew = all(i.__new__ is object.__new__ for i in bases)
if defineNew:
def n(cls, *args, **kwds):
if len(args) == 1 and isinstance(args[0], cls) and not kwds:
return args[0]
else:
return super(res, cls).__new__(cls)
namespace["__new__"] = n
realInit = namespace.get("__init__")
if realInit is not None or defineNew:
#functools.wraps(realInit)
def i(self, *args, **kwds):
if len(args) != 1 or args[0] is not self:
if realInit is None:
return super(res, self).__init__(*args, **kwds)
else:
return realInit(self, *args, **kwds)
namespace["__init__"] = i
res = type.__new__(cls, name, bases, namespace)
return res
class Foo(metaclass=IdempotentCast):
...
That metaclass adds a __new__ method unless there exists a base class that already added a __new__ method. So for class hierarchies where some such class extends another such class, the __new__ method will get added but once. It also wraps the constructor to perform the check whether the first argument is identical to self (thanks to the answer by Serge Ballesta for pointing out this simple check). Otherwise it calls the original constructor, or the base constructor if no constructor had been defined.
Quite a bit of code, but you only need that once, and can use it to introduce these semantics for as many types as you want. If you only need this for a single class, other answers may be more appropriate.
Related
I'm looking for a means to trace all calls on an instance of a class, including calls to dunder methods like __iter__, __getattribute__ etc. I've tried using sys.setprofile, the autologging package, but failed : it doesn't seem to be able to trace calls to dundermethods, or attributes etc.
My usecase is the following. Python's duck typing is awesome, and I would need to be able to "mock" objects, implementing a class only with the necessary methods (including magic or dunders). As I understand duck typing (tell me if i'm misleading), it's basically the fact that the following code will work for any object instance_of_class as long as it implements the methods necessary for iterables (__iter__ and __next__ for example) and that elements returned implements object.__getattribute__('name') which result has __repr__ method, for the printing
import ExternalClass
ExternalClass = my_awesome_decorator_to_trace_them_all(ExternalClass)
instance_of_class = ExternalClass(...)
for element in instance_of_class:
print(element.name)
So, I would like to be able to trace variables (or the original class, and then every instances will get traced), using for example a decorator or anything, that would show the list of all methods called. In the previous example for ExternalClass, it would be:
method __init__ called with arguments ...
method __new__ called with arguments ...
method __iter__ called with arguments ...
method __next__ called with arguments ...
Thanks for the help
--- Update with #martineau efficient and elegant solution
From #martineau's working code on How to wrap every method of a class?
from functools import wraps
from types import FunctionType
def wrapper(method):
#wraps(method)
def wrapped(*args, **kwargs):
class_name = args[0].__class__.__name__
func_name = method.__name__
print('calling {}.{}()... '.format(class_name, func_name))
return method(*args, **kwargs)
return wrapped
class MetaClass(type):
def __new__(meta, classname, bases, classDict):
newClassDict = {}
for attributeName, attribute in classDict.items():
if isinstance(attribute, FunctionType):
# replace it with a wrapped version
attribute = wrapper(attribute)
newClassDict[attributeName] = attribute
return type.__new__(meta, classname, bases, newClassDict)
class SimpleList(object, metaclass=MetaClass):
def __init__(self):
super().__init__()
self.list_elem = list(range(1))
def __iter__(self):
self.index = 0
return self
def __next__(self):
this_index = self.index
if this_index >= len(self.list_elem):
raise StopIteration()
self.index += 1
return self.list_elem[this_index]
simple_list = SimpleList()
print(simple_list)
for element in simple_list:
pass
prints the following
calling SimpleList.__init__()...
<__main__.SimpleList object at 0x1054ce9a0>
calling SimpleList.__iter__()...
calling SimpleList.__next__()...
calling SimpleList.__next__()...
So I am trying to override __new__ and let it exist as a factory to create
derived instances. After reading a bit on SO, I am under the impression that I should be calling __new__ on the derived instance as well.
BaseThing
class BaseThing:
def __init(self, name, **kwargs):
self.name = name
# methods to be derived
ThingFactory
class Thing(BaseThing):
def __new__(cls, name, **kwargs):
if name == 'A':
return A.__new__(name, **kwargs)
if name == 'B':
return B.__new__(name, **kwargs)
def __init__(self, *args, **kwargs):
super().__init__(name, **kwargs)
# methods to be implemented by concrete class (same as those in base)
A
class A(BaseThing):
def __init__(self, name, **kwargs):
super().__init__(name, **kwargs)
B
class B(BaseThing):
def __init__(self, name, **kwargs):
super().__init__(name, **kwargs)
what I am expecting was that it'd just work.
>>> a = Thing('A')
gives me TypeError: object.__new__(X): X is not a type object (str)
I am bit confused by this; when I just return a concrete instance of derived classes, it just worked. i.e.
def __new__(cls, name, **kwargs):
if name == 'A':
return A(name)
if name == 'B':
return B(name)
I don't think this is the correct way to return in __new__; it may duplicate the calls to __init__.
when I am checking signatures of __new__ in object it seems be this one:
#staticmethod # known case of __new__
def __new__(cls, *more): # known special case of object.__new__
""" Create and return a new object. See help(type) for accurate signature. """
pass
I didn't expect this was the one; I'd expect it came with args and kwargs as well. I must have done something wrong here.
it seems to me that I need to inherit object directly in my base but could anyone explain the correct way of doing it?
You're calling __new__ wrong. If you want your __new__ to create an instance of a subclass, you don't call the subclass's __new__; you call the superclass's __new__ as usual, but pass it the subclass as the first argument:
instance = super().__new__(A)
I can't guarantee that this will be enough to fix your problems, since the code you've posted wouldn't reproduce the error you claim; it has other problems that would have caused a different error first (infinite recursion). Particularly, if A and B don't really descend from Thing, that needs different handling.
Part 1
I have a setup where I have a set of classes that I want to mock, my idea was that in the cases where I want to do this I pass a mock keyword argument into the constructor and in __new__ intercept this and instead pass back a mocked version of that object.
It looks like this (Edited the keyword lookup after #mgilsons suggestion):
class RealObject(object):
def __new__(cls, *args, **kwargs):
if kwargs.pop('mock', None):
return MockRealObject()
return super(RealObect, cls).__new__(cls, *args, **kwargs)
def __init__(self, whatever = None):
'''
Constructor
'''
#stuff happens
I then call the constructor like this:
ro = RealObject(mock = bool)
The issue I have here is that I get the following error when bool is False:
TypeError: __init__() got an unexpected keyword argument 'mock'
This works if I add mock as a keyword argument to __init__ but what I am asking if this is possible to avoid. I even pop the mock from the kwargs dict.
This is also a question about the design. Is there a better way to do this? (of course!) I wanted to try doing it this way, without using a factory or a superclass or anything. But still, should I use another keyword maybe? __call__?
Part 2 based on jsbueno's answer
So I wanted to extract the metaclass and the __new__ function into a separate module. I did this:
class Mockable(object):
def __new__(cls, *args, **kwargs):
if kwargs.pop('mock', None):
mock_cls = eval('{0}{1}'.format('Mock',cls.__name__))
return super(mock_cls, mock_cls).__new__(mock_cls)
return super(cls, cls).__new__(cls,*args, **kwargs)
class MockableMetaclass(type):
def __call__(self, *args, **kwargs):
obj = self.__new__(self, *args, **kwargs)
if "mock" in kwargs:
del kwargs["mock"]
obj.__init__(*args, **kwargs)
return obj
And I have defined in a separate module the classes RealObject and MockRealObject.
I have two problems now:
If MockableMetaclass and Mockable are not in the same module as the RealObject class the eval will raise a NameError if I provide mock = True.
If mock = False the code will enter into an endless recursion that ends in an impressive RuntimeError: maximum recursion depth exceeded while calling a Python objec. I'm guessing this is due to RealObject's superclass no longer being object but instead Mockable.
How can I fix these problems? is my approach incorrect? Should I instead have Mockable as a decorator? I tried that but that didn't seem to work since __new__ of an instance is only read-only it seems.
This is a job for the metaclass! :-)
The code responsible to call both __new__ and __init__ when instantiating a Python new-style object lies in the __call__method for the class metaclass. (or the semantically equivalent to that).
In other words - when you do:
RealObject() - what is really called is the RealObject.__class__.__call__ method.
Since without declaring a explicit metaclass, the metaclass is type, it is type.__call__ which is called.
Most recipes around dealing with metaclasses deal with subclassing the __new__ method - automating actions when the class is created. But overriding __call__ we can take actions when the class is instantiated, instead.
In this case, all that is needed is to remove the "mock" keyword parameter, if any, before calling __init__:
class MetaMock(type):
def __call__(cls, *args, **kw):
obj = cls.__new__(cls, *args, **kw)
if "mock" in kw:
del kw["mock"]
obj.__init__(*args, **kw)
return obj
class RealObject(metaclass=MetaMock):
...
A subclass is pretty much essential, since __new__ always passes the arguments to the constructor call to the __init__ method. If you add a subclass via a class decorator as a mixin then you can intercept the mock argument in the subclass __init__:
def mock_with(mock_cls):
class MockMixin(object):
def __new__(cls, *args, **kwargs):
if kwargs.pop('mock'):
return mock_cls()
return super(MockMixin, cls).__new__(cls, *args, **kwargs)
def __init__(self, *args, **kwargs):
kwargs.pop('mock')
super(MockMixin, self).__init__(*args, **kwargs)
def decorator(real_cls):
return type(real_cls.__name__, (MockMixin, real_cls), {})
return decorator
class MockRealObject(object):
pass
#mock_with(MockRealObject)
class RealObject(object):
def __init__(self, whatever=None):
pass
r = RealObject(mock=False)
assert isinstance(r, RealObject)
m = RealObject(mock=True)
assert isinstance(m, MockRealObject)
The alternative is for the subclass __new__ method to return RealObject(cls, *args, **kwargs); in that case, since the returned object isn't an instance of the subclass. However in that case the isinstance check will fail.
I have the following python code:
class FooMeta(type):
def __setattr__(self, name, value):
print name, value
return super(FooMeta, self).__setattr__(name, value)
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
I would have expected __setattr__ of the meta class being called for both FOO and a. However, it is not called at all. When I assign something to Foo.whatever after the class has been defined the method is called.
What's the reason for this behaviour and is there a way to intercept the assignments that happen during the creation of the class? Using attrs in __new__ won't work since I'd like to check if a method is being redefined.
A class block is roughly syntactic sugar for building a dictionary, and then invoking a metaclass to build the class object.
This:
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
Comes out pretty much as if you'd written:
d = {}
d['__metaclass__'] = FooMeta
d['FOO'] = 123
def a(self):
pass
d['a'] = a
Foo = d.get('__metaclass__', type)('Foo', (object,), d)
Only without the namespace pollution (and in reality there's also a search through all the bases to determine the metaclass, or whether there's a metaclass conflict, but I'm ignoring that here).
The metaclass' __setattr__ can control what happens when you try to set an attribute on one of its instances (the class object), but inside the class block you're not doing that, you're inserting into a dictionary object, so the dict class controls what's going on, not your metaclass. So you're out of luck.
Unless you're using Python 3.x! In Python 3.x you can define a __prepare__ classmethod (or staticmethod) on a metaclass, which controls what object is used to accumulate attributes set within a class block before they're passed to the metaclass constructor. The default __prepare__ simply returns a normal dictionary, but you could build a custom dict-like class that doesn't allow keys to be redefined, and use that to accumulate your attributes:
from collections import MutableMapping
class SingleAssignDict(MutableMapping):
def __init__(self, *args, **kwargs):
self._d = dict(*args, **kwargs)
def __getitem__(self, key):
return self._d[key]
def __setitem__(self, key, value):
if key in self._d:
raise ValueError(
'Key {!r} already exists in SingleAssignDict'.format(key)
)
else:
self._d[key] = value
def __delitem__(self, key):
del self._d[key]
def __iter__(self):
return iter(self._d)
def __len__(self):
return len(self._d)
def __contains__(self, key):
return key in self._d
def __repr__(self):
return '{}({!r})'.format(type(self).__name__, self._d)
class RedefBlocker(type):
#classmethod
def __prepare__(metacls, name, bases, **kwargs):
return SingleAssignDict()
def __new__(metacls, name, bases, sad):
return super().__new__(metacls, name, bases, dict(sad))
class Okay(metaclass=RedefBlocker):
a = 1
b = 2
class Boom(metaclass=RedefBlocker):
a = 1
b = 2
a = 3
Running this gives me:
Traceback (most recent call last):
File "/tmp/redef.py", line 50, in <module>
class Boom(metaclass=RedefBlocker):
File "/tmp/redef.py", line 53, in Boom
a = 3
File "/tmp/redef.py", line 15, in __setitem__
'Key {!r} already exists in SingleAssignDict'.format(key)
ValueError: Key 'a' already exists in SingleAssignDict
Some notes:
__prepare__ has to be a classmethod or staticmethod, because it's being called before the metaclass' instance (your class) exists.
type still needs its third parameter to be a real dict, so you have to have a __new__ method that converts the SingleAssignDict to a normal one
I could have subclassed dict, which would probably have avoided (2), but I really dislike doing that because of how the non-basic methods like update don't respect your overrides of the basic methods like __setitem__. So I prefer to subclass collections.MutableMapping and wrap a dictionary.
The actual Okay.__dict__ object is a normal dictionary, because it was set by type and type is finicky about the kind of dictionary it wants. This means that overwriting class attributes after class creation does not raise an exception. You can overwrite the __dict__ attribute after the superclass call in __new__ if you want to maintain the no-overwriting forced by the class object's dictionary.
Sadly this technique is unavailable in Python 2.x (I checked). The __prepare__ method isn't invoked, which makes sense as in Python 2.x the metaclass is determined by the __metaclass__ magic attribute rather than a special keyword in the classblock; which means the dict object used to accumulate attributes for the class block already exists by the time the metaclass is known.
Compare Python 2:
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
def a(self):
pass
Being roughly equivalent to:
d = {}
d['__metaclass__'] = FooMeta
d['FOO'] = 123
def a(self):
pass
d['a'] = a
Foo = d.get('__metaclass__', type)('Foo', (object,), d)
Where the metaclass to invoke is determined from the dictionary, versus Python 3:
class Foo(metaclass=FooMeta):
FOO = 123
def a(self):
pass
Being roughly equivalent to:
d = FooMeta.__prepare__('Foo', ())
d['Foo'] = 123
def a(self):
pass
d['a'] = a
Foo = FooMeta('Foo', (), d)
Where the dictionary to use is determined from the metaclass.
There are no assignments happening during the creation of the class. Or: they are happening, but not in the context you think they are. All class attributes are collected from class body scope and passed to metaclass' __new__, as the last argument:
class FooMeta(type):
def __new__(self, name, bases, attrs):
print attrs
return type.__new__(self, name, bases, attrs)
class Foo(object):
__metaclass__ = FooMeta
FOO = 123
Reason: when the code in the class body executes, there's no class yet. Which means there's no opportunity for metaclass to intercept anything yet.
Class attributes are passed to the metaclass as a single dictionary and my hypothesis is that this is used to update the __dict__ attribute of the class all at once, e.g. something like cls.__dict__.update(dct) rather than doing setattr() on each item. More to the point, it's all handled in C-land and simply wasn't written to call a custom __setattr__().
It's easy enough to do whatever you want to the attributes of the class in your metaclass's __init__() method, since you're passed the class namespace as a dict, so just do that.
During the class creation, your namespace is evaluated to a dict and passed as an argument to the metaclass, together with the class name and base classes. Because of that, assigning a class attribute inside the class definition wouldn't work the way you expect. It doesn't create an empty class and assign everything. You also can't have duplicated keys in a dict, so during class creation attributes are already deduplicated. Only by setting an attribute after the class definition you can trigger your custom __setattr__.
Because the namespace is a dict, there's no way for you to check duplicated methods, as suggested by your other question. The only practical way to do that is parsing the source code.
I'd like to create a Python class decorator (*) that would be able to seamlessly wrap all method types the class might have: instance, class and static.
This is the code I have for now, with the parts that break it commented:
def wrapItUp(method):
def wrapped(*args, **kwargs):
print "This method call was wrapped!"
return method(*args, **kwargs)
return wrapped
dundersICareAbout = ["__init__", "__str__", "__repr__"]#, "__new__"]
def doICareAboutThisOne(cls, methodName):
return (callable(getattr(cls, methodName))
and (not (methodName.startswith("__") and methodName.endswith("__"))
or methodName in dundersICareAbout))
def classDeco(cls):
myCallables = ((aname, getattr(cls, aname)) for aname in dir(cls) if doICareAboutThisOne(cls, aname))
for name, call in myCallables:
print "*** Decorating: %s.%s(...)" % (cls.__name__, name)
setattr(cls, name, wrapItUp(call))
return cls
#classDeco
class SomeClass(object):
def instanceMethod(self, p):
print "instanceMethod: p =", p
#classmethod
def classMethod(cls, p):
print "classMethod: p =", p
#staticmethod
def staticMethod(p):
print "staticMethod: p =", p
instance = SomeClass()
instance.instanceMethod(1)
#SomeClass.classMethod(2)
#instance.classMethod(2)
#SomeClass.staticMethod(3)
#instance.staticMethod(3)
I'm having two issues trying to make this work:
When iterating over all callables, how do I find out if it is of an instance, class or static type?
How to I overwrite the method with a proper wrapped version of it that is invoked correctly for each of those cases?
Currently, this code generates different TypeErrors depending on what commented snippet is uncommented, like:
TypeError: unbound method wrapped() must be called with SomeClass instance as first argument (got int instance instead)
TypeError: classMethod() takes exactly 2 arguments (3 given)
(*): The same problem is much simpler if you're decorating the methods directly.
Because methods are wrappers for functions, to apply a decorator to a method on a class after the class has been constructed, you have to:
Extract the underlying function from the method using its im_func attribute.
Decorate the function.
Re-apply the wrapper.
Overwrite the attribute with the wrapped, decorated function.
It is difficult to distinguish a classmethod from a regular method once the #classmethod decorator has been applied; both kinds of methods are of type instancemethod. However, you can check the im_self attribute and see whether it is None. If so, it's a regular instance method; otherwise it's a classmethod.
Static methods are simple functions (the #staticmethod decorator merely prevents the usual method wrapper from being applied). So you don't have to do anything special for these, it looks like.
So basically your algorithm looks like this:
Get the attribute.
Is it callable? If not, proceed to the next attribute.
Is its type types.MethodType? If so, it is either a class method or an instance method.
If its im_self is None, it is an instance method. Extract the underlying function via the im_func attribute, decorate that, and re-apply the instance method: meth = types.MethodType(func, None, cls)
If its im_self is not None, it is a class method. Exctract the underlying function via im_func and decorate that. Now you have to reapply the classmethod decorator but you can't because classmethod() doesn't take a class, so there's no way to specify what class it will be attached to. Instead you have to use the instance method decorator: meth = types.MethodType(func, cls, type). Note that the type here is the actual built-in, type.
If its type is not types.MethodType then it is a static method or other non-bound callable, so just decorate it.
Set the new attribute back onto the class.
These change somewhat in Python 3 -- unbound methods are functions there, IIRC. In any case this will probably need to be completely rethought there.
There is an undocumented function, inspect.classify_class_attrs, which can tell you which attributes are classmethods or staticmethods. Under the hood, it uses isinstance(obj, staticmethod) and isinstance(obj, classmethod) to classify static and class methods. Following that pattern, this works in both Python2 and Python3:
def wrapItUp(method,kind='method'):
if kind=='static method':
#staticmethod
def wrapped(*args, **kwargs):
return _wrapped(*args,**kwargs)
elif kind=='class method':
#classmethod
def wrapped(cls,*args, **kwargs):
return _wrapped(*args,**kwargs)
else:
def wrapped(self,*args, **kwargs):
return _wrapped(self,*args,**kwargs)
def _wrapped(*args, **kwargs):
print("This method call was wrapped!")
return method(*args, **kwargs)
return wrapped
def classDeco(cls):
for name in (name
for name in dir(cls)
if (callable(getattr(cls,name))
and (not (name.startswith('__') and name.endswith('__'))
or name in '__init__ __str__ __repr__'.split()))
):
method = getattr(cls, name)
obj = cls.__dict__[name] if name in cls.__dict__ else method
if isinstance(obj, staticmethod):
kind = "static method"
elif isinstance(obj, classmethod):
kind = "class method"
else:
kind = "method"
print("*** Decorating: {t} {c}.{n}".format(
t=kind,c=cls.__name__,n=name))
setattr(cls, name, wrapItUp(method,kind))
return cls
#classDeco
class SomeClass(object):
def instanceMethod(self, p):
print("instanceMethod: p = {}".format(p))
#classmethod
def classMethod(cls, p):
print("classMethod: p = {}".format(p))
#staticmethod
def staticMethod(p):
print("staticMethod: p = {}".format(p))
instance = SomeClass()
instance.instanceMethod(1)
SomeClass.classMethod(2)
instance.classMethod(2)
SomeClass.staticMethod(3)
instance.staticMethod(3)