Related
I'm trying to add a __set_name__ hook to the descriptor produced by functools.wraps inside a decorator, but it is never called and I don't see any error messages:
import functools
def wrap(fn):
"""Decorator."""
#functools.wraps(fn)
def w(*args, **kwargs):
return fn(*args, **kwargs)
# This never gets called.
def set_name(self, obj, name):
print(f"inside __set_name__: {self=}, {obj=}, {name=}")
w.__set_name__ = set_name.__get__(w)
return w
class Foo:
#wrap
def foo(self):
pass
From what I understand, wrap() is called and its return value bound to the foo variable in the class's execution frame before the Foo class is created, so the __set_name__ hook should be in place by the time Python looks for it. So why isn't it being called?
Whenever Python looks for magic methods, it looks on the type of the object, not the instance. What you've done is take a function object (the return value of functools.wrap, in this case) and assign something on its __dict__. But for efficiency (and correctness, in some cases), special methods like __set_name__ bypass __dict__ and look on the type object directly. See Special method lookup for details and a rationale.
To make your code work, you need to create a custom callable class (i.e. a class which defines a function called __call__), define __set_name__ on that class, and then make w an instance of that class.
In Python, I can create a class method using the #classmethod decorator:
>>> class C:
... #classmethod
... def f(cls):
... print(f'f called with cls={cls}')
...
>>> C.f()
f called with cls=<class '__main__.C'>
Alternatively, I can use a normal (instance) method on a metaclass:
>>> class M(type):
... def f(cls):
... print(f'f called with cls={cls}')
...
>>> class C(metaclass=M):
... pass
...
>>> C.f()
f called with cls=<class '__main__.C'>
As shown by the output of C.f(), these two approaches provide similar functionality.
What are the differences between using #classmethod and using a normal method on a metaclass?
As classes are instances of a metaclass, it is not unexpected that an "instance method" on the metaclass will behave like a classmethod.
However, yes, there are differences - and some of them are more than semantic:
The most important difference is that a method in the metaclass is not "visible" from a class instance. That happens because the attribute lookup in Python (in a simplified way - descriptors may take precedence) search for an attribute in the instance - if it is not present in the instance, Python then looks in that instance's class, and then the search continues on the superclasses of the class, but not on the classes of the class. The Python stdlib make use of this feature in the abc.ABCMeta.register method.
That feature can be used for good, as methods related with the class themselves are free to be re-used as instance attributes without any conflict (but a method would still conflict).
Another difference, though obvious, is that a method declared in the metaclass can be available in several classes, not otherwise related - if you have different class hierarchies, not related at all in what they deal with, but want some common functionality for all classes, you'd have to come up with a mixin class, that would have to be included as base in both hierarchies (say for including all classes in an application registry). (NB. the mixin may sometimes be a better call than a metaclass)
A classmethod is a specialized "classmethod" object, while a method in the metaclass is an ordinary function.
So, it happens that the mechanism that classmethods use is the "descriptor protocol". While normal functions feature a __get__ method that will insert the self argument when they are retrieved from an instance, and leave that argument empty when retrieved from a class, a classmethod object have a different __get__, that will insert the class itself (the "owner") as the first parameter in both situations.
This makes no practical differences most of the time, but if you want access to the method as a function, for purposes of adding dynamically adding decorator to it, or any other, for a method in the metaclass meta.method retrieves the function, ready to be used, while you have to use cls.my_classmethod.__func__ to retrieve it from a classmethod (and then you have to create another classmethod object and assign it back, if you do some wrapping).
Basically, these are the 2 examples:
class M1(type):
def clsmethod1(cls):
pass
class CLS1(metaclass=M1):
pass
def runtime_wrap(cls, method_name, wrapper):
mcls = type(cls)
setattr(mcls, method_name, wrapper(getatttr(mcls, method_name)))
def wrapper(classmethod):
def new_method(cls):
print("wrapper called")
return classmethod(cls)
return new_method
runtime_wrap(cls1, "clsmethod1", wrapper)
class CLS2:
#classmethod
def classmethod2(cls):
pass
def runtime_wrap2(cls, method_name, wrapper):
setattr(cls, method_name, classmethod(
wrapper(getatttr(cls, method_name).__func__)
)
)
runtime_wrap2(cls1, "clsmethod1", wrapper)
In other words: apart from the important difference that a method defined in the metaclass is visible from the instance and a classmethod object do not, the other differences, at runtime will seem obscure and meaningless - but that happens because the language does not need to go out of its way with special rules for classmethods: Both ways of declaring a classmethod are possible, as a consequence from the language design - one, for the fact that a class is itself an object, and another, as a possibility among many, of the use of the descriptor protocol which allows one to specialize attribute access in an instance and in a class:
The classmethod builtin is defined in native code, but it could just be coded in pure python and would work in the exact same way. The 5 line class bellow can be used as a classmethod decorator with no runtime differences to the built-in #classmethod" at all (though distinguishable through introspection such as calls toisinstance, and evenrepr` of course):
class myclassmethod:
def __init__(self, func):
self.__func__ = func
def __get__(self, instance, owner):
return lambda *args, **kw: self.__func__(owner, *args, **kw)
And, beyond methods, it is interesting to keep in mind that specialized attributes such as a #property on the metaclass will work as specialized class attributes, just the same, with no surprising behavior at all.
When you phrase it like you did in the question, the #classmethod and metaclasses may look similar but they have rather different purposes. The class that is injected in the #classmethod's argument is usually used for constructing an instance (i.e. an alternative constructor). On the other hand, the metaclasses are usually used to modify the class itself (e.g. like what Django does with its models DSL).
That is not to say that you can't modify the class inside a classmethod. But then the question becomes why didn't you define the class in the way you want to modify it in the first place? If not, it might suggest a refactor to use multiple classes.
Let's expand the first example a bit.
class C:
#classmethod
def f(cls):
print(f'f called with cls={cls}')
Borrowing from the Python docs, the above will expand to something like the following:
class ClassMethod(object):
"Emulate PyClassMethod_Type() in Objects/funcobject.c"
def __init__(self, f):
self.f = f
def __get__(self, obj, klass=None):
if klass is None:
klass = type(obj)
def newfunc(*args):
return self.f(klass, *args)
return newfunc
class C:
def f(cls):
print(f'f called with cls={cls}')
f = ClassMethod(f)
Note how __get__ can take either an instance or the class (or both), and thus you can do both C.f and C().f. This is unlike the metaclass example you give which will throw an AttributeError for C().f.
Moreover, in the metaclass example, f does not exist in C.__dict__. When looking up the attribute f with C.f, the interpreter looks at C.__dict__ and then after failing to find, looks at type(C).__dict__ (which is M.__dict__). This may matter if you want the flexibility to override f in C, although I doubt this will ever be of practical use.
In your example, the difference would be in some other classes that will have M set as their metaclass.
class M(type):
def f(cls):
pass
class C(metaclass=M):
pass
class C2(metaclass=M):
pass
C.f()
C2.f()
class M(type):
pass
class C(metaclass=M):
#classmethod
def f(cls):
pass
class C2(metaclass=M):
pass
C.f()
# C2 does not have 'f'
Here is more on metaclasses
What are some (concrete) use-cases for metaclasses?
Both #classmethod and Metaclass are different.
Everything in python is an object. Every thing means every thing.
What is Metaclass ?
As said every thing is an object. Classes are also objects in fact classes are instances of other mysterious objects formally called as meta-classes. Default metaclass in python is "type" if not specified
By default all classes defined are instances of type.
Classes are instances of Meta-Classes
Few important points are to understand metioned behaviour
As classes are instances of meta classes.
Like every instantiated object, like objects(instances) get their attributes from class. Class will get it's attributes from Meta-Class
Consider Following Code
class Meta(type):
def foo(self):
print(f'foo is called self={self}')
print('{} is instance of {}: {}'.format(self, Meta, isinstance(self, Meta)))
class C(metaclass=Meta):
pass
C.foo()
Where,
class C is instance of class Meta
"class C" is class object which is instance of "class Meta"
Like any other object(instance) "class C" has access it's attributes/methods defined in it's class "class Meta"
So, decoding "C.foo()" . "C" is instance of "Meta" and "foo" is method calling through instance of "Meta" which is "C".
First argument of method "foo" is reference to instance not class unlike "classmethod"
We can verify as if "class C" is instance of "Class Meta
isinstance(C, Meta)
What is classmethod?
Python methods are said to be bound. As python imposes the restriction that method has to be invoked with instance only.
Sometimes we might want to invoke methods directly through class without any instance (much like static members in java) with out having to create any instance.By default instance is required to call method. As a workaround python provides built-in function classmethod to bind given method to class instead of instance.
As class methods are bound to class. It takes at least one argument which is reference to class itself instead of instance (self)
if built-in function/decorator classmethod is used. First argument
will be reference to class instead of instance
class ClassMethodDemo:
#classmethod
def foo(cls):
print(f'cls is ClassMethodDemo: {cls is ClassMethodDemo}')
As we have used "classmethod" we call method "foo" without creating any instance as follows
ClassMethodDemo.foo()
Above method call will return True. Since first argument cls is indeed reference to "ClassMethodDemo"
Summary:
Classmethod's receive first argument which is "a reference to class(traditionally referred as cls) itself"
Methods of meta-classes are not classmethods. Methods of Meta-classes receive first argument which is "a reference to instance(traditionally referred as self) not class"
I'm just trying to streamline one of my classes and have introduced some functionality in the same style as the flyweight design pattern.
However, I'm a bit confused as to why __init__ is always called after __new__. I wasn't expecting this. Can anyone tell me why this is happening and how I can implement this functionality otherwise? (Apart from putting the implementation into the __new__ which feels quite hacky.)
Here's an example:
class A(object):
_dict = dict()
def __new__(cls):
if 'key' in A._dict:
print "EXISTS"
return A._dict['key']
else:
print "NEW"
return super(A, cls).__new__(cls)
def __init__(self):
print "INIT"
A._dict['key'] = self
print ""
a1 = A()
a2 = A()
a3 = A()
Outputs:
NEW
INIT
EXISTS
INIT
EXISTS
INIT
Why?
Use __new__ when you need to control
the creation of a new instance.
Use
__init__ when you need to control initialization of a new instance.
__new__ is the first step of instance creation. It's called first, and is
responsible for returning a new
instance of your class.
In contrast,
__init__ doesn't return anything; it's only responsible for initializing the
instance after it's been created.
In general, you shouldn't need to
override __new__ unless you're
subclassing an immutable type like
str, int, unicode or tuple.
From April 2008 post: When to use __new__ vs. __init__? on mail.python.org.
You should consider that what you are trying to do is usually done with a Factory and that's the best way to do it. Using __new__ is not a good clean solution so please consider the usage of a factory. Here's a good example: ActiveState Fᴀᴄᴛᴏʀʏ ᴘᴀᴛᴛᴇʀɴ Recipe.
__new__ is static class method, while __init__ is instance method.
__new__ has to create the instance first, so __init__ can initialize it. Note that __init__ takes self as parameter. Until you create instance there is no self.
Now, I gather, that you're trying to implement singleton pattern in Python. There are a few ways to do that.
Also, as of Python 2.6, you can use class decorators.
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance
#singleton
class MyClass:
...
In most well-known OO languages, an expression like SomeClass(arg1, arg2) will allocate a new instance, initialise the instance's attributes, and then return it.
In most well-known OO languages, the "initialise the instance's attributes" part can be customised for each class by defining a constructor, which is basically just a block of code that operates on the new instance (using the arguments provided to the constructor expression) to set up whatever initial conditions are desired. In Python, this corresponds to the class' __init__ method.
Python's __new__ is nothing more and nothing less than similar per-class customisation of the "allocate a new instance" part. This of course allows you to do unusual things such as returning an existing instance rather than allocating a new one. So in Python, we shouldn't really think of this part as necessarily involving allocation; all that we require is that __new__ comes up with a suitable instance from somewhere.
But it's still only half of the job, and there's no way for the Python system to know that sometimes you want to run the other half of the job (__init__) afterwards and sometimes you don't. If you want that behavior, you have to say so explicitly.
Often, you can refactor so you only need __new__, or so you don't need __new__, or so that __init__ behaves differently on an already-initialised object. But if you really want to, Python does actually allow you to redefine "the job", so that SomeClass(arg1, arg2) doesn't necessarily call __new__ followed by __init__. To do this, you need to create a metaclass, and define its __call__ method.
A metaclass is just the class of a class. And a class' __call__ method controls what happens when you call instances of the class. So a metaclass' __call__ method controls what happens when you call a class; i.e. it allows you to redefine the instance-creation mechanism from start to finish. This is the level at which you can most elegantly implement a completely non-standard instance creation process such as the singleton pattern. In fact, with less than 10 lines of code you can implement a Singleton metaclass that then doesn't even require you to futz with __new__ at all, and can turn any otherwise-normal class into a singleton by simply adding __metaclass__ = Singleton!
class Singleton(type):
def __init__(self, *args, **kwargs):
super(Singleton, self).__init__(*args, **kwargs)
self.__instance = None
def __call__(self, *args, **kwargs):
if self.__instance is None:
self.__instance = super(Singleton, self).__call__(*args, **kwargs)
return self.__instance
However this is probably deeper magic than is really warranted for this situation!
To quote the documentation:
Typical implementations create a new instance of the class by invoking
the superclass's __new__() method using "super(currentclass,
cls).__new__(cls[, ...])"with appropriate arguments and then
modifying the newly-created instance as necessary before returning it.
...
If __new__() does not return an instance of cls, then the new
instance's __init__() method will not be invoked.
__new__() is intended mainly to allow subclasses of immutable
types (like int, str, or tuple) to customize instance creation.
I realize that this question is quite old but I had a similar issue.
The following did what I wanted:
class Agent(object):
_agents = dict()
def __new__(cls, *p):
number = p[0]
if not number in cls._agents:
cls._agents[number] = object.__new__(cls)
return cls._agents[number]
def __init__(self, number):
self.number = number
def __eq__(self, rhs):
return self.number == rhs.number
Agent("a") is Agent("a") == True
I used this page as a resource http://infohost.nmt.edu/tcc/help/pubs/python/web/new-new-method.html
When __new__ returns instance of the same class, __init__ is run afterwards on returned object. I.e. you can NOT use __new__ to prevent __init__ from being run. Even if you return previously created object from __new__, it will be double (triple, etc...) initialized by __init__ again and again.
Here is the generic approach to Singleton pattern which extends vartec answer above and fixes it:
def SingletonClass(cls):
class Single(cls):
__doc__ = cls.__doc__
_initialized = False
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(Single, cls).__new__(cls, *args, **kwargs)
return cls._instance
def __init__(self, *args, **kwargs):
if self._initialized:
return
super(Single, self).__init__(*args, **kwargs)
self.__class__._initialized = True # Its crucial to set this variable on the class!
return Single
Full story is here.
Another approach, which in fact involves __new__ is to use classmethods:
class Singleton(object):
__initialized = False
def __new__(cls, *args, **kwargs):
if not cls.__initialized:
cls.__init__(*args, **kwargs)
cls.__initialized = True
return cls
class MyClass(Singleton):
#classmethod
def __init__(cls, x, y):
print "init is here"
#classmethod
def do(cls):
print "doing stuff"
Please pay attention, that with this approach you need to decorate ALL of your methods with #classmethod, because you'll never use any real instance of MyClass.
I think the simple answer to this question is that, if __new__ returns a value that is the same type as the class, the __init__ function executes, otherwise it won't. In this case your code returns A._dict('key') which is the same class as cls, so __init__ will be executed.
class M(type):
_dict = {}
def __call__(cls, key):
if key in cls._dict:
print 'EXISTS'
return cls._dict[key]
else:
print 'NEW'
instance = super(M, cls).__call__(key)
cls._dict[key] = instance
return instance
class A(object):
__metaclass__ = M
def __init__(self, key):
print 'INIT'
self.key = key
print
a1 = A('aaa')
a2 = A('bbb')
a3 = A('aaa')
outputs:
NEW
INIT
NEW
INIT
EXISTS
NB As a side effect M._dict property automatically becomes accessible from A as A._dict so take care not to overwrite it incidentally.
An update to #AntonyHatchkins answer, you probably want a separate dictionary of instances for each class of the metatype, meaning that you should have an __init__ method in the metaclass to initialize your class object with that dictionary instead of making it global across all the classes.
class MetaQuasiSingleton(type):
def __init__(cls, name, bases, attibutes):
cls._dict = {}
def __call__(cls, key):
if key in cls._dict:
print('EXISTS')
instance = cls._dict[key]
else:
print('NEW')
instance = super().__call__(key)
cls._dict[key] = instance
return instance
class A(metaclass=MetaQuasiSingleton):
def __init__(self, key):
print 'INIT'
self.key = key
print()
I have gone ahead and updated the original code with an __init__ method and changed the syntax to Python 3 notation (no-arg call to super and metaclass in the class arguments instead of as an attribute).
Either way, the important point here is that your class initializer (__call__ method) will not execute either __new__ or __init__ if the key is found. This is much cleaner than using __new__, which requires you to mark the object if you want to skip the default __init__ step.
__new__ should return a new, blank instance of a class. __init__ is then called to initialise that instance. You're not calling __init__ in the "NEW" case of __new__, so it's being called for you. The code that is calling __new__ doesn't keep track of whether __init__ has been called on a particular instance or not nor should it, because you're doing something very unusual here.
You could add an attribute to the object in the __init__ function to indicate that it's been initialised. Check for the existence of that attribute as the first thing in __init__ and don't proceed any further if it has been.
Digging little deeper into that!
The type of a generic class in CPython is type and its base class is Object (Unless you explicitly define another base class like a metaclass). The sequence of low level calls can be found here. The first method called is the type_call which then calls tp_new and then tp_init.
The interesting part here is that tp_new will call the Object's (base class) new method object_new which does a tp_alloc (PyType_GenericAlloc) which allocates the memory for the object :)
At that point the object is created in memory and then the __init__ method gets called. If __init__ is not implemented in your class then the object_init gets called and it does nothing :)
Then type_call just returns the object which binds to your variable.
One should look at __init__ as a simple constructor in traditional OO languages. For example, if you are familiar with Java or C++, the constructor is passed a pointer to its own instance implicitly. In the case of Java, it is the this variable. If one were to inspect the byte code generated for Java, one would notice two calls. The first call is to an "new" method, and then next call is to the init method (which is the actual call to the user defined constructor). This two step process enables creation of the actual instance before calling the constructor method of the class which is just another method of that instance.
Now, in the case of Python, __new__ is a added facility that is accessible to the user. Java does not provide that flexibility, due to its typed nature. If a language provided that facility, then the implementor of __new__ could do many things in that method before returning the instance, including creating a totally new instance of a unrelated object in some cases. And, this approach also works out well for especially for immutable types in the case of Python.
However, I'm a bit confused as to why __init__ is always called after __new__.
I think the C++ analogy would be useful here:
__new__ simply allocates memory for the object. The instance variables of an object needs memory to hold it, and this is what the step __new__ would do.
__init__ initialize the internal variables of the object to specific values (could be default).
Referring to this doc:
When subclassing immutable built-in types like numbers and strings,
and occasionally in other situations, the static method __new__ comes
in handy. __new__ is the first step in instance construction, invoked
before __init__.
The __new__ method is called with the class as its
first argument; its responsibility is to return a new instance of that
class.
Compare this to __init__: __init__ is called with an instance
as its first argument, and it doesn't return anything; its
responsibility is to initialize the instance.
There are situations
where a new instance is created without calling __init__ (for example
when the instance is loaded from a pickle). There is no way to create
a new instance without calling __new__ (although in some cases you can
get away with calling a base class's __new__).
Regarding what you wish to achieve, there also in same doc info about Singleton pattern
class Singleton(object):
def __new__(cls, *args, **kwds):
it = cls.__dict__.get("__it__")
if it is not None:
return it
cls.__it__ = it = object.__new__(cls)
it.init(*args, **kwds)
return it
def init(self, *args, **kwds):
pass
you may also use this implementation from PEP 318, using a decorator
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance
#singleton
class MyClass:
...
Now I've got the same problem, and for some reasons I decided to avoid decorators, factories and metaclasses. I did it like this:
Main file
def _alt(func):
import functools
#functools.wraps(func)
def init(self, *p, **k):
if hasattr(self, "parent_initialized"):
return
else:
self.parent_initialized = True
func(self, *p, **k)
return init
class Parent:
# Empty dictionary, shouldn't ever be filled with anything else
parent_cache = {}
def __new__(cls, n, *args, **kwargs):
# Checks if object with this ID (n) has been created
if n in cls.parent_cache:
# It was, return it
return cls.parent_cache[n]
else:
# Check if it was modified by this function
if not hasattr(cls, "parent_modified"):
# Add the attribute
cls.parent_modified = True
cls.parent_cache = {}
# Apply it
cls.__init__ = _alt(cls.__init__)
# Get the instance
obj = super().__new__(cls)
# Push it to cache
cls.parent_cache[n] = obj
# Return it
return obj
Example classes
class A(Parent):
def __init__(self, n):
print("A.__init__", n)
class B(Parent):
def __init__(self, n):
print("B.__init__", n)
In use
>>> A(1)
A.__init__ 1 # First A(1) initialized
<__main__.A object at 0x000001A73A4A2E48>
>>> A(1) # Returned previous A(1)
<__main__.A object at 0x000001A73A4A2E48>
>>> A(2)
A.__init__ 2 # First A(2) initialized
<__main__.A object at 0x000001A7395D9C88>
>>> B(2)
B.__init__ 2 # B class doesn't collide with A, thanks to separate cache
<__main__.B object at 0x000001A73951B080>
Warning: You shouldn't initialize Parent, it will collide with other classes - unless you defined separate cache in each of the children, that's not what we want.
Warning: It seems a class with Parent as grandparent behaves weird. [Unverified]
Try it online!
The __init__ is called after __new__ so that when you override it in a subclass, your added code will still get called.
If you are trying to subclass a class that already has a __new__, someone unaware of this might start by adapting the __init__ and forwarding the call down to the subclass __init__. This convention of calling __init__ after __new__ helps that work as expected.
The __init__ still needs to allow for any parameters the superclass __new__ needed, but failing to do so will usually create a clear runtime error. And the __new__ should probably explicitly allow for *args and '**kw', to make it clear that extension is OK.
It is generally bad form to have both __new__ and __init__ in the same class at the same level of inheritance, because of the behavior the original poster described.
However, I'm a bit confused as to why __init__ is always called after __new__.
Not much of a reason other than that it just is done that way. __new__ doesn't have the responsibility of initializing the class, some other method does (__call__, possibly-- I don't know for sure).
I wasn't expecting this. Can anyone tell me why this is happening and how I implement this functionality otherwise? (apart from putting the implementation into the __new__ which feels quite hacky).
You could have __init__ do nothing if it's already been initialized, or you could write a new metaclass with a new __call__ that only calls __init__ on new instances, and otherwise just returns __new__(...).
The simple reason is that the new is used for creating an instance, while init is used for initializing the instance. Before initializing, the instance should be created first. That's why new should be called before init.
When instantiating a class, first, __new__() is called to create the instance of a class, then __init__() is called to initialize the instance.
__new__():
Called to create a new instance of class cls. ...
If __new__() is invoked during object construction and it returns an
instance of cls, then the new instance’s __init__() method will be
invoked like __init__(self[, ...]), ...
__init__():
Called after the instance has been created (by __new__()), ...
Because __new__() and __init__() work together in constructing objects
(__new__() to create it, and __init__() to customize it), ...
For example, when instantiating Teacher class, first, __new__() is called to create the instance of Teacher class, then __init__() is called to initialize the instance as shown below:
class Teacher:
def __init__(self, name):
self.name = name
class Student:
def __init__(self, name):
self.name = name
obj = Teacher("John") # Instantiation
print(obj.name)
This is the output:
<class '__main__.Teacher'>
John
And, using __new__() of the instance of Teacher class, we can create the instance of Student class as shown below:
# ...
obj = Teacher("John")
print(type(obj))
print(obj.name)
obj = obj.__new__(Student) # Creates the instance of "Student" class
print(type(obj))
Now, the instance of Student class is created as shown below:
<class '__main__.Teacher'>
<__main__.Teacher object at 0x7f4e3950bf10>
<class '__main__.Student'> # Here
Next, if we try to get the value of name variable from **the instance of Student class as shown below:
obj = Teacher("John")
print(type(obj))
print(obj.name)
obj = obj.__new__(Student)
print(type(obj))
print(obj.name) # Tries to get the value of "name" variable
The error below occurs because the instance of Student class has not been initialized by __init__() yet:
AttributeError: 'Student' object has no attribute 'name'
So, we initialize the instance of Student class as shown below:
obj = Teacher("John")
print(type(obj))
print(obj.name)
obj = obj.__new__(Student)
print(type(obj))
obj.__init__("Tom") # Initializes the instance of "Student" class
print(obj.name)
Then, we can get the value of name variable from the instance of Student class as shown below:
<class '__main__.Teacher'>
John
<class '__main__.Student'>
Tom # Here
People have already detailed the question and answer both use some examples like singleton etc. See the code below:
__instance = None
def __new__(cls):
if cls.__instance is None:
cls.__instance = object.__new__(cls)
return cls.__instance
I got the above code from this link, it has detailed overview of new vs init. Worth reading!
I'm just trying to streamline one of my classes and have introduced some functionality in the same style as the flyweight design pattern.
However, I'm a bit confused as to why __init__ is always called after __new__. I wasn't expecting this. Can anyone tell me why this is happening and how I can implement this functionality otherwise? (Apart from putting the implementation into the __new__ which feels quite hacky.)
Here's an example:
class A(object):
_dict = dict()
def __new__(cls):
if 'key' in A._dict:
print "EXISTS"
return A._dict['key']
else:
print "NEW"
return super(A, cls).__new__(cls)
def __init__(self):
print "INIT"
A._dict['key'] = self
print ""
a1 = A()
a2 = A()
a3 = A()
Outputs:
NEW
INIT
EXISTS
INIT
EXISTS
INIT
Why?
Use __new__ when you need to control
the creation of a new instance.
Use
__init__ when you need to control initialization of a new instance.
__new__ is the first step of instance creation. It's called first, and is
responsible for returning a new
instance of your class.
In contrast,
__init__ doesn't return anything; it's only responsible for initializing the
instance after it's been created.
In general, you shouldn't need to
override __new__ unless you're
subclassing an immutable type like
str, int, unicode or tuple.
From April 2008 post: When to use __new__ vs. __init__? on mail.python.org.
You should consider that what you are trying to do is usually done with a Factory and that's the best way to do it. Using __new__ is not a good clean solution so please consider the usage of a factory. Here's a good example: ActiveState Fᴀᴄᴛᴏʀʏ ᴘᴀᴛᴛᴇʀɴ Recipe.
__new__ is static class method, while __init__ is instance method.
__new__ has to create the instance first, so __init__ can initialize it. Note that __init__ takes self as parameter. Until you create instance there is no self.
Now, I gather, that you're trying to implement singleton pattern in Python. There are a few ways to do that.
Also, as of Python 2.6, you can use class decorators.
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance
#singleton
class MyClass:
...
In most well-known OO languages, an expression like SomeClass(arg1, arg2) will allocate a new instance, initialise the instance's attributes, and then return it.
In most well-known OO languages, the "initialise the instance's attributes" part can be customised for each class by defining a constructor, which is basically just a block of code that operates on the new instance (using the arguments provided to the constructor expression) to set up whatever initial conditions are desired. In Python, this corresponds to the class' __init__ method.
Python's __new__ is nothing more and nothing less than similar per-class customisation of the "allocate a new instance" part. This of course allows you to do unusual things such as returning an existing instance rather than allocating a new one. So in Python, we shouldn't really think of this part as necessarily involving allocation; all that we require is that __new__ comes up with a suitable instance from somewhere.
But it's still only half of the job, and there's no way for the Python system to know that sometimes you want to run the other half of the job (__init__) afterwards and sometimes you don't. If you want that behavior, you have to say so explicitly.
Often, you can refactor so you only need __new__, or so you don't need __new__, or so that __init__ behaves differently on an already-initialised object. But if you really want to, Python does actually allow you to redefine "the job", so that SomeClass(arg1, arg2) doesn't necessarily call __new__ followed by __init__. To do this, you need to create a metaclass, and define its __call__ method.
A metaclass is just the class of a class. And a class' __call__ method controls what happens when you call instances of the class. So a metaclass' __call__ method controls what happens when you call a class; i.e. it allows you to redefine the instance-creation mechanism from start to finish. This is the level at which you can most elegantly implement a completely non-standard instance creation process such as the singleton pattern. In fact, with less than 10 lines of code you can implement a Singleton metaclass that then doesn't even require you to futz with __new__ at all, and can turn any otherwise-normal class into a singleton by simply adding __metaclass__ = Singleton!
class Singleton(type):
def __init__(self, *args, **kwargs):
super(Singleton, self).__init__(*args, **kwargs)
self.__instance = None
def __call__(self, *args, **kwargs):
if self.__instance is None:
self.__instance = super(Singleton, self).__call__(*args, **kwargs)
return self.__instance
However this is probably deeper magic than is really warranted for this situation!
To quote the documentation:
Typical implementations create a new instance of the class by invoking
the superclass's __new__() method using "super(currentclass,
cls).__new__(cls[, ...])"with appropriate arguments and then
modifying the newly-created instance as necessary before returning it.
...
If __new__() does not return an instance of cls, then the new
instance's __init__() method will not be invoked.
__new__() is intended mainly to allow subclasses of immutable
types (like int, str, or tuple) to customize instance creation.
I realize that this question is quite old but I had a similar issue.
The following did what I wanted:
class Agent(object):
_agents = dict()
def __new__(cls, *p):
number = p[0]
if not number in cls._agents:
cls._agents[number] = object.__new__(cls)
return cls._agents[number]
def __init__(self, number):
self.number = number
def __eq__(self, rhs):
return self.number == rhs.number
Agent("a") is Agent("a") == True
I used this page as a resource http://infohost.nmt.edu/tcc/help/pubs/python/web/new-new-method.html
When __new__ returns instance of the same class, __init__ is run afterwards on returned object. I.e. you can NOT use __new__ to prevent __init__ from being run. Even if you return previously created object from __new__, it will be double (triple, etc...) initialized by __init__ again and again.
Here is the generic approach to Singleton pattern which extends vartec answer above and fixes it:
def SingletonClass(cls):
class Single(cls):
__doc__ = cls.__doc__
_initialized = False
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(Single, cls).__new__(cls, *args, **kwargs)
return cls._instance
def __init__(self, *args, **kwargs):
if self._initialized:
return
super(Single, self).__init__(*args, **kwargs)
self.__class__._initialized = True # Its crucial to set this variable on the class!
return Single
Full story is here.
Another approach, which in fact involves __new__ is to use classmethods:
class Singleton(object):
__initialized = False
def __new__(cls, *args, **kwargs):
if not cls.__initialized:
cls.__init__(*args, **kwargs)
cls.__initialized = True
return cls
class MyClass(Singleton):
#classmethod
def __init__(cls, x, y):
print "init is here"
#classmethod
def do(cls):
print "doing stuff"
Please pay attention, that with this approach you need to decorate ALL of your methods with #classmethod, because you'll never use any real instance of MyClass.
I think the simple answer to this question is that, if __new__ returns a value that is the same type as the class, the __init__ function executes, otherwise it won't. In this case your code returns A._dict('key') which is the same class as cls, so __init__ will be executed.
class M(type):
_dict = {}
def __call__(cls, key):
if key in cls._dict:
print 'EXISTS'
return cls._dict[key]
else:
print 'NEW'
instance = super(M, cls).__call__(key)
cls._dict[key] = instance
return instance
class A(object):
__metaclass__ = M
def __init__(self, key):
print 'INIT'
self.key = key
print
a1 = A('aaa')
a2 = A('bbb')
a3 = A('aaa')
outputs:
NEW
INIT
NEW
INIT
EXISTS
NB As a side effect M._dict property automatically becomes accessible from A as A._dict so take care not to overwrite it incidentally.
An update to #AntonyHatchkins answer, you probably want a separate dictionary of instances for each class of the metatype, meaning that you should have an __init__ method in the metaclass to initialize your class object with that dictionary instead of making it global across all the classes.
class MetaQuasiSingleton(type):
def __init__(cls, name, bases, attibutes):
cls._dict = {}
def __call__(cls, key):
if key in cls._dict:
print('EXISTS')
instance = cls._dict[key]
else:
print('NEW')
instance = super().__call__(key)
cls._dict[key] = instance
return instance
class A(metaclass=MetaQuasiSingleton):
def __init__(self, key):
print 'INIT'
self.key = key
print()
I have gone ahead and updated the original code with an __init__ method and changed the syntax to Python 3 notation (no-arg call to super and metaclass in the class arguments instead of as an attribute).
Either way, the important point here is that your class initializer (__call__ method) will not execute either __new__ or __init__ if the key is found. This is much cleaner than using __new__, which requires you to mark the object if you want to skip the default __init__ step.
__new__ should return a new, blank instance of a class. __init__ is then called to initialise that instance. You're not calling __init__ in the "NEW" case of __new__, so it's being called for you. The code that is calling __new__ doesn't keep track of whether __init__ has been called on a particular instance or not nor should it, because you're doing something very unusual here.
You could add an attribute to the object in the __init__ function to indicate that it's been initialised. Check for the existence of that attribute as the first thing in __init__ and don't proceed any further if it has been.
Digging little deeper into that!
The type of a generic class in CPython is type and its base class is Object (Unless you explicitly define another base class like a metaclass). The sequence of low level calls can be found here. The first method called is the type_call which then calls tp_new and then tp_init.
The interesting part here is that tp_new will call the Object's (base class) new method object_new which does a tp_alloc (PyType_GenericAlloc) which allocates the memory for the object :)
At that point the object is created in memory and then the __init__ method gets called. If __init__ is not implemented in your class then the object_init gets called and it does nothing :)
Then type_call just returns the object which binds to your variable.
One should look at __init__ as a simple constructor in traditional OO languages. For example, if you are familiar with Java or C++, the constructor is passed a pointer to its own instance implicitly. In the case of Java, it is the this variable. If one were to inspect the byte code generated for Java, one would notice two calls. The first call is to an "new" method, and then next call is to the init method (which is the actual call to the user defined constructor). This two step process enables creation of the actual instance before calling the constructor method of the class which is just another method of that instance.
Now, in the case of Python, __new__ is a added facility that is accessible to the user. Java does not provide that flexibility, due to its typed nature. If a language provided that facility, then the implementor of __new__ could do many things in that method before returning the instance, including creating a totally new instance of a unrelated object in some cases. And, this approach also works out well for especially for immutable types in the case of Python.
However, I'm a bit confused as to why __init__ is always called after __new__.
I think the C++ analogy would be useful here:
__new__ simply allocates memory for the object. The instance variables of an object needs memory to hold it, and this is what the step __new__ would do.
__init__ initialize the internal variables of the object to specific values (could be default).
Referring to this doc:
When subclassing immutable built-in types like numbers and strings,
and occasionally in other situations, the static method __new__ comes
in handy. __new__ is the first step in instance construction, invoked
before __init__.
The __new__ method is called with the class as its
first argument; its responsibility is to return a new instance of that
class.
Compare this to __init__: __init__ is called with an instance
as its first argument, and it doesn't return anything; its
responsibility is to initialize the instance.
There are situations
where a new instance is created without calling __init__ (for example
when the instance is loaded from a pickle). There is no way to create
a new instance without calling __new__ (although in some cases you can
get away with calling a base class's __new__).
Regarding what you wish to achieve, there also in same doc info about Singleton pattern
class Singleton(object):
def __new__(cls, *args, **kwds):
it = cls.__dict__.get("__it__")
if it is not None:
return it
cls.__it__ = it = object.__new__(cls)
it.init(*args, **kwds)
return it
def init(self, *args, **kwds):
pass
you may also use this implementation from PEP 318, using a decorator
def singleton(cls):
instances = {}
def getinstance():
if cls not in instances:
instances[cls] = cls()
return instances[cls]
return getinstance
#singleton
class MyClass:
...
Now I've got the same problem, and for some reasons I decided to avoid decorators, factories and metaclasses. I did it like this:
Main file
def _alt(func):
import functools
#functools.wraps(func)
def init(self, *p, **k):
if hasattr(self, "parent_initialized"):
return
else:
self.parent_initialized = True
func(self, *p, **k)
return init
class Parent:
# Empty dictionary, shouldn't ever be filled with anything else
parent_cache = {}
def __new__(cls, n, *args, **kwargs):
# Checks if object with this ID (n) has been created
if n in cls.parent_cache:
# It was, return it
return cls.parent_cache[n]
else:
# Check if it was modified by this function
if not hasattr(cls, "parent_modified"):
# Add the attribute
cls.parent_modified = True
cls.parent_cache = {}
# Apply it
cls.__init__ = _alt(cls.__init__)
# Get the instance
obj = super().__new__(cls)
# Push it to cache
cls.parent_cache[n] = obj
# Return it
return obj
Example classes
class A(Parent):
def __init__(self, n):
print("A.__init__", n)
class B(Parent):
def __init__(self, n):
print("B.__init__", n)
In use
>>> A(1)
A.__init__ 1 # First A(1) initialized
<__main__.A object at 0x000001A73A4A2E48>
>>> A(1) # Returned previous A(1)
<__main__.A object at 0x000001A73A4A2E48>
>>> A(2)
A.__init__ 2 # First A(2) initialized
<__main__.A object at 0x000001A7395D9C88>
>>> B(2)
B.__init__ 2 # B class doesn't collide with A, thanks to separate cache
<__main__.B object at 0x000001A73951B080>
Warning: You shouldn't initialize Parent, it will collide with other classes - unless you defined separate cache in each of the children, that's not what we want.
Warning: It seems a class with Parent as grandparent behaves weird. [Unverified]
Try it online!
The __init__ is called after __new__ so that when you override it in a subclass, your added code will still get called.
If you are trying to subclass a class that already has a __new__, someone unaware of this might start by adapting the __init__ and forwarding the call down to the subclass __init__. This convention of calling __init__ after __new__ helps that work as expected.
The __init__ still needs to allow for any parameters the superclass __new__ needed, but failing to do so will usually create a clear runtime error. And the __new__ should probably explicitly allow for *args and '**kw', to make it clear that extension is OK.
It is generally bad form to have both __new__ and __init__ in the same class at the same level of inheritance, because of the behavior the original poster described.
However, I'm a bit confused as to why __init__ is always called after __new__.
Not much of a reason other than that it just is done that way. __new__ doesn't have the responsibility of initializing the class, some other method does (__call__, possibly-- I don't know for sure).
I wasn't expecting this. Can anyone tell me why this is happening and how I implement this functionality otherwise? (apart from putting the implementation into the __new__ which feels quite hacky).
You could have __init__ do nothing if it's already been initialized, or you could write a new metaclass with a new __call__ that only calls __init__ on new instances, and otherwise just returns __new__(...).
The simple reason is that the new is used for creating an instance, while init is used for initializing the instance. Before initializing, the instance should be created first. That's why new should be called before init.
When instantiating a class, first, __new__() is called to create the instance of a class, then __init__() is called to initialize the instance.
__new__():
Called to create a new instance of class cls. ...
If __new__() is invoked during object construction and it returns an
instance of cls, then the new instance’s __init__() method will be
invoked like __init__(self[, ...]), ...
__init__():
Called after the instance has been created (by __new__()), ...
Because __new__() and __init__() work together in constructing objects
(__new__() to create it, and __init__() to customize it), ...
For example, when instantiating Teacher class, first, __new__() is called to create the instance of Teacher class, then __init__() is called to initialize the instance as shown below:
class Teacher:
def __init__(self, name):
self.name = name
class Student:
def __init__(self, name):
self.name = name
obj = Teacher("John") # Instantiation
print(obj.name)
This is the output:
<class '__main__.Teacher'>
John
And, using __new__() of the instance of Teacher class, we can create the instance of Student class as shown below:
# ...
obj = Teacher("John")
print(type(obj))
print(obj.name)
obj = obj.__new__(Student) # Creates the instance of "Student" class
print(type(obj))
Now, the instance of Student class is created as shown below:
<class '__main__.Teacher'>
<__main__.Teacher object at 0x7f4e3950bf10>
<class '__main__.Student'> # Here
Next, if we try to get the value of name variable from **the instance of Student class as shown below:
obj = Teacher("John")
print(type(obj))
print(obj.name)
obj = obj.__new__(Student)
print(type(obj))
print(obj.name) # Tries to get the value of "name" variable
The error below occurs because the instance of Student class has not been initialized by __init__() yet:
AttributeError: 'Student' object has no attribute 'name'
So, we initialize the instance of Student class as shown below:
obj = Teacher("John")
print(type(obj))
print(obj.name)
obj = obj.__new__(Student)
print(type(obj))
obj.__init__("Tom") # Initializes the instance of "Student" class
print(obj.name)
Then, we can get the value of name variable from the instance of Student class as shown below:
<class '__main__.Teacher'>
John
<class '__main__.Student'>
Tom # Here
People have already detailed the question and answer both use some examples like singleton etc. See the code below:
__instance = None
def __new__(cls):
if cls.__instance is None:
cls.__instance = object.__new__(cls)
return cls.__instance
I got the above code from this link, it has detailed overview of new vs init. Worth reading!
I'm trying to understand how new instances of a Python class should be created when the creation process can either be via the constructor or via the __new__ method. In particular, I notice that when using the constructor, the __init__ method will be automatically called after __new__, while when invoking __new__ directly the __init__ class will not automatically be called. I can force __init__ to be called when __new__ is explicitly called by embedding a call to __init__ within __new__, but then __init__ will end up getting called twice when the class is created via the constructor.
For example, consider the following toy class, which stores one internal property, namely a list object called data: it is useful to think of this as the start of a vector class.
class MyClass(object):
def __new__(cls, *args, **kwargs):
obj = object.__new__(cls, *args, **kwargs)
obj.__init__(*args, **kwargs)
return obj
def __init__(self, data):
self.data = data
def __getitem__(self, index):
return self.__new__(type(self), self.data[index])
def __repr__(self):
return repr(self.data)
A new instance of the class can be created either using the constructor (not actually sure if that is the right terminology in Python), something like
x = MyClass(range(10))
or via slicing, which you can see invokes a call to __new__ in the __getitem__ method.
x2 = x[0:2]
In the first instance, __init__ will be called twice (both via the explicit call within __new__ and then again automatically), and once in the second instance. Obviously I would only like __init__ to be invoked once in any case. Is there a standard way to do this in Python?
Note that in my example I could get rid of the __new__ method and redefine __getitem__ as
def __getitem__(self, index):
return MyClass(self.data[index])
but then this would cause a problem if I later want to inherit from MyClass, because if I make a call like child_instance[0:2] I will get back an instance of MyClass, not the child class.
First, some basic facts about __new__ and __init__:
__new__ is a constructor.
__new__ typically returns an instance of cls, its first argument.
By __new__ returning an instance of cls, __new__ causes Python to call __init__.
__init__ is an initializer. It modifies the instance (self)
returned by __new__. It does not need to return self.
When MyClass defines:
def __new__(cls, *args, **kwargs):
obj = object.__new__(cls, *args, **kwargs)
obj.__init__(*args, **kwargs)
return obj
MyClass.__init__ gets called twice. Once from calling obj.__init__ explicitly, and a second time because __new__ returned obj, an instance of cls. (Since the first argument to object.__new__ is cls, the instance returned is an instance of MyClass so obj.__init__ calls MyClass.__init__, not object.__init__.)
The Python 2.2.3 release notes has an interesting comment, which sheds light on when to use __new__ and when to use __init__:
The __new__ method is called with the class as its first argument; its
responsibility is to return a new instance of that class.
Compare this to __init__:__init__ is called with an instance as its
first argument, and it doesn't return anything; its responsibility is
to initialize the instance.
All this is done so that immutable types can preserve their
immutability while allowing subclassing.
The immutable types (int, long, float, complex, str, unicode, and
tuple) have a dummy __init__, while the mutable types (dict, list,
file, and also super, classmethod, staticmethod, and property) have a
dummy __new__.
So, use __new__ to define immutable types, and use __init__ to define mutable types. While it is possible to define both, you should not need to do so.
Thus, since MyClass is mutable, you should only define __init__:
class MyClass(object):
def __init__(self, data):
self.data = data
def __getitem__(self, index):
return type(self)(self.data[index])
def __repr__(self):
return repr(self.data)
x = MyClass(range(10))
x2 = x[0:2]
There are a couple of things that shouldn't be done:
Call __init__ from __new__
Call __new__ directly in a method
As you have already seen, both the __new__ and the __init__ methods are automatically called when creating an object of a given class. Using them directly would break this functionality (calling __init__ inside another __init__ is allowed though, as it can be seen in the example below).
You can get the class of the object in any method getting the __class__ attribute as in the following example:
class MyClass(object):
def __new__(cls, *args, **kwargs):
# Customized __new__ implementation here
return obj
def __init__(self, data):
super(MyClass, self).__init__(self)
self.data = data
def __getitem__(self, index):
cls = self.__class__
return cls(self.data[index])
def __repr__(self):
return repr(self.data)
x = MyClass(range(10))
x2 = x[0:2]
When you create an instance of a class with MyClass(args), the default instance creation sequence is as follows:
new_instance = MyClass.__new__(args) is invoked to get a new "blank" instance
new_instance.__init__(args) is invoked (new_instance is the instance returned from the call to __new__ as above) to initialise the attributes of the new instance [1]
new_instance is returned as the result of MyClass(args)
From this, it is clear to see that calling MyClass.__new__ yourself will not result in __init__ being called, so you'll end up with an uninitialised instance. It's equally clear that putting a call to __init__ into __new__ will also not be correct, as then MyClass(args) will call __init__ twice.
The source of your problem is this:
I'm trying to understand how new instances of a Python class should be
created when the creation process can either be via the constructor or
via the new method
The creation process should not normally be via the __new__ method at all. __new__ is a part of the normal instance creation protocol, so you shouldn't expect it to invoke the whole protocol for you.
One (bad) solution would be to implement this protocol by hand yourself; instead of:
def __getitem__(self, index):
return self.__new__(type(self), self.data[index])
you could have:
def __getitem__(self, index):
new_item = self.__new__(type(self), self.data[index])
new_item.__init__(self.data[index])
return new_item
But really, what you want to do is not mess with __new__ at all. The default __new__ is fine for your case, and the default instance creation protocol is fine for you case, so you should neither implement __new__ nor call it directly.
What you want is to create a new instance of the class the normal way, by calling the class. If there's no inheritance going on and you don't think there ever will be, simply replace self.__new__(type(self), self.data[index]) with MyClass(self.data[index]).
If you think there might one day be subclasses of MyClass that would want to create instances of the subclass through slicing rather than MyClass, then you need to dynamically get the class of self and invoke that. You already know how to do this, because you used it in your program! type(self) will return the type (class) of self, which you then can invoke exactly as you would invoke it directly through MyClass: type(self)(self.data[index]).
As an aside, the point of __new__ is when you want to customise the process of getting a "new" blank instance of a class before it is initialised. Almost all of the time, this is completely unnecessary and the default __new__ is fine.
You only need __new__ in two circumstances:
You're have an unusual "allocation" scheme, where you might return an existing instance rather than create a genuinely new one (the only way to actually create a new instance is to delegate to the ultimate default implementation of __new__ anyway).
You're implementing a subclass of an immutable builtin type. Since the immutable builtin types can't be modified after creation (because they're immutable), they must be initialised as they're created rather than afterwards in __init__.
As a generalisation of point (1), you can make __new__ return whatever you like (not necessarily an instance of the class) to make invoking a class behave in some arbitrarily bizarre manner. This seems like it would almost always be more confusing than helpful, though.
[1] I believe in fact the protocol is slightly more complex; __init__ is only invoked on the value returned by __new__ if it's an instance of the class that was invoked to start the process. However it's very unusual for this not to be the case.