(Python) Closure created when it wasn't expected - python

I got an unexpected closure when creating a nested class. I suspect that this is something related to metaclasses, super, or both. It is definitely related to how closures get created. I am using python2.7.
Here are five simplified examples that demonstrate the same problem that I am seeing (they all build off the first):
EXAMPLE 1:
class Metaclass(type):
def __init__(self, name, bases, dict):
self.CONST = 5
class Base(object):
__metaclass__=Metaclass
def __init__(self):
"Set things up."
class Subclass(Base):
def __init__(self, name):
super(Subclass, self).__init__(self)
self.name = name
def other(self, something): pass
class Test(object):
def setup(self):
class Subclass(Base):
def __init__(self, name):
super(Subclass, self).__init__(self)
self.name = name
def other(self, something): pass
self.subclass = Subclass
class Subclass2(Base):
def __init__(self, name):
super(Subclass, self).__init__(self)
self.subclass2 = Subclass2
"0x%x" % id(Metaclass)
# '0x8257f74'
"0x%x" % id(Base)
# '0x825814c'
t=Test()
t.setup()
"0x%x" % id(t.subclass)
# '0x8258e8c'
"0x%x" % id(t.subclass2)
# '0x825907c'
t.subclass.__init__.__func__.__closure__
# (<cell at 0xb7d33d4c: Metaclass object at 0x8258e8c>,)
t.subclass.other.__func__.__closure__
# None
t.subclass2.__init__.__func__.__closure__
# (<cell at 0xb7d33d4c: Metaclass object at 0x8258e8c>,)
Subclass.__init__.__func__.__closure__
# None
EXAMPLE 2:
class Test2(object):
def setup(self):
class Subclass(Base):
def __init__(self, name):
self.name = name
def other(self, something): pass
self.subclass = Subclass
t2=Test2()
t2.setup()
t2.subclass.__init__.__func__.__closure__
# None
EXAMPLE 3:
class Test3(object):
def setup(self):
class Other(object):
def __init__(self):
super(Other, self).__init__()
self.other = Other
class Other2(object):
def __init__(self): pass
self.other2 = Other2
t3=Test3()
t3.setup()
"0x%x" % id(t3.other)
# '0x8259734'
t3.other.__init__.__func__.__closure__
# (<cell at 0xb7d33e54: type object at 0x8259734>,)
t3.other2.__init__.__func__.__closure__
# None
EXAMPLE 4:
class Metaclass2(type): pass
class Base2(object):
__metaclass__=Metaclass2
def __init__(self):
"Set things up."
class Base3(object):
__metaclass__=Metaclass2
class Test4(object):
def setup(self):
class Subclass2(Base2):
def __init__(self, name):
super(Subclass2, self).__init__(self)
self.subclass2 = Subclass2
class Subclass3(Base3):
def __init__(self, name):
super(Subclass3, self).__init__(self)
self.subclass3 = Subclass3
class Subclass4(Base3):
def __init__(self, name):
super(Subclass4, self).__init__(self)
self.subclass4 = Subclass4
"0x%x" % id(Metaclass2)
# '0x8259d9c'
"0x%x" % id(Base2)
# '0x825ac9c'
"0x%x" % id(Base3)
# '0x825affc'
t4=Test4()
t4.setup()
"0x%x" % id(t4.subclass2)
# '0x825b964'
"0x%x" % id(t4.subclass3)
# '0x825bcac'
"0x%x" % id(t4.subclass4)
# '0x825bff4'
t4.subclass2.__init__.__func__.__closure__
# (<cell at 0xb7d33d04: Metaclass2 object at 0x825b964>,)
t4.subclass3.__init__.__func__.__closure__
# (<cell at 0xb7d33e9c: Metaclass2 object at 0x825bcac>,)
t4.subclass4.__init__.__func__.__closure__
# (<cell at 0xb7d33ddc: Metaclass2 object at 0x825bff4>,)
EXAMPLE 5:
class Test5(object):
def setup(self):
class Subclass(Base):
def __init__(self, name):
Base.__init__(self)
self.subclass = Subclass
t5=Test5()
t5.setup()
"0x%x" % id(t5.subclass)
# '0x8260374'
t5.subclass.__init__.__func__.__closure__
# None
Here is what I understand (referencing examples):
Metaclasses are inherited, so Subclass gets Base’s metaclass.
Only __init__ is affected, Subclass.other method is not (#1).
Removing Subclass.other does not make a difference (#1).
Removing self.name=name from Subclass.__init__ does not make a difference (#1).
The object in the closure cell is not a function.
The object is not Metaclass or Base, but some object of type Metaclass, just like Base is (#1).
The object is actually an object of the type of the nested Subclass (#1).
The closure cells for t1.subclass.__init__ and t1.subclass2.__init__ are the same, even though they are from two different classes (#1).
When I do not nest the creation of Subclass (#1) then there is no closure created.
When I do not call super(...).__init__ in Subclass.init__ no closure is created (#2).
If I assign no __metaclass__ and inherit from object then the same behavior shows up (#3).
The object in the closure cell for t3.other.__init__ is t3.other (#3).
The same behavior happens if the metaclass has no __init__ (#4).
The same behavior happens if the Base has no __init__ (#4).
The closure cells for the three subclasses in example 4 are all different and each matches the corresponding class (#4).
When super(...).__init__ is replaced with Base.__init__(self), the closure disappears (#5).
Here is what I do not understand:
Why does a closure get set for __init__?
Why doesn't the closure get set for other?
Why is the object in the closure cell set to the class to which __init__ belongs?
Why does this only happen when super(...).__init__ is called?
Why doesn't this happen when Base.__init__(self) is called?
Does this actually have anything at all to do with using metaclasses (probably, since the default metaclass is type)?
Thanks for the help!
-eric
(Update) Here is something that I found then (based on Jason's insight):
def something1():
print "0x%x" % id(something1)
def something2():
def something3():
print "0x%x" % id(something1)
print "0x%x" % id(something2)
print "0x%x" % id(something3)
return something3
return something2
something1.__closure__
# None
something1().__closure__
# 0xb7d4056c
# (<cell at 0xb7d33eb4: function object at 0xb7d40df4>,)
something1()().__closure__
# 0xb7d4056c
# (<cell at 0xb7d33fec: function object at 0xb7d40e64>, <cell at 0xb7d33efc: function object at 0xb7d40e2c>)
something1()()()
# 0xb7d4056c
# 0xb7d4056c
# 0xb7d40e9c
# 0xb7d40ed4
First, a function's name is in scope within its own body. Second, functions get closures for the functions in which they are defined if they reference those functions.
I hadn't realized that the function name was in scope like that. The same goes for classes. When a class is defined within a function's scope, any references to that class name inside the class's methods cause the class to bound in a closure on that method's function, like so:
def test():
class Test(object):
def something(self):
print Test
return Test
test()
# <class '__main__.Test'>
test().something.__func__.__closure__
# (<cell at 0xb7d33c2c: type object at 0x825e304>,)
However, since closures cannot be created on non-functions the following fails:
def test():
class Test(object):
SELF=Test
def something(self):
print Test
return Test
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# File "<stdin>", line 2, in test
# File "<stdin>", line 3, in Test
# NameError: free variable 'Test' referenced before assignment in enclosing scope
Good stuff!

Why does a closure get set for __init__?
It refers to a local variable (namely Subclass) in the enclosing function (namely setup).
Why doesn't the closure get set for other?
Because it doesn't refer to any local variables (or parameters) in any enclosing functions.
Why is the object in the closure cell set to the class to which __init__ belongs?
That is the value of the enclosing variable being referred to.
Why does this only happen when super(...).__init__ is called?
Why doesn't this happen when Base.__init__(self) is called?
Because Base is not a local variable in any enclosing function.
Does this actually have anything at all to do with using metaclasses?
No.

Related

Initialized variable not accessable inside of the class

class DemoClass:
def __init__(self):
self.name = "Marko"
def some_method(self):
print(self.name)
print(self.name) # NameError: name 'self' is not defined ???
my_object = DemoClass()
Why does this happen? Didn't I initialize the self.name variable in the init method which I think it means that it should be accessable in the entire class?
class DemoClass:
def __init__(self):
self.name = "Marko"
def some_method(self):
print(self.name)
my_object = DemoClass()
my_object.some_method()
do like this bro then only you can print the name.
You call the print() function with a class attribute (name) as argument. Even though the attribute is defined when Python executes the print() line, the class attributes exist in the local scope of the class and are accessible only to the class members or through the class namespace (e.g. my_object.name in a different scope where my_object is defined).

python extending a memoized class gives a compilation error

I have given up memoization of a class as a bag-of-worms that I didn't want to explore and here is one example of why. The question I ask is "how does one extend or inherit from a memoized class" but it's very possible I have made a mistake. The memoize class below is a cut-down version of the one by brandizzi in How can I memoize a class instantiation in Python? and googling the subject finds more involved such classes.
class memoize(object):
def __init__(self, cls):
self.cls = cls
# I didn't understand why this was needed
self.__dict__.update(cls.__dict__)
# bit about static methods not needed
def __call__(self, *args):
try:
self.cls.instances
except:
self.cls.instances = {}
key = '//'.join(map(str, args))
if key not in self.cls.instances:
self.cls.instances[key] = self.cls(*args)
return self.cls.instances[key]
class Foo():
def __init__(self,val):
self.val = val
def __repr__(self):
return "{}<{},{}>".format(self.__class__.__name__,self.val,id(self))
class Bar(Foo):
def __init__(self,val):
super().__init__(val)
f1,f2,f3 = [Foo(i) for i in (0,0,1)]
print([f1,f2,f3])
b1,b2,b3 = [Bar(i) for i in (0,0,1)]
print([b1,b2,b3])
# produces exactly what I expect
# [Foo<0,3071981964>, Foo<0,3071982092>, Foo<1,3071982316>]
# [Bar<0,3071983340>, Bar<0,3071983404>, Bar<1,3071983436>]
Foo = memoize(Foo)
f1,f2,f3 = [Foo(i) for i in (0,0,1)]
print([f1,f2,f3])
b1,b2,b3 = [Bar(i) for i in (0,0,1)]
print([b1,b2,b3])
# and now Foo has been memoized so Foo(0) always produces the same object
# [Foo<0,3071725804>, Foo<0,3071725804>, Foo<1,3071726060>]
# [Bar<0,3071711916>, Bar<0,3071711660>, Bar<1,3071725644>]
# this produces a compilation error that I don't understand
class Baz(Foo):
def __init__(self,val):
super().__init__(val)
# Traceback (most recent call last):
# File "/tmp/foo.py", line 49, in <module>
# class Baz(Foo):
# TypeError: __init__() takes 2 positional arguments but 4 were given
This "recipe" is indeed a very bad idea - once you rebind Foo to memoize(Foo), Foo is a memoize instance and not class Foo anymore. This breaks all expectations wrt/ python's type and the whole object model. In this case, it about how the class statement works. Actually, this:
class Titi():
x = 42
def toto(self):
print(self.x)
is syntactic sugar for:
def toto(self):
print(self.x)
Titi = type("Titi", (object,), {x:42, toto:toto})
del toto
Note that this happens at runtime (like everything in Python except parsing / bytecode compilation), and that type is a class so calling type creates a new class which is a type instance (this is named a 'metaclass' - the class of a class - and type is the default metaclass).
So with Foo being now a memoize instance instead of a Type instance, and since memoize is not a proper metaclass (it's __init__ methods signature is incompatible), the whole thing just cannot work.
To get this to work, you'd have to make memoize a proper metaclass (this is a simplified example assuming a single arg named param but it can be generalized if you want to):
class FooType(type):
def __new__(meta, name, bases, attrs):
if "_instances" not in attrs:
attrs["_instances"] = dict()
return type.__new__(meta, name, bases, attrs)
def __call__(cls, param):
if param not in cls._instances:
cls._instances[param] = super(FooType, cls).__call__(param)
return cls._instances[param]
class Foo(metaclass=FooType):
def __init__(self, param):
self._param = param
print("%s init(%s)" % (self, param))
def __repr__(self):
return "{}<{},{}>".format(self.__class__.__name__, self._param, id(self))
class Bar(Foo):
pass
f1,f2,f3 = [Foo(i) for i in (0,0,1)]
print([f1,f2,f3])
b1,b2,b3 = [Bar(i) for i in (0,0,1)]
print([b1,b2,b3])

Get caller class instance (object) inside method that was called?

Let say I have
class A(object):
def m1(self):
B().m2()
class B(object):
def m2(self):
#return object of caller instance
#let say 'a' is instance object this method was called from
return a
a = A().m1()
You can pass the caller instance, and make it a parameter of the called function itself. Something like -
class A(object):
def m1(self):
B().m2(self)
class B(object):
def m2(self, obj):
#return object of caller instance
#let say 'a' is instance object this method was called from
return obj
a = A().m1()

Python Class Name as Class Variable

I'm working as an application with classes and subclasses. For each class, both super and sub, there is a class variable called label. I would like the label variable for the super class to default to the class name. For example:
class Super():
label = 'Super'
class Sub(Super):
label = 'Sub'
Rather than manually type out the variable for each class, is it possible to derive the variable from the class name in the super class and have it automatically populated for the subclasses?
class Super():
label = # Code to get class name
class Sub(Super)
pass
# When inherited Sub.label == 'Sub'.
The reason for this is that this will be the default behavior. I'm also hoping that if I can get the default behavior, I can override it later by specifying an alternate label.
class SecondSub(Super):
label = 'Pie' # Override the default of SecondSub.label == 'SecondSub'
I've tried using __name__, but that's not working and just gives me '__main__'.
I would like to use the class variable label in #classmethod methods. So I would like to be able to reference the value without having to actually create a Super() or Sub() object, like below:
class Super():
label = # Magic
#classmethod
def do_something_with_label(cls):
print(cls.label)
you can return self.__class__.__name__ in label as a property
class Super:
#property
def label(self):
return self.__class__.__name__
class Sub(Super):
pass
print Sub().label
alternatively you could set it in the __init__ method
def __init__(self):
self.label = self.__class__.__name__
this will obviously only work on instantiated classes
to access the class name inside of a class method you would need to just call __name__ on the cls
class XYZ:
#classmethod
def my_label(cls):
return cls.__name__
print XYZ.my_label()
this solution might work too (snagged from https://stackoverflow.com/a/13624858/541038)
class classproperty(object):
def __init__(self, fget):
self.fget = fget
def __get__(self, owner_self, owner_cls):
return self.fget(owner_cls)
class Super(object):
#classproperty
def label(cls):
return cls.__name__
class Sub(Super):
pass
print Sub.label #works on class
print Sub().label #also works on an instance
class Sub2(Sub):
#classmethod
def some_classmethod(cls):
print cls.label
Sub2.some_classmethod()
You can use a descriptor:
class ClassNameDescriptor(object):
def __get__(self, obj, type_):
return type_.__name__
class Super(object):
label = ClassNameDescriptor()
class Sub(Super):
pass
class SecondSub(Super):
label = 'Foo'
Demo:
>>> Super.label
'Super'
>>> Sub.label
'Sub'
>>> SecondSub.label
'Foo'
>>> Sub().label
'Sub'
>>> SecondSub().label
'Foo'
If class ThirdSub(SecondSub) should have ThirdSub.label == 'ThirdSub' instead of ThirdSub.label == 'Foo', you can do that with a bit more work. Assigning label at the class level will be inherited, unless you use a metaclass (which is a lot more hassle than it's worth for this), but we can have the label descriptor look for a _label attribute instead:
class ClassNameDescriptor(object):
def __get__(self, obj, type_):
try:
return type_.__dict__['_label']
except KeyError:
return type_.__name__
Demo:
>>> class SecondSub(Super):
... _label = 'Foo'
...
>>> class ThirdSub(SecondSub):
... pass
...
>>> SecondSub.label
'Foo'
>>> ThirdSub.label
'ThirdSub'
A metaclass might be useful here.
class Labeller(type):
def __new__(meta, name, bases, dct):
dct.setdefault('label', name)
return super(Labeller, meta).__new__(meta, name, bases, dct)
# Python 2
# class Super(object):
# __metaclass__ = Labeller
class Super(metaclass=Labeller):
pass
class Sub(Super):
pass
class SecondSub(Super):
label = 'Pie'
class ThirdSub(SecondSub):
pass
Disclaimer: when providing a custom metaclass for your class, you need to make sure it is compatible with whatever metaclass(es) are used by any class in its ancestry. Generally, this means making sure your metaclass inherits from all the other metaclasses, but it can be nontrivial to do so. In practice, metaclasses aren't so commonly used, so it's usually just a matter of subclassing type, but it's something to be aware of.
As of Python 3.6, the cleanest way to achieve this is with __init_subclass__ hook introduced in PEP 487. It is much simpler (and easier to manage with respect to inheritance) than using a metaclass.
class Base:
#classmethod
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
if 'label' not in cls.__dict__: # Check if label has been set in the class itself, i.e. not inherited from any of its superclasses
cls.label = cls.__name__ # If not, default to class's __name__
class Sub1(Base):
pass
class Sub2(Base):
label = 'Custom'
class SubSub(Sub2):
pass
print(Sub1.label) # Sub1
print(Sub2.label) # Custom
print(SubSub.label) # SubSub

create instance of class within class definition

I'm trying to have a default instance of a class. I want to have
class Foo():
def __init__(self):
....
_default = Foo()
#staticmethod
def get_default():
return _default
However _default = Foo() leads to NameError: name 'Foo' is not defined
Foo does not exist until the class definition is finalized. You can easily refer to it after the class definition, though:
class Foo(object):
def __init__(self):
# ....
Foo.default_instance = Foo()
Note also that I have removed the superfluous getter method in favor of a plain old attribute.
You can also solve the problem with a decorator:
def defaultinstance(Class):
Class.default_instance = Class()
return Class
#defaultinstance
class Foo(object):
# ...
Or, gratuitously, with a metaclass:
def defaultmeta(name, bases, attrs):
Class = type(name, bases, attrs)
Class.default_instance = Class()
return Class
# Python 2.x usage
class Foo(object):
__metaclass__ = defaultmeta
# ...
# Python 3.x usage
class Foo(metaclass=defaultmeta):
# ...
When might you might want to use each method?
Use the post-definition class attribute assignment for one-offs
Use the decorator if you want the same behavior in a lot of unrelated classes and to "hide" the implementation of it (it's not really hidden, or even that complicated, here, though)
Use the metaclass if you want the behavior to be inheritable in which case it's not really gratuitous. :-)
You cannot refer to a class that doesn't yet exist. Within the class definition body, the Foo class is not yet created.
Add the attribute after the class has been created:
class Foo():
def __init__(self):
....
#staticmethod
def get_default():
return Foo._default
Foo._default = Foo()
Note that you also need to alter the get_default() static method; the class body doesn't form a scope, so you cannot reach _default as a non-local from get_default().
You are now, however, repeating yourself a lot. Reduce repetition a little by making get_default() a classmethod instead:
class Foo():
def __init__(self):
....
#classmethod
def get_default(cls):
return cls._default
Foo._default = Foo()
or create the default on first call:
class Foo():
def __init__(self):
....
#classmethod
def get_default(cls):
if not hasattr(cls, '_default'):
cls._default = cls()
return cls._default
You may lazily initialize your default instance.
class Foo(object):
_default = None
#staticmethod
def get_default():
if not Foo._default:
Foo._default = Foo()
return Foo._default

Categories

Resources