Develop Reference

Why isn't __bases__ accessible in the class body?

Class objects have a __bases__ (and a __base__) attribute:
>>> class Foo(object):
... pass
...
>>> Foo.__bases__
(<class 'object'>,)
Sadly, these attributes aren't accessible in the class body, which would be very convenient for accessing parent class attributes without having to hard-code the name:
class Foo:
cls_attr = 3
class Bar(Foo):
cls_attr = __base__.cls_attr + 2
# throws NameError: name '__base__' is not defined
Is there a reason why __bases__ and __base__ can't be accessed in the class body?
(To be clear, I'm asking if this is a conscious design decision. I'm not asking about the implementation; I know that __bases__ is a descriptor in type and that this descriptor can't be accessed until a class object has been created. I want to know why python doesn't create __bases__ as a local variable in the class body.)

I want to know why python doesn't create __bases__ as a local variable in the class body
As you know, class is mostly a shortcut for type.__new__() - when the runtime hits a class statements, it executes all statements at the top-level of the class body, collects all resulting bindings in a dedicated namespace dict, calls type() with the concrete metaclass, the class name, the base classes and the namespace dict, and binds the resulting class object to the class name in the enclosing scope (usually but not necessarily the module's top-level namespace).
The important point here is that it's the metaclass responsabilty to build the class object, and to allow for class object creation customisations, the metaclass must be free to do whatever it wants with its arguments. Most often a custom metaclass will mainly work on the attrs dict, but it must also be able to mess with the bases argument. Now since the metaclass is only invoked AFTER the class body statements have been executed, there's no way the runtime can reliably expose the bases in the class body scope since those bases could be modified afterward by the metaclass.
There are also some more philosophical considerations here, notably wrt/ explicit vs implicit, and as shx2 mentions, Python designers try to avoid magic variables popping out of the blue. There are indeed a couple implementation variables (__module__ and, in py3, __qualname__) that are "automagically" defined in the class body namespace, but those are just names, mostly intended as additional debugging / inspection informations for developers) and have absolutely no impact on the class object creation nor on its properties, behaviour and whatnots.
As always with Python, you have to consider the whole context (the execution model, the object model, how the different parts work together etc) to really understand the design choices. Whether you agree with the whole design and philosophy is another debate (and one that doesn't belong here), but you can be sure that yes, those choices are "conscious design decisions".

I am not answering as to why it was decided to be implemented the way it was, I'm answering why it wasn't implemented as a "local variable in the class body":
Simply because nothing in python is a local variable magically defined in the class body. Python doesn't like names magically appearing out of nowhere.

It's because it's simply is not yet created.
Consider the following:
>>> class baselessmeta(type):
... def __new__(metaclass, class_name, bases, classdict):
... return type.__new__(
... metaclass,
... class_name,
... (), # I can just ignore all the base
... {}
... )
...
>>> class Baseless(int, metaclass=baselessmeta):
... # imaginary print(__bases__, __base__)
... ...
...
>>> Baseless.__bases__
(<class 'object'>,)
>>> Baseless.__base__
<class 'object'>
>>>
What should the imaginary print result in?
Every python class is created via the type metaclass one way or another.
You have the int argument for the type() in bases argument, yet you do not know what is the return value is going to be. You may use that directly as a base in your metaclass, or you may return another base with your LOC.
Just realized your to be clear part and now my answer is useless haha. Oh welp.

Accessing the parameters of a constructor from a metaclass

TL;DR -
I have a class that uses a metaclass.
I would like to access the parameters of the object's constructor from the metaclass, just before the initialization process, but I couldn't find a way to access those parameters.
How can I access the constructor's parameters from the metaclass function __new__?
In order to practice the use of metaclasses in python, I would like to create a class that would be used as the supercomputer "Deep Thought" from the book "The Hitchhiker's Guide to the Galaxy".
The purpose of my class would be to store the various queries the supercomputer gets from users.
At the bottom line, it would just get some arguments and store them.
If one of the given arguments is number 42 or the string "The answer to life, the universe, and everything", I don't want to create a new object but rather return a pointer to an existing object.
The idea behind this is that those objects would be the exact same so when using the is operator to compare those two, the result would be true.
In order to be able to use the is operator and get True as an answer, I would need to make sure those variables point to the same object. So, in order to return a pointer to an existing object, I need to intervene in the middle of the initialization process of the object. I cannot check the given arguments at the constructor itself and modify the object's inner-variables accordingly because it would be too late: If I check the given parameters only as part of the __init__ function, those two objects would be allocated on different portions of the memory (they might be equal but won't return True when using the is operator).
I thought of doing something like that:
class SuperComputer(type):
answer = 42
def __new__(meta, name, bases, attributes):
# Check if args contains the number "42"
# or has the string "The answer to life, the universe, and everything"
# If so, just return a pointer to an existing object:
return SuperComputer.answer
# Else, just create the object as it is:
return super(SuperComputer, meta).__new__(meta, name, bases, attributes)
class Query(object):
__metaclass__ = SuperComputer
def __init__(self, *args, **kwargs):
self.args = args
for key, value in kwargs.items():
setattr(self, key, value)
def main():
number = Query(42)
string = Query("The answer to life, the universe, and everything")
other = Query("Sunny", "Sunday", 123)
num2 = Query(45)
print number is string # Should print True
print other is string # Should print False
print number is num2 # Should print False
if __name__ == '__main__':
main()
But I'm stuck on getting the parameters from the constructor.
I saw that the __new__ method gets only four arguments:
The metaclass instance itself, the name of the class, its bases, and its attributes.
How can I send the parameters from the constructor to the metaclass?
What can I do in order to achieve my goal?

You don't need a metaclass for that.
The fact is __init__ is not the "constructor" of an object in Python, rather, it is commonly called an "initializator" . The __new__ is closer to the role of a "constructor" in other languages, and it is not available only for the metaclass - all classes have a __new__ method. If it is not explicitly implemented, the object.__new__ is called directly.
And actually, it is object.__new__ which creates a new object in Python. From pure Python code, there is no other possible way to create an object: it will always go through there. That means that if you implement the __new__ method on your own class, you have the option of not creating a new instance, and instead return another pre-existing instance of the same class (or any other object).
You only have to keep in mind that: if __new__ returns an instance of the same class, then the default behavior is that __init__ is called on the same instance. Otherwise, __init__ is not called.
It is also worth noting that in recent years some recipe for creating "singletons" in Python using metaclasses became popular - it is actually an overkill approach,a s overriding __new__ is also preferable for creating singletons.
In your case, you just need to have a dictionary with the parameters you want to track as your keys, and check if you create a new instance or "recycle" one whenever __new__ runs. The dictionary may be a class attribute, or a global variable at module level - that is your pick:
class Recycler:
_instances = {}
def __new__(cls, parameter1, ...):
if parameter1 in cls._instances:
return cls._instances[parameter1]
self = super().__new__(cls) # don't pass remaining parameters to object.__new__
_instances[parameter1] = self
return self
If you'd have any code in __init__ besides that, move it to __new__ as well.
You can have a baseclass with this behavior and have a class hierarchy without needing to re-implement __new__ for every class.
As for a metaclass, none of its methods are called when actually creating a new instance of the classes created with that metaclass. It would only be of use to automatically insert this behavior, by decorating or creating a fresh __new__ method, on classes created with that metaclass. Since this behavior is easier to track, maintain, and overall to combine with other classes just using ordinary inheritance, no need for a metaclass at all.

Why does __prepare__ take name and bases as parameters

I'm in the process of migrating from 2.7 to 3.x and I'm trying to understand the __prepare__ method for metaclasses introduced in PEP3115.
In most of the examples I've seen, implementations of this method ignore the parameters (name, bases, and **kwargs) simply returns a custom dictionary that does something interesting to the namespace provided to the __new__ and __init__ methods for the metaclass. Even the example in PEP3115 does nothing with the parameters.
I don't doubt that there is some good reason for the signature of __prepare__ but I haven't seen the use case.
What are some good examples that demonstrate the rational for making the signature of __prepare__ take these parameters?

__prepare__ will create a namespace for the class like you said, so we can do some logic inside of it like this:
class MyMeta(type):
#classmethod
def __prepare__(metacls, klass_name, bases):
namespace = {'s': 'my string',
'description': 'N/A'}
if klass_name.endswith('Base'):
namespace.update({'description': 'base class'})
return namespace
class KlassBase(metaclass=MyMeta):
def __init__(self, value):
self.value = value
class SubKlass(KlassBase):
def __init__(self, value):
super().__init__(value)
print(KlassBase(5).s)
print(KlassBase(5).description)
print(SubKlass(5).s)
print(SubKlass(5).description)
And you got:
my string
base class
my string
N/A
The reason why we don't do it, because same things could be done in other part of the meta class like : __new__, __init__, or be overrided by the latter. So most of time, we won't do it in __prepare__
The following is an image of the class creation work-flow, which is much more clearer:
[sorry I cannot find the original source of this pic]

When you look at __prepare__, and the state of Python class creation mechanism at the time, it is pretty much clear that what was really needed was a mechanism to enable attribute order preservation.
This way, one would be able to create classes that would describe data records with ordered fields, which is pretty much what humans would expect when describing a record. (Try to imagine a form that each time it is rendered, it shuffles the field ordes, so half the time you will be filling in the country you live in before the country).
Instead of a fixed mechanism to just enable the class body namespace to be an collections.OrderedDict, they came up with __prepare__ which enables this easily, with a single line returning a new OrderedDict instance.
But __prepare__ can have so many uses and abuses, that I think no one really thought of all the possibilities. The parameters ou mentioned are avaliable at the time it is called, and since it exists, there is no reason whatsoever for they not to be passed to the function. Why to cripple one of knowing the class' name inside the __prepare__ function?
So, it is just a powerful and flexible mechanism put in place, and not necessarily all possible use cases were thought of when making it. The "Ordered Attributes" thing n the other hand is so important that it became the default for Python 3.6, with PEP 520, even without any custom metaclass declaration.
__prepare__ receivog bases, for example, would allow one to pre-populate the namespace with certain objects that would be found in the superclass namespaces, overriding the inheritance attribute access, for example.
Or it could simply check the class name against a pre-existing registry, and either raise or pre-populate stuff from there.
One super-easy usage would be to pre-populate __name__ for example, so that class attributes could make use of it:
import collections
class M(type):
#classmethod
def __prepare__(metacls, name, bases):
ns = collections.OrderedDict()
ns["__name__"] = name
class T(metaclass=M):
__table__ = f"_{__name__.lower()}"
trivia
Due to the way functions work as methods in Python 3, one interesting related thing that is not documented anywhere is that if __prepare__ is not explicitly decorated to be a classmethod, it works as a staticmethod, and the target class name is passed directly in the first parameter to it.

What is the best practice for metaclass methods that call each other?

I'm trying to build a metaclass, and I was just thinking I was really getting it. And I want to have class methods for each of the instances of this metaclass.
class MyMeta(type):
"""A metaclass"""
def __new__(mcs, name, bases, attributes):
pass
def _foo(cls):
pass
def _bar(cls):
cls._foo()
When I run pylint on it, the cls._foo has difficulty:
[pylint] E1120:No value for argument 'cls' in unbound method call
When I try running the code (My code is more complex than this) it appears to be running fine, and doing what I'm expecting it to do. So how am I supposed to fix this case? What does it mean precisely.
It sounds like other errors related to not properly declaring things as #staticmethods, but I can't mark this as a #classmethod because then it would be a Metaclass method.
Related searches seem to be talking about places where people are dynamically adding constructors or something, and I don't think this is that case. Am I misunderstanding something still about metaclasses?
What I really want is class methods that call each other defined in a metaclass. Is there a better way to do this?

[pylint] E1120:No value for argument 'cls' in unbound method call
As far as your metaclasss is concerned, your _foo and _bar methods are just ordinary Python methods - which means Python will fill in a reference to each class (metaclass instance) automatically when calling these methods.
And, as this is a metaclass, for clarity, semantics, and readability, it is much better to do as you did: name their first parameter as cls instead of self.
The problem is just that pylint does not knw about that: it is likely it is expecting a hard-coded self.
Your code, in this respect is irretouchable.
All you have to do is to add a meta-comment fo that pylint ignore these lines - fortunatelly, Pylint allows it at block level, unlike some tools where you have to mark that in each line.
class MyMeta(type):
"""A metaclass"""
# pylint: disable=no-value-for-parameter
def __new__(mcs, name, bases, attributes):
pass
def _foo(cls):
pass
def _bar(cls):
cls._foo()
Your example metaclass, with the "meta-comment" so that pylint ignores your non-error.

How is the __class__ cell value set in class methods?

Looking at the documentation of the super type in Python 3.5, it notes that super(…) is the same as super(__class__, «first argument to function»). To my surprise, I wrote a method that returned __class__ – and it actually worked:
>>> class c:
... def meth(self): return __class__
...
>>> c().meth()
<class '__main__.c'>
Apparently, __class__ is a free variable assigned by the closure of the function:
>>> c.meth.__code__.co_freevars
('__class__',)
>>> c.meth.__closure__
(<cell at 0x7f6346a91048: type object at 0x55823b17f3a8>,)
I'd like to know under what circumstances that free variable is associated in the closure. I know that if I assign a function to a variable as part of creating a class it doesn't happen.
>>> def meth2(self): return __class__
...
>>> meth2.__code__.co_freevars
()
Even if I create a new class and as part of that creation assign some attribute to meth2, meth2 doesn't somehow magically gain a free variable that gets filled in.
That's unsurprising, because part of this appears to depend on the lexical state of the compiler at the time that the code is compiled.
I'd like to confirm that the conditions necessary for __class__ to be treated as a free variable are simply:
A reference to __class__ in the code block; and
The def containing the __class__ reference is lexically within a class declaration block.
I'd further like to understand what the conditions necessary for that variable getting filled in correctly are. It appears – at least from the Python 3.6 documentation – that something like type.__new__(…) is involved somehow. I haven't been able to understand for sure how type comes into play and how this all interacts with metaclasses that do not ultimately call type.__new__(…).
I'm particularly confused because I didn't think that at the time the namespace's __setattr__ method was used to assign the attribute containing the method to the method function (as it exists on the ultimately-constructed class object). I know that this namespace object exists because it was either constructed implicitly by the use of the class statement, or explicitly by the metaclass's __prepare__ method – but as best I can tell, the metaclass constructs the class object that populates __class__ after the function object is set as a value within the class namespace.

In the docs for Python’s data model, § 3.3.3.6 – “Creating the class object” – you will find the following:
[The] class object is the one that will be referenced by the
zero-argument form of super(). __class__ is an implicit closure
reference created by the compiler if any methods in a class body refer
to either __class__ or super. This allows the zero argument form
of super() to correctly identify the class being defined based on
lexical scoping, while the class or instance that was used to make
the current call is identified based on the first argument passed to
the method.
…emphasis is mine. This confirms your two putative criteria for a __class__ closure happening: a “__class__” reference in the method def, which itself is defined inside a class statement.
But then, the next ¶ in “Creating the class object” goes on to say:
CPython implementation detail: In CPython 3.6 and later, the __class__ cell is passed to the metaclass as a __classcell__ entry
in the class namespace. If present, this must be propagated up to the
type.__new__ call in order for the class to be initialized
correctly. Failing to do so will result in a RuntimeError in Python
3.8.
… emphasis is theirs. This means that if you are employing a metaclass with a __new__ method – in order to dictate the terms by which classes so designated are created – for example like e.g.:
class Meta(type):
def __new__(metacls, name, bases, attributes, **kwargs):
# Or whatever:
if '__slots__' not in attributes:
attributes['__slots__'] = tuple()
# Call up, creating and returning the new class:
return super().__new__(metacls, name,
bases,
attributes,
**kwargs)
… that last super(…).__new__(…) call is effectively calling type.__new__(…). In real life, there might be some other ancestral “__new__(…)” methods that get called between here and there, if your metaclass inherits from other metaclasses (like, e.g. abc.ABCMeta). Effectively, though, inside your Meta.__new__(…) method, between the method entry point, the super(…).__new__(…) call, and return-ing the new class object, you can inspect or set the value of the eventual __class__ cell variable through attributes['__classcell__']†.
Now as for whether this is at all useful: I don’t know. I have been programming in python for ten years; I totally use metaclasses‡, like, absolutely all the time (for better or for worse); and in the course of doing so I have never done any of the following things:
reassigned a __class__ attribute;
inspected the __class__ cell variable of anything; nor
messed around with this supposed __classcell__ namespace entry, in like any capacity
… Naturally, your programming experience will be different from mine, who knows what one does. It is not that any one of those aforementioned stratagems are de facto problematic, necessarily. But I am no stranger to bending Python’s type systems and metaprogramming facilities to my whim, and these particular things have never presented themselves as partiuclarly useful, especially once you are working within the general context of metaclasses, and what they do.
By which I suppose I mean, tl;dr: you are on the cusp of figuring out the basics of metaclasses and what they can do – do press on and experiment, but do investigate the topic with depth as well as breath. Indeed!
† – In reading through code examples of this sort, you’ll often find what my snippet here calls the attributes dictionary referred to as namespace or ns, or similar. It’s all the same stuff.
‡ – …and ABCs and mixins and class decorators and __init_subclass__(…) and the abuse of __mro_entries__(…) for personal gain; et cetera, ad nauseum