I have a class family for which I need to be able to iterate through attributes of type: Metric.
The family consists of an abstract base class parent and child classes. The child classes will all have varying number of class attributes of type Metric, and they inherit an __iter__ method from the parent class that allows me to iterate through the attributes.
I am using iterable attributes rather than a dict because I want my objects to be typed, but I need to be able to call metrics in sequence, and by name.
So I need to be able to do:
Metrics.metric_1
and
for metric in Metrics:
My question is, how do I correctly hint in the base class that there are a variable number of attributes of the same type?
I'm currently using a couple of attribute hints with an ellipsis:
class MetricsBase(ABC):
metric_1: Metric
metric_2: Metric
...
#classmethod
def __iter__(cls):
for attr, value in cls.__dict__.items():
if not attr.startswith("__"):
yield value
class MetricChild(MetricsBase):
metric_1 = Metric(x)
metric_2 = Metric(y)
metric_3 = Metric(z)
But I'm not sure if this is pythonic or correct, and wondering if there is a neater way of doing this.
Many thanks for any input!
I am not answering how to "fix" static type checking on that.
That said, this is ok as Python code, hence "pythonic" . the problem is that you want to use static type checking on it - and you are using a dynamic meta programming technique there. Static type checking is not meant to check this (in a way of saying, it can only handle a small subset of what would be "pythonic"). Maybe there is a way to "solve" this - but if you can, just mark the static type checkers to skip that, and spare hours yourself hours of meaningless work (since it won't change how the code works)
More important than that, that __iter__ method won't work for the class itself, regardless of you marking it as a #classmethod. (It will work fot the instances, despite you doing so, though). If you want to iterate on the class, you will have to resort to a metaclass:
import abc
class MetricMeta(abc.ABCMeta):
def __iter__(cls):
# this will make _instances of this metaclass__ iterable
for attr, value in cls.__dict__.items():
if not attr.startswith("__"):
yield value
class MetricsBase(metaclass=MetricsMeta):
metric_1: Metric
metric_2: Metric
...
Type chekers actually, should, supposedly, not need one to expliclty annotate all variables, reducing Python to a subset of Pascal or the like. If you simply type in your class attributes in each subclass, attributing a Metric instance to then it should work, without the need to explictly annotate each one with a :Metric.
They will certainly complain when you try to iterate over a class with a statement like for metric in Metrics:, but that is easily resolvable by asserting to it explicitly that the class is iterable, using typing.cast. No tool (at least not yet) will be able to "see" that the metaclass you are using feature an __iter__ method that enables the class itself to be iterable.
from typing import cast
from collections.abc import Iterable
...
for metric in cast(Iterable, metrics):
...
Related
I'm in the process of migrating from 2.7 to 3.x and I'm trying to understand the __prepare__ method for metaclasses introduced in PEP3115.
In most of the examples I've seen, implementations of this method ignore the parameters (name, bases, and **kwargs) simply returns a custom dictionary that does something interesting to the namespace provided to the __new__ and __init__ methods for the metaclass. Even the example in PEP3115 does nothing with the parameters.
I don't doubt that there is some good reason for the signature of __prepare__ but I haven't seen the use case.
What are some good examples that demonstrate the rational for making the signature of __prepare__ take these parameters?
__prepare__ will create a namespace for the class like you said, so we can do some logic inside of it like this:
class MyMeta(type):
#classmethod
def __prepare__(metacls, klass_name, bases):
namespace = {'s': 'my string',
'description': 'N/A'}
if klass_name.endswith('Base'):
namespace.update({'description': 'base class'})
return namespace
class KlassBase(metaclass=MyMeta):
def __init__(self, value):
self.value = value
class SubKlass(KlassBase):
def __init__(self, value):
super().__init__(value)
print(KlassBase(5).s)
print(KlassBase(5).description)
print(SubKlass(5).s)
print(SubKlass(5).description)
And you got:
my string
base class
my string
N/A
The reason why we don't do it, because same things could be done in other part of the meta class like : __new__, __init__, or be overrided by the latter. So most of time, we won't do it in __prepare__
The following is an image of the class creation work-flow, which is much more clearer:
[sorry I cannot find the original source of this pic]
When you look at __prepare__, and the state of Python class creation mechanism at the time, it is pretty much clear that what was really needed was a mechanism to enable attribute order preservation.
This way, one would be able to create classes that would describe data records with ordered fields, which is pretty much what humans would expect when describing a record. (Try to imagine a form that each time it is rendered, it shuffles the field ordes, so half the time you will be filling in the country you live in before the country).
Instead of a fixed mechanism to just enable the class body namespace to be an collections.OrderedDict, they came up with __prepare__ which enables this easily, with a single line returning a new OrderedDict instance.
But __prepare__ can have so many uses and abuses, that I think no one really thought of all the possibilities. The parameters ou mentioned are avaliable at the time it is called, and since it exists, there is no reason whatsoever for they not to be passed to the function. Why to cripple one of knowing the class' name inside the __prepare__ function?
So, it is just a powerful and flexible mechanism put in place, and not necessarily all possible use cases were thought of when making it. The "Ordered Attributes" thing n the other hand is so important that it became the default for Python 3.6, with PEP 520, even without any custom metaclass declaration.
__prepare__ receivog bases, for example, would allow one to pre-populate the namespace with certain objects that would be found in the superclass namespaces, overriding the inheritance attribute access, for example.
Or it could simply check the class name against a pre-existing registry, and either raise or pre-populate stuff from there.
One super-easy usage would be to pre-populate __name__ for example, so that class attributes could make use of it:
import collections
class M(type):
#classmethod
def __prepare__(metacls, name, bases):
ns = collections.OrderedDict()
ns["__name__"] = name
class T(metaclass=M):
__table__ = f"_{__name__.lower()}"
trivia
Due to the way functions work as methods in Python 3, one interesting related thing that is not documented anywhere is that if __prepare__ is not explicitly decorated to be a classmethod, it works as a staticmethod, and the target class name is passed directly in the first parameter to it.
I've got two class trees in my Python code:
BaseComponent BaseSeries
/ | \ |
Resistor Capacitor Inductor ESeries
The BaseSeries class tree implements preferred numbers such as the E-series, and generates sets of values between a pair of powers (e.g. [1.0, 2.2, 4.7, 10, 22, 47, 100, 220, 470] for the E3 series with exponents between 1 to 3).
By default, ESeries and any other instance of BaseSeries creates sequences of float objects. I'd like to use these classes to instead create sequences of Resistor, Capacitor and Inductor objects. Ideally, the individual Resistor, Capacitor, Inductor and ESeries classes would remain usable on their own (i.e. not rely on methods being implemented by other classes).
This sounds like a job for multiple inheritance, but I'm a bit confused about how best to implement this in Python (3). Ideally I'd like to just define something like:
class ResistorESeries(Resistor, ESeries):
pass
class CapacitorESeries(Capacitor, ESeries):
pass
class InductorESeries(Inductor, ESeries):
pass
in order to create classes that yield sequences of resistors, capacitors and inductors, but I don't know how best to tell BaseSeries instances to create objects of type Resistor, Capacitor and Inductor. I can think of two ways, but I can't decide which one is best, and I have a feeling there is a simpler, more Pythonic way that I'm missing:
have BaseSeries contain a property or variable pointing to the element type (e.g. Resistor) set either by the constructor, a class variable in the child class (e.g. Resistor.ELEMENT_TYPE = Resistor) or with an abstract property provided by the child class:
class BaseSeries(object):
...
def elements(self):
# loop over numbers in this series
for v in self.values():
yield self.element_type(v)
#property
#abc.abstractmethod
def element_type(self):
return NotImplemented
class ESeries(BaseSeries):
....
class BaseComponent(object):
...
#property
def element_type(self):
return self
class Resistor(BaseComponent):
...
class ResistorESeries(Resistor, ESeries):
# now BaseSeries' `element_type` property is provided by `BaseComponent`
pass
This would mean ESeries cannot be used on its own as a concrete object, as it does not implement this property/variable, which is not ideal.
use self when creating elements in BaseSeries, where self will, as long as Resistor is earlier in the method resolution order, refer to the desired element:
class BaseSeries(object):
...
def elements(self):
# loop over numbers in this series
for v in self.values():
# self here would refer to `Resistor` in
# `ResistorESeries` instances
yield self(v)
class ESeries(BaseSeries):
....
class BaseComponent(object):
...
class Resistor(BaseComponent):
...
class ResistorESeries(Resistor, ESeries):
pass
This has the downside that, in instances of ESeries without being used as a mix-in, self will refer to itself, which does not support the correct __init__ signature.
So, does anyone have an idea of how best to do this in a Pythonic way, with maximum ability to reuse classes on their own?
You are likely mixing some concepts there - notably "instances" and "classes" - your example calls that do self(v) are perplexing.
I can't see from your design why the classes on the BaseComponent tree would need to be inherited along the BaseSeries tree: can't the component type simply be an attribute on the BaseSeries class?
It is simply a matter of using a class attribute, and in the code suggested on your first attempt, use a prosaic if statement.
class BaseSeries:
component = None
def elements(self):
# loop over numbers in this series
for v in self.values():
yield self.component(v) if self.component else v
class Capacitor(BaseComponent):
...
class CapacitorSeries(BaseSeries):
component = Capacitor
If you think you need multiple inheritance, you can just go for your idea of using a property, and use the same "if" statement there. But if both hierarchies are that ortogonal, I don't see why force the use of multiple inheritance, just because the language permits it.
Maybe you prefer to have it the other way around: a factory method on the component tree that will take an ESeries class as input, and extract the values from that ...
Anyway, you are not making clear the disctinction of classes and instances there. Do you need to have a way to produce several subclasses of
"CapacitorESeries", each class for a different value?
Or would you need just instances of "Capacitors", each for a different value produced on the series?
class BaseComponent:
...
#classmethod
def series_factory(self, series):
for value in series.values():
yield self.__class__(value)
Of course, there could be use cases for really needing classes for everything you claim, including the factories for series of classes, but your use of self as a callable in your snippets suggests that your stance on that is not that solid.
In that case, first, you need all methods to make proper use of super. even if they ar enot supposed to exist across both hierarchies, using super will just call the proper method on the superclasses. But for methods like __init__ this is just needed.
If you design a proper __init__ method using super, and always using named parameters, your second strategy will work out of the box, just fixing the instantiating call (to something other than self(v). Using named parameters and passing the remaining parameters to super will ensure each class in the tree consumes what it needs of those parameters - and when Python gets to the root of both your hierarchies and calls object's __init__, no parameter is remaining,
class BaseSeries:
def __init__(self, value_min, value_max, **kwargs):
self.value_min = value_min
selfvalue_max = value_max
super().__init__(**kwargs)
def elements(self):
# loop over numbers in this series
for v in self.values():
yield self.__class__(value_min = self.value_min, value_max=self.value_max, value=value)
class BaseComponent:
def __init__(self, value, **kwargs):
self.value = value
...
class CapacitorESeries(Capacitor, Eseries):
pass
So for a Django project, I would really like to be able to generate and display tables (not based on querysets) dynamically without needing to know the contents or schema beforehand.
It looks like the django-tables2 app provides nice functionality for rendering tables, but it requires that you either explicitly declare column names by declaring attributes on a custom-defined Table subclass or else provide a model for it infer the columns.
I.e, to use a column named "name", you'd do:
class NameTable(tables.Table):
name = tables.Column()
The Tables class does not provide a method for adding columns post-facto because, from reading the source, it seems to use a metaclass that sweeps the class attributes on __new__ and locks them in.
It seemed like very simple metaprogramming would be an elegant solution. I defined a basic class factory that accepts column names are arguments:
def define_table(columns):
class klass(tables.Table): pass
for col in columns:
setattr(klass, col, tables.Column())
return klass
Sadly this does not work. If I run `
x = define_table(["foo", "bar"])(data)
x.foo
x.bar
I get back:
<django_tables2.columns.base.Column object at 0x7f34755af5d0>
<django_tables2.columns.base.Column object at 0x7f347577f750>
But if I list the columns:
print x.base_columns
I get back nothing i.e. {}
I realize that there are probably simpler solutions (e.g. just bite the bullet and define every possible data configuration in code, or don't use django-tables2 and roll my own), but I am now treating this as an opportunity to learn more about meta programming, so I would really like to make this work this way.
Any idea what I'm wrong doing wrong? My theory is that the __new__ method (which is redefined in the metaclass Table uses) is getting invoked when klass is defined rather than when it's instantiated, so by the time I tack on the attributes it's too late. But that violates my understanding of when __new__ should happen. Otherwise, I'm struggling to understand how the metaclass __new__ can tell the difference between defined-in-code attributes vs. dynamically defined ones.
Thanks!
You're on the right track here, but instead of creating a barebones class and adding attributes to it, you should use the type() built-in function. The reason it's not working the way you're trying, is because the metaclass has already done its work.
Using type() allows you to construct a new class with your own attributes, while setting the base class. Meaning - you get to describe the fields you want as a blueprint to your class, allowing the Tables metaclass to take over after your definition.
Here's an example of using type() with django. I've used this myself for my own project (with some slight variations) but it should give you a nice place to start from, considering you're already almost there.
def define_table(columns):
attrs = dict((c, tables.Column()) for c in columns)
klass = type('DynamicTable', (tables.Table,), attrs)
return klass
You're confusing the __new__ of a "regular" class with the __new__ of a metaclass. As you note, Table relies on __new__ method on its metaclass. The metaclass is indeed invoked when the class is defined. The class is itself an instance of the metaclass, so defining the class is instantiating the metaclass. (In this case, Table is an instance of DeclarativeColumnMetaClass.) So by the time the class is define, it's too late.
One possible solution is to write a Table subclass that has some method refreshColumns or the like. You could adapt the code from DeclarativeColumnMetaclass.__new__ to essentially make refreshColumns do the same magic again. Then you could call refreshColumns() on your new class.
is there any possible to override class, or undeclare class or redeclare a Class in python?
Yes, just declare it again:
class Foo(object): x = 1
class Foo(object): x = 2
The above code will not raise any error, and the name Foo will refer to the second class declared. Note however, that the class declared by the first declaration will still exist if anything refers to it, e.g. an instance, or a derived class.
This means that existing instances will not change class when you declare a new class with the same name, and existing subclasses will not magically inherit from the new class.
Probably the simplest method to deal with subclasses is to also re-declare them, so they inherit from the "renewed" base class. An alternative would be to mess with their __bases__ property, although I can't tell you if that would have unexpected results (there will almost certainly be some corner cases where this would not work).
As to existing instances, it is possible to re-assign their __class__ property with a new class. This does present two issues - first you have to find them (see this question: Printing all instances of a class), and second of all, items stored in instance __dict__ or __slots__ properties will still be there in those instances. If that is not something that should happen with your new class definition, you will have to write appropriate code to handle that as part of the transformation.
IN summary, it's unlikely to be worth it except in quite simple cases. If you need complete uptime for a running system, you might be better using a replication-based approach to achieve code changes.
Update: If this is the kind of thing you know you're going to do, another solution would be to use the strategy pattern.
Undeclare a class using del className as usual.
Might be a n00b question, but I currently have a class that implements an iterator so I can do something like
for i in class():
but I want to be able to access the class by index as well like
class()[1]
How can I do that?
Thanks!
The current accepted answer from #Ignacio Vazquez-Abrams is sufficient. However, others interested in this question may want to consider inheriting their class from an abstract base class (ABC) (such as those found in the standard module collections.abc). This does a number of things (there are probably others as well):
ensures that all of the methods you need to treat your object "like a ____" are there
it is self-documenting, in that someone reading your code is able to instantly know that you intend your object to "act like a ____".
allows isinstance(myobject,SomeABC) to work correctly.
often provides methods auto-magically so we don't have to define them ourselves
(Note that, in addition to the above, creating your own ABC can allow you to test for the presence of a specific method or set of methods in any object, and based on this to declare that object to be a subclass of the ABC, even if the object does not inherit from the ABCdirectly. See this answer for more information.)
Example: implement a read-only, list-like class using ABC
Now as an example, let's choose and implement an ABC for the class in the original question. There are two requirements:
the class is iterable
access the class by index
Obviously, this class is going to be some kind of collection. So what we will do is look at our menu of collection ABC's to find the appropriate ABC (note that there are also numeric ABCs). The appropriate ABC is dependent upon which abstract methods we wish to use in our class.
We see that an Iterable is what we are after if we want to use the method __iter__(), which is what we need in order to do things like for o in myobject:. However, an Iterable does not include the method __getitem__(), which is what we need in order to do things like myobject[i]. So we'll need to use a different ABC.
On down the collections.abc menu of abstract base classes, we see that a Sequence is the simplest ABC to offer the functionality we require. And - would you look at that - we get Iterable functionality as a mixin method - which means we don't have to define it ourselves - for free! We also get __contains__, __reversed__, index, and count. Which, if you think about it, are all things that should be included in any indexed object. If you had forgotten to include them, users of your code (including, potentially, yourself!) might get pretty annoyed (I know I would).
However, there is a second ABC that also offers this combination of functionality (iterable, and accessible by []): a Mapping. Which one do we want to use?
We recall that the requirement is to be able to access the object by index (like a list or a tuple), i.e. not by key (like a dict). Therefore, we select Sequence instead of Mapping.
Sidebar: It's important to note that a Sequence is read-only (as is a Mapping), so it will not allow us to do things like myobject[i] = value, or random.shuffle(myobject). If we want to be able do things like that, we need to continue down the menu of ABCs and use a MutableSequence (or a MutableMapping), which will require implementing several additional methods.
Example Code
Now we are able to make our class. We define it, and have it inherit from Sequence.
from collections.abc import Sequence
class MyClass(Sequence):
pass
If we try to use it, the interpreter will tell us which methods we need to implement before it can be used (note that the methods are also listed on the Python docs page):
>>> myobject = MyClass()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MyClass with abstract methods __getitem__, __len__
This tells us that if we go ahead and implement __getitem__ and __len__, we'll be able to use our new class. We might do it like this in Python 3:
from collections.abc import Sequence
class MyClass(Sequence):
def __init__(self,L):
self.L = L
super().__init__()
def __getitem__(self, i):
return self.L[i]
def __len__(self):
return len(self.L)
# Let's test it:
myobject = MyClass([1,2,3])
try:
for idx,_ in enumerate(myobject):
print(myobject[idx])
except Exception:
print("Gah! No good!")
raise
# No Errors!
It works!
Implement both __iter__() and __getitem__() et alia methods.