Develop Reference

Python: How to annotate a variable number of iterable attributes?

I have a class family for which I need to be able to iterate through attributes of type: Metric.
The family consists of an abstract base class parent and child classes. The child classes will all have varying number of class attributes of type Metric, and they inherit an __iter__ method from the parent class that allows me to iterate through the attributes.
I am using iterable attributes rather than a dict because I want my objects to be typed, but I need to be able to call metrics in sequence, and by name.
So I need to be able to do:
Metrics.metric_1
and
for metric in Metrics:
My question is, how do I correctly hint in the base class that there are a variable number of attributes of the same type?
I'm currently using a couple of attribute hints with an ellipsis:
class MetricsBase(ABC):
metric_1: Metric
metric_2: Metric
...
#classmethod
def __iter__(cls):
for attr, value in cls.__dict__.items():
if not attr.startswith("__"):
yield value
class MetricChild(MetricsBase):
metric_1 = Metric(x)
metric_2 = Metric(y)
metric_3 = Metric(z)
But I'm not sure if this is pythonic or correct, and wondering if there is a neater way of doing this.
Many thanks for any input!

I am not answering how to "fix" static type checking on that.
That said, this is ok as Python code, hence "pythonic" . the problem is that you want to use static type checking on it - and you are using a dynamic meta programming technique there. Static type checking is not meant to check this (in a way of saying, it can only handle a small subset of what would be "pythonic"). Maybe there is a way to "solve" this - but if you can, just mark the static type checkers to skip that, and spare hours yourself hours of meaningless work (since it won't change how the code works)
More important than that, that __iter__ method won't work for the class itself, regardless of you marking it as a #classmethod. (It will work fot the instances, despite you doing so, though). If you want to iterate on the class, you will have to resort to a metaclass:
import abc
class MetricMeta(abc.ABCMeta):
def __iter__(cls):
# this will make _instances of this metaclass__ iterable
for attr, value in cls.__dict__.items():
if not attr.startswith("__"):
yield value
class MetricsBase(metaclass=MetricsMeta):
metric_1: Metric
metric_2: Metric
...
Type chekers actually, should, supposedly, not need one to expliclty annotate all variables, reducing Python to a subset of Pascal or the like. If you simply type in your class attributes in each subclass, attributing a Metric instance to then it should work, without the need to explictly annotate each one with a :Metric.
They will certainly complain when you try to iterate over a class with a statement like for metric in Metrics:, but that is easily resolvable by asserting to it explicitly that the class is iterable, using typing.cast. No tool (at least not yet) will be able to "see" that the metaclass you are using feature an __iter__ method that enables the class itself to be iterable.
from typing import cast
from collections.abc import Iterable
...
for metric in cast(Iterable, metrics):
...

Inhibiting a method called on a dataclass member

My dataclass has a field that holds an array of data in a custom type (actually it is a PyROOT std vector). However, for the user it is supposed to be visible as a list. This is simple enough with dataclass getters and setters, that convert the vector to list and vice versa. However, this works only if the user initialises the field with a full list. If the user wants to append to the list, it, obviously, doesn't work, as there is no permanent list associated with the field.
I wonder if there is a way to inhibit the ".append()" call on the field and call instead the vector's push_back()? Or perhaps there is a good Pythonic way to deal with it in general?
The context is, that I need the dataclass fields in the PyROOT format, as later I am storing the data in ROOT TTrees. However, I am creating this interface, so that the user does not need to know ROOT to use the dataclass. I know that I could create both the vector and the list that would hold the same data, but that seems like a waste of memory, and I am not certain how to update the vector each time the list is modified, anyway.

According to the Python Docs, “Lists are mutable sequences, typically used to store collections of homogeneous items (where the precise degree of similarity will vary by application).” (emphasis added)
With that in mind, I would start off with something like this:
from collections.abc import MutableSequence
class ListLike(MutableSequence):
def __init__(self):
self.backing_data = object() # Replace with the type your using
ListLike()
When you run that code, you’ll get the error: TypeError: Can't instantiate abstract class ListLike with abstract methods __delitem__, __getitem__, __len__, __setitem__, insert. Once you implement those methods, you’ll have have a type that acts a lot like list, but isn’t.
To make ListLikes act even more like lists, use this code to compare the two:
example_list = list()
example_list_like = ListLike()
list_attributes = [attribute for attribute in dir(example_list)]
list_like_attributes = [attribute for attribute in dir(example_list_like)]
for attribute in list_attributes:
if attribute not in list_like_attributes:
print(f"ListLikes don't have {attribute}")
print("-----------")
for attribute in list_like_attributes:
if attribute not in list_attributes:
print(f"lists don't have {attribute}")
and change your implementation accordingly.

Python object's __iter__not getting called

I have a Python 2.7 class (call it Child), that is a child of another class (Parent) that is itself a subclass of dict.
I'm trying to define __iter__ in Child in the hopes that when someone does a dict(child_object) I can control how it is converted to a dict. I must be misunderstanding something though, because it seems the dict() call is bypassing calling __iter__ completely and is instead going to the underlying dict.
I did some research and from the dict() method's docs I see that it may be seeing the object as a mapping first, and therefore using that instead of the iterable's __iter__. Is that the case, and if so, is there a way I can overwrite some method that is being called on the mapping?

When you use dict() on a mapping (or use dictionary.update(...) passing in a mapping), then Python will not use __iter__. Python looks for a .keys() method to detect mappings. In that case, if the mapping happens to be a dict or a subclass of dict, then a fast path is picked that copies key-value pairs directly from the underlying C structures. You can't prevent this with custom Python functions.
Put differently, if you must define a custom mapping type that lets you control how dict() copies key-value pairs from it, you should not subclass dict. Implement your own mapping type by subclassing collections.Mapping or collections.MutableMapping or a class from the UserDict module.

How to define a "callable" parameter in a Python docstring?

Consider an implementation of filterNot (basically the opposite of filter):
def filterNot(f, sequence):
return filter(lambda x: not f(x), sequence)
The parameter f can be a "function" or a "method" or a lambda -- or even an object whose class defines __call__.
Now consider a line of docstring for this parameter:
:param ??? f: Should return True for each element to be abandoned
Now, what should go in place of ??? -- how should the type of parameter f be referred to in a docstring. callable is the obvious choice (and what I would dictate if I were calling the shots :P) but is there an established convention?

Yes, the term callable is the one to use here.
The abstract base class Callable exists in collections.abc - abstract base classes can be best thought of as interfaces (although more like they dynamic ones in Go than those in Java, for example) - they define an interface, and any class that has the given functions is defined as inheriting from that abstract base class (whether they did so explicitly or not) - this means anything you would usefully pass into a function like this would be a subclass of Callable, making the use of the term completely correct here. Just as you might say Iterable.
It is definitely the term used by most people in when talking informally about Python code, and anyone reading your code should understand what you mean.
The callable() built-in (that got removed for a while in 3.x, then added back) does the check for function-like objects, and this further reinforces the name as the best choice where you are looking for function-like objects.

Python class accessible by iterator and index

Might be a n00b question, but I currently have a class that implements an iterator so I can do something like
for i in class():
but I want to be able to access the class by index as well like
class()[1]
How can I do that?
Thanks!

The current accepted answer from #Ignacio Vazquez-Abrams is sufficient. However, others interested in this question may want to consider inheriting their class from an abstract base class (ABC) (such as those found in the standard module collections.abc). This does a number of things (there are probably others as well):
ensures that all of the methods you need to treat your object "like a ____" are there
it is self-documenting, in that someone reading your code is able to instantly know that you intend your object to "act like a ____".
allows isinstance(myobject,SomeABC) to work correctly.
often provides methods auto-magically so we don't have to define them ourselves
(Note that, in addition to the above, creating your own ABC can allow you to test for the presence of a specific method or set of methods in any object, and based on this to declare that object to be a subclass of the ABC, even if the object does not inherit from the ABCdirectly. See this answer for more information.)
Example: implement a read-only, list-like class using ABC
Now as an example, let's choose and implement an ABC for the class in the original question. There are two requirements:
the class is iterable
access the class by index
Obviously, this class is going to be some kind of collection. So what we will do is look at our menu of collection ABC's to find the appropriate ABC (note that there are also numeric ABCs). The appropriate ABC is dependent upon which abstract methods we wish to use in our class.
We see that an Iterable is what we are after if we want to use the method __iter__(), which is what we need in order to do things like for o in myobject:. However, an Iterable does not include the method __getitem__(), which is what we need in order to do things like myobject[i]. So we'll need to use a different ABC.
On down the collections.abc menu of abstract base classes, we see that a Sequence is the simplest ABC to offer the functionality we require. And - would you look at that - we get Iterable functionality as a mixin method - which means we don't have to define it ourselves - for free! We also get __contains__, __reversed__, index, and count. Which, if you think about it, are all things that should be included in any indexed object. If you had forgotten to include them, users of your code (including, potentially, yourself!) might get pretty annoyed (I know I would).
However, there is a second ABC that also offers this combination of functionality (iterable, and accessible by []): a Mapping. Which one do we want to use?
We recall that the requirement is to be able to access the object by index (like a list or a tuple), i.e. not by key (like a dict). Therefore, we select Sequence instead of Mapping.
Sidebar: It's important to note that a Sequence is read-only (as is a Mapping), so it will not allow us to do things like myobject[i] = value, or random.shuffle(myobject). If we want to be able do things like that, we need to continue down the menu of ABCs and use a MutableSequence (or a MutableMapping), which will require implementing several additional methods.
Example Code
Now we are able to make our class. We define it, and have it inherit from Sequence.
from collections.abc import Sequence
class MyClass(Sequence):
pass
If we try to use it, the interpreter will tell us which methods we need to implement before it can be used (note that the methods are also listed on the Python docs page):
>>> myobject = MyClass()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MyClass with abstract methods __getitem__, __len__
This tells us that if we go ahead and implement __getitem__ and __len__, we'll be able to use our new class. We might do it like this in Python 3:
from collections.abc import Sequence
class MyClass(Sequence):
def __init__(self,L):
self.L = L
super().__init__()
def __getitem__(self, i):
return self.L[i]
def __len__(self):
return len(self.L)
# Let's test it:
myobject = MyClass([1,2,3])
try:
for idx,_ in enumerate(myobject):
print(myobject[idx])
except Exception:
print("Gah! No good!")
raise
# No Errors!
It works!

Implement both __iter__() and __getitem__() et alia methods.