Develop Reference

mypy fails to narrow down generic types in `TypeGuard`-based conditionals

Trying to create a type-pure container in Python, I've stumbled upon type-narrowing failure of generic types by mypy when creating a signature of the __setitem__ method.
Before we dive in, I'm using:
mypy 0.991
python 3.10.6.
First, let's consider a basic example.
Let us define two types, A and B, and a type variable AB that can be either A or B (but not a Union[A, B]):
from __future__ import annotations
import typing as t
from collections import abc
from itertools import tee
from typing_extensions import reveal_type
class A:
...
class B:
...
AB = t.TypeVar('AB', A, B) # Strictly A or B
Now, suppose we want to define a function or method with the following signature:
def fn(arg: AB | abc.Iterable[AB]):
The function's body will depend on whether AB or abc.Iterable[AB] are provided.
To check that, let's introduce the following utilities:
T = t.TypeVar('T')
def is_iterable_of(
s: abc.Iterable[t.Any], _type: t.Type[T]
) -> t.TypeGuard[abc.Iterable[T]]:
return all(isinstance(x, _type) for x in s)
def is_type(x: t.Any, _type: t.Type[T]) -> t.TypeGuard[T]:
return isinstance(x, _type)
def is_a_or_b(x: t.Any) -> t.TypeGuard[A | B]:
return isinstance(x, A) or isinstance(x, B)
def is_iterable_of_ab(s: abc.Iterable[t.Any]) -> t.TypeGuard[abc.Iterable[AB]]:
ss = tee(s)
return any(
is_iterable_of(_s, _t) for _s, _t in zip(ss, [A, B])
)
I think only is_iterable_of_ab might be a little confusing: it checks whether the provided iterable contains all A types or all B types. To view what TypeGuard types are doing, please refer to the mypy docs.
We apply our type-narrowing functions in fn as follows:
def fn(arg: A | B | abc.Iterable[AB]):
if is_a_or_b(arg):
# Process single instance
reveal_type(arg) # A | B
else:
# Process multiple instances
reveal_type(arg) # (A | B | Iterable[A]) | (A | B | Iterable[B])
assert is_iterable_of_ab(arg)
Mypy produces the following output (you can check the full example's code in mypy playground):
main.py:46: note: Revealed type is "Union[__main__.A, __main__.B]"
main.py:49: note: Revealed type is "Union[__main__.A, __main__.B, typing.Iterable[__main__.A]]"
main.py:49: note: Revealed type is "Union[__main__.A, __main__.B, typing.Iterable[__main__.B]]"
main.py:50: error: Argument 1 to "is_iterable_of_ab" has incompatible type "Union[A, B, Iterable[A]]"; expected "Iterable[Any]" [arg-type]
main.py:50: error: Argument 1 to "is_iterable_of_ab" has incompatible type "Union[A, B, Iterable[B]]"; expected "Iterable[Any]" [arg-type]
What is confusing here is that the call to is_a_or_b, while narrows down the type to A | B correctly, doesn't lead to mypy recognizing that if the conditional is False, the provided arg must have the abc.Iterable[AB] type. As a result, any functionality inside the else block that depends on arg being Iterable[AB] will raise an error since mypy thinks arg is can still be A | B.
Note that this also fails:
def fn2(arg: A | B | abc.Iterable[AB]):
if is_a_or_b(arg):
# Process single instance
reveal_type(arg) # A | B
elif is_iterable_of_ab(arg): # error: incompatible type
# Process multiple instances
reveal_type(arg) # wrong type: Iterable[A]
else:
raise TypeError()
So, how could I correctly use type-narrowing here?
For a more realistic example indicated at the beginning, consider making a type pure list class:
class TypePureList(t.Generic[AB]):
def __init__(self, items: abc.Iterable[AB]):
if not isinstance(items, list):
items = list(items)
self._items: list[AB] = items
#t.overload
def __setitem__(self, index: t.SupportsIndex, value: AB):
...
#t.overload
def __setitem__(self, index: slice, value: abc.Iterable[AB]):
...
def __setitem__(self, index: t.SupportsIndex | slice, value: AB | abc.Iterable[AB]):
if len(self._items) == 0:
raise ValueError
if isinstance(index, t.SupportsIndex):
# Check that the value type is correct by itself
assert is_a_or_b(value)
# Check that the value type matches the existing type
assert is_iterable_of_ab([self._items[0], value])
reveal_type(value) # A | B
else:
# Index must be slice
# Value must be Iterable
reveal_type(index) # slice
reveal_type(value) # (A | Iterable[A]) | (B | Iterable[B])
# Check that the iterable contains correct types
value, value1, value2 = tee(value, 3) # error: incompatible type
assert is_iterable_of_ab(value1)
# Check if the iterable types match the existing type
assert is_iterable_of_ab([self._items[0], next(value2)])
# error: invalid syntax -- index is Union[SupportsIndex, slice]
# error: incompatible types in assignment
self._items.__setitem__(index, value)
Firstly, mypy fails to use the overloading information, but that seems to be an ongoing issue (see #7858).
Secondly, same as above, mypy fails to narrow down the types following the assertion that if index has SupportsIndex type, then value must be Iterable[AB] due to assert is_a_or_b(value).

What is the difference between data class and class (Python)? [duplicate]

With PEP 557 data classes are introduced into python standard library.
They make use of the #dataclass decorator and they are supposed to be "mutable namedtuples with default" but I'm not really sure I understand what this actually means and how they are different from common classes.
What exactly are python data classes and when is it best to use them?

Data classes are just regular classes that are geared towards storing state, rather than containing a lot of logic. Every time you create a class that mostly consists of attributes, you make a data class.
What the dataclasses module does is to make it easier to create data classes. It takes care of a lot of boilerplate for you.
This is especially useful when your data class must be hashable; because this requires a __hash__ method as well as an __eq__ method. If you add a custom __repr__ method for ease of debugging, that can become quite verbose:
class InventoryItem:
'''Class for keeping track of an item in inventory.'''
name: str
unit_price: float
quantity_on_hand: int = 0
def __init__(
self,
name: str,
unit_price: float,
quantity_on_hand: int = 0
) -> None:
self.name = name
self.unit_price = unit_price
self.quantity_on_hand = quantity_on_hand
def total_cost(self) -> float:
return self.unit_price * self.quantity_on_hand
def __repr__(self) -> str:
return (
'InventoryItem('
f'name={self.name!r}, unit_price={self.unit_price!r}, '
f'quantity_on_hand={self.quantity_on_hand!r})'
def __hash__(self) -> int:
return hash((self.name, self.unit_price, self.quantity_on_hand))
def __eq__(self, other) -> bool:
if not isinstance(other, InventoryItem):
return NotImplemented
return (
(self.name, self.unit_price, self.quantity_on_hand) ==
(other.name, other.unit_price, other.quantity_on_hand))
With dataclasses you can reduce it to:
from dataclasses import dataclass
#dataclass(unsafe_hash=True)
class InventoryItem:
'''Class for keeping track of an item in inventory.'''
name: str
unit_price: float
quantity_on_hand: int = 0
def total_cost(self) -> float:
return self.unit_price * self.quantity_on_hand
The same class decorator can also generate comparison methods (__lt__, __gt__, etc.) and handle immutability.
namedtuple classes are also data classes, but are immutable by default (as well as being sequences). dataclasses are much more flexible in this regard, and can easily be structured such that they can fill the same role as a namedtuple class.
The PEP was inspired by the attrs project, which can do even more (including slots, validators, converters, metadata, etc.).
If you want to see some examples, I recently used dataclasses for several of my Advent of Code solutions, see the solutions for day 7, day 8, day 11 and day 20.
If you want to use dataclasses module in Python versions < 3.7, then you could install the backported module (requires 3.6) or use the attrs project mentioned above.

Overview
The question has been addressed. However, this answer adds some practical examples to aid in the basic understanding of dataclasses.
What exactly are python data classes and when is it best to use them?
code generators: generate boilerplate code; you can choose to implement special methods in a regular class or have a dataclass implement them automatically.
data containers: structures that hold data (e.g. tuples and dicts), often with dotted, attribute access such as classes, namedtuple and others.
"mutable namedtuples with default[s]"
Here is what the latter phrase means:
mutable: by default, dataclass attributes can be reassigned. You can optionally make them immutable (see Examples below).
namedtuple: you have dotted, attribute access like a namedtuple or a regular class.
default: you can assign default values to attributes.
Compared to common classes, you primarily save on typing boilerplate code.
Features
This is an overview of dataclass features (TL;DR? See the Summary Table in the next section).
What you get
Here are features you get by default from dataclasses.
Attributes + Representation + Comparison
import dataclasses
#dataclasses.dataclass
##dataclasses.dataclass() # alternative
class Color:
r : int = 0
g : int = 0
b : int = 0
These defaults are provided by automatically setting the following keywords to True:
#dataclasses.dataclass(init=True, repr=True, eq=True)
What you can turn on
Additional features are available if the appropriate keywords are set to True.
Order
#dataclasses.dataclass(order=True)
class Color:
r : int = 0
g : int = 0
b : int = 0
The ordering methods are now implemented (overloading operators: < > <= >=), similarly to functools.total_ordering with stronger equality tests.
Hashable, Mutable
#dataclasses.dataclass(unsafe_hash=True) # override base `__hash__`
class Color:
...
Although the object is potentially mutable (possibly undesired), a hash is implemented.
Hashable, Immutable
#dataclasses.dataclass(frozen=True) # `eq=True` (default) to be immutable
class Color:
...
A hash is now implemented and changing the object or assigning to attributes is disallowed.
Overall, the object is hashable if either unsafe_hash=True or frozen=True.
See also the original hashing logic table with more details.
What you don't get
To get the following features, special methods must be manually implemented:
Unpacking
#dataclasses.dataclass
class Color:
r : int = 0
g : int = 0
b : int = 0
def __iter__(self):
yield from dataclasses.astuple(self)
Optimization
#dataclasses.dataclass
class SlottedColor:
__slots__ = ["r", "b", "g"]
r : int
g : int
b : int
The object size is now reduced:
>>> imp sys
>>> sys.getsizeof(Color)
1056
>>> sys.getsizeof(SlottedColor)
888
In some circumstances, __slots__ also improves the speed of creating instances and accessing attributes. Also, slots do not allow default assignments; otherwise, a ValueError is raised.
See more on slots in this blog post.
Summary Table
+----------------------+----------------------+----------------------------------------------------+-----------------------------------------+
| Feature | Keyword | Example | Implement in a Class |
+----------------------+----------------------+----------------------------------------------------+-----------------------------------------+
| Attributes | init | Color().r -> 0 | __init__ |
| Representation | repr | Color() -> Color(r=0, g=0, b=0) | __repr__ |
| Comparision* | eq | Color() == Color(0, 0, 0) -> True | __eq__ |
| | | | |
| Order | order | sorted([Color(0, 50, 0), Color()]) -> ... | __lt__, __le__, __gt__, __ge__ |
| Hashable | unsafe_hash/frozen | {Color(), {Color()}} -> {Color(r=0, g=0, b=0)} | __hash__ |
| Immutable | frozen + eq | Color().r = 10 -> TypeError | __setattr__, __delattr__ |
| | | | |
| Unpacking+ | - | r, g, b = Color() | __iter__ |
| Optimization+ | - | sys.getsizeof(SlottedColor) -> 888 | __slots__ |
+----------------------+----------------------+----------------------------------------------------+-----------------------------------------+
+These methods are not automatically generated and require manual implementation in a dataclass.
* __ne__ is not needed and thus not implemented.
Additional features
Post-initialization
#dataclasses.dataclass
class RGBA:
r : int = 0
g : int = 0
b : int = 0
a : float = 1.0
def __post_init__(self):
self.a : int = int(self.a * 255)
RGBA(127, 0, 255, 0.5)
# RGBA(r=127, g=0, b=255, a=127)
Inheritance
#dataclasses.dataclass
class RGBA(Color):
a : int = 0
Conversions
Convert a dataclass to a tuple or a dict, recursively:
>>> dataclasses.astuple(Color(128, 0, 255))
(128, 0, 255)
>>> dataclasses.asdict(Color(128, 0, 255))
{'r': 128, 'g': 0, 'b': 255}
Limitations
Lacks mechanisms to handle starred arguments
Working with nested dataclasses can be complicated
References
R. Hettinger's talk on Dataclasses: The code generator to end all code generators
T. Hunner's talk on Easier Classes: Python Classes Without All the Cruft
Python's documentation on hashing details
Real Python's guide on The Ultimate Guide to Data Classes in Python 3.7
A. Shaw's blog post on A brief tour of Python 3.7 data classes
E. Smith's github repository on dataclasses

From the PEP specification:
A class decorator is provided which inspects a class definition for
variables with type annotations as defined in PEP 526, "Syntax for
Variable Annotations". In this document, such variables are called
fields. Using these fields, the decorator adds generated method
definitions to the class to support instance initialization, a repr,
comparison methods, and optionally other methods as described in the
Specification section. Such a class is called a Data Class, but
there's really nothing special about the class: the decorator adds
generated methods to the class and returns the same class it was
given.
The #dataclass generator adds methods to the class that you'd otherwise have to define yourself like __repr__, __init__, __lt__, and __gt__.

Consider this simple class Foo
from dataclasses import dataclass
#dataclass
class Foo:
def bar():
pass
Here is the dir() built-in comparison. On the left-hand side is the Foo without the #dataclass decorator, and on the right is with the #dataclass decorator.
Here is another diff, after using the inspect module for comparison.

Programmatic generation of Literal options

MyPy's Literal type can be super useful for defining available options. Is it possible to generate a literal type programmatically, e.g. from a canonical registry?
e.g.
class Dispatcher():
func_reg = {
'f1': my_func,
'f2': new_func,
'f3': shoe_func,
}
def dispatch(cls, func_name: Literal[*func_reg.keys()]) -> Whatever:
pass

Unfortunately, the answer is no.
According to the mypy documentation:
Literal types may contain one or more literal bools, ints, strs, bytes, and enum values. However, literal types cannot contain arbitrary expressions: types like Literal[my_string.trim()], Literal[x > 3], or Literal[3j + 4] are all illegal.

As #BrokenBenchmark notes, it is not possible to auto-generate Literal types. However, if the end goal is just to require specific values generated from some kind of function registry, we can hack it with enum.Enum.
To quote PEP 586
rather than entirely special-casing enums, we can instead treat them
as being approximately equivalent to the union of their values...
the Status enum could be treated as being approximately equivalent to Literal[Status.SUCCESS, Status.INVALID_DATA, Status.FATAL_ERROR]
Here, functions are "registered" by adding an enum value that is an exact uppercasing of the function name to the FuncNames enum. This is not a pretty or robust solution, but it runs, it supports single-location registration of a function for type-checked dispatch, and mypy handles the required enum values as expected.
from enum import Enum, auto
def f():
return "f"
def g():
return "g"
def h():
return "h"
class Dispatcher():
# Build the enum used to register the functions
class FuncNames(Enum):
"""
The enum names here _must_ be exact uppercase-ings of the function
names. The names will be lowercased and evaluated to register their
associated functions
"""
F = auto()
G = auto()
H = auto()
# NOTE: The functional syntax works just as well
# FuncNames = Enum('FuncNames', 'F G H')
# Comprehensions can't access names defined in the class block,
# so use a standard for loop
func_reg = dict()
for name in list(FuncNames):
func_reg[eval(f"FuncNames.{name.name}")] = eval(str(name.name).lower())
#classmethod
def dispatch(cls, func_name: FuncNames):
"""
Prints the return from a registered function.
Can only be called with an item from FuncNames
"""
print(cls.func_reg[func_name]())
Dispatcher.dispatch(Dispatcher.FuncNames.F)
Dispatcher.dispatch(Dispatcher.FuncNames.G)
Dispatcher.dispatch(Dispatcher.FuncNames.H)
# Dispatcher.dispatch(Dispatcher.FuncNames.I) -> "FuncNames has no attribute I"
# Dispatcher.dispatch(Dispatcher2.FuncNames) -> "incompatible type"
# Dispatcher.dispatch('MyPy hates me!') -> "incompatible type"
Interestingly, though it feels cleaner to generate the enum itself from a list of the functions themselves, MyPy chokes on this.
class Dispatcher2():
# Build an enum used to register these (the actual functions)
funcs_to_register = [f, g, h]
enum_names = [func.__name__.upper() for func in funcs_to_register]
joined = ' '.join(enum_names)
FuncNames = Enum('FuncNames', joined)
func_reg = dict()
for name in enum_names:
func_reg[eval(f"FuncNames.{name}")] = eval(name.lower())
#classmethod
def dispatch(cls, func_name: FuncNames):
"""
Prints the return from a registered function.
Can only be called with an item from FuncNames
"""
print(cls.func_reg[func_name]())
Dispatcher2.dispatch(Dispatcher2.FuncNames.F)
Dispatcher2.dispatch(Dispatcher2.FuncNames.G)
Dispatcher2.dispatch(Dispatcher2.FuncNames.H)
The above runs as expected, but mypy presumably can't infer the values present in the enum unless it is statically defined, so it errors.
> mypy enums_typing.py
enums_typing.py:19: error: Enum() expects a string, tuple, list or dict literal as the second argument
enums_typing.py:36: error: "Type[FuncNames]" has no attribute "F"
enums_typing.py:37: error: "Type[FuncNames]" has no attribute "G"
enums_typing.py:38: error: "Type[FuncNames]" has no attribute "H"
Found 4 errors in 1 file (checked 1 source file)
TLDR:
In order to define a fixed set of choices that MyPy can check against, you must define them statically. It may then be possible to use those statically-defined choices to programmatically build your function registry.

Python: Naming types and assigning variables outside a function from within a function

I'd like to be able to easily create new types, plus (optionally) add some information about them (say some docs, and a set of variable names they often come under).
The straightforward way to do this would be:
from typing import Any, NewType, Union, List, Iterable, Optional
Key = NewType('Key', Any)
Key._aka = set(['key', 'k'])
Val = NewType('Val', Union[int, float, List[Union[int, float]]])
Val.__doc__ = "A number or list of numbers."
But there's two reasons I don't like this:
I have to copy paste the name of the new type I'm making three times (not D.R.Y. and prone to mistakes)
I don't like to "externalize" the assignment of optional additional information (_aka and __doc__)
So I came up with this:
from typing import Any, NewType, Union, List, Iterable, Optional
def new_type(name, tp, doc: Optional[str]=None, aka: Optional[Iterable]=None):
"""Make a new type with (optional) doc and (optional) aka, set of var names it often appears as"""
new_tp = NewType(name, tp)
if doc is not None:
setattr(new_tp, '__doc__', doc)
if aka is not None:
setattr(new_tp, '_aka', set(aka))
globals()[name] = new_tp # is this dangerous? Does scope need to be considered more carefully?
which then gives me the interface I'd like:
new_type('Key', Any, aka=['key', 'k'])
new_type('Val', Union[int, float, List[Union[int, float]]], doc="A number or list of numbers.")
But I'm not sure of that globals()[name] = new_tp thing. It seems it would be benign if I'm defining my types in the top level of a module, but not sure how this would fair in some edge case nested scopes situation.

The normal way you create a new type is to just write a new class:
class Key:
def __init__(self, key: object) -> None:
self._key = key
self._aka = set(['key', 'k'])
class SqlString(str):
"""A custom string which has been verified to be valid SQL."""
Note that this approach avoids the DRY and scoping concerns that you had.
You use NewType only for when you don't want to add any extra attributes or a docstring -- doing Foo = NewType('Foo', X) is basically like doing class Foo(X): pass except with slightly less runtime overhead.
(More precisely, type checkers will treat Foo = NewType('Foo', X) as if it were written like class Foo(X): pass, but at runtime what actually happens Foo = lambda x: x -- Foo is the identity function.)
We do run into a complication with your second Union example, however, since Unions are not subclassable (which makes that NewType illegal, as per PEP 484).
Instead, I would personally just do the following:
# A number or list of numbers
Val = Union[int, float, List[Union[int, float]]]
IMO since types are supposed to be invisible at runtime, I think it makes sense to just not bother attaching runtime-available documentation.

Transform a "collection" (list, set, single object, ...) into a new collection of the same type in Python

First of all, I use the terms "container" and "collection" in a very general way, not linked to any Python terminology.
I have the following function in Python. It takes a list of ids idlist and returns a list of objects from objs corresponding to those ids. That works fine:
def findObj(idlist, objs):
return [next(o for o in objs if id == o.id) for id in idlist]
The problem is: idlist and the return value should not necessarily need to be a Python list. I would wish that idlist could also be one plain id, and the function would return a single object. Or idlist would be a Python set and the function would return a set of objects.
How can I achieve that I can use various "container" types (including a plain id) for idlist and get returned the same "container" type?

I argue that what you have in mind is not a good API.
It's simpler, more robust and less error prone to have the function return a specific type and let the user handle the eventual conversion.
In particular I'd prefer making the function lazy using a generator:
def find_obj(ids, objs):
try:
for id in ids:
yield next(o for o in objs if o.id == id)
except TypeError:
# ids not iterable? assume it is a plain id
yield next(o for o in objs if o.id == ids)
Now a user can just do:
list(find_obj(...))
set(find_obj(...))
next(find_obj(...)) # 1 element
And obtain the thing he wants.
Added benefits:
Explicit is better than implicit: here the type conversion is explicit. Image code where the calls are of the kind find_obj(some_var_defined_elsewhere, objects) now how do you know which type will be returned if the definition of the input is not near there?
you can pass a type X as input and convert to type Y without wasting intermediate space and doing an unneccessary conversion
No special cases needed. The caller can provide an input that doesn't follow the usual way to construct containers (note that there is no standard way to build a container)
Alternative that special cases the single id case:
def find_obj(ids, objs):
try:
return (next(o for o in objs if o.id == id) for id in ids)
except TypeError:
for o in objs:
if o.id == id:
return o
The above returns a generator when given a sequence of ids and returns a plain object (instead of a 1-element-generator) when passed in a single id.
Finally: most of the time (but not always) sequences have a constructor that accepts an iterable and builds the container with those elements. This means that:
type(some_iterable)(something for el in some_iterable)
will produce a container of the same type as some_iterable.
Note that some classes require a list instead of a generic iterable so you'd have to use type(some_iterable)([<as-before>]) and other containers do not have such a constructor. In this last case only the caller could perform the conversion. The first solution handles this nicely without any special case.
You could generalize more the function by adding a parameter to perform the conversion:
def find_obj(ids, objs, converter=None):
if converter is None:
converter = type(ids)
try:
return converter(next(o for o in objs if o.id == id) for id in ids)
except TypeError:
return next(o for o in objs if o.id == ids)
in this way the caller can customize the conversion if he's dealing with strange types.
An added note: in python we use "duck typing", i.e. just use the object as if it was of the correct type and if it raises an exception fallback to do other stuff. In some cases it's simpler to first check for support of certain operations, in that cases you could use isinstance with the abstract base classes found in collections.abc to see if an object is Iterable, a Sequence, a Mapping etc.

This should work if your plain object's id is passed (for idliist) as int (not string)
def findObj(idlist, objs):
t = type(idlist)
try:
iter(idlist)
return t([next(o for o in objs if id == o.id) for id in idlist])
except TypeError, te: #not iterator. So single id and object
return idlist if idlist == objs.id else False

You can change your code to:
import collections
def findObj(idlist, objs):
if isinstance(idlist, collections.Iterable):
return type(idlist)([next(o for o in objs if id == o.id) for id in idlist])
else:
#cover case when idlist is just plain object
pass