Develop Reference

In a python iterator select other iterator based on conditions

In python I have an iterator returning an infinite string of indices in a fixed range [0, N] called Sampler. Actually I have a list of those and all they do is return indices in the range [0, N_0], [N_0, N_1], ..., [N_{n-1}, N_n].
What I now want to do is first select one of these iterators based on the length of their range, so I have a weights list [N_0, N_1 - N_0, ...] and I select one of these with:
iterator_idx = random.choices(range(len(weights)), weights=weights/weights.sum())[0]
Next, what I want to do is create an iterator which randomly selects one of the iterators and selects a batch of M samples.
class BatchSampler:
def __init__(self, M):
self.M = M
self.weights = [weight_list]
self.samplers = [list_of_iterators]
]
self._batch_samplers = [
self.batch_sampler(sampler) for sampler in self.samplers
]
def batch_sampler(self, sampler):
batch = []
for batch_idx in sampler:
batch.append(batch_idx)
if len(batch) == self.M:
yield batch
if len(batch) > 0:
yield batch
def __iter__(self):
# First select one of the datasets.
iterator_idx = random.choices(
range(len(self.weights)), weights=self.weights / self.weights.sum()
)[0]
return self._batch_samplers[iterator_idx]
The issue with this is that iter() only seems to be called once, so only the first time iterator_idx is selected. Obviously this is wrong... What is the way around this?
This is a possible case when you would have multiple datasets in pytorch, but you want to sample only batches from one of the datasets.

Seems to me that you want to define your own container type.
I'll try to provide examples of a few standard ways to do so
(hopefully without missing too many details);
you should be able to reuse one of these simple examples,
into your own class.
Using just __getitem__ (support indexing & looping):
object.__getitem__
Called to implement evaluation of self[key].
class MyContainer:
def __init__(self, sequence):
self.elements = sequence # Just something to work with.
def __getitem__(self, key):
# If we're delegating to sequences like built-in list,
# invalid indices are handled automatically by them
# (throwing IndexError, as per the documentation).
return self.elements[key]
t = (1, 2, 'a', 'b')
c = MyContainer(t)
elems = [e for e in c]
assert elems == [1, 2, 'a', 'b']
assert c[1:-1] == t[1:-1] == (2, 'a')
Using the iterator protocol:
object.__iter__
object.__iter__(self)
This method is called when an iterator is required for a container. This method should return a new iterator object that can iterate over all the objects in the container. For mappings, it should iterate over the keys of the container.
Iterator objects also need to implement this method; they are required to return themselves. For more information on iterator objects, see Iterator Types.
Iterator Types
container.__iter__()
Return an iterator object. The object is required to support the iterator protocol described below.
The iterator objects themselves are required to support the following two methods, which together form the iterator protocol:
iterator.__iter__()
Return the iterator object itself. This is required to allow both containers and iterators to be used with the for and in statements.
iterator.__next__()
Return the next item from the container. If there are no further items, raise the StopIteration exception.
Once an iterator's __next__() method raises StopIteration, it must continue to do so on subsequent calls.
class MyContainer:
class Iter:
def __init__(self, container):
self.cont = container
self.pos = 0
self.len = len(container.elements)
def __iter__(self): return self
def __next__(self):
if self.pos == self.len: raise StopIteration
curElem = self.cont.elements[self.pos]
self.pos += 1
return curElem
def __init__(self, sequence):
self.elements = sequence # Just something to work with.
def __iter__(self):
return MyContainer.Iter(self)
t = (1, 2, 'a', 'b')
c = MyContainer(t)
elems = [e for e in c]
assert elems == [1, 2, 'a', 'b']
Using a generator:
Generator Types
Python's generators provide a convenient way to implement the iterator protocol. If a container object's iter() method is implemented as a generator, it will automatically return an iterator object (technically, a generator object) supplying the iter() and next() methods.
generator
A function which returns a generator iterator. It looks like a normal function except that it contains yield expressions for producing a series of values usable in a for-loop or that can be retrieved one at a time with the next() function.
Usually refers to a generator function, but may refer to a generator iterator in some contexts.
generator iterator
An object created by a generator function.
6.2.9. Yield expressions
Using a yield expression in a function's body causes that function to be a generator
class MyContainer:
def __init__(self, sequence):
self.elements = sequence # Just something to work with.
def __iter__(self):
for e in self.elements: yield e
t = (1, 2, 'a', 'b')
c = MyContainer(t)
elems = [e for e in c]
assert elems == [1, 2, 'a', 'b']

How to remove `duplicates' in list of instances

I have a list of instances of a certain class. This list contains `duplicates', in the sense that duplicates share the exact same attributes. I want to remove the duplicates from this list.
I can check whether two instances share the same attributes by using
class MyClass:
def __eq__(self, other) :
return self.__dict__ == other.__dict__
I could of course iterate through the whole list of instances and compare them element by element to remove duplicates, but I was wondering if there is a more pythonic way to do this, preferably using the in operator + list comprehension.

sets (no order)
A set cannot contain duplicate elements. list(set(content)) will deduplicate a list. This is not too inefficient and is probably one of the better ways to do it :P You will need to define a __hash__ function for your class though, which must be the same for equal elements and different for unequal elements for this to work. Note that the hash value must obey the aforementioned rule but otherwise it may change between runs without causing issues.
index function (stable order)
You could do lambda l: [l[index] for index in range(len(l)) if index == l.index(l[index])]. This only keeps elements that are the first in the list.
in operator (stable order)
def uniquify(content):
result = []
for element in content:
if element not in result:
result.append(element)
return result
This will keep appending elements to the output list unless they are already in the output list.

A little more on the set approach. You can safely implement a hash by delegating to a tuple's hash - just hash a tuple of all the attributes you want to look at. You will also need to define an __eq__ that behaves properly.
class MyClass:
def __init__(self, a, b, c):
self.a = a
self.b = b
self.c = c
def __eq__(self, other):
return (self.a, self.b, self.c) == (other.a, other.b, other.c)
def __hash__(self):
return hash((self.a, self.b, self.c))
def __repr__(self):
return "MyClass({!r}, {!r}, {!r})".format(self.a, self.b, self.c)
As you're doing so much tuple construction, you could just make your class iterable:
def __iter__(self):
return iter((self.a, self.b, self.c))
This enables you to call tuple on self instead of laboriously doing .a, .b, .c etc.
You can then do something like this:
def unordered_elim(l):
return list(set(l))
If you want to preserve ordering, you can use an OrderedDict instead:
from collections import OrderedDict
def ordered_elim(l):
return list(OrderedDict.fromkeys(l).keys())
This should be faster than using in or index, while still preserving ordering. You can test it something like this:
data = [MyClass("this", "is a", "duplicate"),
MyClass("first", "unique", "datum"),
MyClass("this", "is a", "duplicate"),
MyClass("second", "unique", "datum")]
print(unordered_elim(data))
print(ordered_elim(data))
With this output:
[MyClass('first', 'unique', 'datum'), MyClass('second', 'unique', 'datum'), MyClass('this', 'is a', 'duplicate')]
[MyClass('this', 'is a', 'duplicate'), MyClass('first', 'unique', 'datum'), MyClass('second', 'unique', 'datum')]
NB if any of your attributes aren't hashable, this won't work, and you'll either need to work around it (change a list to a tuple) or use a slow, n ^ 2 approach like in.

Change what the *splat and **splatty-splat operators do to my object

How do you override the result of unpacking syntax *obj and **obj?
For example, can you somehow create an object thing which behaves like this:
>>> [*thing]
['a', 'b', 'c']
>>> [x for x in thing]
['d', 'e', 'f']
>>> {**thing}
{'hello world': 'I am a potato!!'}
Note: the iteration via __iter__ ("for x in thing") returns different elements from the *splat unpack.
I had a look inoperator.mul and operator.pow, but those functions only concern usages with two operands, like a*b and a**b, and seem unrelated to splat operations.

* iterates over an object and uses its elements as arguments. ** iterates over an object's keys and uses __getitem__ (equivalent to bracket notation) to fetch key-value pairs. To customize *, simply make your object iterable, and to customize **, make your object a mapping:
class MyIterable(object):
def __iter__(self):
return iter([1, 2, 3])
class MyMapping(collections.Mapping):
def __iter__(self):
return iter('123')
def __getitem__(self, item):
return int(item)
def __len__(self):
return 3
If you want * and ** to do something besides what's described above, you can't. I don't have a documentation reference for that statement (since it's easier to find documentation for "you can do this" than "you can't do this"), but I have a source quote. The bytecode interpreter loop in PyEval_EvalFrameEx calls ext_do_call to implement function calls with * or ** arguments. ext_do_call contains the following code:
if (!PyDict_Check(kwdict)) {
PyObject *d;
d = PyDict_New();
if (d == NULL)
goto ext_call_fail;
if (PyDict_Update(d, kwdict) != 0) {
which, if the ** argument is not a dict, creates a dict and performs an ordinary update to initialize it from the keyword arguments (except that PyDict_Update won't accept a list of key-value pairs). Thus, you can't customize ** separately from implementing the mapping protocol.
Similarly, for * arguments, ext_do_call performs
if (!PyTuple_Check(stararg)) {
PyObject *t = NULL;
t = PySequence_Tuple(stararg);
which is equivalent to tuple(args). Thus, you can't customize * separately from ordinary iteration.
It'd be horribly confusing if f(*thing) and f(*iter(thing)) did different things. In any case, * and ** are part of the function call syntax, not separate operators, so customizing them (if possible) would be the callable's job, not the argument's. I suppose there could be use cases for allowing the callable to customize them, perhaps to pass dict subclasses like defaultdict through...

I did succeed in making an object that behaves how I described in my question, but I really had to cheat. So just posting this here for fun, really -
class Thing:
def __init__(self):
self.mode = 'abc'
def __iter__(self):
if self.mode == 'abc':
yield 'a'
yield 'b'
yield 'c'
self.mode = 'def'
else:
yield 'd'
yield 'e'
yield 'f'
self.mode = 'abc'
def __getitem__(self, item):
return 'I am a potato!!'
def keys(self):
return ['hello world']
The iterator protocol is satisfied by a generator object returned from __iter__ (note that a Thing() instance itself is not an iterator, though it is iterable). The mapping protocol is satisfied by the presence of keys() and __getitem__. Yet, in case it wasn't already obvious, you can't call *thing twice in a row and have it unpack a,b,c twice in a row - so it's not really overriding splat like it pretends to be doing.

Python: Passing mutable(?) object to method

I'm trying to implement a class with a method that calls another method with an object that's part of the class where the lowest method mutates the object. My implementation is a little more complicated, so I'll post just some dummy code so you can see what I'm talking about:
class test:
def __init__(self,list):
self.obj = list
def mult(self, x, n):
x = x*n
def numtimes(self, n):
self.mult(self.obj, n)
Now, if I create an object of this type and run the numtimes method, it won't update self.obj:
m = test([1,2,3,4])
m.numtimes(3)
m.obj #returns [1,2,3,4]
Whereas I'd like it to give me [1,2,3,4,1,2,3,4,1,2,3,4]
Basically, I need to pass self.obj to the mult method and have it mutate self.obj so that when I call m.obj, I'll get [1,2,3,4,1,2,3,4,1,2,3,4] instead of [1,2,3,4].
I feel like this is just a matter of understanding how python passes objects as arguments to methods (like it's making a copy of the object, and instead I need to use a pointer), but maybe not. I'm new to python and could really use some help here.
Thanks in advance!!

Allow me to take on the bigger subject of mutability.
Lists are mutable objects, and support both mutable operations, and immutable operations. That means, operations that change the list in-place, and operations that return a new list. Tuples, for contrast, only are only immutable.
So, to multiply a list, you can choose two methods:
a *= b
This is a mutable operation, that will change 'a' in-place.
a = a * b
This is an immutable operation. It will evaluate 'a*b', create a new list with the correct value, and assign 'a' to that new list.
Here, already, lies a solution to your problem. But, I suggest you read on a bit. When you pass around lists (and other objects) as parameters, you are only passing a new reference, or "pointer" to that same list. So running mutable operations on that list will also change the one that you passed. The result might be a very subtle bug, when you write:
>>> my_list = [1,2,3]
>>> t = test(my_list)
>>> t.numtimes(2)
>>> my_list
[1,2,3,1,2,3] # Not what you intended, probably!
So here's my final recommendation. You can choose to use mutable operations, that's fine. But then create a new copy from your arguments, as such:
def __init__(self,l):
self.obj = list(l)
OR use immutable operations, and reassign them to self:
def mult(self, x, n):
self.x = x*n
Or do both, there's no harm in being extra safe :)

The multiplication x * n creates a new instance and does not alter the existing list. See here:
a = [1]
print (id (a) )
a = a * 2
print (id (a) )
This should work:
class test:
def __init__(self,list):
self.obj = list
def mult(_, x, n):
x *= n
def numtimes(self, n):
self.mult(self.obj, n)

Python, lazy list

Is it possible to have a list be evaluated lazily in Python?
For example
a = 1
list = [a]
print list
#[1]
a = 2
print list
#[1]
If the list was set to evaluate lazily then the final line would be [2]

The concept of "lazy" evaluation normally comes with functional languages -- but in those you could not reassign two different values to the same identifier, so, not even there could your example be reproduced.
The point is not about laziness at all -- it is that using an identifier is guaranteed to be identical to getting a reference to the same value that identifier is referencing, and re-assigning an identifier, a bare name, to a different value, is guaranteed to make the identifier refer to a different value from them on. The reference to the first value (object) is not lost.
Consider a similar example where re-assignment to a bare name is not in play, but rather any other kind of mutation (for a mutable object, of course -- numbers and strings are immutable), including an assignment to something else than a bare name:
>>> a = [1]
>>> list = [a]
>>> print list
[[1]]
>>> a[:] = [2]
>>> print list
[[2]]
Since there is no a - ... that reassigns the bare name a, but rather an a[:] = ... that reassigns a's contents, it's trivially easy to make Python as "lazy" as you wish (and indeed it would take some effort to make it "eager"!-)... if laziness vs eagerness had anything to do with either of these cases (which it doesn't;-).
Just be aware of the perfectly simple semantics of "assigning to a bare name" (vs assigning to anything else, which can be variously tweaked and controlled by using your own types appropriately), and the optical illusion of "lazy vs eager" might hopefully vanish;-)

Came across this post when looking for a genuine lazy list implementation, but it sounded like a fun thing to try and work out.
The following implementation does basically what was originally asked for:
from collections import Sequence
class LazyClosureSequence(Sequence):
def __init__(self, get_items):
self._get_items = get_items
def __getitem__(self, i):
return self._get_items()[i]
def __len__(self):
return len(self._get_items())
def __repr__(self):
return repr(self._get_items())
You use it like this:
>>> a = 1
>>> l = LazyClosureSequence(lambda: [a])
>>> print l
[1]
>>> a = 2
>>> print l
[2]
This is obviously horrible.

Python is not really very lazy in general.
You can use generators to emulate lazy data structures (like infinite lists, et cetera), but as far as things like using normal list syntax, et cetera, you're not going to have laziness.

That is a read-only lazy list where it only needs a pre-defined length and a cache-update function:
import copy
import operations
from collections.abc import Sequence
from functools import partialmethod
from typing import Dict, Union
def _cmp_list(a: list, b: list, op, if_eq: bool, if_long_a: bool) -> bool:
"""utility to implement gt|ge|lt|le class operators"""
if a is b:
return if_eq
for ia, ib in zip(a, b):
if ia == ib:
continue
return op(ia, ib)
la, lb = len(a), len(b)
if la == lb:
return if_eq
if la > lb:
return if_long_a
return not if_long_a
class LazyListView(Sequence):
def __init__(self, length):
self._range = range(length)
self._cache: Dict[int, Value] = {}
def __len__(self) -> int:
return len(self._range)
def __getitem__(self, ix: Union[int, slice]) -> Value:
length = len(self)
if isinstance(ix, slice):
clone = copy.copy(self)
clone._range = self._range[slice(*ix.indices(length))] # slicing
return clone
else:
if ix < 0:
ix += len(self) # negative indices count from the end
if not (0 <= ix < length):
raise IndexError(f"list index {ix} out of range [0, {length})")
if ix not in self._cache:
... # update cache
return self._cache[ix]
def __iter__(self) -> dict:
for i, _row_ix in enumerate(self._range):
yield self[i]
__eq__ = _eq_list
__gt__ = partialmethod(_cmp_list, op=operator.gt, if_eq=False, if_long_a=True)
__ge__ = partialmethod(_cmp_list, op=operator.ge, if_eq=True, if_long_a=True)
__le__ = partialmethod(_cmp_list, op=operator.le, if_eq=True, if_long_a=False)
__lt__ = partialmethod(_cmp_list, op=operator.lt, if_eq=False, if_long_a=False)
def __add__(self, other):
"""BREAKS laziness and returns a plain-list"""
return list(self) + other
def __mul__(self, factor):
"""BREAKS laziness and returns a plain-list"""
return list(self) * factor
__radd__ = __add__
__rmul__ = __mul__
Note that this class is discussed also in this SO.