Develop Reference

How can I modify my class to input stuff into my lists and find the mean of the class?

So I have created a class Calculate where I will define two functions plus(self,x) and avg_over_totalnum(self) that will modify a list __init__(self, items).
plus(self, x) will add either x or the members of x to the list self.items.
Here x is one of the following: (1) a number, (2) a list, or (3) a tuple. If x is a number, it is added to the list directly. If x is a list or a tuple, its members are individually added to the list.
For instance if self.items == [5]
if x == 45, plus(self, x) will return [5,45]
if x == [4,5], the output will be [5,4,5]
avg_over_totalnum(self) will compute and return the mean of the data maintained in the
collection
Here is my attempt at the code so far, which doesn't work at all...
class Calculate:
def __init__(self, items):
self.items = list(items)
def plus(self,x):
if x in 123456789:
return self.items + x
else:
return self.items.append(x)
def avg_over_totalnum(self):
return (sum(self.items))/(len(self.items))
What changes should I make in my code??

Your plus method is a bit weird. It doesn't modify self.items in-place, so self.items don't get updated with x. Also, your if-statement is checking if x is in an integer, which doesn't make sense. Change plus with the function below:
def plus(self,x):
if isinstance(x,(list,tuple)):
self.items.extend(list(x))
else:
self.items.append(x)
return self.items
Also to avoid ZeroDivisionError:
def avg_over_totalnum(self):
return (sum(self.items))/(len(self.items)) if self.items else 0
Then it works fine.
c = Calculate([5])
print(c.plus(4)) # [5, 4]
print(c.plus([3,2])) # [5, 4, 3, 2]
print(c.avg_over_totalnum()) # 3.5

Impact of removing a list item on reversed() in python

As far as I know, reversed() function gives an iterator and works just like iter() but will give the items in reverse order. However I faced a strange behavior from the object that gets back from reversed() function.
By looking at:
lst = ['a', 'b', 'c', 'd']
iter_lst = iter(lst)
lst.remove('c')
print(list(iter_lst))
output : ['a', 'b', 'd']
It's just as expected. but:
lst = ['a', 'b', 'c', 'd']
rev_iter_lst = reversed(lst)
lst.remove('c')
print(list(rev_iter_lst))
output : []
Shouldn't it be : ['d', 'b', 'a'] ?
Is it something in implementation of __reversed__() method in list object or __next__() method in the iterator object that prevents this ? I mean if something changes in original list it won't produce reverse sequence maybe...
Update: I've posted an answer which is a possible fix to it here, I've tested that but I'm unaware if there is a situation that this implementation would give unexcepted result.

according to list.__reversed__ source code, the iterator remembers the last index address and returns the iterator which remembers the last index address.
now when you remove an item it will shift all the indexes and makes the last address point to nowhere and it will return an empty list because there is nothing to iterate over.
let me describe more:
consider following list: lst = ['a','b','c']
also let's assume lst[0] is at the 100 and each character is one byte so, the character 'c' is in the 102.
when you create a revered iterator it will remember 102 as the start point.
in the next step, we omit 'b' now the character 'c' is in address 101.
and when you ask the iterator to iterate, it will start to look at position 102. what it will found there? literally nothing and obviously it will return an empty list.
I hope this can be helpful :)
EDIT: the word address is not correct. I must use index instead...

So after the discussion with #kkasra12 I ended up implementing Python's pseudo-list object (without doing all the necessary checkings) to mimic it's behavior and I just focused on reverse() operation. Here is my class:
class MyList:
def __init__(self, n):
self.length = n
self._seq = list(range(n))
#property
def seq(self):
return self._seq
def __len__(self):
return self.length
def __getitem__(self, item):
return self._seq[item]
def __setitem__(self, idx, value):
self._seq[idx] = value
def __reversed__(self):
return ReverseIterator(self)
def __str__(self):
return str(self._seq)
def append(self, v):
self._seq.append(v)
self.length += 1
def remove(self, v):
self._seq.remove(v)
self.length -= 1
And my ReverseIterator:
class ReverseIterator:
def __init__(self, org):
self.org = org
self._index = org.length
def __iter__(self):
return self
def __next__(self):
if 0 < self._index:
try:
item = self.org.seq[self._index - 1]
self._index -= 1
return item
except IndexError:
raise StopIteration()
else:
raise StopIteration()
The result:
obj = MyList(6)
iter_obj = iter(obj)
obj.remove(2)
print(list(iter_obj))
print('-----------------------')
obj = MyList(6)
rev_iter_obj = reversed(obj)
obj.remove(2)
print(list(rev_iter_obj))
output :
[0, 1, 3, 4, 5]
-----------------------
[]
By commenting those remove statements above, we can see that it works like original list object.
Then I created new SmartReverseIterator iterator which can handle if an item is removed from the original object and can generate the values on the fly just like how iter() wokred on the list in OP.
The only thing should be considered is if an item is removed(self._index would be smaller than original object's length), the self._index should be reset.
class SmartReverseIterator:
def __init__(self, org):
self.org = org
self._index = org.length
def __iter__(self):
return self
def __next__(self):
if 0 < self._index:
try:
item = self.org.seq[self._index - 1]
return item
except IndexError:
self._index = self.org.length
item = self.org.seq[self._index - 1]
return item
finally:
self._index -= 1
else:
raise StopIteration()
By changing the __reversed__ method on MyList to return this new iterator, the result is going to be:
obj = MyList(6)
iter_obj = iter(obj)
obj.remove(2)
print(list(iter_obj))
print('-----------------------')
obj = MyList(6)
rev_iter_obj = reversed(obj)
obj.remove(2)
print(list(rev_iter_obj))
Output:
[0, 1, 3, 4, 5]
-----------------------
[5, 4, 3, 1, 0]
I wanted to know if there is any downside to this, or in other words why python decided not to implement __reversed__ method on list objects like this to result exactly how iter() can generate values if an item is removed.
In which situation we would see issues ?

The list call will iterate the reverse iterator, whose index < PyList_GET_SIZE(seq) check here will fail because you shrunk seq in the meantime, and thus won't yield a value but stop:
listreviter_next(listreviterobject *it)
{
(some checks)
index = it->it_index;
if (index>=0 && index < PyList_GET_SIZE(seq)) {
(decrease the index and return the element)
}
(stop the iteration)
}

How can I have multiple iterators over a single python iterable at the same time?

I would like to compare all elements in my iterable object combinatorically with each other. The following reproducible example just mimics the functionality of a plain list, but demonstrates my problem. In this example with a list of ["A","B","C","D"], I would like to get the following 16 lines of output, every combination of each item with each other. A list of 100 items should generate 100*100=10,000 lines.
A A True
A B False
A C False
... 10 more lines ...
D B False
D C False
D D True
The following code seemed like it should do the job.
class C():
def __init__(self):
self.stuff = ["A","B","C","D"]
def __iter__(self):
self.idx = 0
return self
def __next__(self):
self.idx += 1
if self.idx > len(self.stuff):
raise StopIteration
else:
return self.stuff[self.idx - 1]
thing = C()
for x in thing:
for y in thing:
print(x, y, x==y)
But after finishing the y-loop, the x-loop seems done, too, even though it's only used the first item in the iterable.
A A True
A B False
A C False
A D False
After much searching, I eventually tried the following code, hoping that itertools.tee would allow me two independent iterators over the same data:
import itertools
thing = C()
thing_one, thing_two = itertools.tee(thing)
for x in thing_one:
for y in thing_two:
print(x, y, x==y)
But I got the same output as before.
The real-world object this represents is a model of a directory and file structure with varying numbers of files and subdirectories, at varying depths into the tree. It has nested links to thousands of members and iterates correctly over them once, just like this example. But it also does expensive processing within its many internal objects on-the-fly as needed for comparisons, which would end up doubling the workload if I had to make a complete copy of it prior to iterating. I would really like to use multiple iterators, pointing into a single object with all the data, if possible.
Edit on answers: The critical flaw in the question code, pointed out in all answers, is the single internal self.idx variable being unable to handle multiple callers independently. The accepted answer is the best for my real class (oversimplified in this reproducible example), another answer presents a simple, elegant solution for simpler data structures like the list presented here.

It's actually impossible to make a container class that is it's own iterator. The container shouldn't know about the state of the iterator and the iterator doesn't need to know the contents of the container, it just needs to know which object is the corresponding container and "where" it is. If you mix iterator and container different iterators will share state with each other (in your case the self.idx) which will not give the correct results (they read and modify the same variable).
That's the reason why all built-in types have a seperate iterator class (and even some have an reverse-iterator class):
>>> l = [1, 2, 3]
>>> iter(l)
<list_iterator at 0x15e360c86d8>
>>> reversed(l)
<list_reverseiterator at 0x15e360a5940>
>>> t = (1, 2, 3)
>>> iter(t)
<tuple_iterator at 0x15e363fb320>
>>> s = '123'
>>> iter(s)
<str_iterator at 0x15e363fb438>
So, basically you could just return iter(self.stuff) in __iter__ and drop the __next__ altogether because list_iterator knows how to iterate over the list:
class C:
def __init__(self):
self.stuff = ["A","B","C","D"]
def __iter__(self):
return iter(self.stuff)
thing = C()
for x in thing:
for y in thing:
print(x, y, x==y)
prints 16 lines, like expected.
If your goal is to make your own iterator class, you need two classes (or 3 if you want to implement the reversed-iterator yourself).
class C:
def __init__(self):
self.stuff = ["A","B","C","D"]
def __iter__(self):
return C_iterator(self)
def __reversed__(self):
return C_reversed_iterator(self)
class C_iterator:
def __init__(self, parent):
self.idx = 0
self.parent = parent
def __iter__(self):
return self
def __next__(self):
self.idx += 1
if self.idx > len(self.parent.stuff):
raise StopIteration
else:
return self.parent.stuff[self.idx - 1]
thing = C()
for x in thing:
for y in thing:
print(x, y, x==y)
works as well.
For completeness, here's one possible implementation of the reversed-iterator:
class C_reversed_iterator:
def __init__(self, parent):
self.parent = parent
self.idx = len(parent.stuff) + 1
def __iter__(self):
return self
def __next__(self):
self.idx -= 1
if self.idx <= 0:
raise StopIteration
else:
return self.parent.stuff[self.idx - 1]
thing = C()
for x in reversed(thing):
for y in reversed(thing):
print(x, y, x==y)
Instead of defining your own iterators you could use generators. One way was already shown in the other answer:
class C:
def __init__(self):
self.stuff = ["A","B","C","D"]
def __iter__(self):
yield from self.stuff
def __reversed__(self):
yield from self.stuff[::-1]
or explicitly delegate to a generator function (that's actually equivalent to the above but maybe more clear that it's a new object that is produced):
def C_iterator(obj):
for item in obj.stuff:
yield item
def C_reverse_iterator(obj):
for item in obj.stuff[::-1]:
yield item
class C:
def __init__(self):
self.stuff = ["A","B","C","D"]
def __iter__(self):
return C_iterator(self)
def __reversed__(self):
return C_reverse_iterator(self)
Note: You don't have to implement the __reversed__ iterator. That was just meant as additional "feature" of the answer.

Your __iter__ is completely broken. Instead of actually making a fresh iterator on every call, it just resets some state on self and returns self. That means you can't actually have more than one iterator at a time over your object, and any call to __iter__ while another loop over the object is active will interfere with the existing loop.
You need to actually make a new object. The simplest way to do that is to use yield syntax to write a generator function. The generator function will automatically return a new iterator object every time:
class C(object):
def __init__(self):
self.stuff = ['A', 'B', 'C', 'D']
def __iter__(self):
for thing in self.stuff:
yield thing

Recursively calling an object method that returns an iterator of itself

I'm currently writing a project that requires third party code that uses a method that returns an iterator of itself, an example of how this would look in my code:
def generate():
for x in obj.children():
for y in x.children():
for z in y.children():
yield z.thing
Currently this simply clutters my code, and becomes hard to read after 3 levels. Ideally I'd get it to do something like this:
x = recursive(obj, method="children", repeat=3).thing
Is there a built in way to do this in Python?

Starting from python3.3, you can use the yield from syntax to yield an entire generator expression.
So, you can modify your function a bit, to take a couple of parameters:
def generate(obj, n):
if n == 1:
for x in obj.children():
yield x.thing
else:
for x in obj.children():
yield from generate(x, n - 1)
The yield from expression will yield the entire generator expression of the recursive call.
Call your function like this:
x = generate(obj, 3)
Note that this returns you a generator of x.things.
Based on your particular requirement, here's a more generic version using getattr that works with arbitrary attributes.
def generate(obj, iterable_attr, attr_to_yield, n):
if n == 1:
for x in getattr(obj, iterable_attr):
yield getattr(x, attr_to_yield)
else:
for x in getattr(obj, iterable_attr):
yield from generate(x, iterable_attr, attr_to_yield, n - 1)
And now, call your function as:
x = generate(obj, 'children', 'thing', 3)

If using Python 2.7 you need to keep your own stack of iterables and do the looping:
from operator import methodcaller
def recursive(obj, iterater, yielder, depth):
iterate = methodcaller(iterater)
xs = [iterate(obj)]
while xs:
try:
x = xs[-1].next()
if len(xs) != depth:
xs.append(iterate(x))
else:
yield getattr(x, yielder)
except StopIteration:
xs.pop()
This a specialized case of a more general recursive ichain from iterable function:
def recursive_ichain(iterable_tree):
xs = [iter(iterable_tree)]
while [xs]:
try:
x = xs[-1].next()
if isinstance(x, collections.Iterable):
xs.append(iter(x))
else:
yield x
except StopIteration:
xs.pop()
And some test objects:
class Thing(object):
def __init__(self, thing):
self.thing = thing
class Parent(object):
def __init__(self, *kids):
self.kids = kids
def children(self):
return iter(self.kids)
test_obj = Parent(
Parent(
Parent(Thing('one'), Thing('two'), Thing('three')),
Parent(Thing('four')),
Parent(Thing('five'), Thing('six')),
),
Parent(
Parent(Thing('seven'), Thing('eight')),
Parent(),
Parent(Thing('nine'), Thing('ten')),
)
)
And testing it:
>>>for t in recursive(test_obj, 'children', 'thing', 3):
>>> print t
one
two
three
four
five
six
seven
eight
nine
ten
Personnaly I'd be inclined to change the yield getattr(x, yielder) to yield x to access the leaf objects themselves and explicitly access the thing. i.e.
for leaf in recursive(test_obj, 'children', 3):
print leaf.thing

The yield from example above is good, but I seriously doubt the level/depth param is needed. A simpler / more generic solution that works for any tree:
class Node(object):
def __init__(self, thing, children=None):
self.thing = thing
self._children = children
def children(self):
return self._children if self._children else []
def generate(node):
if node.thing:
yield node.thing
for child in node.children():
yield from generate(child)
node = Node('mr.', [Node('derek', [Node('curtis')]), Node('anderson')])
print(list(generate(node)))
Returns:
$ python3 test.py
['mr.', 'derek', 'curtis', 'anderson']
Note this will return the current node's thing before any of its children's. (IE it expresses itself on the way down the walk.) If you'd prefer it to express itself on the way back up the walk, swap the if and the for statements. (DFS vs BFS) But likely doesn't matter in your case (where I suspect a node has either a thing or children, never both).

How can I sum (make totals) on multiple object attributes with one loop pass?

I want to sum multiple attributes at a time in a single loop:
class Some(object):
def __init__(self, acounter, bcounter):
self.acounter = acounter
self.bcounter = bcounter
someList = [Some(x, x) for x in range(10)]
Can I do something simpler and faster than it?
atotal = sum([x.acounter for x in someList])
btotal = sum([x.bcounter for x in someList])

First off - sum doesn't need a list - you can use a generator expression instead:
atotal = sum(x.acounter for x in someList)
You could write a helper function to do the search of the list once but look up each attribute in turn per item, eg:
def multisum(iterable, *attributes, **kwargs):
sums = dict.fromkeys(attributes, kwargs.get('start', 0))
for it in iterable:
for attr in attributes:
sums[attr] += getattr(it, attr)
return sums
counts = multisum(someList, 'acounter', 'bcounter')
# {'bcounter': 45, 'acounter': 45}

Another alternative (which may not be faster) is to overload the addition operator for your class:
class Some(object):
def __init__(self, acounter, bcounter):
self.acounter = acounter
self.bcounter = bcounter
def __add__(self, other):
if isinstance(other, self.__class__):
return Some(self.acounter+other.acounter, self.bcounter+other.bcounter)
elif isinstance(other, int):
return self
else:
raise TypeError("useful message")
__radd__ = __add__
somelist = [Some(x, x) for x in range(10)]
combined = sum(somelist)
print combined.acounter
print combined.bcounter
This way sum returns a Some object.

I doubt that this is really faster, but you can do it like thus:
First define padd (for "pair add") via:
def padd(p1,p2):
return (p1[0]+p2[0],p1[1]+p2[1])
For example, padd((1,4), (5,10)) = (6,14)
Then use reduce:
atotal, btotal = reduce(padd, ((x.acounter,x.bcounter) for x in someList))
in Python 3 you need to import reduce from functools but IIRC it can be used directly in Python 2.
On edit: For more than 2 attributes you can replace padd by vadd ("vector add") which can handle tuples of arbitrary dimensions:
def vadd(v1,v2):
return tuple(x+y for x,y in zip(v1,v2))
For just 2 attributes it is probably more efficient to hard-wire in the dimension since there is less function-call overhead.

Use this line to accumulate all of the attributes that you wish to sum.
>>> A = ((s.acounter,s.bcounter) for s in someList)
Then use this trick from https://stackoverflow.com/a/19343/47078 to make separate lists of each attribute by themselves.
>>> [sum(x) for x in zip(*A)]
[45, 45]
You can obviously combine the lines, but I thought breaking it apart would be easier to follow here.
And based on this answer, you can make it much more readable by defining an unzip(iterable) method.
def unzip(iterable):
return zip(*iterable)
[sum(x) for x in unzip((s.acounter,s.bcounter) for s in someList)]