This is my code
class A:
pass
def f():
yield A()
def g():
it = f()
next(it).a = next(it, None)
g()
that produces the StopIteration error, caused by next(it).a = next(it, None). Why?
The documentation says that next function does not raise the StopIteration if the default value is provided, and I expected it to get me the first item from the generator (the A instance) and set the a attribute to None.
Because f only yields a single value, you can only call next on it once.
The right hand side of your expression (next(it, None)) is evaluated before the left hand side, and thus exhausts the generator.
Calling next(it).a on the left hand side will then raise StopIteration.
Your f() generator function yields just one value. After that it is exhausted and raises StopIteration.
>>> class A:
... pass
...
>>> def f():
... yield A()
...
>>> generator = f()
>>> generator
<generator object f at 0x10be771f8>
>>> next(generator)
<__main__.A object at 0x10be76f60>
>>> next(generator)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
That's because there is no loop in f() to yield more than once, a generator function does not, on its own, loop, just because you can use it in a loop.
Note that for an assignment, Python executes the right-hand-side expression first, before figuring out what to assign it to. So the next(it, None) is called first, the next(it).a for the assignment is called second.
The body of the f() function is executed just like any other Python function, with the addition of pausing. next() on the generator un-pauses the code, and the code then runs until the next yield statement is executed. That statement then pauses the generator again. If the function instead ends (returns), StopIteration is raised.
In your f() generator that means:
when you call f() a new generator object is created. The function body is paused.
you call next() on it the first time. The code starts running, creates an instance of A() and yields that instance. The function is paused again.
you call next() on it a second time. The code starts running, reaches the end of the function, and returns. StopIteration is raised.
If you add a loop to f(), or simply add a second yield line, your code works:
def f():
yield A()
yield A()
or
def f():
while True:
yield A()
Related
As indicated in the documentation, the default value is returned if the iterator is exhausted. However, in the following program, the g(x) function is not exhausted, and I hope that the error from f(x) would not be processed in the next function.
def f(x) :
if 0 : # to make sure that nothing is generated
yield 10
def g(x) :
yield next(f(x))
# list(g(3))
next(g(3), None)
What I expect:
Traceback (most recent call last):
File "a.py", line 9, in <module>
next(g(3), None)
File "a.py", line 6, in g
yield next(f(x))
StopIteration
What I encountered is that the program was running successfully.
Can I use an alternating approach to achieve the goal? Or can it be fixed in Python?
Edit: The program mentioned above may be modified like this in order to prevent ambiguation.
def f(x) :
if 0 : # to make sure that nothing is generated
yield 10
def g(x) :
f(x).__next__() # g(x) is not exhausted at this time
yield 'something meaningful'
# I hope that the next function will only catch this line
# list(g(3))
next(g(3), None)
next with a default parameter catches the StopIteration no matter the source.
The behavior you're seeing is expected, and maybe better understood using this code:
def justraise():
yield next(iter([])) # raises StopIteration
next(justraise(), None) # None
next(justraise()) # raises StopIteration
Moving to your code - even though the inner use is of next without a default argument, the StopIteration it raised is caught in the outer next with the default argument.
If you have a meaningful exception to raise, you should raise a meaningful exception and not StopIteration which indicates the iteration ended (and not erroneously) - which is what next relies on.
g(x) is an iterator that always yields f(x), which yields Nothing, and raises a StopIteration (in f)
You can check that next(f(some_value)) does throw an exception when called itself.
As will
def g(x):
return next(f(x))
But, you've added the default None, so that g(x) will run, but simply return back None since the iterator is exhausted.
If you remove the None, then you see
In [5]: next(g(3))
---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
<ipython-input-14-05eb86fce40b> in <module>()
----> 1 next(g(3))
<ipython-input-13-a4323284f776> in g(x)
1 def g(x) :
----> 2 yield next(f(x))
3
StopIteration:
I have the following experimental code whose function is similar to the zip built-in. What it tries to do should have been simple and clear, trying to return the zipped tuples one at a time until an IndexError occurs when we stop the generator.
def my_zip(*args):
i = 0
while True:
try:
yield (arg[i] for arg in args)
except IndexError:
raise StopIteration
i += 1
However, when I tried to execute the following code, the IndexError was not caught but instead thrown by the generator:
gen = my_zip([1,2], ['a','b'])
print(list(next(gen)))
print(list(next(gen)))
print(list(next(gen)))
IndexError Traceback (most recent call last)
I:\Software\WinPython-32bit-3.4.2.4\python-3.4.2\my\temp2.py in <module>()
12 print(list(next(gen)))
13 print(list(next(gen)))
---> 14 print(list(next(gen)))
I:\Software\WinPython-32bit-3.4.2.4\python-3.4.2\my\temp2.py in <genexpr>(.0)
3 while True:
4 try:
----> 5 yield (arg[i] for arg in args)
6 except IndexError:
7 raise StopIteration
IndexError: list index out of range
Why is this happening?
Edit:
Thanks #thefourtheye for providing a nice explanation for what's happening above. Now another problem occurs when I execute:
list(my_zip([1,2], ['a','b']))
This line never returns and seems to hang the machine. What's happening now?
The yield yields a generator object everytime and when the generators were created there was no problem at all. That is why try...except in my_zip is not catching anything. The third time when you executed it,
list(arg[2] for arg in args)
this is how it got reduced to (over simplified for our understanding) and now, observe carefully, list is iterating the generator, not the actual my_zip generator. Now, list calls next on the generator object and arg[2] is evaluated, only to find that 2 is not a valid index for arg (which is [1, 2] in this case), so IndexError is raised, and list fails to handle it (it has no reason to handle that anyway) and so it fails.
As per the edit,
list(my_zip([1,2], ['a','b']))
will be evaluated like this. First, my_zip will be called and that will give you a generator object. Then iterate it with list. It calls next on it, and it gets another generator object list(arg[0] for arg in args). Since there is no exception or return encountered, it will call next, to get another generator object list(arg[1] for arg in args) and it keeps on iterating. Remember, the yielded generators are never iterated, so we ll never get the IndexError. That is why the code runs infinitely.
You can confirm this like this,
from itertools import islice
from pprint import pprint
pprint(list(islice(my_zip([1, 2], ["a", 'b']), 10)))
and you will get
[<generator object <genexpr> at 0x7f4d0a709678>,
<generator object <genexpr> at 0x7f4d0a7096c0>,
<generator object <genexpr> at 0x7f4d0a7099d8>,
<generator object <genexpr> at 0x7f4d0a709990>,
<generator object <genexpr> at 0x7f4d0a7095a0>,
<generator object <genexpr> at 0x7f4d0a709510>,
<generator object <genexpr> at 0x7f4d0a7095e8>,
<generator object <genexpr> at 0x7f4d0a71c708>,
<generator object <genexpr> at 0x7f4d0a71c750>,
<generator object <genexpr> at 0x7f4d0a71c798>]
So the code tries to build an infinite list of generator objects.
def my_zip(*args):
i = 0
while True:
try:
yield (arg[i] for arg in args)
except IndexError:
raise StopIteration
i += 1
IndexError is not caught, because (arg[i] for arg in args) is a generator which is not executed immediately, but when you start iterating over it. And you iterate over it in another scope, when you call list((arg[i] for arg in args)):
# get the generator which yields another generator on each iteration
gen = my_zip([1,2], ['a','b'])
# get the second generator `(arg[i] for arg in args)` from the first one
# then iterate over it: list((arg[i] for arg in args))
print(list(next(gen)))
On the first list(next(gen)) i equals 0.
On the second list(next(gen)) i equals 1.
On the third list(next(gen)) i equals 2. And here you get IndexError -- in the outer scope. The line is treated as list(arg[2] for arg in ([1,2], ['a','b']))
Sorry, I'm not able to offer a coherent explanation regarding the failure to catch the exception, however, there's an easy way around it; use a for loop over the length of the shortest sequence:
def my_zip(*args):
for i in range(min(len(arg) for arg in args)):
yield (arg[i] for arg in args)
>>> gen = my_zip([1,2], ["a",'b','c'])
>>> print(list(next(gen)))
[1, 'a']
>>> print(list(next(gen)))
[2, 'b']
>>> print(list(next(gen)))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
Try replacing yield (arg[i] for ...) with the following.
for arg in args:
yield arg[i]
But in case of numbers that causes an exception as 1[1] makes no sense. I suggest replacing arg[i] just with arg.
I have a function called x that produces a generator like this:
a = 5
def x():
global a
if a == 3:
raise Exception("Stop")
a = a - 1
yield a
Then in the python shell I call that function like this:
>>> print x().next()
>>> 4
>>> print x().next()
>>> 3
>>> print x().next()
>>> <python-input-112-f3c02bba26c8> in x()
2 global a
3 if a == 3:
----> 4 raise Exception
5 a = a - 1
6 yield a
Exception:
However, when I call that function and assign it to a variable, it behaves differently:
>>> a = 5
>>> b = x()
>>> print b.next()
>>> 4
>>> print b.next()
>>> ----> 1 b.next()
StopIteration:
How is that even possible? Shouldn't it print out 3 and raise StopIteration in the next iteration?
PS: I know that when I first call the function, the body does not run, just produces a generator. The point I didn't understand is that what changes if I call and assign it to a variable?
In your first example, you were creating a new generator each time:
x().next()
This starts the generator from the top, so the first statement. When a == 3, the exception is raised, otherwise the generator just yields and pauses.
When you assigned your generator later on, the global a started at 5, the code then continued from where it left of until it ends or comes across another yield statement, then ended. When a generator function ends, it raises StopIteration.
Let's break this down into steps:
a = 5.
You create new generator and call .next() on it. The following code is executed:
global a
if a == 3: # False
raise Exception("Stop")
a = a - 1 # a is now 4
yield a
The generator is paused on the last line, and 4 is yielded.
You create a new generator and call .next() on it. a is 4 at the start:
global a
if a == 3: # False
raise Exception("Stop")
a = a - 1 # a is now 3
yield a
The generator is paused on the last line, and 3 is yielded.
You create a new generator and call .next() on it. a is 3 at the start:
global a
if a == 3: # True
raise Exception("Stop")
An exception is raised.
You set a = 5 again.
You create a new generator, store a reference in b and call .next() on it. The following code is executed:
global a
if a == 3: # False
raise Exception("Stop")
a = a - 1 # a is now 4
yield a
The generator is paused on the last line, and 4 is yielded.
You call .next() again on the same, existing generator referenced by b. The code resumes at the paused point.
The function has no more code at that point, and returns. StopIteration is raised.
If you were to use a loop instead, you'd see the difference better:
>>> def looping(stop):
... for i in looping(stop):
... yield i
...
>>> looping(3).next()
0
>>> looping(3).next()
0
Note how each time I create a new generator, the loop starts from the beginning. Store a reference however, and you'll notice it continue instead:
>>> stored = looping(3)
>>> stored.next()
0
>>> stored.next()
1
>>> stored.next()
2
>>> stored.next()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
During the loop, each time the yield expression is executed, the code is paused; calling .next() continues the function where it left of the previous time.
The StopIteration exception is entirely normal; it is how generators communicate that they are done. A for loop looks for this exception to end the loop:
>>> for i in looping(3):
... print i
...
0
1
2
You've not quite got how yield works. I think this example might help:
>>> def a():
... for x in range(5):
... yield x
...
>>> a()
<generator object a at 0xb7f0a9b4>
>>> list(a())
[0, 1, 2, 3, 4]
You normally want to use yield inside a loop, and it has the very distinct behavior of returning a value, then later resuming the loop.
If your example, x always returns a generator that will produce one item only. In your first example, you are calling x multiple times, so you get multiple results. In your second example, where you assign it to a variable, you are only calling it once, so you only get one result.
Also, you should not usually use a global variable in the way you have done.
Paul
There is this code:
def f():
return 3
return (i for i in range(10))
x = f()
print(type(x)) # int
def g():
return 3
for i in range(10):
yield i
y = g()
print(type(y)) # generator
Why f returns int when there is return generator statement? I guess that yield and generator expression both returns generators (at least when the statement return 3 is removed) but are there some other rules of function compilation when there is once generator expression returned and second time when there is yield keyword inside?
This was tested in Python 3.3
As soon as you use a yield statement in a function body, it becomes a generator. Calling a generator function just returns that generator object. It is no longer a normal function; the generator object has taken over control instead.
From the yield expression documentation:
Using a yield expression in a function definition is sufficient to cause that definition to create a generator function instead of a normal function.
When a generator function is called, it returns an iterator known as a generator. That generator then controls the execution of a generator function. The execution starts when one of the generator’s methods is called.
In a regular function, calling that function immediately switches control to that function body, and you are simply testing the result of the function, set by it's return statement. In a generator function, return still signals the end of the generator function, but that results in a StopIteration exception being raised instead. But until you call one of the 4 generator methods (.__next__(), .send(), .throw() or .close()), the generator function body is not executed at all.
For your specific function f(), you have a regular function, that contains a generator. The function itself is nothing special, other that that it exits early when return 3 is executed. The generator expression on the next line stands on its own, it does not influence the function in which it is defined. You can define it without the function:
>>> (i for i in range(10))
<generator object <genexpr> at 0x101472730>
Using a generator expression produces a generator object, just like using yield in a function, then calling that function produces a generator object. So you could have called g() in f() with the same result as using the generator expression:
def f():
return 3
return g()
g() is still a generator function, but using it in f() does not make f() a generator function too. Only yield can do that.
def f():
return 3
return (i for i in range(10))
is the same as
def f():
return 3
The second return statement never gets executed, just by having a generator expression within f does not make it a generator.
def f():
return 3
#unreachable code below
return (i for i in range(10))
I believe what you meant is:
def f():
yield 3
yield from (i for i in range(10))
Returning a generator doesn't make f a generator function. A generator is just an object, and generator objects can be returned by any function. If you want f to be a generator function, you have to use yield inside the function, as you did with g.
In Python 2 there was an error when return was together with yield in a function definition. But for this code in Python 3.3:
def f():
return 3
yield 2
x = f()
print(x.__next__())
there is no error that return is used in function with yield. However when the function __next__ is called then there is thrown exception StopIteration. Why there is not just returned value 3? Is this return somehow ignored?
This is a new feature in Python 3.3. Much like return in a generator has long been equivalent to raise StopIteration(), return <something> in a generator is now equivalent to raise StopIteration(<something>). For that reason, the exception you're seeing should be printed as StopIteration: 3, and the value is accessible through the attribute value on the exception object. If the generator is delegated to using the (also new) yield from syntax, it is the result. See PEP 380 for details.
def f():
return 1
yield 2
def g():
x = yield from f()
print(x)
# g is still a generator so we need to iterate to run it:
for _ in g():
pass
This prints 1, but not 2.
The return value is not ignored, but generators only yield values, a return just ends the generator, in this case early. Advancing the generator never reaches the yield statement in that case.
Whenever a iterator reaches the 'end' of the values to yield, a StopIteration must be raised. Generators are no exception. As of Python 3.3 however, any return expression becomes the value of the exception:
>>> def gen():
... return 3
... yield 2
...
>>> try:
... next(gen())
... except StopIteration as ex:
... e = ex
...
>>> e
StopIteration(3,)
>>> e.value
3
Use the next() function to advance iterators, instead of calling .__next__() directly:
print(next(x))