As indicated in the documentation, the default value is returned if the iterator is exhausted. However, in the following program, the g(x) function is not exhausted, and I hope that the error from f(x) would not be processed in the next function.
def f(x) :
if 0 : # to make sure that nothing is generated
yield 10
def g(x) :
yield next(f(x))
# list(g(3))
next(g(3), None)
What I expect:
Traceback (most recent call last):
File "a.py", line 9, in <module>
next(g(3), None)
File "a.py", line 6, in g
yield next(f(x))
StopIteration
What I encountered is that the program was running successfully.
Can I use an alternating approach to achieve the goal? Or can it be fixed in Python?
Edit: The program mentioned above may be modified like this in order to prevent ambiguation.
def f(x) :
if 0 : # to make sure that nothing is generated
yield 10
def g(x) :
f(x).__next__() # g(x) is not exhausted at this time
yield 'something meaningful'
# I hope that the next function will only catch this line
# list(g(3))
next(g(3), None)
next with a default parameter catches the StopIteration no matter the source.
The behavior you're seeing is expected, and maybe better understood using this code:
def justraise():
yield next(iter([])) # raises StopIteration
next(justraise(), None) # None
next(justraise()) # raises StopIteration
Moving to your code - even though the inner use is of next without a default argument, the StopIteration it raised is caught in the outer next with the default argument.
If you have a meaningful exception to raise, you should raise a meaningful exception and not StopIteration which indicates the iteration ended (and not erroneously) - which is what next relies on.
g(x) is an iterator that always yields f(x), which yields Nothing, and raises a StopIteration (in f)
You can check that next(f(some_value)) does throw an exception when called itself.
As will
def g(x):
return next(f(x))
But, you've added the default None, so that g(x) will run, but simply return back None since the iterator is exhausted.
If you remove the None, then you see
In [5]: next(g(3))
---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
<ipython-input-14-05eb86fce40b> in <module>()
----> 1 next(g(3))
<ipython-input-13-a4323284f776> in g(x)
1 def g(x) :
----> 2 yield next(f(x))
3
StopIteration:
Related
I've learned about iterators and such and discovered this quite interesting way of getting the first element in a list that a condition is applied (and also with default value in case we don't find it):
first_occurence = next((x for x in range(1,10) if x > 5), None)
For me, it seems a very useful, clear way of obtaining the result.
But since I've never seen that in production code, and since next is a little more "low-level" in the python structure I was wondering if that could be bad practice for some reason. Is that the case? and why?
It's fine. It's efficient, it's fairly readable, etc.
If you're expecting a result, or None is a possible result (so using None as a placeholder makes it hard to figure out if you got a result or got the default) it may be better to use the EAFP form rather than providing a default, catching the StopIteration it raises if no item is found, or just letting it bubble up if the problem is from the caller's input not meeting specs (so it's up to them to handle it). It looks even cleaner at point of use that way:
first_occurence = next(x for x in range(1,10) if x > 5)
Alternatively, when None is a valid result, you can use an explicit sentinel object that's guaranteed unique like so:
sentinel = object() # An anonymous object you construct can't possibly appear in the input
first_occurence = next((x for x in range(1,10) if x > 5), sentinel)
if first_occurence is not sentinel: # Compare with is for performance and to avoid broken __eq__ comparing equal to sentinel
A common use case for this one of these constructs to replace a call to any when you not only need to know if any item passed the test, but which item (any can only return True or False, so it's unsuited to finding which item passed).
We can wrap it up in a function to provide an even nicer interface:
_raise = object()
# can pass either an iterable or an iterator
def first(iterable, condition, *, default=_raise, exctype=None):
"""Get the first value from `iterable` which meets `condition`.
Will consume elements from the iterable.
default -> if no element meets the condition, return this instead.
exctype -> if no element meets the condition and there is no default,
raise this kind of exception rather than `StopIteration`.
(It will be chained from the original `StopIteration`.)
"""
try:
# `iter` is idempotent; this makes sure we have an iterator
return next(filter(condition, iter(iterable)))
except StopIteration as e:
if default is not _raise:
return default
if exctype:
raise exctype() from e
raise
Let's test it:
>>> first(range(10), lambda x: x > 5)
6
>>> first(range(10), lambda x: x > 11)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in first
StopIteration
>>> first(range(10), lambda x: x > 11, exctype=ValueError)
Traceback (most recent call last):
File "<stdin>", line 4, in first
StopIteration
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 9, in first
ValueError
>>> first(range(10), lambda x: x > 11, default=None)
>>>
I want to have a generator that may or may not have anything to yield, and if .next() or similar is used it will not have a StopIteration error if none of the conditions to yield are met.
An example:
def A(iterable):
for x in iterable:
if x == 1:
yield True
Which works like the following:
>>> list(A([1,2]))
[True]
>>> A([1]).next()
True
>>> A([2]).next()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
But what I would want is for A([2]).next() to return is None.
generally, in these circumstances you'd use the builtin next function:
my_iterator = A([2])
value = next(my_iterator, None)
In addition to being able to pass an optional "default" value when the iterator is empty, next has the advantage of working on python2.x and python3.x where the name of the method changes to __next__.
How about this:
def A(iterable):
for x in iterable:
yield True if x == 1 else None
I have the following experimental code whose function is similar to the zip built-in. What it tries to do should have been simple and clear, trying to return the zipped tuples one at a time until an IndexError occurs when we stop the generator.
def my_zip(*args):
i = 0
while True:
try:
yield (arg[i] for arg in args)
except IndexError:
raise StopIteration
i += 1
However, when I tried to execute the following code, the IndexError was not caught but instead thrown by the generator:
gen = my_zip([1,2], ['a','b'])
print(list(next(gen)))
print(list(next(gen)))
print(list(next(gen)))
IndexError Traceback (most recent call last)
I:\Software\WinPython-32bit-3.4.2.4\python-3.4.2\my\temp2.py in <module>()
12 print(list(next(gen)))
13 print(list(next(gen)))
---> 14 print(list(next(gen)))
I:\Software\WinPython-32bit-3.4.2.4\python-3.4.2\my\temp2.py in <genexpr>(.0)
3 while True:
4 try:
----> 5 yield (arg[i] for arg in args)
6 except IndexError:
7 raise StopIteration
IndexError: list index out of range
Why is this happening?
Edit:
Thanks #thefourtheye for providing a nice explanation for what's happening above. Now another problem occurs when I execute:
list(my_zip([1,2], ['a','b']))
This line never returns and seems to hang the machine. What's happening now?
The yield yields a generator object everytime and when the generators were created there was no problem at all. That is why try...except in my_zip is not catching anything. The third time when you executed it,
list(arg[2] for arg in args)
this is how it got reduced to (over simplified for our understanding) and now, observe carefully, list is iterating the generator, not the actual my_zip generator. Now, list calls next on the generator object and arg[2] is evaluated, only to find that 2 is not a valid index for arg (which is [1, 2] in this case), so IndexError is raised, and list fails to handle it (it has no reason to handle that anyway) and so it fails.
As per the edit,
list(my_zip([1,2], ['a','b']))
will be evaluated like this. First, my_zip will be called and that will give you a generator object. Then iterate it with list. It calls next on it, and it gets another generator object list(arg[0] for arg in args). Since there is no exception or return encountered, it will call next, to get another generator object list(arg[1] for arg in args) and it keeps on iterating. Remember, the yielded generators are never iterated, so we ll never get the IndexError. That is why the code runs infinitely.
You can confirm this like this,
from itertools import islice
from pprint import pprint
pprint(list(islice(my_zip([1, 2], ["a", 'b']), 10)))
and you will get
[<generator object <genexpr> at 0x7f4d0a709678>,
<generator object <genexpr> at 0x7f4d0a7096c0>,
<generator object <genexpr> at 0x7f4d0a7099d8>,
<generator object <genexpr> at 0x7f4d0a709990>,
<generator object <genexpr> at 0x7f4d0a7095a0>,
<generator object <genexpr> at 0x7f4d0a709510>,
<generator object <genexpr> at 0x7f4d0a7095e8>,
<generator object <genexpr> at 0x7f4d0a71c708>,
<generator object <genexpr> at 0x7f4d0a71c750>,
<generator object <genexpr> at 0x7f4d0a71c798>]
So the code tries to build an infinite list of generator objects.
def my_zip(*args):
i = 0
while True:
try:
yield (arg[i] for arg in args)
except IndexError:
raise StopIteration
i += 1
IndexError is not caught, because (arg[i] for arg in args) is a generator which is not executed immediately, but when you start iterating over it. And you iterate over it in another scope, when you call list((arg[i] for arg in args)):
# get the generator which yields another generator on each iteration
gen = my_zip([1,2], ['a','b'])
# get the second generator `(arg[i] for arg in args)` from the first one
# then iterate over it: list((arg[i] for arg in args))
print(list(next(gen)))
On the first list(next(gen)) i equals 0.
On the second list(next(gen)) i equals 1.
On the third list(next(gen)) i equals 2. And here you get IndexError -- in the outer scope. The line is treated as list(arg[2] for arg in ([1,2], ['a','b']))
Sorry, I'm not able to offer a coherent explanation regarding the failure to catch the exception, however, there's an easy way around it; use a for loop over the length of the shortest sequence:
def my_zip(*args):
for i in range(min(len(arg) for arg in args)):
yield (arg[i] for arg in args)
>>> gen = my_zip([1,2], ["a",'b','c'])
>>> print(list(next(gen)))
[1, 'a']
>>> print(list(next(gen)))
[2, 'b']
>>> print(list(next(gen)))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
Try replacing yield (arg[i] for ...) with the following.
for arg in args:
yield arg[i]
But in case of numbers that causes an exception as 1[1] makes no sense. I suggest replacing arg[i] just with arg.
This is my code
class A:
pass
def f():
yield A()
def g():
it = f()
next(it).a = next(it, None)
g()
that produces the StopIteration error, caused by next(it).a = next(it, None). Why?
The documentation says that next function does not raise the StopIteration if the default value is provided, and I expected it to get me the first item from the generator (the A instance) and set the a attribute to None.
Because f only yields a single value, you can only call next on it once.
The right hand side of your expression (next(it, None)) is evaluated before the left hand side, and thus exhausts the generator.
Calling next(it).a on the left hand side will then raise StopIteration.
Your f() generator function yields just one value. After that it is exhausted and raises StopIteration.
>>> class A:
... pass
...
>>> def f():
... yield A()
...
>>> generator = f()
>>> generator
<generator object f at 0x10be771f8>
>>> next(generator)
<__main__.A object at 0x10be76f60>
>>> next(generator)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
That's because there is no loop in f() to yield more than once, a generator function does not, on its own, loop, just because you can use it in a loop.
Note that for an assignment, Python executes the right-hand-side expression first, before figuring out what to assign it to. So the next(it, None) is called first, the next(it).a for the assignment is called second.
The body of the f() function is executed just like any other Python function, with the addition of pausing. next() on the generator un-pauses the code, and the code then runs until the next yield statement is executed. That statement then pauses the generator again. If the function instead ends (returns), StopIteration is raised.
In your f() generator that means:
when you call f() a new generator object is created. The function body is paused.
you call next() on it the first time. The code starts running, creates an instance of A() and yields that instance. The function is paused again.
you call next() on it a second time. The code starts running, reaches the end of the function, and returns. StopIteration is raised.
If you add a loop to f(), or simply add a second yield line, your code works:
def f():
yield A()
yield A()
or
def f():
while True:
yield A()
I have a function that either returns a tuple or None. How is the Caller supposed to handle that condition?
def nontest():
return None
x,y = nontest()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not iterable
EAFP:
try:
x,y = nontest()
except TypeError:
# do the None-thing here or pass
or without try-except:
res = nontest()
if res is None:
....
else:
x, y = res
How about:
x,y = nontest() or (None,None)
If nontest returns a two-item tuple like it should, then x and y are assigned to the items in the tuple. Otherwise, x and y are each assigned to none. Downside to this is that you can't run special code if nontest comes back empty (the above answers can help you if that is your goal). Upside is that it is clean and easy to read/maintain.
If you can change the function itself, it's probably a better idea to make it raise a relevant exception instead of returning None to signal an error condition. The caller should then just try/except that.
If the None isn't signalling an error condition, you'll want to rethink your semantics altogether.