I have a function called x that produces a generator like this:
a = 5
def x():
global a
if a == 3:
raise Exception("Stop")
a = a - 1
yield a
Then in the python shell I call that function like this:
>>> print x().next()
>>> 4
>>> print x().next()
>>> 3
>>> print x().next()
>>> <python-input-112-f3c02bba26c8> in x()
2 global a
3 if a == 3:
----> 4 raise Exception
5 a = a - 1
6 yield a
Exception:
However, when I call that function and assign it to a variable, it behaves differently:
>>> a = 5
>>> b = x()
>>> print b.next()
>>> 4
>>> print b.next()
>>> ----> 1 b.next()
StopIteration:
How is that even possible? Shouldn't it print out 3 and raise StopIteration in the next iteration?
PS: I know that when I first call the function, the body does not run, just produces a generator. The point I didn't understand is that what changes if I call and assign it to a variable?
In your first example, you were creating a new generator each time:
x().next()
This starts the generator from the top, so the first statement. When a == 3, the exception is raised, otherwise the generator just yields and pauses.
When you assigned your generator later on, the global a started at 5, the code then continued from where it left of until it ends or comes across another yield statement, then ended. When a generator function ends, it raises StopIteration.
Let's break this down into steps:
a = 5.
You create new generator and call .next() on it. The following code is executed:
global a
if a == 3: # False
raise Exception("Stop")
a = a - 1 # a is now 4
yield a
The generator is paused on the last line, and 4 is yielded.
You create a new generator and call .next() on it. a is 4 at the start:
global a
if a == 3: # False
raise Exception("Stop")
a = a - 1 # a is now 3
yield a
The generator is paused on the last line, and 3 is yielded.
You create a new generator and call .next() on it. a is 3 at the start:
global a
if a == 3: # True
raise Exception("Stop")
An exception is raised.
You set a = 5 again.
You create a new generator, store a reference in b and call .next() on it. The following code is executed:
global a
if a == 3: # False
raise Exception("Stop")
a = a - 1 # a is now 4
yield a
The generator is paused on the last line, and 4 is yielded.
You call .next() again on the same, existing generator referenced by b. The code resumes at the paused point.
The function has no more code at that point, and returns. StopIteration is raised.
If you were to use a loop instead, you'd see the difference better:
>>> def looping(stop):
... for i in looping(stop):
... yield i
...
>>> looping(3).next()
0
>>> looping(3).next()
0
Note how each time I create a new generator, the loop starts from the beginning. Store a reference however, and you'll notice it continue instead:
>>> stored = looping(3)
>>> stored.next()
0
>>> stored.next()
1
>>> stored.next()
2
>>> stored.next()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
During the loop, each time the yield expression is executed, the code is paused; calling .next() continues the function where it left of the previous time.
The StopIteration exception is entirely normal; it is how generators communicate that they are done. A for loop looks for this exception to end the loop:
>>> for i in looping(3):
... print i
...
0
1
2
You've not quite got how yield works. I think this example might help:
>>> def a():
... for x in range(5):
... yield x
...
>>> a()
<generator object a at 0xb7f0a9b4>
>>> list(a())
[0, 1, 2, 3, 4]
You normally want to use yield inside a loop, and it has the very distinct behavior of returning a value, then later resuming the loop.
If your example, x always returns a generator that will produce one item only. In your first example, you are calling x multiple times, so you get multiple results. In your second example, where you assign it to a variable, you are only calling it once, so you only get one result.
Also, you should not usually use a global variable in the way you have done.
Paul
Related
According to docs, the send() function:
"Resumes the execution and “sends” a value into the generator function. The value argument becomes the result of the current yield expression. The send() method returns the next value yielded by the generator, or raises StopIteration if the generator exits without yielding another value. When send() is called to start the generator, it must be called with None as the argument, because there is no yield expression that could receive the value."
But i can't understand, why "The value argument becomes the result of the current yield expression" is not happened in the following example:
def gen():
yield 1
x = (yield 42)
print(x)
yield 2
>>>c=gen() #create generator
>>>next(c) #prints '1' and stop execution, which is caused by yield 1
>>>c.send(100) #prints '42', because 'The send() method returns the next value yielded by the generator'
>>>next(c) #prints 'None' and '2'
So why x variable stays 'None' dispite i send 100 to it by c.send(100)? It seems, that yield expression in right hand side works in two steps: first it return the value to generator's caller and the second it returns argument of send function inside generator. And if add extra next(c) before send(42) i'll get expected behavior and programm prints '100'. It's not clear for me from documentation, why these two steps should not happen simultaneously when i call send().
So why x variable stays 'None' dispite i send 100 to it by
c.send(100)?
Because the current yield is not the one you think it is, it's the one before.
When you send 100, the generator is still stopped at yield 1, not yet at yield 42, hence 100 will be the result of yield 1, not of yield 42.
To see this clearer, if you modify the generator in order to retrieve the content of yield 1 in a z variable, you'll see that z does contain 100:
>>> def gen():
... z = yield 1
... x = (yield 42)
... print(x, z)
... yield 2
...
>>> c=gen()
>>> next(c)
1
>>> c.send(100)
42
>>> next(c)
None 100
2
>>>
And that's the reason why an extra next() will print 100. Back to your code:
def gen():
yield 1 # first next(), stops here (before "getting out" of yield)
x = (yield 42) # second next(), stops here (before "getting out" of yield),
# so, when you send(100), then 100 is given to x and execution goes on, so:
print(x) # 100 is printed
yield 2 # ... and 2.
I think I've figured it out.
c = gen()
You create a genertor in the c variable, there none of the generator's code is executed.
next(c)
Then you use the next() function, the next function go to the next yield statement who is yield 1 here.
So this yield returns 1 and stops the generator's execution to try to catch a value.
The next line is
c.send(100), so the yield 1 was waiting for a value and you provided it, but this value isn't saved.
The send method also execute the rest of the generator until the next yield statement who is:
x = (yield 42)
So the yield here return 42 and stops the generator program to try to catch a value. But after that you call
next(c)
And you didn't provided a value so x is None now. Then the rest of the code is executed (until the next yield statement, don't forget)
print(x)
yield 2
So it prints x who is None and then the yield returns 2 and try to catch a value.
Try to write this
c = gen()
next(c)
next(c)
c.send(100)
And it will work (understand why !)
From documentation:
The value argument becomes the result of the current yield expression. The send() method returns the next value yielded by the generator.
assume we have following infinite generator:
def infgen():
a = 0
while True:
a = yield a
a += 1
generato = infgen()
generato.send(None)
output:0
.send(None) is executed almost the same as next()
now we have yeild first value of a = 0 and
code stopped here a = yield a
as value becomes result of the current yield expression
yielded expression is yield a. Result of yielded expresion is a
generato.send(5)
.send(5) we send 5 to a in generator and it continue:
Result of yielded expresion is a = 5
a = 5
a += 1
Go to while loop with a = 5 + 1 = 6
And stops and yield a where a is 6
output:6
I am calling a method and I need a static counter within this method. It's required to parse the elements of the list. The counter will tell which position of the list to lookup at.
For e.g
static_var_with_position = 0
noib_list = [3, 2, 2, 2, 2, 1, 2, 2]
def foo(orig_output, NOB):
# tried two ways
#static_var_with_position += 1 # doesn't work
#global static_var_with_position
#static_var_with_position += 1 # doesn't work either
bit_required = noib_list[static_var_with_position]
converted_output = convert_output(orig_output, NOB, bit_required)
The static_var_with_position value is never incremented. I have commented the two ways I tried to increment value.
In c++ its piece of cake, but I couldn't find anything similar in python so far. Any help will be appreciated :)
Thanks!
Instead of using a global/static counter variable, you could use an iterator:
iterator = iter(noib_list)
def foo(orig_output, NOB):
bit_required = next(iterator)
converted_output = convert_output(orig_output, NOB, bit_required)
The iterator will automatically keep track which is the next element internally.
When the iterator is exhausted (i.e. when you reached the end of the list), next will raise a StopIteration error, so it you do not know when the end is reached, you can use bit_required = next(iterator, None) instead; then just test whether the value is None and you know that the list is exhausted (or just use try/except).
Following this example, you could do the same with your counter :
def inc() :
global global_cpt
global_cpt +=1
print global_cpt
if __name__ == '__main__' :
global_cpt = 0
inc()
inc()
inc()
will print
> 1
> 2
> 3
I don't actually advocate doing this in your case, but it's a little-known hack for creating a "static" variable within a function: put it as a parameter with a mutable default value! You can modify it within the function and it will hold until the next function call, as long as the caller doesn't pass a value for it.
def foo(value=[0]):
value[0] += 1
print(value[0])
>>> foo()
1
>>> foo()
2
>>> foo()
3
>>> foo([906])
907
>>> foo()
4
Is it safe in Python to do something like this (say, in a parser)?
iterator = iter(some_iterable)
for x in iterator:
# do stuff with x
if some_condition(x):
y = next(iterator)
# do stuff with y
I've tested in Python 2 and 3 and it does what I expect, but I wonder whether it's really safe and whether it's idiomatic. The above code should be equivalent to the following rather more verbose alternative:
iterator = iter(some_iterable)
while True:
try:
x = next(iterator)
except StopIteration:
break
# do stuff with x
if some_condition(x):
y = next(iterator)
# do stuff with y
Basically it's always better to keep track of your exceptions and handle them properly. But regarding the difference between the while and for loops in this case when you are calling a next() function within a while loop it's always possible to raise an StopIteration exception, but when you are using a for loop based on the number of next() calls and your iteration it may be different.
For example in this case for even number of iteration it doesn't raise an exception but for odds it does. And the reason is that in even iteration numbers your next is always one item in front of the for loop, while for odds it's not like so.
In [1]: it = iter(range(4))
In [2]: for x in it:
...: print(x)
...: next(it)
...:
0
2
In [3]: it = iter(range(3))
In [4]: for x in it:
print(x)
next(it)
...:
0
2
---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
<ipython-input-4-1b3db7079e29> in <module>()
1 for x in it:
2 print(x)
----> 3 next(it)
4
StopIteration:
I'm playing around with generators and generator expressions and I'm not completely sure that I understand how they work (some reference material):
>>> a = (x for x in range(10))
>>> next(a)
0
>>> next(a)
1
>>> a.send(-1)
2
>>> next(a)
3
So it looks like generator.send was ignored. That makes sense (I guess) because there is no explicit yield expression to catch the sent information ...
However,
>>> a = ((yield x) for x in range(10))
>>> next(a)
0
>>> print next(a)
None
>>> print next(a)
1
>>> print next(a)
None
>>> a.send(-1) #this send is ignored, Why? ... there's a yield to catch it...
2
>>> print next(a)
None
>>> print next(a)
3
>>> a.send(-1) #this send isn't ignored
-1
I understand this is pretty far out there, and I (currently) can't think of a use-case for this (so don't ask;)
I'm mostly just exploring to try to figure out how these various generator methods work (and how generator expressions work in general). Why does my second example alternate between yielding a sensible value and None? Also, Can anyone explain why one of my generator.send's was ignored while the other wasn't?
The confusion here is that the generator expression is doing a hidden yield. Here it is in function form:
def foo():
for x in range(10):
yield (yield x)
When you do a .send(), what happens is the inner yield x gets executed, which yields x. Then the expression evaluates to the value of the .send, and the next yield yields that. Here it is in clearer form:
def foo():
for x in range(10):
sent_value = (yield x)
yield sent_value
Thus the output is very predictable:
>>> a = foo()
#start it off
>>> a.next()
0
#execution has now paused at "sent_value = ?"
#now we fill in the "?". whatever we send here will be immediately yielded.
>>> a.send("yieldnow")
'yieldnow'
#execution is now paused at the 'yield sent_value' expression
#as this is not assigned to anything, whatever is sent now will be lost
>>> a.send("this is lost")
1
#now we're back where we were at the 'yieldnow' point of the code
>>> a.send("yieldnow")
'yieldnow'
#etc, the loop continues
>>> a.send("this is lost")
2
>>> a.send("yieldnow")
'yieldnow'
>>> a.send("this is lost")
3
>>> a.send("yieldnow")
'yieldnow'
EDIT: Example usage. By far the coolest one I've seen so far is twisted's inlineCallbacks function. See here for an article explaining it. The nub of it is it lets you yield functions to be run in threads, and once the functions are done, twisted sends the result of the function back into your code. Thus you can write code that heavily relies on threads in a very linear and intuitive manner, instead of having to write tons of little functions all over the place.
See the PEP 342 for more info on the rationale of having .send work with potential use cases (the twisted example I provided is an example of the boon to asynchronous I/O this change offered).
You're confusing yourself a bit because you actually are generating from two sources: the generator expression (... for x in range(10)) is one generator, but you create another source with the yield. You can see that if do list(a) you'll get [0, None, 1, None, 2, None, 3, None, 4, None, 5, None, 6, None, 7, None, 8, None, 9, None].
Your code is equivalent to this:
>>> def gen():
... for x in range(10):
... yield (yield x)
Only the inner yield ("yield x") is "used" in the generator --- it is used as the value of the outer yield. So this generator iterates back and forth between yielding values of the range, and yielding whatever is "sent" to those yields. If you send something to the inner yield, you get it back, but if you happen to send on an even-numbered iteration, the send is sent to the outer yield and is ignored.
This generator translates into:
for i in xrange(10):
x = (yield i)
yield x
Result of second call to send()/next() are ignored, because you do nothing with result of one of yields.
The generator you wrote is equivalent to the more verbose:
def testing():
for x in range(10):
x = (yield x)
yield x
As you can see here, the second yield, which is implicit in the generator expression, does not save the value you pass it, therefore depending on where the generator execution is blocked the send may or may not work.
Indeed - the send method is meant to work with a generator object that is the result of a co-routine you have explicitly written. It is difficult to get some meaning to it in a generator expression - though it works.
-- EDIT --
I had previously written this, but it is incorrecct, as yield inside generator expressions are predictable across implementations - though not mentioned in any PEP.
generator expressions are not meant to have the yield keyword - I am
not shure the behavior is even defined in this case. We could think a
little and get to what is happening on your expression, to meet from
where those "None"s are coming from. However, assume that as a side
effect of how the yield is implemented in Python (and probably it is
even implementation dependent), not as something that should be so.
The correct form for a generator expression, in a simplified manner is:
(<expr> for <variable> in <sequence> [if <expr>])
so, <expr> is evaluated for each value in the <sequence: - not only is yield uneeded, as you should not use it.
Both yield and the send methods are meant to be used in full co-routines, something like:
def doubler():
value = 0
while value < 100:
value = 2 * (yield value)
And you can use it like:
>>> a = doubler()
>>> # Next have to be called once, so the code will run up to the first "yield"
...
>>> a.next()
0
>>> a.send(10)
20
>>> a.send(20)
40
>>> a.send(23)
46
>>> a.send(51)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
>>>
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
The Python yield keyword explained
Can someone explain to me what the yield statement actually does in this bit of code here:
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a+b
for number in fibonacci(): # Use the generator as an iterator; print number
What I understand so far is, we are defining a function finonacci(), with no parameters?
inside the function we are defining a and b equal to 0 and 1, next, while this is true, we are yielding a. What is this actually doing? Furthermore, while yielding a? a is now equal to b, while b is now equal to a + b.
Next question, for number in fibonacci(), does this mean for every number in the function or what? I'm equally stumped on what yield and 'for number' are actually doing. Obviously I am aware that it means for every number in fibonacci() print number. Am I actually defining number without knowing it?
Thanks, sorry if I'm not clear. BTW, it's for project Euler, if I knew how to program well this would be a breeze but I'm trying to learn this on the fly.
Using yield makes the function a generator.
The generator will continue to yield the a variable on each loop, waiting until the generator's next() method is called to continue on to the next loop iteration.
Or, until you return or StopIteration is raised.
Slightly modified to show use of StopIteration:
>>> def fib():
... a = 0
... b = 1
... while True:
... yield a
... a = b
... b += a
... if a > 100:
... raise StopIteration
...
>>>
>>> for value in fib():
... print value
...
0
1
2
4
8
16
32
64
>>>
>>> # assign the resulting object to 'generator'
>>> generator = fib()
>>> generator.next()
0
>>> generator.next()
1
>>> for value in generator:
... print value
...
2
4
8
16
32
64
>>>
Generators have a special property of being iterables which do not consume memories for their values.
They do this by calculating the new value, when it is required while being iterated.
i.e.
def f():
a = 2
yield a
a += 1
for ele in f():
print ele
would print
2
So you are using a function as an iterable that keeps returning values.
This is especially useful when you require heavy memory usage, and so you cannot afford the use of a list comprehension
i.e.
li = [ele*10 for ele in range(10)]
takes 10 memory spaces for ints as a list
but if you simple want to iterate over it, not access it individually
it would be very memory efficient to instead use
def f():
i=0
while i<10
yield i*10
i += 1
which would use 1 memory space as i keeps being reused
a short cut for this is
ge = (i*10 for i in range(10))
you can do any of the following
for ele in f():
for ele in li:
for ele in ge:
to obtain equivalent results
When the code calls fibonacci a special generator object is created. Please note, that no code gets executed - only a generator object is returned. When you are later calling its next method, the function executes until it encounters a yield statement. The object that is supplied to yield is returned. When you call next method again the function executes again until it encounters a yield. When there are no more yield statements and the end of function is reached, a StopIteration exception is raised.
Please note that the objects inside the function are preserved between the calls to next. It means, when the code continues execution on the next loop, all the objects that were in the scope from which yield was called have their values from the point where a previous next call returned.
The cool thing about generators is that they allow convenient iteration with for loops.
The for loop obtains a generator from the result of fibonacci call and then executes the loop retrieving elements using next method of generatior object until StopIteration exception is encountered.
This answer is a great explanation of the yield statement, and also of iterators and generators.
Specifically here, the first call to fibonaci() will initialize a to 0, b to 1, enter the while loop and return a.
Any next call will start after the yield statement, affect b to a, a+b to b, and then go to the next iteration of the while statement, reach again the yield statement, and return a again.