This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
The Python yield keyword explained
Can someone explain to me what the yield statement actually does in this bit of code here:
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a+b
for number in fibonacci(): # Use the generator as an iterator; print number
What I understand so far is, we are defining a function finonacci(), with no parameters?
inside the function we are defining a and b equal to 0 and 1, next, while this is true, we are yielding a. What is this actually doing? Furthermore, while yielding a? a is now equal to b, while b is now equal to a + b.
Next question, for number in fibonacci(), does this mean for every number in the function or what? I'm equally stumped on what yield and 'for number' are actually doing. Obviously I am aware that it means for every number in fibonacci() print number. Am I actually defining number without knowing it?
Thanks, sorry if I'm not clear. BTW, it's for project Euler, if I knew how to program well this would be a breeze but I'm trying to learn this on the fly.
Using yield makes the function a generator.
The generator will continue to yield the a variable on each loop, waiting until the generator's next() method is called to continue on to the next loop iteration.
Or, until you return or StopIteration is raised.
Slightly modified to show use of StopIteration:
>>> def fib():
... a = 0
... b = 1
... while True:
... yield a
... a = b
... b += a
... if a > 100:
... raise StopIteration
...
>>>
>>> for value in fib():
... print value
...
0
1
2
4
8
16
32
64
>>>
>>> # assign the resulting object to 'generator'
>>> generator = fib()
>>> generator.next()
0
>>> generator.next()
1
>>> for value in generator:
... print value
...
2
4
8
16
32
64
>>>
Generators have a special property of being iterables which do not consume memories for their values.
They do this by calculating the new value, when it is required while being iterated.
i.e.
def f():
a = 2
yield a
a += 1
for ele in f():
print ele
would print
2
So you are using a function as an iterable that keeps returning values.
This is especially useful when you require heavy memory usage, and so you cannot afford the use of a list comprehension
i.e.
li = [ele*10 for ele in range(10)]
takes 10 memory spaces for ints as a list
but if you simple want to iterate over it, not access it individually
it would be very memory efficient to instead use
def f():
i=0
while i<10
yield i*10
i += 1
which would use 1 memory space as i keeps being reused
a short cut for this is
ge = (i*10 for i in range(10))
you can do any of the following
for ele in f():
for ele in li:
for ele in ge:
to obtain equivalent results
When the code calls fibonacci a special generator object is created. Please note, that no code gets executed - only a generator object is returned. When you are later calling its next method, the function executes until it encounters a yield statement. The object that is supplied to yield is returned. When you call next method again the function executes again until it encounters a yield. When there are no more yield statements and the end of function is reached, a StopIteration exception is raised.
Please note that the objects inside the function are preserved between the calls to next. It means, when the code continues execution on the next loop, all the objects that were in the scope from which yield was called have their values from the point where a previous next call returned.
The cool thing about generators is that they allow convenient iteration with for loops.
The for loop obtains a generator from the result of fibonacci call and then executes the loop retrieving elements using next method of generatior object until StopIteration exception is encountered.
This answer is a great explanation of the yield statement, and also of iterators and generators.
Specifically here, the first call to fibonaci() will initialize a to 0, b to 1, enter the while loop and return a.
Any next call will start after the yield statement, affect b to a, a+b to b, and then go to the next iteration of the while statement, reach again the yield statement, and return a again.
Related
Can someone please help me to understand the execution of this code?
from functools import reduce
def foo():
for i in range(10):
yield i
gen = foo()
print (gen == 0, gen.__next__ == 0, gen.__next__ == 1, reduce(lambda a,b:a+b, gen))
My questions are the following:
What are the multiple print arguments doing? What does gen == 0 mean? Isn't the gen a generator object?
Also how is the reduce function remembering what to assign to a and b?
And after gen = foo() is called, does execution just switch between the function and the reduce statement?
If someone could please map out the functionalities of the generator and the reduce function, that would help a lot! Thanks!
As you note, gen is a generator; all of the tests in the print are nonsensical (if they added call parens to get gen.__next__() then the comparison to 0 and 1 would return True I suppose, but by not calling it, it's guaranteed False).
As for reduce, that's in the docs. Each call of the reducer function uses the result of the prior call (stored internally in the reduce function's locals, not visible except when it's passed to the reducer function) as the first argument (on the first call, this is either the optional third argument initializer, or the first call is performed with the first two elements from gen), and the next result from the input iterable as b.
So if __next__ had actually been called (consuming the first two values from gen), the first call would use 2 as a and 3 as b; the result (5) would be a for the next call, with b the next value from gen (4), producing 9, etc. Stepping it out the first few steps. you'd see:
a = 2, b = 3 -> 5
a = 5, b = 4 -> 9
a = 9, b = 5 -> 14
a = 14, b = 6 -> 20
and so on; effectively, it's just using reduce as a slow/ugly form of the sum function.
As for your fourth question: Generators are lazy. Execution switches to the generator when the next value is requested, and when it's produced, the generator is "hibernated" until the next value is requested. So in this case, yes, each call of the lambda passed to reduce corresponds to an additional read from gen (two reads for the first call since it needs to get an accumulator value to start), but it's entirely on-demand; there is no true parallelism going on here; when gen resumes, the code that asked it for a value is paused pending the result; when no value is being requested from it, gen is "frozen" indefinitely (not background processing at all).
According to docs, the send() function:
"Resumes the execution and “sends” a value into the generator function. The value argument becomes the result of the current yield expression. The send() method returns the next value yielded by the generator, or raises StopIteration if the generator exits without yielding another value. When send() is called to start the generator, it must be called with None as the argument, because there is no yield expression that could receive the value."
But i can't understand, why "The value argument becomes the result of the current yield expression" is not happened in the following example:
def gen():
yield 1
x = (yield 42)
print(x)
yield 2
>>>c=gen() #create generator
>>>next(c) #prints '1' and stop execution, which is caused by yield 1
>>>c.send(100) #prints '42', because 'The send() method returns the next value yielded by the generator'
>>>next(c) #prints 'None' and '2'
So why x variable stays 'None' dispite i send 100 to it by c.send(100)? It seems, that yield expression in right hand side works in two steps: first it return the value to generator's caller and the second it returns argument of send function inside generator. And if add extra next(c) before send(42) i'll get expected behavior and programm prints '100'. It's not clear for me from documentation, why these two steps should not happen simultaneously when i call send().
So why x variable stays 'None' dispite i send 100 to it by
c.send(100)?
Because the current yield is not the one you think it is, it's the one before.
When you send 100, the generator is still stopped at yield 1, not yet at yield 42, hence 100 will be the result of yield 1, not of yield 42.
To see this clearer, if you modify the generator in order to retrieve the content of yield 1 in a z variable, you'll see that z does contain 100:
>>> def gen():
... z = yield 1
... x = (yield 42)
... print(x, z)
... yield 2
...
>>> c=gen()
>>> next(c)
1
>>> c.send(100)
42
>>> next(c)
None 100
2
>>>
And that's the reason why an extra next() will print 100. Back to your code:
def gen():
yield 1 # first next(), stops here (before "getting out" of yield)
x = (yield 42) # second next(), stops here (before "getting out" of yield),
# so, when you send(100), then 100 is given to x and execution goes on, so:
print(x) # 100 is printed
yield 2 # ... and 2.
I think I've figured it out.
c = gen()
You create a genertor in the c variable, there none of the generator's code is executed.
next(c)
Then you use the next() function, the next function go to the next yield statement who is yield 1 here.
So this yield returns 1 and stops the generator's execution to try to catch a value.
The next line is
c.send(100), so the yield 1 was waiting for a value and you provided it, but this value isn't saved.
The send method also execute the rest of the generator until the next yield statement who is:
x = (yield 42)
So the yield here return 42 and stops the generator program to try to catch a value. But after that you call
next(c)
And you didn't provided a value so x is None now. Then the rest of the code is executed (until the next yield statement, don't forget)
print(x)
yield 2
So it prints x who is None and then the yield returns 2 and try to catch a value.
Try to write this
c = gen()
next(c)
next(c)
c.send(100)
And it will work (understand why !)
From documentation:
The value argument becomes the result of the current yield expression. The send() method returns the next value yielded by the generator.
assume we have following infinite generator:
def infgen():
a = 0
while True:
a = yield a
a += 1
generato = infgen()
generato.send(None)
output:0
.send(None) is executed almost the same as next()
now we have yeild first value of a = 0 and
code stopped here a = yield a
as value becomes result of the current yield expression
yielded expression is yield a. Result of yielded expresion is a
generato.send(5)
.send(5) we send 5 to a in generator and it continue:
Result of yielded expresion is a = 5
a = 5
a += 1
Go to while loop with a = 5 + 1 = 6
And stops and yield a where a is 6
output:6
A good exercise to test one's understanding of recursion is to write a function that generates all permutations of a string:
def get_perms(to_go, so_far=''):
if not to_go:
return [so_far]
else:
ret = []
for i in range(len(to_go)):
ret += get_perms(to_go[:i] + to_go[i+1:], so_far + to_go[i])
return ret
This code is fine, but we can significantly improve the efficiency from a memory standpoint by using a generator:
def perms_generator(to_go, so_far=''):
if not to_go:
yield so_far
else:
for i in range(len(to_go)):
for perm in perms_generator(to_go[:i] + to_go[i+1:], so_far + to_go[i]):
yield perm
(Note: the last for loop can also be replaced with yield from in python 3.3)
My question: why do we need to iterate over the results of each recursive call? I know that yield returns a generator, but from the statement yield so_far it would seem as though we're getting a string, and not something we would need to iterate over. Rather, it would seem as though we could replace
for perm in perms_generator(to_go[:i] + to_go[i+1:], so_far + to_go[i]):
yield perm
with
yield perms_generator(to_go[:i] + to_go[i+1:], so_far + to_go[i])
Thank you. Please let me know if the title is unclear. I have a feeling the content of this question is related to this SO question.
Remember, any function using yield does not return those values to the caller. Instead a generator object is returned and the code itself is instead paused until you iterate over the generator. Each time a yield is encountered, the code is paused again:
>>> def pausing_generator():
... print 'Top of the generator'
... yield 1
... print 'In the middle'
... yield 2
... print 'At the end'
...
>>> gen = pausing_generator()
>>> gen
<generator object pausing_generator at 0x1081e0d70>
>>> next(gen)
Top of the generator
1
>>> next(gen)
In the middle
2
>>> next(gen)
At the end
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
Calling the pausing_generator() function returned a generator object. Only iterating (using the next() function here) runs the actual code in the function, but pausing execution each time a yield is encountered.
Your perms_generator function returns a such a generator object, and a recursive call would still return a generator object. You could yield the whole generator object, but then you are producing a generator that produces a generator, etc. until you come to the inner-most generator.
You can visualise this with print statements:
>>> def countdown(i):
... if not i:
... return
... yield i
... recursive_result = countdown(i - 1)
... print i, recursive_result
... for recursive_elem in recursive_result:
... yield recursive_elem
...
>>> for i in countdown(5):
... print i
...
5
5 <generator object countdown at 0x1081e0e10>
4
4 <generator object countdown at 0x1081e0e60>
3
3 <generator object countdown at 0x1081e0eb0>
2
2 <generator object countdown at 0x1081e0f00>
1
1 <generator object countdown at 0x1081e0f50>
Here, the recursive calls returned a new generator object; if you wanted to have the elements produced by the generator your only choice is to loop over it and hand the elements down, not the generator object itself.
In Python 3, you can use yield from to delegate to a nested generator, including a recursive call:
def perms_generator(to_go, so_far=''):
if not to_go:
yield so_far
else:
for i, elem in enumerate(to_go):
yield from perms_generator(to_go[:i] + to_go[i+1:], so_far + elem)
When encountering a yield from iteration continues into the recursive call instead of yielding the whole generator object.
The difference between return and yield is that the former just returns a value. The latter means "wrap the value in a generator and then return the generator."
So in all cases, the function perms_generator() returns a generator.
The expression yield perms_generator() again would wrap the result of perms_generator() in a generator, giving you a generator of a generator. That would mean the function would return different things; sometimes, it would be a simple generator and sometimes nested generators. That would be very confusing for the consumer of your code.
I'm playing around with generators and generator expressions and I'm not completely sure that I understand how they work (some reference material):
>>> a = (x for x in range(10))
>>> next(a)
0
>>> next(a)
1
>>> a.send(-1)
2
>>> next(a)
3
So it looks like generator.send was ignored. That makes sense (I guess) because there is no explicit yield expression to catch the sent information ...
However,
>>> a = ((yield x) for x in range(10))
>>> next(a)
0
>>> print next(a)
None
>>> print next(a)
1
>>> print next(a)
None
>>> a.send(-1) #this send is ignored, Why? ... there's a yield to catch it...
2
>>> print next(a)
None
>>> print next(a)
3
>>> a.send(-1) #this send isn't ignored
-1
I understand this is pretty far out there, and I (currently) can't think of a use-case for this (so don't ask;)
I'm mostly just exploring to try to figure out how these various generator methods work (and how generator expressions work in general). Why does my second example alternate between yielding a sensible value and None? Also, Can anyone explain why one of my generator.send's was ignored while the other wasn't?
The confusion here is that the generator expression is doing a hidden yield. Here it is in function form:
def foo():
for x in range(10):
yield (yield x)
When you do a .send(), what happens is the inner yield x gets executed, which yields x. Then the expression evaluates to the value of the .send, and the next yield yields that. Here it is in clearer form:
def foo():
for x in range(10):
sent_value = (yield x)
yield sent_value
Thus the output is very predictable:
>>> a = foo()
#start it off
>>> a.next()
0
#execution has now paused at "sent_value = ?"
#now we fill in the "?". whatever we send here will be immediately yielded.
>>> a.send("yieldnow")
'yieldnow'
#execution is now paused at the 'yield sent_value' expression
#as this is not assigned to anything, whatever is sent now will be lost
>>> a.send("this is lost")
1
#now we're back where we were at the 'yieldnow' point of the code
>>> a.send("yieldnow")
'yieldnow'
#etc, the loop continues
>>> a.send("this is lost")
2
>>> a.send("yieldnow")
'yieldnow'
>>> a.send("this is lost")
3
>>> a.send("yieldnow")
'yieldnow'
EDIT: Example usage. By far the coolest one I've seen so far is twisted's inlineCallbacks function. See here for an article explaining it. The nub of it is it lets you yield functions to be run in threads, and once the functions are done, twisted sends the result of the function back into your code. Thus you can write code that heavily relies on threads in a very linear and intuitive manner, instead of having to write tons of little functions all over the place.
See the PEP 342 for more info on the rationale of having .send work with potential use cases (the twisted example I provided is an example of the boon to asynchronous I/O this change offered).
You're confusing yourself a bit because you actually are generating from two sources: the generator expression (... for x in range(10)) is one generator, but you create another source with the yield. You can see that if do list(a) you'll get [0, None, 1, None, 2, None, 3, None, 4, None, 5, None, 6, None, 7, None, 8, None, 9, None].
Your code is equivalent to this:
>>> def gen():
... for x in range(10):
... yield (yield x)
Only the inner yield ("yield x") is "used" in the generator --- it is used as the value of the outer yield. So this generator iterates back and forth between yielding values of the range, and yielding whatever is "sent" to those yields. If you send something to the inner yield, you get it back, but if you happen to send on an even-numbered iteration, the send is sent to the outer yield and is ignored.
This generator translates into:
for i in xrange(10):
x = (yield i)
yield x
Result of second call to send()/next() are ignored, because you do nothing with result of one of yields.
The generator you wrote is equivalent to the more verbose:
def testing():
for x in range(10):
x = (yield x)
yield x
As you can see here, the second yield, which is implicit in the generator expression, does not save the value you pass it, therefore depending on where the generator execution is blocked the send may or may not work.
Indeed - the send method is meant to work with a generator object that is the result of a co-routine you have explicitly written. It is difficult to get some meaning to it in a generator expression - though it works.
-- EDIT --
I had previously written this, but it is incorrecct, as yield inside generator expressions are predictable across implementations - though not mentioned in any PEP.
generator expressions are not meant to have the yield keyword - I am
not shure the behavior is even defined in this case. We could think a
little and get to what is happening on your expression, to meet from
where those "None"s are coming from. However, assume that as a side
effect of how the yield is implemented in Python (and probably it is
even implementation dependent), not as something that should be so.
The correct form for a generator expression, in a simplified manner is:
(<expr> for <variable> in <sequence> [if <expr>])
so, <expr> is evaluated for each value in the <sequence: - not only is yield uneeded, as you should not use it.
Both yield and the send methods are meant to be used in full co-routines, something like:
def doubler():
value = 0
while value < 100:
value = 2 * (yield value)
And you can use it like:
>>> a = doubler()
>>> # Next have to be called once, so the code will run up to the first "yield"
...
>>> a.next()
0
>>> a.send(10)
20
>>> a.send(20)
40
>>> a.send(23)
46
>>> a.send(51)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
>>>
I'm familiar with yield to return a value thanks mostly to this question
but what does yield do when it is on the right side of an assignment?
#coroutine
def protocol(target=None):
while True:
c = (yield)
def coroutine(func):
def start(*args,**kwargs):
cr = func(*args,**kwargs)
cr.next()
return cr
return start
I came across this, on the code samples of this blog, while researching state machines and coroutines.
The yield statement used in a function turns that function into a "generator" (a function that creates an iterator). The resulting iterator is normally resumed by calling next(). However it is possible to send values to the function by calling the method send() instead of next() to resume it:
cr.send(1)
In your example this would assign the value 1 to c each time.
cr.next() is effectively equivalent to cr.send(None)
You can send values to the generator using the send function.
If you execute:
p = protocol()
p.next() # advance to the yield statement, otherwise I can't call send
p.send(5)
then yield will return 5, so inside the generator c will be 5.
Also, if you call p.next(), yield will return None.
You can find more information here.
yield returns a stream of data as per the logic defined within the generator function.
However, send(val) is a way to pass a desired value from outside the generator function.
p.next() doesn't work in python3, next(p) works (built-in) for both python 2,3
p.next() doesn't work with python 3, gives the following error,however it still works in python 2.
Error: 'generator' object has no attribute 'next'
Here's a demonstration:
def fun(li):
if len(li):
val = yield len(li)
print(val)
yield None
g = fun([1,2,3,4,5,6])
next(g) # len(li) i.e. 6 is assigned to val
g.send(8) # 8 is assigned to val