When i was googling for python yield i found something interesting and I never knew before i.e we can pass value to yield to change the next() value. I hope some of the new pythonist might now aware of this and I am not sure how it works too. So, If any body could explain how it works and how the behavior changes when i send a new index to yield using send().
Here is my snippets :
def print_me (count):
i = 0
while i < count:
val = (yield i)
if val is not None:
i = val
else:
i += 1
gen = print_me(10)
for i in gen:
if i == 5:
gen.send(8)
else:
print i
Here is my output:
0
1
2
3
4
9
Your test code is pretty instructive actually. What happens in a generator function like this is that whenever you call next on the generator, code gets executed until it hits a yield statement. The value is yielded and then execution in the generator "pauses". If you .send some data to the generator, that value is returned from yield. If you don't .send anything, yield returns None.
So in your code, when you .send(8), then it sets val = 8 inside your generator (val = yield i). since val is not None, i is set to 8. .send actually executes until the next yield statement (returning that value -- 8). Then things resume as normal (the next number to be yielded is 9).
Related
So I have the following generator function:
def gen(n=5):
for i in range(n):
n = yield n
for i in gen(3):
print(i)
The result:
3
None
None
I understand the first result of yield is 3. Because I assigned 3 to function argument n. But where are the None in the second and third yield coming from? Is it because in the for-loop, yield n returns None and this None is assigned to n in this line: n = yield n?
This is explained in the documentation of yield expressions, especially this part:
The value of the yield expression after resuming depends on the method
which resumed the execution. If next() is used (typically via
either a for or the next() builtin) then the result is None.
Otherwise, if send() is used, then the result will be the value passed
in to that method.
As you use a for loop, n just gets None as a value when resuming after the first yield. So, from now on, n is None, and this is what will be yielded the last two times.
It seems to me that you answered your own question:
because in the for-loop, yield n returns None and None is assigned to n in this line: n = yield n
You can read more at this answer.
What would be the difference between the following two generator functions?
def get_primes(number):
while True:
if is_prime(number):
number = yield number
number += 1
And:
def get_primes(number):
while True:
if is_prime(number):
yield number
number += 1
As far as I understand, I can call them as:
p = get_primes(0)
# first call works for both
next(p) # or p.send(None)
# second call different for both
next(p) # works for second way only
p.send(14) # works for first way only
I think my issue is I don't really understand how send works and how it's setting the value and all.
If you check out the docs, it says:
Resumes the execution and “sends” a value into the generator function. The value argument becomes the result of the current yield expression.
That may sounds a little cryptic, so perhaps in other words:
Using send() the generator resumes where it yielded and the value you have sent is what yield returns (and can be assigned to any variable). You can also try the following code:
def get_num():
number = 1
while True:
print(number)
number = yield number
g = get_num()
g.send(None) # haven't yielded yet, cannot send a value to it
g.send(2)
g.send(5)
It'll return:
1: value we've initially assigned to number
2: we did send(2) and that is what number = yield ... assigned to number, then we continued, looped back to print() and yielded again.
5: Same thing, but we did send(5).
I am trying to understand the behaviour of the yield statement by building a generator which behaves similarly to the 'enumerate' built-in function but I am witnessing inconsistencies depending on how I iterate through it.
def enumerate(sequence, start=0):
n = start
for elem in sequence:
print("Before the 'yield' statement in the generator, n = {}".format(n))
yield n, elem
n += 1
print("After the 'yield' statement in the generator, n = {}".format(n))
My understanding of generators is that the execution of the code will stop once a yield statement has been reached, upon which it returns a value. This matches what I get with the script below.
a = 'foo'
b = enumerate(a)
n1,v1 = next(b)
print('n1 = {}, v1 = {}\n'.format(n1,v1))
n2,v2 = next(b)
print('n2 = {}, v2 = {}'.format(n2,v2))
In this case, the generator seems to stop exactly at the yield statement and resumes in the n+=1 one with the second 'next' statement:
Before the 'yield' statement in the generator, n = 0
n1 = 0, v1 = f
After the 'yield' statement in the generator, n = 1
Before the 'yield' statement in the generator, n = 1
n2 = 1, v2 = o
However, if I use the for loop below, the generator does not seem to stop at the yield statement.
for n,v in enumerate(a[0:1]):
print('n = {}, v = {}'.format(n,v))
This is what I get:
Before the 'yield' statement in the generator, n = 0
n = 0, v = f
After the 'yield' statement in the generator, n = 1
Edit taking comments into account
I realise I'm iterating over just one element, but I was not expecting to see the very last "After the 'yield' statement in the generator" sentence (which appears even if I iterate over ALL the elements.
print('\n\n')
for n,v in enumerate(a):
print('n = {}, v = {}'.format(n,v))
Before the 'yield' statement in the generator, n = 0
n = 0, v = f
After the 'yield' statement in the generator, n = 1
Before the 'yield' statement in the generator, n = 1
n = 1, v = o
After the 'yield' statement in the generator, n = 2
Before the 'yield' statement in the generator, n = 2
n = 2, v = o
After the 'yield' statement in the generator, n = 3
Why does this happen?
The fundamental issue here is that you are confusing the fact that you know when the generator will be exhausted just by looking at it, with the fact that Python can only know by running the code. When Python reaches the yield that you consider to be the last one, it does not actually know that it is the last one. What if your generator looked like this:
def enumeratex(x, start=0):
for elem in x:
yield start, x
start += 1
yield start, None
Here, for reasons no one will ever know, a final None element is returned after the main generator loop. Python would have no way of knowing that the generator is done until you either
Return from the generator.
Raise an error, in which case everything will grind to a halt.
In versions before Python 3.7, generators could raise StopIteration to indicate termination. In fact, a return statement would be equivalent to either raise StopIteration (if returning None) or raise StopIteration(return_value).
So while the exact manner in which you tell Python to end the generator is up to you, you do have to be explicit about it. A yield does not by itself end the generator.
TL;DR
All of the code in a loop in a generator will always run, even after the last value has been yielded because Python can only know it was the last value by actually executing all the code.
the answer lies in understanding what for loop in python does:
It get the iterator (i.e. iter()) of an object and continues until a StopIteration exception is raised.
StopIteration exception is thrown when the code of the generator is done, meaning getting the return statement which exists the function (could be implicit also).
This is why it doesn't stops at yield, it keeps asking for the next yield until the generator is done.
There's a number of good questions about similar matters, e.g.
python generator "send" function purpose?
What does the "yield" keyword do?
Lets get back to a definition of "send":
Resumes the execution and “sends” a value into the generator function.
The value argument becomes the result of the current yield expression.
The send() method returns the next value yielded by the generator, or
raises StopIteration if the generator exits without yielding another
value. When send() is called to start the generator, it must be called
with None as the argument, because there is no yield expression that
could receive the value
But I feel I am missing something important. Here's my example with 3 send calls, including the initial one with a None value just to initialize a generator:
def multiplier():
while True:
m = yield # Line #3
print('m = ' + str(m)) # Line #4
yield str(m * 2) # Line #5
yield str(m * 3) # Line #6
#------------------------
it = multiplier()
print('it.send(None): ')
print(str(it.send(None)))
print('--------------')
print('it.send(10): ')
print(it.send(10))
print('--------------')
print('it.send(100): ')
print(it.send(100))
print('--------------')
And here's an output:
it.send(None):
None
--------------
it.send(10):
m = 10
20
--------------
it.send(100):
30
--------------
Questions:
What happens exactly when I use it.send(10) in a line #5. If we
follow the definition, the generator execution resumes. Generator
accepts 10 as input value and uses it in a current yield
expression. It is yield str(m * 2) in my example, but then how m
is set to 10. When did that happen. Is that because of the
reference between m and yield in a line #3?
What happens in a line #6 it.send(10) and why output is still 30?
Does it mean that the reference in my previous question only worked
once?
Note:
If I've changed my example and added a line m = yield between lines #5 and #6 and then use next(it) after print(it.send(10)) - in that case the output starts to make sense: 20 and 300
Your generator function has three yield expressions, but you're throwing away the value from two of them (lines 5 and 6). If you did something with the values there, you'd see the 100 being used in the function. If you kept running your example, the fifth time you called send would cause the generator to update m to a new value.
Lets walk through the code that does the send calls in your example, and see what the generator is doing at the same time:
it = multiplier()
At this point the generator object has been created and saved to it. The generator code has not started running yet, it's paused at the start of the function's code.
print(str(it.send(None)))
This starts running the generator function's code. The value sent must be None or you'll get an error. The function never sees that value. It's more common to use next to start up a generator, since next(it) is equivalent to it.send(None).
The generator function runs until line 3, where the first yield appears. Since you're not yielding any particular value, the return value from send is None (which gets printed).
print(it.send(10))
This value gets sent to the generator and becomes the value of the yield expression on line 3. So 10 gets stored as m, and the code prints it out on line 4. The generator function keeps running to line 5, where it reaches the next yield expression. Since it's yielding str(m * 2), the calling code gets "20" and prints that.
print(it.send(100))
The 100 value gets sent into the generator as the value of the yield on line 4. That value is ignored, since you're not using the yield as an expression but as a statement. Just like putting 100 on a line by itself, this is perfectly legal, but maybe not very useful. The code goes on to line 5 where it yields str(m * 3), or "30", which gets printed by the calling code.
That's where your driving code stops, but the generator is still alive, and you could send more values to it (and get more values back). The next value you send to the generator would also be ignored, just like the 100 was, but the value after that would end up as a new m value when the while loop in the generator returned to the top and the line 3 yield was reached.
I suspect that some of your confusion with send in this code has to do with the fact that you're using yield both as an expression and as a statement. Probably you don't want to be doing both. Usually you'll either care about all the values being sent into the generator, or you don't care about any of them. If you want to yield several values together (like n*2 and n*3), you could yield a tuple rather than a single item.
Here's a modified version of your code that I think might be easier for you to play with and understand:
def multiplier():
print("top of generator")
m = yield # nothing to yield the first time, just a value we get
print("before loop, m =", m)
while True:
print("top of loop, m =", m)
m = yield m * 2, m * 3 # we always care about the value we're sent
print("bottom of loop, m =", m)
print("calling generator")
it = multiplier()
print("calling next")
next(it) # this is equivalent to it.send(None)
print("sending 10")
print(it.send(10))
print("sending 20")
print(it.send(20))
print("sending 100")
print(it.send(100))
I'm reading about the yield keyword in python, and trying to understand running this sample:
def countfrom(n):
while True:
print "before yield"
yield n
n += 1
print "after yield"
for i in countfrom(10):
print "enter for loop"
if i <= 20:
print i
else:
break
The output is:
before yield
enter for loop
10
after yield
before yield
enter for loop
11
after yield
before yield
enter for loop
12
after yield
before yield
enter for loop
13
after yield
before yield
enter for loop
14
after yield
before yield
enter for loop
15
after yield
before yield
enter for loop
16
after yield
before yield
enter for loop
17
after yield
before yield
enter for loop
18
after yield
before yield
enter for loop
19
after yield
before yield
enter for loop
20
after yield
before yield
enter for loop
It looks like the yield will return the specified value, and will continue runnning the function till the end (in a parallel thread, maybe). Is my understand correct?
If you could answer this without mentioning "generators", I would be thankful, because I'm trying to understand one at a time.
You can think of it as if the function which yields simply "pauses" when it comes across the yield. The next time you call it, it will resume after the yield keeping the state that it was in when it left.
No, there is only a single thread.
Each iteration of the for loop runs your countFrom function until it yields something, or returns. After the yield, the body of the for loop runs again and then, when a new iteration starts, the countFrom function picks up exactly where it left off and runs again until it yields (or returns).
This modified version of your example will helpfully make it clearer what path execution takes.
def countfrom(n):
while n <= 12:
print "before yield, n = ", n
yield n
n += 1
print "after yield, n = ", n
for i in countfrom(10):
print "enter for loop, i = ", i
print i
print "end of for loop iteration, i = ", i
Output
before yield, n = 10
enter for loop, i = 10
10
end of for loop iteration, i = 10
after yield, n = 11
before yield, n = 11
enter for loop, i = 11
11
end of for loop iteration, i = 11
after yield, n = 12
before yield, n = 12
enter for loop, i = 12
12
end of for loop iteration, i = 12
after yield, n = 13
..you cannot explain the meaning of the yield statement without mentioning generators; it would be like trying to explain what a stone is without mentioning rock. That is: the yield statement is the one responsible to transform a normal function into a generator.
While you find it well documented here: http://docs.python.org/reference/simple_stmts.html#the-yield-statement
..the brief explaination of it is:
When a function using the yield statement is called, it returns a "generator iterator", having a .next() method (the standard for iterable objects)
Each time the .next() method of the generator is called (eg. by iterating the object with a for loop), the function is called until the first yield is encountered. Then the function execution is paused and a value is passed as return value of the .next() method.
The next time .next() is called, the function execution is resumed until the next yield, etc. until the function returns something.
Some advantages in doing this are:
less memory usage since memory is allocated just for the currently yielded value, not the whole list of returned values (as it would be by returning a list of values)
"realtime" results return, as they are produced can be passed to the caller without waiting for the generation end (i used that to return output from a running process)
The function countfrom is not run in a parallel thread. What happens here is that whenever the for-construct asks for the next value, the function will execute until it hits a yield statement. When the next value after that is required, the function resumes execution from where it left off.
And while you asked not to mention "generators", they are so intimately linked with yield that it doesn't really make sense to talk about the separately. What your countfrom function actually returns is a "generator object". It returns this object immediately after it is called, so the function body is not executed at all until something (e.g. a for-loop) requests values from the generator using its method .next().
the yield statement stores the value that you yield, until that function is called again.
so if you call that function (with an iterator) it will run the function another time and give you the value.
the point being that it knows where it left off last time
Python runs until it hits a yield and then stops and freezes execution. It's not continuing to run. It's hitting "after" on the next call to countfrom
It's easy to say that without making reference to generators but the fact is yield and generator are inextricably linked. To really understand it you've got to view them as the same topic.
It's easy to show yourself that what I (and others) have said is true by working with the generator from your example in a more manual way.
A function that yields instead of returning really returns a generator. You can then consume that generator by calling next. You are confused because your loop is taking care of all that in the background for you.
Here it is with the internals opened up:
def countfrom(n):
while n <= 12:
print "before yield, n = ", n
yield n
n += 1
print "after yield, n = ", n
your_generator = countfrom(10)
next(your_generator)
print "see the after yield hasn't shown up yet, it's stopped at the first yield"
next(your_generator)
print "now it woke back up and printed the after... and continued through the loop until it got to back to yield"
next(your_generator)
print "rinse and repeate"
Yield with and without for loop:
def f1():
print('f1')
yield 10
print(f'f1 done')
def generator_with_for_loop():
print(f'generator_with_for_loop')
for f1_gen in f1():
print(f'f1_gen={f1_gen}')
def generator_without_for_loop():
print(f'\ngenerator_without_for_loop')
gen = f1()
print(f'f1_gen={gen}')
print(gen.__next__())
try:
print(gen.__next__())
except StopIteration:
print('done')
if __name__ == '__main__':
generator_with_for_loop()
generator_without_for_loop()
"""
generator_with_for_loop
f1
f1_gen=10
f1 done
generator_without_for_loop
f1_gen=<generator object f1 at 0x7fd7201e54a0>
f1
10
f1 done
done
"""