I'm familiar with yield to return a value thanks mostly to this question
but what does yield do when it is on the right side of an assignment?
#coroutine
def protocol(target=None):
while True:
c = (yield)
def coroutine(func):
def start(*args,**kwargs):
cr = func(*args,**kwargs)
cr.next()
return cr
return start
I came across this, on the code samples of this blog, while researching state machines and coroutines.
The yield statement used in a function turns that function into a "generator" (a function that creates an iterator). The resulting iterator is normally resumed by calling next(). However it is possible to send values to the function by calling the method send() instead of next() to resume it:
cr.send(1)
In your example this would assign the value 1 to c each time.
cr.next() is effectively equivalent to cr.send(None)
You can send values to the generator using the send function.
If you execute:
p = protocol()
p.next() # advance to the yield statement, otherwise I can't call send
p.send(5)
then yield will return 5, so inside the generator c will be 5.
Also, if you call p.next(), yield will return None.
You can find more information here.
yield returns a stream of data as per the logic defined within the generator function.
However, send(val) is a way to pass a desired value from outside the generator function.
p.next() doesn't work in python3, next(p) works (built-in) for both python 2,3
p.next() doesn't work with python 3, gives the following error,however it still works in python 2.
Error: 'generator' object has no attribute 'next'
Here's a demonstration:
def fun(li):
if len(li):
val = yield len(li)
print(val)
yield None
g = fun([1,2,3,4,5,6])
next(g) # len(li) i.e. 6 is assigned to val
g.send(8) # 8 is assigned to val
Related
This question already has answers here:
Return in generator together with yield
(2 answers)
Closed last year.
Why does
yield [cand]
return
lead to different output/behavior than
return [[cand]]
Minimal viable example
uses recursion
the output of the version using yield [1]; return is different than the output of the version using return [[1]]
def foo(i):
if i != 1:
yield [1]
return
yield from foo(i-1)
def bar(i):
if i != 1:
return [[1]]
yield from bar(i-1)
print(list(foo(1))) # [[1]]
print(list(bar(1))) # []
Min viable counter example
does not use recurion
the output of the version using yield [1]; return is the same as the output of the version using return [[1]]
def foo():
yield [1]
return
def foofoo():
yield from foo()
def bar():
return [[1]]
def barbar():
yield from bar()
print(list(foofoo())) # [[1]]
print(list(barbar())) # [[1]]
Full context
I'm solving Leetcode #39: Combination Sum and was wondering why one solution works, but not the other:
Working solution
from functools import cache # requires Python 3.9+
class Solution:
def combinationSum(self, candidates: List[int], target: int) -> List[List[int]]:
#cache
def helper(targ, i=0):
if i == N or targ < (cand := candidates[i]):
return
if targ == cand:
yield [cand]
return
for comb in helper(targ - cand, i):
yield comb + [cand]
yield from helper(targ, i+1)
N = len(candidates)
candidates.sort()
yield from helper(target)
Non-working solution
from functools import cache # requires Python 3.9+
class Solution:
def combinationSum(self, candidates: List[int], target: int) -> List[List[int]]:
#cache
def helper(targ, i=0):
if i == N or targ < (cand := candidates[i]):
return
if targ == cand:
return [[cand]]
for comb in helper(targ - cand, i):
yield comb + [cand]
yield from helper(targ, i+1)
N = len(candidates)
candidates.sort()
yield from helper(target)
Output
On the following input
candidates = [2,3,6,7]
target = 7
print(Solution().combinationSum(candidates, target))
the working solution correctly prints
[[3,2,2],[7]]
while the non-working solution prints
[]
I'm wondering why yield [cand]; return works, but return [[cand]] doesn't.
In a generator function, return just defines the value associated with the StopIteration exception implicitly raised to indicate an iterator is exhausted. It's not produced during iteration, and most iterating constructs (e.g. for loops) intentionally ignore the StopIteration exception (it means the loop is over, you don't care if someone attached random garbage to a message that just means "we're done").
For example, try:
>>> def foo():
... yield 'onlyvalue' # Existence of yield keyword makes this a generator
... return 'returnvalue'
...
>>> f = foo() # Makes a generator object, stores it in f
>>> next(f) # Pull one value from generator
'onlyvalue'
>>> next(f) # There is no other yielded value, so this hits the return; iteration over
--------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
...
StopIteration: 'returnvalue'
As you can see, your return value does get "returned" in a sense (it's not completely discarded), but it's never seen by anything iterating normally, so it's largely useless. Outside of rare cases involving using generators as coroutines (where you're using .send() and .throw() on instances of the generator and manually advancing it with next(genobj)), the return value of a generator won't be seen.
In short, you have to pick one:
Use yield anywhere in a function, and it's a generator (whether or not the code path of a particular call ever reaches a yield) and return just ends generation (while maybe hiding some data in the StopIteration exception). No matter what you do, calling the generator function "returns" a new generator object (which you can loop over until exhausted), it can never return a raw value computed inside the generator function (which doesn't even begin running until you loop over it at least once).
Don't use yield, and return works as expected (because it's not a generator function).
As an example to explain what happens to the return value in normal looping constructs, this is what for x in gen(): effectively expands to a C optimized version of:
__unnamed_iterator = iter(gen())
while True:
try:
x = next(__unnamed_iterator)
except StopIteration: # StopIteration caught here without inspecting it
break # Loop ends, StopIteration exception cleaned even from sys.exc_info() to avoid possible reference cycles
# body of loop goes here
# Outside of loop, there is no StopIteration object left
As you can see, the expanded form of the for loop has to look for a StopIteration to indicate the loop is over, but it doesn't use it. And for anything that's not a generator, the StopIteration never has any associated values; the for loop has no way to report them even if it did (it has to end the loop when it's told iteration is over, and the arguments to StopIteration are explicitly not part of the values iterated anyway). Anything else that consumes the generator (e.g. calling list on it) is doing roughly the same thing as the for loop, ignoring the StopIteration in the same way; nothing except code that specifically expects generators (as opposed to more generalized iterables and iterators) will ever bother to inspect the StopIteration object (at the C layer, there are optimizations that StopIteration objects aren't even produced by most iterators; they return NULL and leave the set exception empty, which all iterator protocol using things know is equivalent to returning NULL and setting a StopIteration object, so for anything but a generator, there isn't even an exception to inspect much of the time).
I would like to have a function that can, optionally, return or yield the result.
Here is an example.
def f(option=True):
...
for...:
if option:
yield result
else:
results.append(result)
if not option:
return results
Of course, this doesn't work, I have tried with python3 and I always get a generator no matter what option value I set.
As far I have understood, python checks the body of the function and if a yield is present, then the result will be a generator.
Is there any way to get around this and make a function that can return or yield at will?
You can't. Any use of yield makes the function a generator.
You could wrap your function with one that uses list() to store all values the generator produces in a list object and returns that:
def f_wrapper(option=True):
gen = f()
if option:
return gen # return the generator unchanged
return list(gen) # return all values of the generator as a list
However, generally speaking, this is bad design. Don't have your functions alter behaviour like this; stick to one return type (a generator or an object) and don't have it switch between the two.
Consider splitting this into two functions instead:
def f():
yield result
def f_as_list():
return list(f())
and use either f() if you need the generator, and f_as_list() if you want to have a list instead.
Since list(), (and next() to access just one value of a generator) are built-in functions, you rarely need to use a wrapper. Just call those functions directly:
# access elements one by one
gen = f()
one_value = next(gen)
# convert the generator to a list
all_values = list(f())
What about this?
def make_f_or_generator(option):
def f():
return "I am a function."
def g():
yield "I am a generator."
if option:
return f
else:
return g
This gives you at least the choice to create a function or a generator.
class based approach
class FunctionAndGenerator:
def __init__(self):
self.counter = 0
def __iter__(self):
return self
# You need a variable to indicate if dunder next should return the string or raise StopIteration.
# Raising StopIteration will stop the loop from iterating more.
# You'll have to teach next to raise StopIteration at some point
def __next__(self):
self.counter += 1
if self.counter > 1 :
raise StopIteration
return f"I'm a generator and I've generated {self.counter} times"
def __call__(self):
return "I'm a function"
x = FunctionAndGenerator()
print(x())
for i in x:
print(i)
I'm a function
I'm a generator and I've generated 1 times
[Program finished]
I have a simple coroutine
def id():
while True:
x = yield
yield x
I can use it to create a generator and prime it with next
gen = id()
next(gen)
for x in gen:
print(x)
This will print None forever. My intuition is that the generator from id would only yield a value once a value has been sent to it, but in this case it's producing None values all by itself.
Can someone explain this behaviour to me? Is the x = yield statement defaulting x to None?
As others have pointed out, calling next(gen) is equivalent to calling gen.send(None). You can see this if you look at the C code for generator objects:
PyDoc_STRVAR(send_doc,
"send(arg) -> send 'arg' into generator,\n\
return next yielded value or raise StopIteration.");
PyObject *
_PyGen_Send(PyGenObject *gen, PyObject *arg)
{
return gen_send_ex(gen, arg, 0);
}
...
static PyObject *
gen_iternext(PyGenObject *gen)
{
return gen_send_ex(gen, NULL, 0);
}
As you can see, iternext (which is what gets called when you call next(generator_object)), calls the exact same method as generator_object.send, only it always passes in a NULL object. Consider that your for x in gen loop is basically the equivalent of doing this:
iterable = iter(gen):
while True:
try:
x = next(iterable)
except StopIteration:
break
print(x)
And you see what the issue is.
It's important to realize that coroutines and generators are really completely separate concepts, even though they're both implemented using the yield keyword. In general, you shouldn't be iterating over a coroutine, and in general you don't need/want to send things into a generator. I highly recommend David Beazley's PyCon coroutine tutorial for more on this and coroutines in general.
The for statement calls the next method of the generator instead of the send method. According to the documentation (Python 2.7) this results in the yield statement returning None:
generator.next()
Starts the execution of a generator function or
resumes it at the last executed yield expression. When a generator
function is resumed with a next() method, the current yield expression
always evaluates to None.
yield is an expression that is only valid inside of functions. The value of a yield expression is None. So your function yields None, then assigns the result of yield (None) to x, then yields x. To demonstrate what is happening better let's yield something from the first yield and see what happens.
def id():
while True:
x = yield 5
yield x
This will now generate the following sequence:
5
None
5
None
5
None
5
None
...
I was experimenting with creating a function that returns an object or works as a generator.
This is a bad idea because, as a best practice, you want functions to reliably return the same types of values, but in the interest of science...
I'm using Python 2, and so range returns a list, and xrange is an iterable (that interestingly also provides a __len__).
def xr(start, stop=None, step=1, gen=True):
if stop is None:
start, stop = 0, start
if gen == True:
for i in xrange(start, stop, step):
yield i
else:
return range(start, stop, step)
I get this error:
File "<stdin>", line 8
SyntaxError: 'return' with argument inside generator
Questions:
Why (beyond the obvious "you can't have both yield and return in a function," assuming that's right) does it do this? Looking at my code, it is not readily apparent why it would be bad to do this.
How would I attempt to get around this? I know I can return xrange instead of yielding each item from xrange, and so I could return a generator created in another function, but is there a better way?
How about using generator expression?
>>> def xr(start, stop=None, step=1, gen=True):
... if stop is None:
... start, stop = 0, start
... if gen == True:
... return (i for i in xrange(start, stop, step)) # <----
... else:
... return range(start, stop, step)
...
>>> xr(2, gen=False)
[0, 1]
>>> xr(2, gen=True)
<generator object <genexpr> at 0x0000000002C1C828>
>>> list(xr(2, gen=True))
[0, 1]
BTW, I would rather define a generator function only. Then use list(xr(..)) if I need a list.
UPDATE Alternatively you can use iter(xrange(start, stop, step)) instead of the generator expression as #DSM commented. See Built-in functions -- iter.
falsetru has given you a way to have a function that returns a generator or a list. I'm answering your other question about why you can't have a return and a yield in the same function.
When you call a generator function, it doesn't actually do anything immediately (see this question). It doesn't execute the function body, but waits until you start iterating over it (or call next on it). Therefore, Python has to know if the function is a generator function or not at the beginning, when you call it, to know whether to run the function body or not.
It doesn't make sense to then have it return some value, because if it's a generator function what it returns is the generator (i.e., the thing you iterate over). If you could have a return inside a generator function, it wouldn't be reached when you called the function, so the function would have to "spontaneously" return a value at some later point (when the return statement was reached as the generator was consumed), which would either be the same as yielding a value at that time, or would be something bizarre and confusing.
I agree with falsetru that it's probably not a great idea to have a function that sometimes returns a generator and sometimes a list. Just call list on the generator if you want a list.
You cannot have both because interpreter - when it encounters keyword yield - treats the function as a generator function. And obviously you cannot have a return statement in a generator function.
Additional point - you don't use generator functions directly, you use them to create (I am tempted to say instantiate) generators.
Simple example
def infinite():
cnt = 1
while True:
yield cnt
cnt += 1
infinite_gen = infinite()
infinite is generator function and infinite_gen is generator
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
The Python yield keyword explained
Can someone explain to me what the yield statement actually does in this bit of code here:
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a+b
for number in fibonacci(): # Use the generator as an iterator; print number
What I understand so far is, we are defining a function finonacci(), with no parameters?
inside the function we are defining a and b equal to 0 and 1, next, while this is true, we are yielding a. What is this actually doing? Furthermore, while yielding a? a is now equal to b, while b is now equal to a + b.
Next question, for number in fibonacci(), does this mean for every number in the function or what? I'm equally stumped on what yield and 'for number' are actually doing. Obviously I am aware that it means for every number in fibonacci() print number. Am I actually defining number without knowing it?
Thanks, sorry if I'm not clear. BTW, it's for project Euler, if I knew how to program well this would be a breeze but I'm trying to learn this on the fly.
Using yield makes the function a generator.
The generator will continue to yield the a variable on each loop, waiting until the generator's next() method is called to continue on to the next loop iteration.
Or, until you return or StopIteration is raised.
Slightly modified to show use of StopIteration:
>>> def fib():
... a = 0
... b = 1
... while True:
... yield a
... a = b
... b += a
... if a > 100:
... raise StopIteration
...
>>>
>>> for value in fib():
... print value
...
0
1
2
4
8
16
32
64
>>>
>>> # assign the resulting object to 'generator'
>>> generator = fib()
>>> generator.next()
0
>>> generator.next()
1
>>> for value in generator:
... print value
...
2
4
8
16
32
64
>>>
Generators have a special property of being iterables which do not consume memories for their values.
They do this by calculating the new value, when it is required while being iterated.
i.e.
def f():
a = 2
yield a
a += 1
for ele in f():
print ele
would print
2
So you are using a function as an iterable that keeps returning values.
This is especially useful when you require heavy memory usage, and so you cannot afford the use of a list comprehension
i.e.
li = [ele*10 for ele in range(10)]
takes 10 memory spaces for ints as a list
but if you simple want to iterate over it, not access it individually
it would be very memory efficient to instead use
def f():
i=0
while i<10
yield i*10
i += 1
which would use 1 memory space as i keeps being reused
a short cut for this is
ge = (i*10 for i in range(10))
you can do any of the following
for ele in f():
for ele in li:
for ele in ge:
to obtain equivalent results
When the code calls fibonacci a special generator object is created. Please note, that no code gets executed - only a generator object is returned. When you are later calling its next method, the function executes until it encounters a yield statement. The object that is supplied to yield is returned. When you call next method again the function executes again until it encounters a yield. When there are no more yield statements and the end of function is reached, a StopIteration exception is raised.
Please note that the objects inside the function are preserved between the calls to next. It means, when the code continues execution on the next loop, all the objects that were in the scope from which yield was called have their values from the point where a previous next call returned.
The cool thing about generators is that they allow convenient iteration with for loops.
The for loop obtains a generator from the result of fibonacci call and then executes the loop retrieving elements using next method of generatior object until StopIteration exception is encountered.
This answer is a great explanation of the yield statement, and also of iterators and generators.
Specifically here, the first call to fibonaci() will initialize a to 0, b to 1, enter the while loop and return a.
Any next call will start after the yield statement, affect b to a, a+b to b, and then go to the next iteration of the while statement, reach again the yield statement, and return a again.