Is there any way to mix recursion and the yield statement? For instance, a infinite number generator (using recursion) would be something like:
def infinity(start):
yield start
# recursion here ...
>>> it = infinity(1)
>>> next(it)
1
>>> next(it)
2
I tried:
def infinity(start):
yield start
infinity(start + 1)
and
def infinity(start):
yield start
yield infinity(start + 1)
But none of them did what I want, the first one stopped after it yielded start and the second one yielded start, then the generator and then stopped.
NOTE: Please, I know you can do this using a while-loop:
def infinity(start):
while True:
yield start
start += 1
I just want to know if this can be done recursively.
Yes, you can do this:
def infinity(start):
yield start
for x in infinity(start + 1):
yield x
This will error out once the maximum recursion depth is reached, though.
Starting from Python 3.3, you'll be able to use
def infinity(start):
yield start
yield from infinity(start + 1)
If you just call your generator function recursively without looping over it or yield from-ing it, all you do is build a new generator, without actually running the function body or yielding anything.
See PEP 380 for further details.
In some cases it might be preferable to use a stack instead of recursion for generators. It should be possible to rewrite a recursive method using a stack and a while loop.
Here's an example of a recursive method which uses a callback and can be rewritten using stack logic:
def traverse_tree(callback):
# Get the root node from somewhere.
root = get_root_node()
def recurse(node):
callback(node)
for child in node.get('children', []):
recurse(child)
recurse(root)
The above method traverses a node tree where each node has a children array which may contain child nodes. As each node is encountered, the callback is issued and the current node is passed to it.
The method could be used this way, printing out some property on each node.
def callback(node):
print(node['id'])
traverse_tree(callback)
Use a stack instead and write the traversal method as a generator
# A stack-based alternative to the traverse_tree method above.
def iternodes():
stack = [get_root_node()]
while stack:
node = stack.pop()
yield node
for child in reversed(node.get('children', [])):
stack.append(child)
(Note that if you want the same traversal order as originally, you need to reverse the order of children because the first child appended to the stack will be the last one popped.)
Now you can get the same behavior as traverse_tree above, but with a generator:
for node in iternodes():
print(node['id'])
This isn't a one-size-fits-all solution but for some generators you might get a nice result substituting stack processing for recursion.
def lprint(a):
if isinstance(a, list):
for i in a:
yield from lprint(i)
else:
yield a
b = [[1, [2, 3], 4], [5, 6, [7, 8, [9]]]]
for i in lprint(b):
print(i)
Related
This question already has answers here:
Return in generator together with yield
(2 answers)
Closed last year.
Why does
yield [cand]
return
lead to different output/behavior than
return [[cand]]
Minimal viable example
uses recursion
the output of the version using yield [1]; return is different than the output of the version using return [[1]]
def foo(i):
if i != 1:
yield [1]
return
yield from foo(i-1)
def bar(i):
if i != 1:
return [[1]]
yield from bar(i-1)
print(list(foo(1))) # [[1]]
print(list(bar(1))) # []
Min viable counter example
does not use recurion
the output of the version using yield [1]; return is the same as the output of the version using return [[1]]
def foo():
yield [1]
return
def foofoo():
yield from foo()
def bar():
return [[1]]
def barbar():
yield from bar()
print(list(foofoo())) # [[1]]
print(list(barbar())) # [[1]]
Full context
I'm solving Leetcode #39: Combination Sum and was wondering why one solution works, but not the other:
Working solution
from functools import cache # requires Python 3.9+
class Solution:
def combinationSum(self, candidates: List[int], target: int) -> List[List[int]]:
#cache
def helper(targ, i=0):
if i == N or targ < (cand := candidates[i]):
return
if targ == cand:
yield [cand]
return
for comb in helper(targ - cand, i):
yield comb + [cand]
yield from helper(targ, i+1)
N = len(candidates)
candidates.sort()
yield from helper(target)
Non-working solution
from functools import cache # requires Python 3.9+
class Solution:
def combinationSum(self, candidates: List[int], target: int) -> List[List[int]]:
#cache
def helper(targ, i=0):
if i == N or targ < (cand := candidates[i]):
return
if targ == cand:
return [[cand]]
for comb in helper(targ - cand, i):
yield comb + [cand]
yield from helper(targ, i+1)
N = len(candidates)
candidates.sort()
yield from helper(target)
Output
On the following input
candidates = [2,3,6,7]
target = 7
print(Solution().combinationSum(candidates, target))
the working solution correctly prints
[[3,2,2],[7]]
while the non-working solution prints
[]
I'm wondering why yield [cand]; return works, but return [[cand]] doesn't.
In a generator function, return just defines the value associated with the StopIteration exception implicitly raised to indicate an iterator is exhausted. It's not produced during iteration, and most iterating constructs (e.g. for loops) intentionally ignore the StopIteration exception (it means the loop is over, you don't care if someone attached random garbage to a message that just means "we're done").
For example, try:
>>> def foo():
... yield 'onlyvalue' # Existence of yield keyword makes this a generator
... return 'returnvalue'
...
>>> f = foo() # Makes a generator object, stores it in f
>>> next(f) # Pull one value from generator
'onlyvalue'
>>> next(f) # There is no other yielded value, so this hits the return; iteration over
--------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
...
StopIteration: 'returnvalue'
As you can see, your return value does get "returned" in a sense (it's not completely discarded), but it's never seen by anything iterating normally, so it's largely useless. Outside of rare cases involving using generators as coroutines (where you're using .send() and .throw() on instances of the generator and manually advancing it with next(genobj)), the return value of a generator won't be seen.
In short, you have to pick one:
Use yield anywhere in a function, and it's a generator (whether or not the code path of a particular call ever reaches a yield) and return just ends generation (while maybe hiding some data in the StopIteration exception). No matter what you do, calling the generator function "returns" a new generator object (which you can loop over until exhausted), it can never return a raw value computed inside the generator function (which doesn't even begin running until you loop over it at least once).
Don't use yield, and return works as expected (because it's not a generator function).
As an example to explain what happens to the return value in normal looping constructs, this is what for x in gen(): effectively expands to a C optimized version of:
__unnamed_iterator = iter(gen())
while True:
try:
x = next(__unnamed_iterator)
except StopIteration: # StopIteration caught here without inspecting it
break # Loop ends, StopIteration exception cleaned even from sys.exc_info() to avoid possible reference cycles
# body of loop goes here
# Outside of loop, there is no StopIteration object left
As you can see, the expanded form of the for loop has to look for a StopIteration to indicate the loop is over, but it doesn't use it. And for anything that's not a generator, the StopIteration never has any associated values; the for loop has no way to report them even if it did (it has to end the loop when it's told iteration is over, and the arguments to StopIteration are explicitly not part of the values iterated anyway). Anything else that consumes the generator (e.g. calling list on it) is doing roughly the same thing as the for loop, ignoring the StopIteration in the same way; nothing except code that specifically expects generators (as opposed to more generalized iterables and iterators) will ever bother to inspect the StopIteration object (at the C layer, there are optimizations that StopIteration objects aren't even produced by most iterators; they return NULL and leave the set exception empty, which all iterator protocol using things know is equivalent to returning NULL and setting a StopIteration object, so for anything but a generator, there isn't even an exception to inspect much of the time).
0
/ \
1 2
/ \ /|\
3 4 5 6 7
I'm trying to return the nodes without children (3,4,5,6,7) from an object using a recursive function.
It works using a global variable and this function:
def find(self, nodes):
if not hasattr(self, 'children'):
nodes.append(self)
else:
for i in self.children:
i.find(nodes)
nodes = []
node.find(nodes) # can be any node (0,1,2,3,4, etc.)
print(nodes)
But I would like to use return in my function. I tried something like this:
def find2(self):
if not hasattr(self, 'children'):
return self
else:
for i in self.children:
return i.find2()
nodes = root.find2()
print(nodes)
But I get only 1 node. I also tried passing an array like on this post: PYTHON Return a list from a recursive function. But I don't get the result I want because tree structure (I guess)...
I'm stuck, can you help me? How to "return the result for each iteration of the recursive function to a variable"? Thank you
This is the input example:
0
/ \
1 2
/ \ /|\
3 4 5 6 7
Think about what you want the (recursive) function to return for each of the nodes:
when called on the root (0), it should return the full result (3, 4, 5, 6, 7)
when called on a leaf node (e.g. 5), it should return that node (5)
for any other node (e.g. 1), it does as if that was a root node of a smaller tree
So, it sometimes returns one result and sometimes many.
The error in your code is here, because the function ends on the first return, it does not return many:
for i in self.children:
return i.find2()
There are two solutions:
make a generator function, or
make a function which returns a list (make it return a list always, even if it has just one element!)
So, option 1:
def find(self):
if not hasattr(self, 'children'):
yield self
else:
for child in self.children:
yield from child.find()
option 2:
def find(self):
if not hasattr(self, 'children'):
return [self]
else:
rtn = []
for child in self.children:
for result in child.find():
rtn.append(result)
return rtn
I prefer the generator. Additionally, I don't particularly like the fact that some of your nodes have a children attribute, while others do not. I would make sure all of them had children, which could be empty or non-empty. Then, the code becomes:
def find_leaves(self):
if self.children:
for child in self.children:
yield from child.find_leaves()
else:
yield self
Note that in your base case, you return one node. That node then get propagated back in the recursion, and eventually returned. I.e. the node you are returned is the first one without children you hit. There are multiple ways to fix this issue (e.g. by returning and concatenating lists), but I think a natural way to do this is to make your function a generator. To do this, simply replace the return by a yield. Then, the return value of the function functions as an iterator, with each iteration yielding the next value that would be "returned" in execution until the function terminates.
You've not provided enough code for me to make a runnable and testable example but here's my guess of what you seek:
def find(self):
if not hasattr(self, 'children'):
return [self]
nodes = []
for child in self.children:
return nodes.extend(child.find())
return nodes
# ...
nodes = root.find()
print(nodes)
I have a simple tree class and I need to be able to iterate over the leaf nodes only.
I add the data to my tree as follows:
p = ParamTree()
p.add('system.screen.timeout')
p.add('system.screen.lock.enabled')
p.add('system.screen.lock.code')
and I want to be able to get timeout, enabled and code in sequence.
If I write a function to print the values, it works as expected:
def print_nodes(tree):
for node in tree.nodes:
if node.has_children():
print_nodes(node)
else:
print(node)
Output:
>>> print_nodes(p)
timeout
enabled
code
How would I go about implementing a generator that does the same thing? I tried replacing print() with yield, but it doesn't work.
def yield_nodes(tree):
for node in tree.nodes:
if node.has_children():
yield_nodes(node)
else:
yield node
Output:
>>> g = yield_nodes(p)
>>> for n in g:
... print(n)
...
>>>
I'm going to assume your yield_nodes generator to be written as follows:
def yield_nodes(tree):
for node in tree.nodes:
if node.has_children():
yield_nodes(node)
else:
yield node
As you might notice, calling yield_nodes returns a generator (that you could iterate over), but you are not actually doing anything with it. My proposed solution would be the following:
def yield_nodes(tree):
for node in tree.nodes:
if node.has_children():
yield from yield_nodes(node)
else:
yield node
I implemented graph traversal as a generator function which yields the node being visited.
Sometimes the user needs to tell the traversal function that the edges outgoing from a particular node shouldn't be followed; in order to support that, the traversal checks the value sent back to it (using generator send() method), and if it's True, regards the node as a leaf for traversal purposes.
The problem is that the simplest user loop is kinda long:
# simplified thanks to #tobias_k
# bfs is the traversal generator function
traversal = bfs(g, start_node)
try:
n = next(traversal)
while True:
# process(n) returns True if don't want to follow edges out of n
n = traversal.send(process(n))
except StopIteration:
pass
Is there any way to improve this?
I thought something like this should work:
for n in bfs(g, start_node):
???.send(process(n))
but I feel I'm missing the knowledge of some python syntax.
I don't see a way to do this in a regular for loop. However, you could create another generator, that iterates another generator, using some "follow-function" to determine whether to follow the current element, thus encapsulating the tricky parts of your code into a separate function.
def checking_generator(generator, follow_function):
try:
x = next(generator)
while True:
yield x
x = generator.send(follow_function(x))
except StopIteration:
pass
for n in checking_generator(bfs(g, start_node), process):
print(n)
I discovered that my question would have had a one-line answer, using the extended "continue" statement proposed in the earlier version of PEP 342:
for n in bfs(g, start_node):
continue process(n)
However, while PEP 342 was accepted, that particular feature was withdrawn after this June 2005 discussion between Raymond and Guido:
Raymond Hettinger said:
Let me go on record as a strong -1 for "continue EXPR". The
for-loop is our most basic construct and is easily understood in its
present form. The same can be said for "continue" and "break" which
have the added advantage of a near zero learning curve for people
migrating from other languages.
Any urge to complicate these basic statements should be seriously
scrutinized and held to high standards of clarity, explainability,
obviousness, usefulness, and necessity. IMO, it fails most of those
tests.
I would not look forward to explaining "continue EXPR" in the tutorial
and think it would stand out as an anti-feature.
[...] The correct argument against "continue EXPR" is that there
are no use cases yet; if there were a good use case, the explanation
would follow easily.
Guido
If python core developers have since changed their mind about the usefulness of extended "continue", perhaps this could be reintroduced into a future PEP. But, given a nearly identical use case as in this question was already discussed in the quoted thread, and wasn't found persuasive, it seems unlikely.
To simplify the client code, you could use an ordinary bsf() generator and check node.isleaf attribute in it:
for node in bfs(g, start_node):
node.isleaf = process(node) # don't follow if `process()` returns True
The disadvantage is that node is mutable. Or you have to pass a shared data structure that tracks leaf nodes: leaf[node] = process(node) where leaf dictionary is passed into bfs() earlier.
If you want to use .send() method explicitly; you have to handle StopIteration. See PEP 479 -- Change StopIteration handling inside generators. You could hide it in a helper function:
def traverse(tree_generator, visitor):
try:
node = next(tree_generator)
while True:
node = tree_generator.send(visitor(node))
except StopIteration:
pass
Example:
traverse(bfs(g, start_node), process)
I don't see this as a frequent use case, consider this as the original generator:
def original_gen():
for x in range(10):
should_break = yield x
if should_break:
break
If the value of should_break is always calculated based on some function call with x then why not just write the generator like this:
def processing_gen(check_f):
for x in range(10):
yield x
should_break = check_f(x)
if should_break:
break
However I usually think of the code that processes the generated values as being written inside the loop (otherwise what is the point of having a loop at all?)
What it really seems you want to do is create a generator where calling the __next__ method really implies send(process(LAST_VALUE)) which can be implemented with a class:
class Followup_generator(): #feel free to use a better name
def __init__(self,generator,following_function):
self.gen = generator
self.process_f = following_function
def __iter__(self):
return self
def __next__(self):
if hasattr(self,"last_value"):
return self.send(self.process_f(self.last_value))
else:
self.last_value = next(self.gen)
return self.last_value
def send(self,arg):
self.last_value = self.gen.send(arg)
return self.last_value
def __getattr__(self,attr):
"forward other lookups to the generator (.throw etc.)"
return getattr(self.gen, attr)
# call signature is the exact same as #tobias_k's checking_generator
traversal = Followup_generator(bfs(g, start_node), process)
for n in traversal:
print(n)
n = traversal.send(DATA) #you'd be able to send extra values to it
However this still doesn't see this as frequently used, I'd be perfectly fine with a while loop, although I'd put the .send call at the top:
traversal = bfs(g, start_node)
send_value = None
while True:
n = traversal.send(send_value)
#code for loop, ending in calculating the next send_value
send_value = process(n)
And you might wrap that in a try: ... except StopIteration:pass although I find that simply waiting for an error to raise is better expressed with a context manager:
class Catch:
def __init__(self,exc_type):
if issubclass(exc_type,BaseException):
self.catch_type = exc_type
else:
raise TypeError("can only catch Exceptions")
def __enter__(self):
return self
def __exit__(self,exc_type,err, tb):
if issubclass(exc_type, self.catch_type):
self.err = err
return True
with Catch(StopIteration):
traversal = bfs(g, start_node)
send_value = None
while True:
n = traversal.send(send_value)
#code for loop, ending in calculating the next send_value
send_value = process(n)
Probably this is the answer to the question from the thread's topic.
Take a look at the additional empty yields statements inside the traversal function and custom send function, that does the magical job.
# tested with Python 3.7
def traversal(n):
for i in range(n):
yield i, '%s[%s] %s' % (' ' * (4 - n), n, i)
stop = yield
if stop:
yield # here's the first part of the magic
else:
yield # the same as above
yield from traversal(int(n / 2))
def send(generator, value):
next(generator) # here's the second part of the magic
generator.send(value)
g = traversal(4)
for i, (num, msg) in enumerate(g):
print('>', i, msg)
stop = num % 2 == 0
send(g, stop)
I've written a small class SettableGenerator which uses a method to receive the value to be send and then forwards it to the actual generator when __next__ is invoked.
With this you can write:
gen = SettableGenerator(bfs(g, start_node))
for n in gen:
gen.set(process(n))
Let's consider the following generator. It generates numbers from 0 to 9. For every generated number, it gets an input and stores it into ret:
def count_to_nine():
# Output: numbers from 0 to 9
# Input: converted numbers
ret = []
for i in range(10):
# Yield a number, get something back
val = (yield i)
# Remember that "something"
ret.append(val)
return ret
You can, indeed, iterate it using next() + send(),
but the best way is to iterate using send() alone:
g = count_to_nine()
value = None # to make sure that the first send() gives a None
while True:
value = g.send(value) # send the previously generated value, get a new one
value = f'#{value}'
Here's the result:
StopIteration: ['#0', '#1', '#2', '#3', '#4', '#5', '#6', '#7', '#8', '#9']
If you want that output, catch the StopIteration and get the result from it.
Cheers!
I am trying to figure out how I can traverse linked list in Python using Recursion.
I know how to traverse linked-lists using common loops such as:
item_cur = my_linked_list.first
while item_cur is not None:
print(item_cur.item)
item_cur = item_cur.next
I was wondering how I could turn this loop into a recursive step.
Thanks
You could do something like this:
def print_linked_list(item):
# base case
if item == None:
return
# lets print the current node
print(item.item)
# print the next nodes
print_linked_list(item.next)
It looks like your linked list has two kinds of parts. You have list nodes, with next and item attributes, and a wrapper object which has an attribute pointing to a the first node. To recursively print the list, you'll want to have two functions, one to handle the wrapper and a helper function to do the recursive processing of the nodes.
def print_list(linked_list): # Non-recursive outer function. You might want
_print_list_helper(linked_list.first) # to update it to handle empty lists nicely!
def _print_list_helper(node): # Recursive helper function, gets passed a
if node is not None: # "node", rather than the list wrapper object.
print(node.item)
_print_list_helper(node.next) # Base case, when None is passed, does nothing
Try this.
class Node:
def __init__(self,val,nxt):
self.val = val
self.nxt = nxt
def reverse(node):
if not node.nxt:
print node.val
return
reverse(node.nxt)
print node.val
n0 = Node(4,None)
n1 = Node(3,n0)
n2 = Node(2,n1)
n3 = Node(1,n2)
reverse(n3)