I'd like to write a generator which can accept a limited number of inputs from yields and then gracefully handle further inputs. What's the best way of catching StopIteration?
I've tried wrapping by inner generator with an outer generator using a yield from expression inside a try-except block, but StopIteration gets raised anyway...
def limited_writer(max_writes):
for i in range(max_writes):
x = yield
print(x) #
def graceful_writer(l):
try:
yield from l
except StopIteration:
# Ideally will have additional handling logic here
raise Exception("Tried to write too much")
l_w = limited_writer(4)
g_w = graceful_writer(w)
g_w.send(None)
for i in range(5):
g_w.send(i)
I'd like the above to raise Exception (but more generally provide a nice way of handling providing too much data), but in fact it still raises StopIteration. What's the best solution?
If you want graceful_writer to keep accepting data that is sent to it via its .send() method, it needs to keep on yielding indefinitely. The try/except block you currently have doesn't actually do anything, the yield from statement already absorbs the StopIteration from limited_writer. The one you are seeing at the top level comes from graceful_writer itself, when it reaches the end of its code.
To avoid that, try using an infinite loop, like this:
def graceful_writer(gen):
yield from gen # send values to wrapped generator for as long as it will take them
while True:
yield # then loop forever, discarding any additional values sent in
Related
I have a decorator:
def remediation_decorator(dec_mthd):
def new_func(*args, **kwargs):
try:
return dec_mthd(*args, **kwargs)
except (KeyError, HTTPError) as err:
print(f'error = {err}... call the remediation function')
return new_func
Inside the generator function, another function is called to raise specific exceptions under certain conditions:
def check(number):
if number == 1:
raise HTTPError
if number == 2:
raise KeyError
This generator function is decorated like so:
#remediation_decorator
def dec_mthd_b(number):
check(number)
for i in range(0,3):
yield i+1
When an exception is raised by the check function, the decorator's except is not hit.
[ins] In [16]: dec_mthd_b(1)
Out[16]: <generator object dec_mthd_b at 0x10e79cc80>
It appears to behave like this because it's a generator function - from Yield expressions:
When a generator function is called, it returns an iterator known as a generator.
(I wonder whether to take this in the literal sense 'it returns the iterator first irrespective of other logic in the function', hence why check() does not raise the exception?)
and,
By suspended, we mean that all local state is retained, including the current bindings of local variables, the instruction pointer, the internal evaluation stack, and the state of any exception handling.
Have I understood this correctly? Please can anyone explain this further?
Yes you got it.
#remediation_decorator is a Syntactic Sugar in python for decorators. I'm going to use the verbose(?) form:
def dec_mthd_b(number):
check(number)
for i in range(0, 3):
yield i + 1
dec_mthd_b = remediation_decorator(dec_mthd_b)
What does this line do ? remediation_decorator is your decorator, it gives you the inner function, in your case new_func.
What is new_func ? It is a normal function, when you call it, it runs the body of the function.
What will return from new_func ? dec_mthd(*args, **kwargs).
Here dec_mthd points to dec_mthd_b and it is a function again. But when you call it, since dec_mthd_b has yield` keyword inside, it gives you back the generator object.
Now here is the point. The body of your inner function, here new_func, is executed without any problem. You got your generator object back. No error is raised...
# this is the result of calling inner function, which gives you the generator object
gen = dec_mthd_b(1)
# Here is where you're going to face the exceptions.
for i in gen:
print(i)
What will happen in the for loop ? Python runs the body of the dec_mthd_b. The error is raised from there...
So in order to catch the exceptions, you have two options, either catch it inside the dec_mthd_b, or in the last for loop.
You have understood it correctly. The below code will throw an exception. When a generator is created nothing gets executed. You need to fetch the next element, hence get the value from generator, then it will raise the exception.
g = dec_mthd_b(1)
next(g) #throws httperror
In fact thats how iteration is done, we repeatadly call next method, until IterationError exception is thrown.
I have a method want to call repeatedly to iterate over, which will raise a StopIteration when it's done (in this case an instance of pyarrow.csv.CSVStreamingReader looping over a large file). I can use it in a for loop like this:
def batch_generator():
while True:
try:
yield reader.read_next_batch()
except StopIteration:
return
for batch in batch_generator():
writer.write_table(batch)
It can be done in a generic way with a user-defined function:
def make_generator(f):
def gen():
while True:
try:
yield f()
except StopIteration:
return
return gen()
for batch in make_generator(reader.read_next_batch):
writer.write_table(batch)
...but I wondered if something like this was possible with standard library functions or with some obscure syntax?
I would assume that the normal iter() function with its second argument should do what you want. As in:
for batch in iter(reader.read_next_batch, None):
...
The answer to your underlying question of how to iterate a CSVStreamingReader is: The CSVStreamingReader is iterable and does just the thing you want:
reader = pyarrow.csv.open_csv(...)
for batch in reader:
...
In general it is really rare for python libraries to return "iterable" things that are not python-iterable. That is always a sensible first thing to try.
I have a scenario with Tornado where I have a coroutine that is called from a non-coroutine or without yielding, yet I need to propagate the exception back.
Imagine the following methods:
#gen.coroutine
def create_exception(with_yield):
if with_yield:
yield exception_coroutine()
else:
exception_coroutine()
#gen.coroutine
def exception_coroutine():
raise RuntimeError('boom')
def no_coroutine_create_exception(with_yield):
if with_yield:
yield create_exception(with_yield)
else:
create_exception(with_yield)
Calling:
try:
# Throws exception
yield create_exception(True)
except Exception as e:
print(e)
will properly raise the exception. However, none of the following raise the exception :
try:
# none of these throw the exception at this level
yield create_exception(False)
no_coroutine_create_exception(True)
no_coroutine_create_exception(False)
except Exception as e:
print('This is never hit)
The latter are variants similar to what my problem is - I have code outside my control calling coroutines without using yield. In some cases, they are not coroutines themselves. Regardless of which scenario, it means that any exceptions they generate are swallowed until Tornado returns them as "future exception not received."
This is pretty contrary to Tornado's intent, their documentation basically states you need to do yield/coroutine through the entire stack in order for it to work as I'm desiring without hackery/trickery.
I can change the way the exception is raised (ie modify exception_coroutine). But I cannot change several of the intermediate methods.
Is there something I can do in order to force the exception to be raised throughout the Tornado stack, even if it is not properly yielded? Basically to properly raise the exception in all of the last three situations?
This is complicated because I cannot change the code that is causing this situation. I can only change exception_coroutine for example in the above.
What you're asking for is impossible in Python because the decision to yield or not is made by the calling function after the coroutine has finished. The coroutine must return without raising an exception so it can be yielded, and after that it is no longer possible for it to raise an exception into the caller's context in the event that the Future is not yielded.
The best you can do is detect the garbage collection of a Future, but this can't do anything but log (this is how the "future exception not retrieved" message works)
If you're curious why this isn't working, it's because no_coroutine_create_exception contains a yield statement. Therefore it's a generator function, and calling it does not execute its code, it only creates a generator object:
>>> no_coroutine_create_exception(True)
<generator object no_coroutine_create_exception at 0x101651678>
>>> no_coroutine_create_exception(False)
<generator object no_coroutine_create_exception at 0x1016516d0>
Neither of the calls above executes any Python code, it only creates generators that must be iterated.
You'd have to make a blocking function that starts the IOLoop and runs it until your coroutine finishes:
def exception_blocking():
return ioloop.IOLoop.current().run_sync(exception_coroutine)
exception_blocking()
(The IOLoop acts as a scheduler for multiple non-blocking tasks, and the gen.coroutine decorator is responsible for iterating the coroutine until completion.)
However, I think I'm likely answering your immediate question but merely enabling you to proceed down an unproductive path. You're almost certainly better off using async code or blocking code throughout instead of trying to mix them.
In Ruby, I can pass a block of code to a method.
For example, I can pass different code blocks to get_schedules_with_retries method.
And invoke the block by calling black.call
I'd like to know how could I implement that logic in Python,
Because I have lots of code blocks, need retry pattern.
I don't like copy paste the retry logic in many code blocks
Example:
def get_schedules_with_retries(&block)
max_retry_count = 3
retry_count = 0
while (retry_count < max_retry_count)
begin
schedules = get_more_raw_schedules
block.call(schedules)
rescue Exception => e
print_error(e)
end
if schedules.count > 0
break
else
retry_count+=1
end
end
return schedules
end
get_schedules_with_retries do |schedules|
# do something here
end
get_schedules_with_retries do |schedules|
# do another thing here
end
In Python, a block is a syntactic feature (an indentation under block opening statements like if or def) and not an object. The feature you expect may be a closure (which can access variables outside of the block), which you can achieve using inner functions, but any callable could be used. Because of how lambda works in Python, the inline function definition you've shown with do |arg| is limited to a single expression.
Here's a rough rewrite of your sample code in Python.
def get_schedules_with_retries(callable, max_retry_count = 3):
retry_count = 0
while retry_count < max_retry_count:
schedules = get_more_raw_schedules()
try:
callable(schedules)
except: # Note: could filter types, bind name etc.
traceback.print_exc()
if schedules.count > 0:
break
else:
retry_count+=1
return schedules
get_schedules_with_retries(lambda schedules: single_expression)
def more_complex_function(schedules):
pass # do another thing here
get_schedules_with_retries(more_complex_function)
One variant uses a for loop to make it clear the loop is finite:
def call_with_retries(callable, args=(), tries=3):
for attempt in range(tries):
try:
result=callable(*args)
break
except:
traceback.print_exc()
continue
else: # break never reached, so function always failed
raise # Reraises the exception we printed above
return result
Frequently when passing callables like this, you'll already have the function you want available somewhere and won't need to redefine it. For instance, methods on objects (bound methods) are perfectly valid callables.
You could do it like this:
def codeBlock(paramter1, parameter2):
print("I'm a code block")
def passMeABlock(block, *args):
block(*args)
#pass the block like this
passMeABlock(codeBlock, 1, 2)
You do so by defining a function, either by using the def statement or a lambda expression.
There are other techniques however, that may apply here. If you need to apply common logic to the input or output of a function, write a decorator. If you need to handle exceptions in a block of code, perhaps creating a context manager is applicable.
Is there a way to prevent StopIteration exceptions from being thrown from unrelated code (without having to catch them manually)?
Example: loop_all wants to loop through the myiter iterator and simply move on when this one has finished. This works unless some_dangerous_method or any other code in myiter raises a StopIteration.
def loop_all():
myiter = myiter()
try:
while True:
next(myiter) # <- I want exactly the StopIteration from this next method
except StopIteration:
pass
def myiter():
some_dangerous_method() # what if this also raises a StopIteration?
for i in some_other_iter():
# here may be more code
yield
Is there a way to make it clear to which StopIteration the code should react to?
If a function you are calling is invoking next(iter), and isn't dealing with StopIteration, then that function has a bug. Fix it.
Perhaps I'm missing something but why not simply this:
def myiter():
try:
some_dangerous_method()
except StopIteration:
pass # or raise a different exception
for i in some_other_iter():
# here may be more code
yield