A python internals inquiry. Can you gurus please help me understand why the following code leads to a "Fatal python error" rather than nicely catching over the recursion limit overflow?
def flowfunc(x):
try:
if x < 0:
flowfunc(x + 3)
flowfunc(x + 5)
else:
flowfunc(x - 1)
except:
print("exception at: x = ", x)
Call it with e.g.:
flowfunc(0)
and all breaks loss with
exception at: x = -1
Fatal Python error: Cannot recover from stack overflow.
...
...
[threads data...]
Seems like it all depends on the 2nd recursive call. The following code will behave nicely:
def flowfunc2(x):
try:
if x < 0:
flowfunc2(x + 3)
# flowfunc2(x + 5)
else:
flowfunc2(x - 1)
except:
print("exception at: x = ", x)
with
flowfunc2(0)
returning (value may change depending on available stack depth):
exception at: x = -1
I'm on python 3.6. Your insight will be appreciated.
Similarity between the two snippets:
You have an oscillating recursive function going between positives and negatives, never really terminating. Once you reach the max recursive depth set by Python, you catch the exception, print the current value.
Difference between the two snippets:
In the first version (where you get the fatal error), after printing the max depth reached exception, instead of winding up your call stack, you just go one level-up and call flowfunc(x + 5) piling up even more calls in the call stack than Python can possibly handle. That's where you get the fatal error. In the second version, after printing the exception you just go straight up returning None to the original caller and terminate the execution of your program. In a way, the printing of exception in the second snippet serves as your base case for recursion since there is nothing else to do in the caller and hence your program terminates gracefully.
So yes, your observation is right, it does depend on the second recursive call.
This is the same error as https://bugs.python.org/issue6028.
And the reason is explained there as well:
"This is normal behaviour, actually. The RuntimeError is raised, but
you catch it in the except clause and then recurse again ad infinitum.
The interpreter realizes that it "cannot recover from stack overflow",
as the message says, and then bails out."
Basically, because of the existence of your second recursive call, the program continues after the error is caught in your first one and it keeps branching and generates infinite RuntimeError.
Related
There is an argument within my team for a code block of an internal using automation script.
The following function makes a network request and returns the result it has got.
The network request may fail due to IOError sometimes, but if the code retries, it will be able connect and get the result again. So the code have a retry mechanism to deal with such error.
But sometimes other errors happen, (like, say timeout or something). From the point of view of requirement, we should exit the script with an error code for such issues, so the calling program will know that something went wrong with the script and, say mail the engineers to let them to handle it.
Here is the code:
def my_func():
retry_times = 5
sleep_time = 5
for i in range(0, 6)
try:
result = network_request()
break
except IOError as err:
if i == retry_times:
log.error
raise
else:
log.warning("Request failed, sleep %s seconds and retry <%s/%s>"
% (sleep_time, i, retry_times))
return result
But one of our team member argues that we should catch all the remaining exceptions, never let the exception to be thrown to the outer most code and finally make the script to exit with an error code and prints a stacktrace printing.
He calls this "exception leak", which violates common coding rule. Here goes his code (lines with "# Added code line" are the changed lines):
def my_func():
retry_times = 5
sleep_time = 5
for i in range(0, 6)
try:
result = network_request()
break
except IOError as err:
if i == retry_times:
log.error
raise
else:
log.warning("Request failed, sleep %s seconds and retry <%s/%s>"
% (sleep_time, i, retry_times))
except Exception: # Added code line
log.error("Error happened: %s" % err) # Added code line
exit(1)
return result
I said that he is doing things that Python has just done:
Print the error log.
Exist the script with non-zero exit code
But he argues that "how do you know that Python has done nothing when you catch the exception?", "leaving uncaught exception will cause memory leak", "any well designed program never leave uncaught exception to the end, it is a common sense".
This has confused me, cause I've never heard about such "common sense", especially for Python (no sure is C++ has this "common sense"). I Googled the topic and found that there is even nobody has asked such question. I've learned from the very beginning that the principle of exception handling is:
Handle those exceptions that your code is able to handle.
Raise those exceptions to the outer scope if you are unable to handle it.
In this case, since this script is an internal using automation script, when such an issue happens, printing a detail stacktrace and exit the script with non-zero error code is exactly what we want, cause the code cannot handle such issue so it should raise this finally to the engineers.
Adding a line which catches an unknown exception, print a error log and then exit the script with non-zero error code likes doing redundantly doing something that has already been done.
So here I'm asking:
1. Will "leaving uncaught exception to the end and finally cause the script to exit" cause any issue like "memory leak" or something?
2. Is there any such "common sense" in Python coding rule to no leave any uncaught exception to the end, providing "print error log" and "exit with non-zero code" is exactly what the script wants to do.
It depends on your other code. Many frameworks (such as Django) have a settings variable DEBUG. If that is set to true relevant data will be disclosed. If not users are displayed a 404 error.
If you are using a linter it tells you that catching unspecified exceptions is a bad practice
I have a scenario where a function calls itself during error handling if a certain condition is met. Something like:
def test(param):
# code
try:
# attempt
except ValueError as e:
if (x):
test(param+1)
I noticed that if I get stuck in a loop of excepts and try to cancel with my keyboard, I get a giant stacktrace. This doesn't seem right.
Is there a better way to handle this?
Edit:
After running this for a while, I got:
RecursionError: maximum recursion depth exceeded while calling a Python object
I am not sure it is related, but I would imagine a recursion depth issue would arise from too many recursive function calls?
Here's one way to repeat your operation on failure, without using recursion.
def dostuff(param):
while True:
# code
try:
# attempt
except ValueError:
if x:
param += 1
continue
break
This way, if attempt is successful, the loop will break. But if it raises a ValueError, and if your x condition (whatever that is) is true, then the body of the loop will be repeated with param incremented by 1.
I guess I'm not the first asking this question, but I haven't found a solution that I could use/understand yet. And the issue is probably not as simple as i first expected.
I think it can be boiled down to two general questions:
1) Is there a way to avoid Python to stop when an error occur and just jump on to the next line of code in the script?
2) Is there a way to make Python execute a line of code if an error occurs? Like, if error then...
My concrete problem:
I have a very large program with a lot of functions and other stuff, which would take forever to adjust individually by using "try" for example (if i understand it correctly)
My program run as a large loop that gather information and keeps running. This means that it does not really matter to me, that my program fails multiple time as long as it keeps running. I can easily handle that some of the information is with error and would just like my program to take a note of it and keep going.
Is there a solution to this?
As you rightly pointed out, the try/catch block in Python is by far your best ally:
for i in range(N):
try: do_foo() ; except: do_other_foo()
try: do_bar() ; except: do_other_bar()
Alternatively, you could also use, in case you didn't need the Exception:
from contextlib import suppress
for i in range(N):
with suppress(Exception):
do_foo()
with suppress(Exception):
do_bar()
Your only possibility is to rely on the try/except clause. Keep in mind that the try/except may use also finally and else (see documentation:
try:
print("problematic code - error NOT raised")
except:
print("code that gets executed only if an error occurs")
else:
print("code that gets executed only if an error does not occur")
finally:
print("code that gets ALWAYS executed")
# OUTPUT:
# problematic code - error NOT raised
# code that gets executed only if an error does not occur
# code that gets ALWAYS executed
or, when an error is raised:
try:
print("problematic code - error raised!")
raise "Terrible, terrible error"
except:
print("code that gets executed only if an error occurs")
else:
print("code that gets executed only if an error does not occur")
finally:
print("code that gets ALWAYS executed")
# OUTPUT:
# problematic code - error raised!
# code that gets executed only if an error occurs
# code that gets ALWAYS executed
I urge to point out, by the way, that ignoring everything makes me shiver:
you really should (at least, more or less) identify which exception can be raised, catch them (except ArithmeticError: ..., check built-in exceptions) and handle them individually. What you're trying to do will probably snowball into an endless chain of problems, and ignoring them will probably create more problems!
I think that this question helps to understand what a robust software is, meanwhile on this one you can see how SO community thinks python exceptions should be handled
I wrote a fibonacci function that infinitely recurses, and while python couldn't detect it and threw errors upon hitting the max recursion limit, when I used try and assert to see if fib(30) was equal to some value, it immediately told me it was not. How did it do this? It seems like it didn't even need to run fib(30).
note:
I realize this only works if I do
try:
assert infiniteFib(30) == 832040
except:
print "done immediately"
When I do just the assert, it produces many errors about too many recursions, but with the try it stops at the first error.
What I'm curious is how does python manage to produce an error so quickly about infinite recursion? Wouldn't it need to hit the limit (which takes a long time) to tell whether it was infinite?
EDIT:
Some requested code, but just to be clear, I DON'T want a solution to the errors (I know it's wrong because I deliberately excluded the base case), I want to know how python produces errors so quickly when it should take much longer (if you do fib(30), clearly it takes a while to hit the max recursion limit, but somehow python produces errors way before then):
def fib(n):
return fib(n-1) + fib(n-2)
try: assert(fib(30) == 832040)
except: print "done immediately"
The reason the code you've shown runs quickly is because it catches the exception that is raised by fib when it hits the recursion limit and doesn't print the traceback. Running to the recursion limit doesn't take very long at all, but formatting and printing hundreds of lines of traceback does.
If you inspect the exception you get, you'll see it is the same RuntimeError you get when you run fib normally, not an AssertionError. Try this, to see what is going on better:
try:
assert(fib(30) == 832040)
except Exception as e:
print("Got an Exception: %r" % e)
It's not done immediately. Your code runs until python reaches maximum recursion depth and maximum recursion depth is set to 1000 by default in python to avoid stack overflow errors.
So, actually your code runs till it reaches recursion depth of 1000 and errors out RuntimeError: maximum recursion depth exceeded. You can verify this by modifying your code as below:
i=0
def fib(n):
global i
i = i + 1
print i
return fib(n-1) + fib(n-2)
assert(fib(30) == 832040)
print i
print "done immediately"
In my machine, i am getting the last i value as 984 before errors out.
I've read three beginner-level Python books, however, I still don't understand exceptions.
Could someone give me a high level explanation?
I guess I understand that exceptions are errors in code or process that cause the code to stop working.
In the old days, when people wrote in assembly language or C, every time you called a function that might fail, you had to check whether it succeeded. So you'd have code like this:
def countlines(path):
f = open(path, 'r')
if not f:
print("Couldn't open", path)
return None
total = 0
for line in f:
value, success = int(line)
if not success:
print(line, "is not an integer")
f.close()
return None
total += value
f.close()
return total
The idea behind exceptions is that you don't worry about those exceptional cases, you just write this:
def countlines(path):
total = 0
with open(path, 'r') as f:
for line in f:
total += int(line)
return total
If Python can't open the file, or turn the line into an integer, it will raise an exception, which will automatically close the file, exit your function, and quit your whole program, printing out useful debugging information.
In some cases, you want to handle an exception instead of letting it quit your program. For example, maybe you want to print the error message and then ask the user for a different filename:
while True:
path = input("Give me a path")
try:
print(countlines(path))
break
except Exception as e:
print("That one didn't work:", e)
Once you know the basic idea that exceptions are trying to accomplish, the tutorial has a lot of useful information.
If you want more background, Wikipedia can help (although the article isn't very useful until you understand the basic idea).
If you still don't understand, ask a more specific question.
The best place to start with that is Python's list of built-in exceptions, since most you'll see derive from that.
Keep in mind that anybody can throw any error they want over anything, and then catch it and dismiss it as well. Here's one quick snippet that uses exceptions for handling instead of if/else where __get_site_file() throws an exception if the file isn't found in any of a list of paths. Despite that particular exception, the code will still work. However, the code would throw an uncaught error that stops execution if the file exists but the permissions don't allow reading.
def __setup_site_conf(self):
# Look for a site.conf in the site folder
try:
path = self.__get_site_file('site.conf')
self.__site_conf = open(path).read()
except EnvironmentError:
self.__site_conf = self.__get_site_conf_from_template()
Python's documentation: http://docs.python.org/2/tutorial/errors.html
For a high-level explanation, say we want to divide varA / varB. We know that varB can't equal 0, but we might not want to perform the check every time we do the division:
if varB != 0:
varA / varB
We can use exceptions to try the block without performing the conditional first, and then handle the behavior of the program based on whether or not something went wrong in the try block. In the following code, if varB == 0, then 'oops' is printed to the console:
try:
varA / varB
except ZeroDivisionError:
print 'oops'
Here is a list of exceptions that can be used: http://docs.python.org/2/library/exceptions.html#exceptions.BaseException
However, if you know how it may fail, you can just open a python console and see what exception is raised:
>>> 1 / 0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ZeroDivisionError: integer division or modulo by zero
Exceptions are unexpected events that occur during the execution of a program. An exception might result from a logical error or an unanticipated situation.
In Python, exceptions (also known as errors) are objects that are raised (or thrown) by code that encounters an unexpected circumstance.
The Python interpreter can also raise an exception should it encounter an unexpected condition, like running out of memory. A raised error may be caught by a surrounding context that “handles” the exception in an appropriate fashion.
If uncaught, an exception causes the interpreter to stop executing the program and to report an appropriate message to the console.
def sqrt(x):
if not isinstance(x, (int, float)):
raise TypeError( x must be numeric )
elif x < 0:
raise ValueError( x cannot be negative )
Exceptions are not necessarily errors. They are things that get raised when the code encounters something it doesn't (immediately) know how to deal with. This may be entirely acceptable, depending on how you make your code. For instance, let's say you ask a user to put in a number. You then try to take that text (string) and convert it to a number (int). If the user put in, let's say, "cat", however, this will raise an exception. You could have your code handle that exception, however, and rather than break, just give the user a small message asking him to try again, and please use a number. Look at this link to see what I'm talking about: http://www.tutorialspoint.com/python/python_exceptions.htm
Also, you usually handle exceptions with a try, except (or catch) block. Example:
try:
integer = int(raw_input("Please enter an integer: "))
except Exception, exc:
print "An error has occured."
print str(exc)
Hope it helps!