Develop Reference

How to ignore exceptions occuring within a call to rglob?

I am using pathlib.Path.rglob to browse through some files in a directory hierarchy. However one of the (sub-sub-...) directory cannot be parsed and an exception is thrown.
import pathlib
for file in pathlib.Path(r'\\path\to\shared\folder').rglob('*.foo'):
pass
# OSError: [WinError 59] An unexpected network error occurred
Catching the exception in the body of the for loop makes no sense as the exeption occurs in rglob. Putting the entire loop within a try block does catch the exception, but then I cannot resume the computation.
Does anybody have a clue about how to do that with pathlib.Path.rglob? A work around is to use good old os.walk but I would be interested in knowing a solution for this supposedly more modern tool.

First of all, try to run filePaths = Path(r'\\path\to\shared\folder').rglob('*.foo') in your python shell to see whether it completes without an error. If so, you can try to catch runtime-exceptions (probably due to unstable network connection) during the iteration.
def try_loop(g):
while True:
try:
yield next(g)
except StopIteration:
break
except OSError as e:
# log error
continue
Then wrap your rglob call in the try_loop function
for file in try_loop(pathlib.Path(r'\\path\to\shared\folder').rglob('*.foo')):
pass
This is inspired by a more general question "Handle an exception thrown in a generator".

Will the uncaught Python exception lead to memory lead or other issues?

There is an argument within my team for a code block of an internal using automation script.
The following function makes a network request and returns the result it has got.
The network request may fail due to IOError sometimes, but if the code retries, it will be able connect and get the result again. So the code have a retry mechanism to deal with such error.
But sometimes other errors happen, (like, say timeout or something). From the point of view of requirement, we should exit the script with an error code for such issues, so the calling program will know that something went wrong with the script and, say mail the engineers to let them to handle it.
Here is the code:
def my_func():
retry_times = 5
sleep_time = 5
for i in range(0, 6)
try:
result = network_request()
break
except IOError as err:
if i == retry_times:
log.error
raise
else:
log.warning("Request failed, sleep %s seconds and retry <%s/%s>"
% (sleep_time, i, retry_times))
return result
But one of our team member argues that we should catch all the remaining exceptions, never let the exception to be thrown to the outer most code and finally make the script to exit with an error code and prints a stacktrace printing.
He calls this "exception leak", which violates common coding rule. Here goes his code (lines with "# Added code line" are the changed lines):
def my_func():
retry_times = 5
sleep_time = 5
for i in range(0, 6)
try:
result = network_request()
break
except IOError as err:
if i == retry_times:
log.error
raise
else:
log.warning("Request failed, sleep %s seconds and retry <%s/%s>"
% (sleep_time, i, retry_times))
except Exception: # Added code line
log.error("Error happened: %s" % err) # Added code line
exit(1)
return result
I said that he is doing things that Python has just done:
Print the error log.
Exist the script with non-zero exit code
But he argues that "how do you know that Python has done nothing when you catch the exception?", "leaving uncaught exception will cause memory leak", "any well designed program never leave uncaught exception to the end, it is a common sense".
This has confused me, cause I've never heard about such "common sense", especially for Python (no sure is C++ has this "common sense"). I Googled the topic and found that there is even nobody has asked such question. I've learned from the very beginning that the principle of exception handling is:
Handle those exceptions that your code is able to handle.
Raise those exceptions to the outer scope if you are unable to handle it.
In this case, since this script is an internal using automation script, when such an issue happens, printing a detail stacktrace and exit the script with non-zero error code is exactly what we want, cause the code cannot handle such issue so it should raise this finally to the engineers.
Adding a line which catches an unknown exception, print a error log and then exit the script with non-zero error code likes doing redundantly doing something that has already been done.
So here I'm asking:
1. Will "leaving uncaught exception to the end and finally cause the script to exit" cause any issue like "memory leak" or something?
2. Is there any such "common sense" in Python coding rule to no leave any uncaught exception to the end, providing "print error log" and "exit with non-zero code" is exactly what the script wants to do.

It depends on your other code. Many frameworks (such as Django) have a settings variable DEBUG. If that is set to true relevant data will be disclosed. If not users are displayed a 404 error.
If you are using a linter it tells you that catching unspecified exceptions is a bad practice

Stop Python code without an error

I have a piece of code which is not in a function, say
x = 5
y = 10
if x > 5:
print("stopping")
What can I put after the print statement to stop the code from running further? Sys.exit() works, but raises an error that I don't want in the program. I want it to quietly stop the code as if it had reached the end of the main loop. Thanks.

As JBernardo pointed out, sys.exit() raises an exception. This exception is SystemExit. When it is not handled by the user code, the interpreter exits cleanly (a debugger debugging the program can catch it and keep control of the program, thanks to this mechanism, for instance)—as opposed to os._exit(), which is an unconditional abortion of the program.
This exception is not caught by except Exception:, because SystemExit does not inherit from Exception. However, it is caught by a naked except: clause.
So, if your program sees an exception, you may want to catch fewer exceptions by using except Exception: instead of except:. That said, catching all exceptions is discouraged, because this might hide real problems, so avoid it if you can, by making the except clause (if any) more specific.
My understanding of why this SystemExit exception mechanism is useful is that the user code goes through any finally clause after a sys.exit() found in an except clause: files can be closed cleanly, etc.; then the interpreter catches any SystemExit that was not caught by the user and exits for good (a debugger would instead catch it so as to keep the interpreter running and obtain information about the program that exited).

You can do what you're looking for by doing this:
import os
os._exit(1)

sys.exit() which is equivalent to sys.exit(0) means exit with success. sys.exit(1) or sys.exit("Some message") means exit with failure. Both cases raise a SystemExit exception. In fact when your program exists normally it is exactly like sys.exit(0) has been called.

When I ran across this thread, I was looking for a way to exit the program without an error, without an exception, have the code show as 'PASSED', and continue running other tests files. The solution, for me, was to use the return statement.
.
.
.
if re.match("^[\s\S]*gdm-simple-slave[\s\S]*$", driver.find_element_by_css_selector("BODY").text) == None:
print "Identifiers object gdm-simple-slave not listed in table"
return
else:
driver.find_element_by_xpath("//input[#value='gdm-simple-slave']").click()
.
.
.
That allowed me to run multiple programs while keeping the debugger running...
test_logsIdentifiersApache2EventWindow.py#16::test_LogsIdentifiersApache2EventWi
ndow **PASSED**
test_logsIdentifiersAudispdEventWindow.py#16::test_LogsIdentifiersAudispdEventWi
ndow **PASSED**
test_logsIdentifiersGdmSimpleSlaveEventWindow.py#16::test_LogsIdentifiersGdmSimp
leSlaveEventWindow Identifiers object gdm-simple-slave not listed in table
**PASSED**
test_logsIdentifiersAuditdEventWindow.py#16::test_LogsIdentifiersAuditdEventWind
ow **PASSED**

Use try-except statements.
a = [1, 2, 3, 4, 5]
for x in xrange(0,5):
try:
print a[x+1] #this is a faulty statement for test purposes
except:
exit()
print "This is the end of the program."
Output:
> python test.py
2
3
4
5
No errors printed, despite the error raised.

Using sys.exit or SystemExit; when to use which?

Some programmers use sys.exit, others use SystemExit.
What is the difference?
When do I need to use SystemExit or sys.exit inside a function?
Example:
ref = osgeo.ogr.Open(reference)
if ref is None:
raise SystemExit('Unable to open %s' % reference)
or:
ref = osgeo.ogr.Open(reference)
if ref is None:
print('Unable to open %s' % reference)
sys.exit(-1)

No practical difference, but there's another difference in your example code - print goes to standard out, but the exception text goes to standard error (which is probably what you want).

sys.exit(s) is just shorthand for raise SystemExit(s), as described in the former's docstring; try help(sys.exit). So, instead of either one of your example programs, you can do
sys.exit('Unable to open %s' % reference)

There are 3 exit functions, in addition to raising SystemExit.
The underlying one is os._exit, which requires 1 int argument, and exits immediately with no cleanup. It's unlikely you'll ever want to touch this one, but it is there.
sys.exit is defined in sysmodule.c and just runs PyErr_SetObject(PyExc_SystemExit, exit_code);, which is effectively the same as directly raising SystemExit. In fine detail, raising SystemExit is probably faster, since sys.exit requires an LOAD_ATTR and CALL_FUNCTION vs RAISE_VARARGS opcalls. Also, raise SystemExit produces slightly smaller bytecode (4bytes less), (1 byte extra if you use from sys import exit since sys.exit is expected to return None, so includes an extra POP_TOP).
The last exit function is defined in site.py, and aliased to exit or quit in the REPL. It's actually an instance of the Quitter class (so it can have a custom __repr__, so is probably the slowest running. Also, it closes sys.stdin prior to raising SystemExit, so it's recommended for use only in the REPL.
As for how SystemExit is handled, it eventually causes the VM to call os._exit, but before that, it does some cleanup. It also runs atexit._run_exitfuncs() which runs any callbacks registered via the atexit module. Calling os._exit directly bypasses the atexit step.

My personal preference is that at the very least SystemExit is raised (or even better - a more meaningful and well documented custom exception) and then caught as close to the "main" function as possible, which can then have a last chance to deem it a valid exit or not. Libraries/deeply embedded functions that have sys.exit is just plain nasty from a design point of view. (Generally, exiting should be "as high up" as possible)

According to documentation sys.exit(s) effectively does raise SystemExit(s), so it's pretty much the same thing.

While the difference has been answered by many answers, Cameron Simpson makes an interesting point in https://mail.python.org/pipermail/python-list/2016-April/857869.html:
TL;DR: It's better to just raise a "normal" exception, and use SystemExit or sys.exit only at the top levels of a script.
I m on python 2.7 and Linux , I have a simple code need suggestion if I
I could replace sys.exit(1) with raise SystemExit .
==Actual code==
def main():
try:
create_logdir()
create_dataset()
unittest.main()
except Exception as e:
logging.exception(e)
sys.exit(EXIT_STATUS_ERROR)
if __name__ == '__main__': main()
==Changed Code==
def main():
try:
create_logdir()
create_dataset()
unittest.main()
except Exception as e:
logging.exception(e)
raise SystemExit
if __name__ == '__main__':
main()
I am against both of these personally. My preferred pattern is like
this:
def main(argv):
try:
...
except Exception as e:
logging.exception(e)
return 1
if __name__ == '__main__':
sys.exit(main(sys.argv))
Notice that main() is back to being a normal function with normal
returns.
Also, most of us would avoid the "except Exception" and just let a top
level except bubble out: that way you get a stack backtrace for
debugging. I agree it prevents logging the exception and makes for
uglier console output, but I think it is a win. And if you do want
to log the exception there is always this:
try:
... except Exception as e:
logging.exception(e)
raise
to recite the exception into the log and still let it bubble out
normally.
The problem with the "except Exception" pattern is that it catches and
hides
every exception, not merely the narrow set of specific exceptions that you understand.
Finally, it is frowned upon to raise a bare Exception class. In
python 3 I believe it is actually forbidden, so it is nonportable
anyway. But even In Python to it is best to supply an Exception
instance, not the class:
raise SystemExit(1)
All the functions in try block have exception bubbled out using raise
Example for create_logdir() here is the function definition
def create_logdir():
try:
os.makedirs(LOG_DIR)
except OSError as e:
sys.stderr.write("Failed to create log directory...Exiting !!!")
raise
print "log file: " + corrupt_log
return True
def main():
try:
create_logdir()
except Exception as e:
logging.exception(e)
raise SystemExit
(a) In case if create_logdir() fails we will get the below error ,is
this fine or do I need to improve this code.
Failed to create log directory...Exiting !!!ERROR:root:[Errno 17] File
exists: '/var/log/dummy'
Traceback (most recent call last):
File "corrupt_test.py", line 245, in main
create_logdir()
File "corrupt_test.py", line 53, in create_logdir
os.makedirs(LOG_DIR)
File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
OSError: [Errno 17] File exists: '/var/log/dummy'
I prefer the bubble out approach, perhap with a log or warning
messages as you have done, eg:
logging.exception("create_logdir failed: makedirs(%r): %s" %
(LOG_DIR, e)) raise
(Also not that that log message records more context: context is very
useful when debugging problems.)
For very small scripts sys.stderr.write is ok, but in general any of
your functions that turned out to be generally useful might migrate
into a library in order to be reused; consider that stderr is not
always the place for messages; instead reading for the logging module
with error() or wanr() or exception() as appropriate. There is more
scope for configuring where the output goes that way without wiring
it into your inner functions.
Can I have just raise , instead of SystemExit or sys.exit(1) . This
looks wrong to me
def main():
try:
create_logdir()
except Exception as e
logging.exception(e)
raise
This is what I would do, myself.
Think: has the exception been "handled", meaning has the situation
been dealt with because it was expected? If not, let the exception
bubble out so that the user knows that something not understood by
the program has occurred.
Finally, it is generally bad to SystemExit or sys.exit() from inside
anything other than the outermost main() function. And I resist it
even there; the main function, if written well, may often be called
from somewhere else usefully, and that makes it effectively a library
function (it has been reused). Such a function should not
unilaterally abort the program. How rude! Instead, let the exception
bubble out: perhaps the caller of main() expects it and can handle
it. By aborting and not "raise"ing, you have deprived the caller of
the chance to do something appropriate, even though you yourself
(i.e. "main") do not know enough context to handle the exception.
So I am for "raise" myself. And then only because you want to log the
error. If you didn't want to log the exception you could avoid the
try/except entirely and have simpler code: let the caller worry
about unhandled exceptions!

SystemExit is an exception, which basically means that your progam had a behavior such that you want to stop it and raise an error. sys.exit is the function that you can call to exit from your program, possibily giving a return code to the system.
EDIT: they are indeed the same thing, so the only difference is in the logic behind in your program. An exception is some kind of "unwanted" behaviour, whether a call to a function is, from a programmer point of view, more of a "standard" action.

Why can't I handle a KeyboardInterrupt in python?

I'm writing python 2.6.6 code on windows that looks like this:
try:
dostuff()
except KeyboardInterrupt:
print "Interrupted!"
except:
print "Some other exception?"
finally:
print "cleaning up...."
print "done."
dostuff() is a function that loops forever, reading a line at a time from an input stream and acting on it. I want to be able to stop it and clean up when I hit ctrl-c.
What's happening instead is that the code under except KeyboardInterrupt: isn't running at all. The only thing that gets printed is "cleaning up...", and then a traceback is printed that looks like this:
Traceback (most recent call last):
File "filename.py", line 119, in <module>
print 'cleaning up...'
KeyboardInterrupt
So, exception handling code is NOT running, and the traceback claims that a KeyboardInterrupt occurred during the finally clause, which doesn't make sense because hitting ctrl-c is what caused that part to run in the first place! Even the generic except: clause isn't running.
EDIT: Based on the comments, I replaced the contents of the try: block with sys.stdin.read(). The problem still occurs exactly as described, with the first line of the finally: block running and then printing the same traceback.
EDIT #2:
If I add pretty much anything after the read, the handler works. So, this fails:
try:
sys.stdin.read()
except KeyboardInterrupt:
...
But this works:
try:
sys.stdin.read()
print "Done reading."
except KeyboardInterrupt:
...
Here's what's printed:
Done reading. Interrupted!
cleaning up...
done.
So, for some reason, the "Done reading." line is printed, even though the exception occurred on the previous line. That's not really a problem - obviously I have to be able to handle an exception anywhere inside the "try" block. However, the print doesn't work normally - it doesn't print a newline afterwards like it's supposed to! The "Interruped" is printed on the same line... with a space before it, for some reason...? Anyway, after that the code does what it's supposed to.
It seems to me that this is a bug in handling an interrupt during a blocked system call.

Asynchronous exception handling is unfortunately not reliable (exceptions raised by signal handlers, outside contexts via C API, etc). You can increase your chances of handling the async exception properly if there is some coordination in the code about what piece of code is responsible for catching them (highest possible in the call stack seems appropriate except for very critical functions).
The called function (dostuff) or functions further down the stack may itself have a catch for KeyboardInterrupt or BaseException that you didn't/couldn't account for.
This trivial case worked just fine with python 2.6.6 (x64) interactive + Windows 7 (64bit):
>>> import time
>>> def foo():
... try:
... time.sleep(100)
... except KeyboardInterrupt:
... print "INTERRUPTED!"
...
>>> foo()
INTERRUPTED! #after pressing ctrl+c
EDIT:
Upon further investigation, I tried what I believe is the example that others have used to reproduce the issue. I was lazy so I left out the "finally"
>>> def foo():
... try:
... sys.stdin.read()
... except KeyboardInterrupt:
... print "BLAH"
...
>>> foo()
This returns immediately after hitting CTRL+C. The interesting thing happened when I immediately tried to call foo again:
>>> foo()
Traceback (most recent call last):
File "c:\Python26\lib\encodings\cp437.py", line 14, in decode
def decode(self,input,errors='strict'):
KeyboardInterrupt
The exception was raised immediately without me hitting CTRL+C.
This would seem to make sense - it appears that we are dealing with nuances in how asynchronous exceptions are handled in Python. It can take several bytecode instructions before the async exception is actually popped and then raised within the current execution context. (That's the behavior that I've seen when playing with it in the past)
See the C API: http://docs.python.org/c-api/init.html#PyThreadState_SetAsyncExc
So this somewhat explains why KeyboardInterrupt gets raised in the context of the execution of the finally statement in this example:
>>> def foo():
... try:
... sys.stdin.read()
... except KeyboardInterrupt:
... print "interrupt"
... finally:
... print "FINALLY"
...
>>> foo()
FINALLY
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in foo
KeyboardInterrupt
There could be some crazy mixing of custom signal handlers mixed with the interpreter's standard KeyboardInterrupt/CTRL+C handler that's resulting in this sort of behavior. For example, the read() call sees the signal and bails, but it re-raises the signal after deregistering it's handler. I wouldn't know for sure without inspecting the interpreter codebase.
This is why I generally shy away from making use of async exceptions....
EDIT 2
I think there's a good case for a bug report.
Again more theories...(just based on reading code) See the file object source: http://svn.python.org/view/python/branches/release26-maint/Objects/fileobject.c?revision=81277&view=markup
file_read calls Py_UniversalNewlineFread(). fread can return with an error with errno = EINTR (it performs its own signal handling). In this case Py_UniversalNewlineFread() bails but does not perform any signal checking with PyErr_CheckSignals() so that the handlers can get called synchronously. file_read clears the file error but also does not call PyErr_CheckSignals().
See getline() and getline_via_fgets() for examples of how it's used. The pattern is documented in this bug report for a similar issue: ( http://bugs.python.org/issue1195 ). So it seems that the signal is handled at an indeterminate time by the interpreter.
I guess there's little value in diving any deeper since it's still not clear whether the sys.stdin.read() example is a proper analog of your "dostuff()" function. (there could be multiple bugs at play)

sys.stdin.read() is a system call and so the behavior is going to be different for each system. For windows 7 I think what is happening is that the input is being buffered and so you're getting where sys.stdin.read() is returning everything up to the Ctrl-C and as soon as you access sys.stdin again it'll send the "Ctrl-C".
try the following,
def foo():
try:
print sys.stdin.read()
print sys.stdin.closed
except KeyboardInterrupt:
print "Interrupted!"
This suggests that there is a buffering of stdin going on that is causing another call to stdin to recognize the keyboard input
def foo():
try:
x=0
while 1:
x += 1
print x
except KeyboardInterrupt:
print "Interrupted!"
there doesn't appear to be a problem.
Is dostuff() reading from stdin?

Having similar problem and this is my workaround:
try:
some_blocking_io_here() # CTRL-C to interrupt
except:
try:
print() # any i/o will get the second KeyboardInterrupt here?
except:
real_handler_here()

Here's a guess about what's happening:
pressing Ctrl-C breaks the "print" statement (for whatever reason... bug? Win32 limitation?)
pressing Ctrl-C also throws the first KeyboardInterrupt, in dostuff()
The exception handler runs and tries to print "Interrupted", but the "print" statement is broken and throws another KeyboardInterrupt.
The finally clause runs and tries to print "cleaning up....", but the "print" statement is broken and throws yet another KeyboardInterrupt.

This works for me:
import sys
if __name__ == "__main__":
try:
sys.stdin.read()
print "Here"
except KeyboardInterrupt:
print "Worked"
except:
print "Something else"
finally:
print "Finally"
Try putting a line outside of your dostuff() function or move the loop condition outside of the function. For example:
try:
while True:
dostuff()
except KeyboardInterrupt:
print "Interrupted!"
except:
print "Some other exception?"
finally:
print "cleaning up...."
print "done."

def foo():
try:
x=0
while 1:
x+=1
print (x)
except KeyboardInterrupt:
print ("interrupted!!")
foo()
That works fine.