Develop Reference

Python: On error in thread, how do I restart program from the beginning

I am somewhat new to python and error handling. I am running a program that connects to a data source and continuously receives market data, every so often I encounter an "Exception in thread Thread" which I think happens because an issue with the data connection occurs, so the computation that the program should be doing can't be completed. I think I can resolve this problem by restarting the program from the beginning if the connection fails. Something like... if this type of error is encountered, go back to the beginning and try to reconnect, and if that doesn't fix the problem after a few attempts, stop trying. The thread is currently started in the App class, when the class is initialized, so going back to the first line would solve the problem, I'm just not sure how I would do this. My main code is fairly complex, but is structured like so:
if __name__ == '__main__':
my_app = App(args)
while my_app.market_open():
do some computation
do some more computation, etc.
my_app.disconnect()

Define a function def main():
and within it, do the (re)connection stuff.
Then you write the famous
if __name__ == '__main__' :
main()
To restart from anywhere within your program, call main() from wherever you are! In this case, you put everything under/after if __name__ == "__main__" into main()s definition and from inside main() call main()

You might also try putting your calculations into a try / except loop to catch the error and redirect the code to a new function call? You will need to specify the error:
try:
while my_app.market_open():
do some computation
do some more computation, etc.
except WhatEverYourErrorIsHere:
while my_app.market_open():
do some computation again?
Here is example running different functions in two threads every minute. Again, you will need to specify the Error to catch:
import schedule, threading, datetime
def someCallableFunction():
try:
print('running someCallableFunction\n')
except ConnectionError:
print(datetime.utcnow(), 'External Connection Error')
return
def someOtherCallableFunction():
try:
print('running someOtherCallableFunction\n')
except ConnectionError:
print(datetime.utcnow(), 'External Connection Error')
return
def run_threaded(job_func):
job_thread = threading.Thread(target=job_func)
job_thread.start()
schedule.every(1).minutes.at(':00').do(run_threaded, someCallableFunction)
schedule.every(1).minutes.at(':00').do(run_threaded,
someOtherCallableFunction)
while True:
schedule.run_pending()

How to stop the execution of a function right after the exception happened? [Python]

Let's say I have following code:
def main():
try:
int('string_for_ValueError')
except ValueError:
print('How to stop further execution right here?')
print('Executed')
main()
As you can see, print('Executed') line will be executed every time no matter what. My goal is to stop the execution of the current function right after except ValueError is caught.
So, the question is - how to do that?
UPDATE
This function is a part of the multithreaded algorithm. So, if this function cannot be executed because of except ValueError - this function should stop and does not return anything. But other threads should work after that.

You can simple raise an error with a message to stop execution. If you are using try except you basically want the program not to stop from execution when errors found.
If you really want to do that, you can do it this way,
but you can do it several ways. My question is why you are catching this error if you want to stop the program?
EDIT AFTER YOUR UPDATE: You can add return to your function.
def main():
try:
int('string_for_ValueError')
except ValueError:
print('How to stop further execution right here?')
return "to something"
print('this line not executed because function sees the return')
main()

You can quit a Python script using exit("Failure") where the string is an optional message.

Stop Python code without an error

I have a piece of code which is not in a function, say
x = 5
y = 10
if x > 5:
print("stopping")
What can I put after the print statement to stop the code from running further? Sys.exit() works, but raises an error that I don't want in the program. I want it to quietly stop the code as if it had reached the end of the main loop. Thanks.

As JBernardo pointed out, sys.exit() raises an exception. This exception is SystemExit. When it is not handled by the user code, the interpreter exits cleanly (a debugger debugging the program can catch it and keep control of the program, thanks to this mechanism, for instance)—as opposed to os._exit(), which is an unconditional abortion of the program.
This exception is not caught by except Exception:, because SystemExit does not inherit from Exception. However, it is caught by a naked except: clause.
So, if your program sees an exception, you may want to catch fewer exceptions by using except Exception: instead of except:. That said, catching all exceptions is discouraged, because this might hide real problems, so avoid it if you can, by making the except clause (if any) more specific.
My understanding of why this SystemExit exception mechanism is useful is that the user code goes through any finally clause after a sys.exit() found in an except clause: files can be closed cleanly, etc.; then the interpreter catches any SystemExit that was not caught by the user and exits for good (a debugger would instead catch it so as to keep the interpreter running and obtain information about the program that exited).

You can do what you're looking for by doing this:
import os
os._exit(1)

sys.exit() which is equivalent to sys.exit(0) means exit with success. sys.exit(1) or sys.exit("Some message") means exit with failure. Both cases raise a SystemExit exception. In fact when your program exists normally it is exactly like sys.exit(0) has been called.

When I ran across this thread, I was looking for a way to exit the program without an error, without an exception, have the code show as 'PASSED', and continue running other tests files. The solution, for me, was to use the return statement.
.
.
.
if re.match("^[\s\S]*gdm-simple-slave[\s\S]*$", driver.find_element_by_css_selector("BODY").text) == None:
print "Identifiers object gdm-simple-slave not listed in table"
return
else:
driver.find_element_by_xpath("//input[#value='gdm-simple-slave']").click()
.
.
.
That allowed me to run multiple programs while keeping the debugger running...
test_logsIdentifiersApache2EventWindow.py#16::test_LogsIdentifiersApache2EventWi
ndow **PASSED**
test_logsIdentifiersAudispdEventWindow.py#16::test_LogsIdentifiersAudispdEventWi
ndow **PASSED**
test_logsIdentifiersGdmSimpleSlaveEventWindow.py#16::test_LogsIdentifiersGdmSimp
leSlaveEventWindow Identifiers object gdm-simple-slave not listed in table
**PASSED**
test_logsIdentifiersAuditdEventWindow.py#16::test_LogsIdentifiersAuditdEventWind
ow **PASSED**

Use try-except statements.
a = [1, 2, 3, 4, 5]
for x in xrange(0,5):
try:
print a[x+1] #this is a faulty statement for test purposes
except:
exit()
print "This is the end of the program."
Output:
> python test.py
2
3
4
5
No errors printed, despite the error raised.

Using sys.exit or SystemExit; when to use which?

Some programmers use sys.exit, others use SystemExit.
What is the difference?
When do I need to use SystemExit or sys.exit inside a function?
Example:
ref = osgeo.ogr.Open(reference)
if ref is None:
raise SystemExit('Unable to open %s' % reference)
or:
ref = osgeo.ogr.Open(reference)
if ref is None:
print('Unable to open %s' % reference)
sys.exit(-1)

No practical difference, but there's another difference in your example code - print goes to standard out, but the exception text goes to standard error (which is probably what you want).

sys.exit(s) is just shorthand for raise SystemExit(s), as described in the former's docstring; try help(sys.exit). So, instead of either one of your example programs, you can do
sys.exit('Unable to open %s' % reference)

There are 3 exit functions, in addition to raising SystemExit.
The underlying one is os._exit, which requires 1 int argument, and exits immediately with no cleanup. It's unlikely you'll ever want to touch this one, but it is there.
sys.exit is defined in sysmodule.c and just runs PyErr_SetObject(PyExc_SystemExit, exit_code);, which is effectively the same as directly raising SystemExit. In fine detail, raising SystemExit is probably faster, since sys.exit requires an LOAD_ATTR and CALL_FUNCTION vs RAISE_VARARGS opcalls. Also, raise SystemExit produces slightly smaller bytecode (4bytes less), (1 byte extra if you use from sys import exit since sys.exit is expected to return None, so includes an extra POP_TOP).
The last exit function is defined in site.py, and aliased to exit or quit in the REPL. It's actually an instance of the Quitter class (so it can have a custom __repr__, so is probably the slowest running. Also, it closes sys.stdin prior to raising SystemExit, so it's recommended for use only in the REPL.
As for how SystemExit is handled, it eventually causes the VM to call os._exit, but before that, it does some cleanup. It also runs atexit._run_exitfuncs() which runs any callbacks registered via the atexit module. Calling os._exit directly bypasses the atexit step.

My personal preference is that at the very least SystemExit is raised (or even better - a more meaningful and well documented custom exception) and then caught as close to the "main" function as possible, which can then have a last chance to deem it a valid exit or not. Libraries/deeply embedded functions that have sys.exit is just plain nasty from a design point of view. (Generally, exiting should be "as high up" as possible)

According to documentation sys.exit(s) effectively does raise SystemExit(s), so it's pretty much the same thing.

While the difference has been answered by many answers, Cameron Simpson makes an interesting point in https://mail.python.org/pipermail/python-list/2016-April/857869.html:
TL;DR: It's better to just raise a "normal" exception, and use SystemExit or sys.exit only at the top levels of a script.
I m on python 2.7 and Linux , I have a simple code need suggestion if I
I could replace sys.exit(1) with raise SystemExit .
==Actual code==
def main():
try:
create_logdir()
create_dataset()
unittest.main()
except Exception as e:
logging.exception(e)
sys.exit(EXIT_STATUS_ERROR)
if __name__ == '__main__': main()
==Changed Code==
def main():
try:
create_logdir()
create_dataset()
unittest.main()
except Exception as e:
logging.exception(e)
raise SystemExit
if __name__ == '__main__':
main()
I am against both of these personally. My preferred pattern is like
this:
def main(argv):
try:
...
except Exception as e:
logging.exception(e)
return 1
if __name__ == '__main__':
sys.exit(main(sys.argv))
Notice that main() is back to being a normal function with normal
returns.
Also, most of us would avoid the "except Exception" and just let a top
level except bubble out: that way you get a stack backtrace for
debugging. I agree it prevents logging the exception and makes for
uglier console output, but I think it is a win. And if you do want
to log the exception there is always this:
try:
... except Exception as e:
logging.exception(e)
raise
to recite the exception into the log and still let it bubble out
normally.
The problem with the "except Exception" pattern is that it catches and
hides
every exception, not merely the narrow set of specific exceptions that you understand.
Finally, it is frowned upon to raise a bare Exception class. In
python 3 I believe it is actually forbidden, so it is nonportable
anyway. But even In Python to it is best to supply an Exception
instance, not the class:
raise SystemExit(1)
All the functions in try block have exception bubbled out using raise
Example for create_logdir() here is the function definition
def create_logdir():
try:
os.makedirs(LOG_DIR)
except OSError as e:
sys.stderr.write("Failed to create log directory...Exiting !!!")
raise
print "log file: " + corrupt_log
return True
def main():
try:
create_logdir()
except Exception as e:
logging.exception(e)
raise SystemExit
(a) In case if create_logdir() fails we will get the below error ,is
this fine or do I need to improve this code.
Failed to create log directory...Exiting !!!ERROR:root:[Errno 17] File
exists: '/var/log/dummy'
Traceback (most recent call last):
File "corrupt_test.py", line 245, in main
create_logdir()
File "corrupt_test.py", line 53, in create_logdir
os.makedirs(LOG_DIR)
File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
OSError: [Errno 17] File exists: '/var/log/dummy'
I prefer the bubble out approach, perhap with a log or warning
messages as you have done, eg:
logging.exception("create_logdir failed: makedirs(%r): %s" %
(LOG_DIR, e)) raise
(Also not that that log message records more context: context is very
useful when debugging problems.)
For very small scripts sys.stderr.write is ok, but in general any of
your functions that turned out to be generally useful might migrate
into a library in order to be reused; consider that stderr is not
always the place for messages; instead reading for the logging module
with error() or wanr() or exception() as appropriate. There is more
scope for configuring where the output goes that way without wiring
it into your inner functions.
Can I have just raise , instead of SystemExit or sys.exit(1) . This
looks wrong to me
def main():
try:
create_logdir()
except Exception as e
logging.exception(e)
raise
This is what I would do, myself.
Think: has the exception been "handled", meaning has the situation
been dealt with because it was expected? If not, let the exception
bubble out so that the user knows that something not understood by
the program has occurred.
Finally, it is generally bad to SystemExit or sys.exit() from inside
anything other than the outermost main() function. And I resist it
even there; the main function, if written well, may often be called
from somewhere else usefully, and that makes it effectively a library
function (it has been reused). Such a function should not
unilaterally abort the program. How rude! Instead, let the exception
bubble out: perhaps the caller of main() expects it and can handle
it. By aborting and not "raise"ing, you have deprived the caller of
the chance to do something appropriate, even though you yourself
(i.e. "main") do not know enough context to handle the exception.
So I am for "raise" myself. And then only because you want to log the
error. If you didn't want to log the exception you could avoid the
try/except entirely and have simpler code: let the caller worry
about unhandled exceptions!

SystemExit is an exception, which basically means that your progam had a behavior such that you want to stop it and raise an error. sys.exit is the function that you can call to exit from your program, possibily giving a return code to the system.
EDIT: they are indeed the same thing, so the only difference is in the logic behind in your program. An exception is some kind of "unwanted" behaviour, whether a call to a function is, from a programmer point of view, more of a "standard" action.

Why can't I handle a KeyboardInterrupt in python?

I'm writing python 2.6.6 code on windows that looks like this:
try:
dostuff()
except KeyboardInterrupt:
print "Interrupted!"
except:
print "Some other exception?"
finally:
print "cleaning up...."
print "done."
dostuff() is a function that loops forever, reading a line at a time from an input stream and acting on it. I want to be able to stop it and clean up when I hit ctrl-c.
What's happening instead is that the code under except KeyboardInterrupt: isn't running at all. The only thing that gets printed is "cleaning up...", and then a traceback is printed that looks like this:
Traceback (most recent call last):
File "filename.py", line 119, in <module>
print 'cleaning up...'
KeyboardInterrupt
So, exception handling code is NOT running, and the traceback claims that a KeyboardInterrupt occurred during the finally clause, which doesn't make sense because hitting ctrl-c is what caused that part to run in the first place! Even the generic except: clause isn't running.
EDIT: Based on the comments, I replaced the contents of the try: block with sys.stdin.read(). The problem still occurs exactly as described, with the first line of the finally: block running and then printing the same traceback.
EDIT #2:
If I add pretty much anything after the read, the handler works. So, this fails:
try:
sys.stdin.read()
except KeyboardInterrupt:
...
But this works:
try:
sys.stdin.read()
print "Done reading."
except KeyboardInterrupt:
...
Here's what's printed:
Done reading. Interrupted!
cleaning up...
done.
So, for some reason, the "Done reading." line is printed, even though the exception occurred on the previous line. That's not really a problem - obviously I have to be able to handle an exception anywhere inside the "try" block. However, the print doesn't work normally - it doesn't print a newline afterwards like it's supposed to! The "Interruped" is printed on the same line... with a space before it, for some reason...? Anyway, after that the code does what it's supposed to.
It seems to me that this is a bug in handling an interrupt during a blocked system call.

Asynchronous exception handling is unfortunately not reliable (exceptions raised by signal handlers, outside contexts via C API, etc). You can increase your chances of handling the async exception properly if there is some coordination in the code about what piece of code is responsible for catching them (highest possible in the call stack seems appropriate except for very critical functions).
The called function (dostuff) or functions further down the stack may itself have a catch for KeyboardInterrupt or BaseException that you didn't/couldn't account for.
This trivial case worked just fine with python 2.6.6 (x64) interactive + Windows 7 (64bit):
>>> import time
>>> def foo():
... try:
... time.sleep(100)
... except KeyboardInterrupt:
... print "INTERRUPTED!"
...
>>> foo()
INTERRUPTED! #after pressing ctrl+c
EDIT:
Upon further investigation, I tried what I believe is the example that others have used to reproduce the issue. I was lazy so I left out the "finally"
>>> def foo():
... try:
... sys.stdin.read()
... except KeyboardInterrupt:
... print "BLAH"
...
>>> foo()
This returns immediately after hitting CTRL+C. The interesting thing happened when I immediately tried to call foo again:
>>> foo()
Traceback (most recent call last):
File "c:\Python26\lib\encodings\cp437.py", line 14, in decode
def decode(self,input,errors='strict'):
KeyboardInterrupt
The exception was raised immediately without me hitting CTRL+C.
This would seem to make sense - it appears that we are dealing with nuances in how asynchronous exceptions are handled in Python. It can take several bytecode instructions before the async exception is actually popped and then raised within the current execution context. (That's the behavior that I've seen when playing with it in the past)
See the C API: http://docs.python.org/c-api/init.html#PyThreadState_SetAsyncExc
So this somewhat explains why KeyboardInterrupt gets raised in the context of the execution of the finally statement in this example:
>>> def foo():
... try:
... sys.stdin.read()
... except KeyboardInterrupt:
... print "interrupt"
... finally:
... print "FINALLY"
...
>>> foo()
FINALLY
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in foo
KeyboardInterrupt
There could be some crazy mixing of custom signal handlers mixed with the interpreter's standard KeyboardInterrupt/CTRL+C handler that's resulting in this sort of behavior. For example, the read() call sees the signal and bails, but it re-raises the signal after deregistering it's handler. I wouldn't know for sure without inspecting the interpreter codebase.
This is why I generally shy away from making use of async exceptions....
EDIT 2
I think there's a good case for a bug report.
Again more theories...(just based on reading code) See the file object source: http://svn.python.org/view/python/branches/release26-maint/Objects/fileobject.c?revision=81277&view=markup
file_read calls Py_UniversalNewlineFread(). fread can return with an error with errno = EINTR (it performs its own signal handling). In this case Py_UniversalNewlineFread() bails but does not perform any signal checking with PyErr_CheckSignals() so that the handlers can get called synchronously. file_read clears the file error but also does not call PyErr_CheckSignals().
See getline() and getline_via_fgets() for examples of how it's used. The pattern is documented in this bug report for a similar issue: ( http://bugs.python.org/issue1195 ). So it seems that the signal is handled at an indeterminate time by the interpreter.
I guess there's little value in diving any deeper since it's still not clear whether the sys.stdin.read() example is a proper analog of your "dostuff()" function. (there could be multiple bugs at play)

sys.stdin.read() is a system call and so the behavior is going to be different for each system. For windows 7 I think what is happening is that the input is being buffered and so you're getting where sys.stdin.read() is returning everything up to the Ctrl-C and as soon as you access sys.stdin again it'll send the "Ctrl-C".
try the following,
def foo():
try:
print sys.stdin.read()
print sys.stdin.closed
except KeyboardInterrupt:
print "Interrupted!"
This suggests that there is a buffering of stdin going on that is causing another call to stdin to recognize the keyboard input
def foo():
try:
x=0
while 1:
x += 1
print x
except KeyboardInterrupt:
print "Interrupted!"
there doesn't appear to be a problem.
Is dostuff() reading from stdin?

Having similar problem and this is my workaround:
try:
some_blocking_io_here() # CTRL-C to interrupt
except:
try:
print() # any i/o will get the second KeyboardInterrupt here?
except:
real_handler_here()

Here's a guess about what's happening:
pressing Ctrl-C breaks the "print" statement (for whatever reason... bug? Win32 limitation?)
pressing Ctrl-C also throws the first KeyboardInterrupt, in dostuff()
The exception handler runs and tries to print "Interrupted", but the "print" statement is broken and throws another KeyboardInterrupt.
The finally clause runs and tries to print "cleaning up....", but the "print" statement is broken and throws yet another KeyboardInterrupt.

This works for me:
import sys
if __name__ == "__main__":
try:
sys.stdin.read()
print "Here"
except KeyboardInterrupt:
print "Worked"
except:
print "Something else"
finally:
print "Finally"
Try putting a line outside of your dostuff() function or move the loop condition outside of the function. For example:
try:
while True:
dostuff()
except KeyboardInterrupt:
print "Interrupted!"
except:
print "Some other exception?"
finally:
print "cleaning up...."
print "done."

def foo():
try:
x=0
while 1:
x+=1
print (x)
except KeyboardInterrupt:
print ("interrupted!!")
foo()
That works fine.