Develop Reference

Is there a way to implement a program termination protocol?

In my case, I am handling databases throughout the entire runtime of my program, and so I need to keep my 'cursor' open for the entire program. Is there anyway I can implement a termination protocol, so that when I terminate its execution or an error arises, I am able to run this quick piece of code that simply closes the cursor (I am using python sockets btw).
I would suspect I could do something like this:
if __name__ == "__main__":
Menu()
cursor.close()
However, the only reason that this doesn't work in my case is that Menu is simply starting up threads, and so its execution continues on, returning me back to cursor.close() whilst my program continues to run.
I'm not sure if there is a way to get around this problem.

Yes, you could use the signal library in python to achieve some of this functionality, in particular, capturing program termination as well interrupts to the program like ctrl + c. Example:
# a function to register the signal handlers
# once the program terminates or is halted by an interrupt like ctrl + c it executes the quit_gracefully function
def register_signal_handler():
signal.signal(signal.SIGINT, quit_gracefully)
signal.signal(signal.SIGTERM, quit_gracefully)
return
def quit_gracefully():
# close connections etc.
in case of a different error you could use a try-except block which handles the error and runs the quit_gracefully function in the except.
try:
# some code
except:
quit_gracefully()
EDIT:
this is a good post on signal. How do I capture SIGINT in Python?

You can also use the atexit module: https://docs.python.org/3/library/atexit.html.
Something like this:
import atexit
#atexit.register
def close_cursor():
print("Closing cursor before exiting.")
cursor.close()

Catch all exceptions in a class

I'm wondering if anybody would have an idea to catch all exceptions in a running thread. My program is started as follow, by a service
def main():
global RUNNING
signal.signal(signal.SIGINT, stopHandler)
signal.signal(signal.SIGTERM, stopHandler)
projectAlice = ProjectAlice()
try:
while RUNNING:
time.sleep(0.1)
except KeyboardInterrupt:
pass
finally:
projectAlice.onStop()
_logger.info('Project Alice stopped, see you soon!')
So a CTRL-C or a signal can stop it. ProjectAlice runs forever and answers to mqtt topics that are sent by Snips. It uses paho-mqtt with loop_forever. As it's pretty large, errors can occur, even though they shouldn't. I cover as many as I can, but today, as an exemple, google-translate started to throw out errors, because it can't use the service anymore (free...). Unhandled errors.... So the thread crashes and ProjectAlice is left as is. I would like to, as it's possible per exemple in Java, to super catch all exceptions and work further from there

Here's a simple solution to override the python exception hook, thus enabling you to handle uncaught exceptions:
import sys
def my_custom_exception_hook(exctype, value, tb):
print('Yo, do stuff here, handle specific exceptions and raise others or whatever')
and before your actual code starts do:
sys.excepthook = my_custom_exception_hook

A simple except Exception: will catch all exceptions except KeyboardInterrupt and SystemExit within the same thread.
You'll have to have the try: except ...: block within the code that is run in the thread to catch exceptions occurring in the thread.

How to handle a specific uncaught exception?

I would like to handle one specific exception in my script in a single place without resorting to a try/exception everytime*. I was hoping that the code below would do this:
import sys
def handle(exc_type, exc_value, exc_traceback):
if issubclass(exc_type, ValueError):
print("ValueError handled here and the script continues")
return
# follow default behaviour for the exception
sys.__excepthook__(exc_type, exc_value, exc_traceback)
sys.excepthook = handle
print("hello")
raise ValueError("wazaa")
print("world")
a = 1/0
The idea was that ValueError would be handled "manually" and the script would continue running (return to the script). For any other error (ZeroDivisionError in the case above), the normal traceback and script crash would ensue.
What happens is
$ python scratch_13.py
hello
ValueError handled here and the script continues
Process finished with exit code 1
The documentation mentions that (emphasis mine)
When an exception is raised and uncaught, the interpreter calls
sys.excepthook with three arguments, the exception class, exception
instance, and a traceback object. In an interactive session this
happens just before control is returned to the prompt; in a Python
program this happens just before the program exits.
which would mean that when I am in handler() it is already too late as the script has decided to die anyway and my only possibility is to influence how the traceback will look like.
Is there a way to ignore a specific exception globally in a script ?
* this is for a debugging context where the exception would normally be raised and crash the script (in production) but in some specific cases (a dev platform for instance), this specific exception needs to just be discarded. Otherwise I would have put a try/exception clause everywhere where the issue could arise.

One way to do it is to use contextlib.suppress and have a global tuple of suppressed Exceptions:
suppressed = (ValueError,)
And then anywhere where the error might occure you just wrap it in with suppress(*suppressed):
print("hello")
with suppress(*suppressed): # gets ignored
raise ValueError("wazaa")
print("world")
a = 1/0 # raise ZeroDivisionError
And then in production you just change suppressed to ():
suppressed = ()
print("hello")
with suppress(*suppressed):
raise ValueError("wazaa") # raises the error
print("world")
a = 1/0 # doesn't get executed
I think this is the best you can do. You can't ignore the exception completly globally, but you can make it so you only have to change on place.

Unhandled exceptions in PyQt5

Have a look at the following MWE.
import sys
from PyQt5.QtWidgets import QMainWindow, QPushButton, QApplication
class MainWindow(QMainWindow):
def __init__(self, parent=None):
super().__init__(parent)
self.button = QPushButton('Bham!')
self.setCentralWidget(self.button)
self.button.clicked.connect(self.btnClicked)
def btnClicked(self):
print(sys.excepthook)
raise Exception
#import traceback
#sys.excepthook = traceback.print_exception
if __name__ == '__main__':
app = QApplication(sys.argv)
mainWindow = MainWindow()
mainWindow.show()
app.exec_()
I have a number of questions. I don't know if they are all related (I guess so), so forgive me if they are not.
When I run the above code from the terminal, all is fine. The program runs, if I click the button it prints the traceback and dies. If I run it inside an IDE (I tested Spyder and PyCharm), the traceback is not displayed. Any idea why? Essentially the same question was raised in other posts also on SO, here and here. Please don't mark this as a duplicate of either of those; please read on.
By adding the commented lines, the traceback is again displayed properly. However, they also have the nasty side effect that the app does no longer terminate on unhandled exceptions! I have no idea why this happens, as AFAIK excepthook only prints the traceback, it cannot prevent the program from exiting. At the moment it is called, it's too late for rescue.
Also, I don't understand how Qt comes into play here, as exceptions that are not thrown inside a slot still crash the app as I would expect. No matter if I change excepthook or not, PyQt does not seem to override it as well (at least the print seems to suggest so).
FYI, I am using Python 3.5 with PyQt 5.6, and I am aware of the changes in the exception handling introduced in PyQt 5.5. If those are indeed the cause for the behaviour above, I would be glad hear some more detailed explanations.

When an exception happens inside a Qt slot, it's C++ which called into your Python code. Since Qt/C++ know nothing about Python exceptions, you only have two possibilities:
Print the exception and return some default value to C++ (like 0, "" or NULL), possibly with unintended side effects. This is what PyQt < 5.5 does.
Print the exception and then call qFatal() or abort(), causing the application to immediately exit inside C++. That's what PyQt >= 5.5 does, except when you have a custom excepthook set.
The reason Python still doesn't terminate is probably because it can't, as it's inside some C++ code. The reason your IDE isn't showing the stack is probably because it doesn't deal with the abort() correctly - I'd suggest opening a bug against the IDE for that.

Whilst #the-compiler's answer is correct in explaining why it happens, I thought I might provide a workaround if you'd like these exceptions to be raised in a more pythony way.
I decorate any slots with this decorator, which catches any exceptions in the slot and saves them to a a global variable:
exc_info = None
def pycrash(func):
"""Decorator that quits the qt mainloop and stores sys.exc_info. We will then
raise it outside the qt mainloop, this is a cleaner crash than Qt just aborting as
it does if Python raises an exception during a callback."""
def f(*args, **kwargs):
global exc_info
try:
return func(*args, **kwargs)
except:
if exc_info is None # newer exceptions don't replace the first one
exc_info = sys.exc_info()
qapplication.exit()
return f
Then just after my QApplication's exec(), I check the global variable and raise if there's anything there:
qapplication.exec_()
if exc_info is not None:
type, value, traceback = exc_info
raise value.with_traceback(traceback)
This is not ideal because quitting the mainloop doesn't stop other slots higher in the stack from still completing, and if the failed slot affects them, they might see some unexpected state. But IMHO it's still much better than PyQt just aborting with no cleanup.

Catching KeyboardInterrupt in Python during program shutdown

I'm writing a command line utility in Python which, since it is production code, ought to be able to shut down cleanly without dumping a bunch of stuff (error codes, stack traces, etc.) to the screen. This means I need to catch keyboard interrupts.
I've tried using both a try catch block like:
if __name__ == '__main__':
try:
main()
except KeyboardInterrupt:
print 'Interrupted'
sys.exit(0)
and catching the signal itself (as in this post):
import signal
import sys
def sigint_handler(signal, frame):
print 'Interrupted'
sys.exit(0)
signal.signal(signal.SIGINT, sigint_handler)
Both methods seem to work quite well during normal operation. However, if the interrupt comes during cleanup code at the end of the application, Python seems to always print something to the screen. Catching the interrupt gives
^CInterrupted
Exception KeyboardInterrupt in <bound method MyClass.__del__ of <path.to.MyClass object at 0x802852b90>> ignored
whereas handling the signal gives either
^CInterrupted
Exception SystemExit: 0 in <Finalize object, dead> ignored
or
^CInterrupted
Exception SystemExit: 0 in <bound method MyClass.__del__ of <path.to.MyClass object at 0x802854a90>> ignored
Not only are these errors ugly, they're not very helpful (especially to an end user with no source code)!
The cleanup code for this application is fairly big, so there's a decent chance that this issue will be hit by real users. Is there any way to catch or block this output, or is it just something I'll have to deal with?

Checkout this thread, it has some useful information about exiting and tracebacks.
If you are more interested in just killing the program, try something like this (this will take the legs out from under the cleanup code as well):
if __name__ == '__main__':
try:
main()
except KeyboardInterrupt:
print('Interrupted')
try:
sys.exit(130)
except SystemExit:
os._exit(130)
[Edited to change the exit code as suggested in comments. 130 is the code typically returned on Linux for a script terminated by Ctrl-C. We may not be on Linux, but the important thing is to return a non-zero value, and 130 is as good as any.]

You could ignore SIGINTs after shutdown starts by calling signal.signal(signal.SIGINT, signal.SIG_IGN) before you start your cleanup code.