Is there any way to run one last command before a running Python script is stopped by being killed by some other script, keyboard interrupt etc.
import time
try:
time.sleep(10)
finally:
print "clean up"
clean up
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
KeyboardInterrupt
If you need to catch other OS level interrupts, look at the signal module:
http://docs.python.org/library/signal.html
Signal Example
from signal import *
import sys, time
def clean(*args):
print "clean me"
sys.exit(0)
for sig in (SIGABRT, SIGBREAK, SIGILL, SIGINT, SIGSEGV, SIGTERM):
signal(sig, clean)
time.sleep(10)
You could use the atexit module. With it, you can register a function which will be called at program termination. An example from here: http://docs.python.org/library/atexit.html
try:
_count = int(open("/tmp/counter").read())
except IOError:
_count = 0
def incrcounter(n):
global _count
_count = _count + n
def savecounter():
open("/tmp/counter", "w").write("%d" % _count)
import atexit
atexit.register(savecounter)
You can also pass positional and keyword parameters to the function you want to call at program termination.
Note that there are a few circumstances listed in the docs in which your handler won't be called:
Note: The functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called.
As such, you may want to also register a signal handler.
import signal
import sys
import time
def cleanup(*args):
print 'Exiting'
sys.exit(0)
signal.signal(signal.SIGINT, cleanup)
signal.signal(signal.SIGTERM, cleanup)
while True:
time.sleep(60) # less busy loop
WIth apologies to 'Unknown' for taking their answer and correcting it as though it was my own answer, but my edits were rejected.
The approved answer contains an error that will cause a segfault.
You cannot use sys.exit() in a signal handler, but you can use os._exit so that it becomes:
from signal import *
import os, time
def clean(*args):
print "clean me"
os._exit(0)
for sig in (SIGABRT, SIGINT, SIGTERM):
signal(sig, clean)
time.sleep(10)
SIGBREAK may be used if the target platform is Windows.
Depending on the use case and the need to cleanup in the event of fatal errors - you may add SIGSEGV and SIGILL but generally this is not advised since the program state may be such that you create an infinite loop.
Use the atexit module to register a function that will be called at the end.
import atexit
atexit.register(some_function)
Related
When I press Ctrl+C the call jumps into signal_handler as expected, but the greenlets are not getting killed as they continue the process.
# signal handler to process after catch ctrl+c command
def signal_handler(signum, frame):
print("Inside Signal Handler")
gevent.sleep(10)
print("Signal Handler After sleep")
gevent.joinall(maingreenlet)
gevent.killall(maingreenlet,block=True,timeout=10)
gevent.kill(block=True)
sys.exit(0)
def main():
signal.signal(signal.SIGINT, signal_handler) // Catching Ctrl+C
try:
maingreenlet = [] // Creating a list of greenlets
while True:
for key,profileval in profile.items():
maingreenlet.append(gevent.spawn(key,profileval)) # appending all grrenlets to list
gevent.sleep(0)
except (Error) as e:
log.exception(e)
raise
if __name__ == "__main__":
main()
The main reason your code is not working is because the variable maingreenlet is defined inside the main function, and is out of the scope of the signal_handler function which tries to access it. Your code should throw an error like this:
NameError: global name 'maingreenlet' is not defined
If you were to move the line maingreenlet = [] into the global scope, i.e. anywhere outside of the two def blocks, the greenlets should get killed without problem.
Of course that's after you fix the other issues in your code, like using // instead of # to start the comments, or calling the function gevent.kill with the wrong parameter. (you didn't specify your gevent version but I assume the current version 1.3.7) Actually this function call is redundant after you call gevent.killall.
Learn to use a Python debugger liker pdb or rpdb2 to help you debug your code. It'll save your precious time in the long run.
Main Program
import os,pymsgbox
try:
c=os.getpid()
with open("sample.txt","w") as fp:
fp.write(str(c))
while True:
print ""
except Exception,e:
print e
finally:
pymsgbox.alert("Done")
File which actually kill process
import os,signal
with open("sample.txt","r") as fp:
pid=fp.read()
os.kill(int(pid),signal.SIGTERM)
In finally block i want to close all open connections.So how to call finally block
Tried this:
import atexit
def final_sub():
pymsgbox.alert("Done")
atexit.register(final_sub)
The atexit module defines a single function to register cleanup functions. More info could be found here: atexit — Exit handlers
also.. instead of calling atexit.register, you can do the same via decorator:
#atexit.register
def final_sub():
pymsgbox.alert("Done")
But, there was a discussion here
Go through it, and atexit may not work in your case. So you need a handler for the SIGTERM signal.
Try to add this code at the start of your file and "finally" starts to work as you want on SIGTERM. Hadler raises normal SystemExit on SIGTERM, so Finally works:
import signal
class SigTerm(SystemExit): pass
def sigterm(sig,frm): raise SigTerm
signal.signal(15,sigterm)
So, the full code will be something like this:
import os
import pymsgbox
import signal
class SigTerm(SystemExit): pass
def sigterm(sig,frm): raise SigTerm
signal.signal(15,sigterm)
def final_sub():
pymsgbox.alert("Done")
try:
c=os.getpid()
with open("sample.txt","w") as fp:
fp.write(str(c))
while True:
print("")
except Exception as e:
print(e)
finally:
final_sub()
I have just moved on to python3 as a result of its concurrent futures module. I was wondering if I could get it to detect errors. I want to use concurrent futures to parallel program, if there are more efficient modules please let me know.
I do not like multiprocessing as it is too complicated and not much documentation is out. It would be great however if someone could write a Hello World without classes only functions using multiprocessing to parallel compute so that it is easy to understand.
Here is a simple script:
from concurrent.futures import ThreadPoolExecutor
def pri():
print("Hello World!!!")
def start():
try:
while True:
pri()
except KeyBoardInterrupt:
print("YOU PRESSED CTRL+C")
with ThreadPoolExecutor(max_workers=3) as exe:
exe.submit(start)
The above code was just a demo, of how CTRL+C will not work to print the statement.
What I want is to be able to call a function is an error is present. This error detection must be from the function itself.
Another example
import socket
from concurrent.futures import ThreadPoolExecutor
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
def con():
try:
s.connect((x,y))
main()
except: socket.gaierror
err()
def err():
time.sleep(1)
con()
def main():
s.send("[+] Hello")
with ThreadPoolExecutor as exe:
exe.submit(con)
Way too late to the party, but maybe it'll help someone else...
I'm pretty sure the original question was not really answered. Folks got hung up on the fact that user5327424 was using a keyboard interrupt to raise an exception when the point was that the exception (however it was caused) was not raised. For example:
import concurrent.futures
def main():
numbers = range(10)
with concurrent.futures.ThreadPoolExecutor() as executor:
results = {executor.submit(raise_my_exception, number): number for number in numbers}
def raise_my_exception(number):
print('Proof that this function is getting called. %s' % number)
raise Exception('This never sees the light of day...')
main()
When the example code above is executed, you will see the text inside the print statement displayed on the screen, but you will never see the exception. This is because the results of each thread are held in the results object. You need to iterate that object to get to your exceptions. The following example shows how to access the results.
import concurrent.futures
def main():
numbers = range(10)
with concurrent.futures.ThreadPoolExecutor() as executor:
results = {executor.submit(raise_my_exception, number): number for number in numbers}
for result in results:
# This will cause the exception to be raised (but only the first one)
print(result.result())
def raise_my_exception(number):
print('Proof that this function is getting called. %s' % number)
raise Exception('This will be raised once the results are iterated.')
main()
I'm not sure I like this behavior or not, but it does allow the threads to fully execute, regardless of the exceptions encountered inside the individual threads.
Here's a solution. I'm not sure you like it, but I can't think of any other. I've modified your code to make it work.
from concurrent.futures import ThreadPoolExecutor
import time
quit = False
def pri():
print("Hello World!!!")
def start():
while quit is not True:
time.sleep(1)
pri()
try:
pool = ThreadPoolExecutor(max_workers=3)
pool.submit(start)
while quit is not True:
print("hei")
time.sleep(1)
except KeyboardInterrupt:
quit = True
Here are the points:
When you use with ThreadPoolExecutor(max_workers=3) as exe, it waits until all tasks have been done. Have a look at Doc
If wait is True then this method will not return until all the pending futures are done executing and the resources associated with the executor have been freed. If wait is False then this method will return immediately and the resources associated with the executor will be freed when all pending futures are done executing. Regardless of the value of wait, the entire Python program will not exit until all pending futures are done executing.
You can avoid having to call this method explicitly if you use the with statement, which will shutdown the Executor (waiting as if Executor.shutdown() were called with wait set to True)
It's like calling join() on a thread.
That's why I replaced it with:
pool = ThreadPoolExecutor(max_workers=3)
pool.submit(start)
Main thread must be doing "work" to be able to catch a Ctrl+C. So you can't just leave main thread there and exit, the simplest way is to run an infinite loop
Now that you have a loop running in main thread, when you hit CTRL+C, program will enter the except KeyboardInterrupt block and set quit=True. Then your worker thread can exit.
Strictly speaking, this is only a workaround. It seems to me it's impossible to have another way for this.
Edit
I'm not sure what's bothering you, but you can catch exception in another thread without problem:
import socket
import time
from concurrent.futures import ThreadPoolExecutor
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
def con():
try:
raise socket.gaierror
main()
except socket.gaierror:
print("gaierror occurred")
err()
def err():
print("err invoked")
time.sleep(1)
con()
def main():
s.send("[+] Hello")
with ThreadPoolExecutor(3) as exe:
exe.submit(con)
Output
gaierror occurred
err invoked
gaierror occurred
err invoked
gaierror occurred
err invoked
gaierror occurred
...
In a multi-threaded design, I want to do some clean steps when the program exits abnormally. The running thread should clean up the current task and then quit, rather than be killed immediately and leave some dirty data. I found that using threading module could not catch KeyInterrupt exception.
Here is my test code:
#!/usr/bin/env python3
from time import sleep
def do_sth():
print("I'm doing something...")
sleep(10)
if __name__ == "__main__":
do_sth()
Python will raise KeyInterrupt exception when I press CTRL-c
$ Python3 test.py
I'm doing something ...
^C
Traceback (most recent call last):
File "test.py", line 10, in <module>
do_sth ()
File "test.py", line 7, in do_sth
sleep (10)
KeyboardInterrupt
So I can catch this exception.
def do_sth ():
try:
print ("I'm doing something ...")
sleep (10)
except (KeyboardInterrupt, SystemExit):
print ("I'm doing some clean steps and exit.")
But when I use threading module, this exception is not raised at all.
#!/usr/bin/env python3
from time import sleep
import threading
def do_sth():
print("I'm doing something...")
sleep(10)
if __name__ == '__main__':
t = threading.Thread(target=do_sth)
t.start()
t.join()
result:
$ python3 test.py
I'm doing something...
^C
The running thread has been killed directly and no exception is raised.
How do I deal with this?
One way is to handle KeyboardInterrupt exceptions.
Another thing to do in such scenarios is to manage the state of your application across all threads.
One of the solutions is to add support for Signals in your code. It allows graceful handling of the shutting down of your process.
Here's one simple setup for that:
class SignalHandler:
continue_running = True
def __init__(self):
signal.signal(signal.SIGUSR2, self.signal_handler)
signal.signal(signal.SIGTERM, self.signal_handler)
signal.signal(signal.SIGINT, self.signal_handler)
logging.info("SignalHandler::Init")
def signal_handler(self, num, stack):
logging.warning('Received signal %d in %s' % (num, threading.currentThread()))
SignalHandler.continue_running = False
logging.warning("Time to SHUT DOWN ALL MODULES")
All threads would try and utilise the status from SignalHandler.continue_running to ensure that they all know when to stop.
If somebody tried to kill this python process by calling kill -2 [PID] for example - all threads will come to know about the need to shut down.
I wrote those test python code as following:
import signal
import time
import os
def handler(signum, frame):
print "do whatever, like call thread.interrupt_main()"
return
signal.signal(signal.SIGINT, handler)
while 1:
try:
time.sleep(10)
except:
os.kill(int(os.getpid()), signal.SIGINT)
pass
when i excute this test code on windows, the process print "do whatever, like call thread.interrupt_main()", then exit;
on linux, it works correctly.
why on windows it not work?
http://docs.python.org/2/library/os.html#os.kill
Windows: The signal.CTRL_C_EVENT and signal.CTRL_BREAK_EVENT signals
are special signals which can only be sent to console processes which
share a common console window, e.g., some subprocesses. Any other
value for sig will cause the process to be unconditionally killed by
the TerminateProcess API, and the exit code will be set to sig.
Try following code:
import time
while 1:
try:
time.sleep(1)
except KeyboardInterrupt:
print "do whatever, like call thread.interrupt_main()"