Develop Reference

Is there any graceful way to interrupt a python concurrent future result() call?

The only mechanism I can find for handling a keyboard interrupt is to poll. Without the while loop below, the signal processing never happens and the process hangs forever.
Is there any graceful mechanism for allowing a keyboard interrupt to function when given a concurrent future object?
Putting polling loops all over my code base seems to defeat the purpose of using futures at all.
More info:
Waiting on the future in the main thread in Windows blocks all signal handling, even if it's fully cancellable and even if it has not "started" yet. The word "exiting" doesn't even print. So 'cancellability' is only part (the easy part) of the issue.
In my real code, I obtain futures via executors (run coro threadsafe, in this case), this was just a simplified example
import concurrent.futures
import signal
import time
import sys
fut = concurrent.futures.Future()
def handler(signum, frame):
print("exiting")
fut.cancel()
signal.signal(signal.SIGINT, orig)
sys.exit()
orig = signal.signal(signal.SIGINT, handler)
# a time sleep is fully interruptible with a signal... but a future isnt
# time.sleep(100)
while True:
try:
fut.result(.03)
except concurrent.futures.TimeoutError:
pass

Defeating the purpose or not, it is how futures currently work in Python.
First of all, directly instantiating a Future() should only be done for testing purposes, normally you would obtain an instance by submitting work to an executor.
Furthermore, you cannot really cancel() a future cleanly that is executing in a thread; attempting to do so will make cancel() return False.
Indeed, in the following test I get could cancel: False printed out:
import concurrent.futures
import signal
import time
import sys
def task(delay):
time.sleep(delay)
return delay
def handler(signum, frame):
print("exiting")
print("could cancel:", fut.cancel())
raise RuntimeError("if in doubt, use brute force")
signal.signal(signal.SIGINT, handler)
with concurrent.futures.ThreadPoolExecutor() as executor:
fut = executor.submit(task, 240)
try:
print(fut.result())
except Exception as ex:
print(f"fut.result() ==> {type(ex).__name__}: {ex}")
If I also raise an exception in my signal handler, that exception is caught when trying to fetch the result, and I'm also seeing fut.result() ==> RuntimeError: if in doubt, use brute force printed out. However, that does not exit the executor loop immediately either, because the task is still running there.
Interestingly, clicking Ctrl-C a couple more times would eventually break even the cleanup loop, and the program would exit, but it's probably not what you're after. You might also be able to kill off futures more freely by employing a ProcessPoolExecutor, but .cancel() would still return False for running futures.
In that light, I think your approach to poll result() is not an unreasonable one. If possible, you could also move your program to asyncio where you would be able to cancel tasks at the yield points or I/O, or somehow make your task itself react to user input by exiting earlier, potentially based on information from a signal.
For instance, here I'm setting a global variable from my interrupt handler, which is then polled from my task:
import concurrent.futures
import signal
import time
import sys
interrupted = False
def task(delay):
slept = 0
for _ in range(int(delay)):
time.sleep(1)
slept += 1
if interrupted:
print("interrupted, wrapping up work prematurely")
break
return slept
def handler(signum, frame):
global interrupted
print("exiting")
print("could cancel:", fut.cancel())
interrupted = True
signal.signal(signal.SIGINT, handler)
with concurrent.futures.ThreadPoolExecutor() as executor:
fut = executor.submit(task, 40)
try:
print(fut.result())
except Exception as ex:
print(f"fut.result() ==> {type(ex).__name__}: {ex}")
Now I'm able to interrupt my work in a more fine grained fashion:
^Cexiting
could cancel: False
interrupted, wrapping up work prematurely
5
In addition, you might also be able to split your work into many smaller tasks, then you could cancel any futures that aren't running yet, also improving responsiveness to SIGINT or other types of user input.

OK, I wrote a solution to this based on digging in cypython source and some bug reports - but it's not pretty.
If you want to be able to interrupt a future, especially on Windows, the following seems to work:
#contextlib.contextmanager
def interrupt_futures(futures): # pragma: no cover
"""Allows a list of futures to be interrupted.
If an interrupt happens, they will all have their exceptions set to KeyboardInterrupt
"""
# this has to be manually tested for now, because the tests interfere with the test runner
def do_interr(*_):
for ent in futures:
try:
ent.set_exception(KeyboardInterrupt)
except:
# if the future is already resolved or cancelled, ignore it
pass
return 1
if sys.platform == "win32":
from ctypes import wintypes # pylint: disable=import-outside-toplevel
kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
CTRL_C_EVENT = 0
CTRL_BREAK_EVENT = 1
HANDLER_ROUTINE = ctypes.WINFUNCTYPE(wintypes.BOOL, wintypes.DWORD)
#HANDLER_ROUTINE
def handler(ctrl):
if ctrl == CTRL_C_EVENT:
handled = do_interr()
elif ctrl == CTRL_BREAK_EVENT:
handled = do_interr()
else:
handled = False
# If not handled, call the next handler.
return handled
if not kernel32.SetConsoleCtrlHandler(handler, True):
raise ctypes.WinError(ctypes.get_last_error())
was = signal.signal(signal.SIGINT, do_interr)
yield
signal.signal(signal.SIGINT, was)
# restore default handler
kernel32.SetConsoleCtrlHandler(handler, False)
else:
was = signal.signal(signal.SIGINT, do_interr)
yield
signal.signal(signal.SIGINT, was)
This allows you to do this:
with interrupt_futures([fut]):
fut.result()
For the duration of that call, interrupt signals will be intercepted and will result in the future raising a KeyboardInterrupt to the caller requesting the result - instead of simply ignoring all interrupts.

Python click command exit flow

I have this python click CLI design (wificli.py). At the end of command execution, it prints only respective print messages.
For example, when executed command python3.7 wificli.py png, it prints only png and when executed command python3.7 wificli.py terminal it prints only terminal.
As I have shown, I am expecting that it would also print End of start function and End of the main function but it is not. The idea is that to do clean up of resources only at one place rather than at each exit point of the respective command.
import click
#click.group()
#click.option('--ssid', help='WiFi network name.')
#click.option('--security', type=click.Choice(['WEP', 'WPA', '']))
#click.option('--password', help='WiFi password.')
#click.pass_context
def main(ctx, ssid: str, security: str = '', password: str = ''):
ctx.obj['ssid'] = ssid
ctx.obj['security'] = security
ctx.obj['password'] = password
#main.command()
#click.pass_context
def terminal(ctx):
print('terminal')
#main.command()
#click.option('--filename', help='full path to the png file')
#click.pass_context
def png(ctx, filename, scale: int = 10):
print('png')
def start():
main(obj={})
print('End of start function')
if __name__ == '__main__':
start()
print('End of main function')
When executed

As you've not asked a specific question, I can only post what worked for me with the reasoning behind it, and if this is not what you are looking for, I apologize in advance.
#main.resultcallback()
def process_result(result, **kwargs):
print('End of start function')
click.get_current_context().obj['callback']()
def start():
main(obj={'callback': lambda: print('End of main function')})
So, the resultcallback seems to be the suggested way of handling the termination of the group, and the invoked command. In our case, it prints End of start function, because at that point, the start function has finished executing, so we are wrapping up before terminating main. Then, it retrieves the callback passed in via the context, and executes that.
I am not sure if this is the idiomatic way of doing it, but it seems to have the intended behaviour.
For the result callback, a similar question was answered here
As to what exactly is causing this behaviour, and this is only a guess based on some quick experimentation with placing yield in the group or the command, I suspect some kind of thread/processor is spawned to handle the execution of the group and its command.
Hope this helps!

click's main() always raises a SystemExit. Quoting the documentation:
This will always terminate the application after a call. If this is not wanted, SystemExit needs to be caught.
In your example, change start() to:
def start():
try:
main(obj={})
except SystemExit as err:
# re-raise unless main() finished without an error
if err.code:
raise
print('End of start function')

See the click docs here also this answer here
# Main Runtime call at bottom of your code
start(standalone_mode=False)
# or e.g
main(standalone_mode=False)

How to detect exceptions in concurrent.futures in Python3?

I have just moved on to python3 as a result of its concurrent futures module. I was wondering if I could get it to detect errors. I want to use concurrent futures to parallel program, if there are more efficient modules please let me know.
I do not like multiprocessing as it is too complicated and not much documentation is out. It would be great however if someone could write a Hello World without classes only functions using multiprocessing to parallel compute so that it is easy to understand.
Here is a simple script:
from concurrent.futures import ThreadPoolExecutor
def pri():
print("Hello World!!!")
def start():
try:
while True:
pri()
except KeyBoardInterrupt:
print("YOU PRESSED CTRL+C")
with ThreadPoolExecutor(max_workers=3) as exe:
exe.submit(start)
The above code was just a demo, of how CTRL+C will not work to print the statement.
What I want is to be able to call a function is an error is present. This error detection must be from the function itself.
Another example
import socket
from concurrent.futures import ThreadPoolExecutor
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
def con():
try:
s.connect((x,y))
main()
except: socket.gaierror
err()
def err():
time.sleep(1)
con()
def main():
s.send("[+] Hello")
with ThreadPoolExecutor as exe:
exe.submit(con)

Way too late to the party, but maybe it'll help someone else...
I'm pretty sure the original question was not really answered. Folks got hung up on the fact that user5327424 was using a keyboard interrupt to raise an exception when the point was that the exception (however it was caused) was not raised. For example:
import concurrent.futures
def main():
numbers = range(10)
with concurrent.futures.ThreadPoolExecutor() as executor:
results = {executor.submit(raise_my_exception, number): number for number in numbers}
def raise_my_exception(number):
print('Proof that this function is getting called. %s' % number)
raise Exception('This never sees the light of day...')
main()
When the example code above is executed, you will see the text inside the print statement displayed on the screen, but you will never see the exception. This is because the results of each thread are held in the results object. You need to iterate that object to get to your exceptions. The following example shows how to access the results.
import concurrent.futures
def main():
numbers = range(10)
with concurrent.futures.ThreadPoolExecutor() as executor:
results = {executor.submit(raise_my_exception, number): number for number in numbers}
for result in results:
# This will cause the exception to be raised (but only the first one)
print(result.result())
def raise_my_exception(number):
print('Proof that this function is getting called. %s' % number)
raise Exception('This will be raised once the results are iterated.')
main()
I'm not sure I like this behavior or not, but it does allow the threads to fully execute, regardless of the exceptions encountered inside the individual threads.

Here's a solution. I'm not sure you like it, but I can't think of any other. I've modified your code to make it work.
from concurrent.futures import ThreadPoolExecutor
import time
quit = False
def pri():
print("Hello World!!!")
def start():
while quit is not True:
time.sleep(1)
pri()
try:
pool = ThreadPoolExecutor(max_workers=3)
pool.submit(start)
while quit is not True:
print("hei")
time.sleep(1)
except KeyboardInterrupt:
quit = True
Here are the points:
When you use with ThreadPoolExecutor(max_workers=3) as exe, it waits until all tasks have been done. Have a look at Doc
If wait is True then this method will not return until all the pending futures are done executing and the resources associated with the executor have been freed. If wait is False then this method will return immediately and the resources associated with the executor will be freed when all pending futures are done executing. Regardless of the value of wait, the entire Python program will not exit until all pending futures are done executing.
You can avoid having to call this method explicitly if you use the with statement, which will shutdown the Executor (waiting as if Executor.shutdown() were called with wait set to True)
It's like calling join() on a thread.
That's why I replaced it with:
pool = ThreadPoolExecutor(max_workers=3)
pool.submit(start)
Main thread must be doing "work" to be able to catch a Ctrl+C. So you can't just leave main thread there and exit, the simplest way is to run an infinite loop
Now that you have a loop running in main thread, when you hit CTRL+C, program will enter the except KeyboardInterrupt block and set quit=True. Then your worker thread can exit.
Strictly speaking, this is only a workaround. It seems to me it's impossible to have another way for this.
Edit
I'm not sure what's bothering you, but you can catch exception in another thread without problem:
import socket
import time
from concurrent.futures import ThreadPoolExecutor
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
def con():
try:
raise socket.gaierror
main()
except socket.gaierror:
print("gaierror occurred")
err()
def err():
print("err invoked")
time.sleep(1)
con()
def main():
s.send("[+] Hello")
with ThreadPoolExecutor(3) as exe:
exe.submit(con)
Output
gaierror occurred
err invoked
gaierror occurred
err invoked
gaierror occurred
err invoked
gaierror occurred
...

What is wrong with this Python Multiprocessing Code?

I am trying to create some multiprocessing code for my project. I have created a snippet of the things that I want to do. However its not working as per my expectations. Can you please let me know what is wrong with this.
from multiprocessing import Process, Pipe
import time
class A:
def __init__(self,rpipe,spipe):
print "In the function fun()"
def run(self):
print"in run method"
time.sleep(5)
message = rpipe.recv()
message = str(message).swapcase()
spipe.send(message)
workers = []
my_pipe_1 = Pipe(False)
my_pipe_2 = Pipe(False)
proc_handle = Process(target = A, args=(my_pipe_1[0], my_pipe_2[1],))
workers.append(proc_handle)
proc_handle.run()
my_pipe_1[1].send("hello")
message = my_pipe_2[0].recv()
print message
print "Back in the main function now"
The trace back displayed when i press ctrl-c:
^CTraceback (most recent call last):
File "sim.py", line 22, in <module>
message = my_pipe_2[0].recv()
KeyboardInterrupt
When I run this above code, the main process does not continue after calling "proc_handle.run". Why is this?

You've misunderstood how to use Process. You're creating a Process object, and passing it a class as target, but target is meant to be passed a callable (usually a function) that Process.run then executes. So in your case it's just instantiating A inside Process.run, and that's it.
You should instead make your A class a Process subclass, and just instantiate it directly:
#!/usr/bin/python
from multiprocessing import Process, Pipe
import time
class A(Process):
def __init__(self,rpipe,spipe):
print "In the function fun()"
super(A, self).__init__()
self.rpipe = rpipe
self.spipe = spipe
def run(self):
print"in run method"
time.sleep(5)
message = self.rpipe.recv()
message = str(message).swapcase()
self.spipe.send(message)
if __name__ == "__main__":
workers = []
my_pipe_1 = Pipe(False)
my_pipe_2 = Pipe(False)
proc_handle = A(my_pipe_1[0], my_pipe_2[1])
workers.append(proc_handle)
proc_handle.start()
my_pipe_1[1].send("hello")
message = my_pipe_2[0].recv()
print message
print "Back in the main function now"
mgilson was right, though. You should call start(), not run(), to make A.run execute in a child process.
With these changes, the program works fine for me:
dan#dantop:~> ./mult.py
In the function fun()
in run method
HELLO
Back in the main function now

Taking a stab at this one, I think it's because you're calling proc_handle.run() instead of proc_handle.start().
The former is the activity that the process is going to do -- the latter actually arranges for run to be called on a separate process. In other words, you're never forking the process, so there's no other process for my_pipe_1[1] to communicate with so it hangs.

How to run one last function before getting killed in Python?

Is there any way to run one last command before a running Python script is stopped by being killed by some other script, keyboard interrupt etc.

import time
try:
time.sleep(10)
finally:
print "clean up"
clean up
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
KeyboardInterrupt
If you need to catch other OS level interrupts, look at the signal module:
http://docs.python.org/library/signal.html
Signal Example
from signal import *
import sys, time
def clean(*args):
print "clean me"
sys.exit(0)
for sig in (SIGABRT, SIGBREAK, SIGILL, SIGINT, SIGSEGV, SIGTERM):
signal(sig, clean)
time.sleep(10)

You could use the atexit module. With it, you can register a function which will be called at program termination. An example from here: http://docs.python.org/library/atexit.html
try:
_count = int(open("/tmp/counter").read())
except IOError:
_count = 0
def incrcounter(n):
global _count
_count = _count + n
def savecounter():
open("/tmp/counter", "w").write("%d" % _count)
import atexit
atexit.register(savecounter)
You can also pass positional and keyword parameters to the function you want to call at program termination.
Note that there are a few circumstances listed in the docs in which your handler won't be called:
Note: The functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called.
As such, you may want to also register a signal handler.

import signal
import sys
import time
def cleanup(*args):
print 'Exiting'
sys.exit(0)
signal.signal(signal.SIGINT, cleanup)
signal.signal(signal.SIGTERM, cleanup)
while True:
time.sleep(60) # less busy loop

WIth apologies to 'Unknown' for taking their answer and correcting it as though it was my own answer, but my edits were rejected.
The approved answer contains an error that will cause a segfault.
You cannot use sys.exit() in a signal handler, but you can use os._exit so that it becomes:
from signal import *
import os, time
def clean(*args):
print "clean me"
os._exit(0)
for sig in (SIGABRT, SIGINT, SIGTERM):
signal(sig, clean)
time.sleep(10)
SIGBREAK may be used if the target platform is Windows.
Depending on the use case and the need to cleanup in the event of fatal errors - you may add SIGSEGV and SIGILL but generally this is not advised since the program state may be such that you create an infinite loop.

Use the atexit module to register a function that will be called at the end.
import atexit
atexit.register(some_function)