I have a Python script for automating simple tasks. Its main loop looks like this:
while True:
input = download_task_input()
if input:
output = process_task(input)
upload_task_output(output)
sleep(60)
Some local files are altered during task processing. They are modified when the task is started, and restored back to proper state when the task is done, or if exception is caught. Restoring these files on program exit is very important to me: leaving them in altered state causes some trouble later that I'd like to avoid.
When I want to terminate the script, I hit Ctrl+C. It raises KeyboardInterrupt exception which both stops task processing and triggers files restoration. However, if I hit Ctrl+Break, the program is simply terminated: if a task is being processed at this moment, then local files are left in altered state (which is undesirable).
The question: I'm worried about the situation when Windows OS is shutdown by pressing the Power button. Is it possible to make Python handle it exactly like it handles Ctrl+C? I.e. I'd like to detect OS shutdown in Python script and raise Python exception on the main thread.
I know it is possible to call SetConsoleCtrlHandler function from WinAPI and install own handler for situations like Ctrl+C, Ctrl+Break, Shutdown, etc. However, this handler seems to be executed in additional thread, and raising exception in it does not achieve anything. On the other hand, Python itself supposedly uses the same WinAPI feature to raise KeyboardInterrupt on the main thread on Ctrl+C, so it should be doable.
This is not a serious automation script, so I don't mind if a solution is hacky or not 100% reliable.
When running Python from a Linux shell (same behavior observed in both bash and ksh), and generating a SIGINT with a Ctl-C keypress, I have discovered behavior that I am unable to understand, and which has frustrated me considerably.
When I press Ctl-C, the Python process appropriately terminates, but the shell continues to the next command on the line, as shown in the following console capture:
$ python -c "import time; time.sleep(100)"; echo END
^CTraceback (most recent call last):
File "<string>", line 1, in <module>
KeyboardInterrupt
END
In contrast, I had expected, and would like, that the shell processes the signal in such a way that execution does not continue to the next command on the line, as I see when I call the sleep function from a bash subshell instead of from Python.
For example, I would expect the above capture to appear more similar to the following:
$ bash -c "sleep 100"; echo END
^C
Python 2 and 3 are installed on my system, and while the above capture was generated running Python 2, both behave the same way.
My best explanation is that when I press Ctl-C while the Python process is running, the signal somehow goes directly to the Python process, whereas normally it is handled by the calling shell, then propagated to the subprocess. However, I have no idea why or how Python is causing this difference.
The examples above are trivial tests but the behavior is also observed in real-world uses. Installing custom signal handlers does not resolve the issue.
After considerable digging I found a few loosely related questions on Stack Overflow that eventually led me to an article describing the proper handling of SIGINT. (The most relevant section is How to be a proper program.)
From this information, I was able to solve the problem. Without it, I would have never have come close.
The solution is best illustrated by beginning with a Bash script that cannot be terminated by a keyboard interrupt, but which does hide the ugly stack trace from Python's KeyboardInterrupt exception.
A basic example might appear as follows:
#!/usr/bin/env bash
echo "Press Ctrl-C to stop... No sorry it won't work."
while true
do
python -c '
import time, signal
signal.signal(signal.SIGINT, signal.SIG_IGN)
time.sleep(100)
'
done
For the outer script to process the interrupt, the following change is required:
echo "Press Ctrl-C to stop..."
while true
do
python -c '
import time, signal, os
signal.signal(signal.SIGINT, signal.SIG_DFL)
time.sleep(100)
'
done
However, the solution makes it impossible to use a custom handler (for example, to perform cleanup). If doing so is required, then a more sophisticated approach is needed.
The required change is illustrated as follows:
#!/usr/bin/env bash
echo "Press [CTRL+C] to stop ..."
while true
do
python -c '
import time, sys, signal, os
def handle_int(signum, frame):
# Cleanup code here
signal.signal(signum, signal.SIG_DFL)
os.kill(os.getpid(), signum)
signal.signal(signal.SIGINT, handle_int)
time.sleep(100)
'
done
The reason appears to be that unless the inner process terminates through executing the default SIGINT handler provided by the system, the parent bash process does not realize that the child has terminated because of a keyboard interrupt, and does not itself terminate.
I have not fully understood all the ancillary issues quite yet, such as whether the parent process is not receiving the SIGINT from the system, or is receiving a signal, but ignoring it. I also have no idea what the default handler does or how the parent detects that it was called. If I am able to learn more, I will offer an update.
I must advance the question of whether the current behavior of Python should be considered a design flaw in Python. I have seen various manifestations of this issue over the years when calling Python from a shell script, but have not had the luxury of investigation until now. I have not found a single article through a web search, however, on the topic. If the issue does represent a flaw, it surprised me to observe that not many developers are affected.
The behavior of any program that gets a CTRL+C is up to that program. Usually the behavior is to exit, but some programs might just abort some internal procedure instead of stopping the whole program. It's even possible (though it may be considered bad manners) for a program to ignore the keystroke completely.
The behavior of the program is defined by the signal handlers it has set up. The C library provides default signal handlers (which do things like exit on SIGTERM and SIGINT), but a program can provide its own handlers that will run instead. Not all signals allow arbitrary responses. For instance, SIGSEGV (a seg-fault) requires the program to exit, though it can configure its signal handlers to make a core dump or not. SIGKILL can't be handled at all (the OS kernel takes care of it).
To customize signal handlers in Python, you'll want to use the signal module from the standard library. You can call signal.signal to set your own signal handler function for any of the signals defined by your system's C library. Typing CTRL+C is going to send SIGINT on any UNIX-based system, so that's probably what you'll want to handle if you want your own behavior.
Try something like this:
import signal
import sys
import time
def interrupt_handler(sig, frame):
sys.exit(1)
signal.signal(signal.SIGINT, interrupt_handler)
time.sleep(100)
If you run this script and interrupt it with CTRL+C, it should exit silently, just like your bash script does.
You could explicitly handle it on the bash side in a script file like this:
if python -c "import time; time.sleep(100)"; then
echo END
fi
or, more aggressively,
python -c "import time; time.sleep(100)"
[[ $? -ne 0 ]] && exit
echo END
$? is the return status code of the previous command. Where a status code of 0 means it exited fine, and anything else was an error. So, we use the short-circuit nature of && to succinctly exit if the previous command fails.
(See https://unix.stackexchange.com/questions/186826/parent-script-continues-when-child-exits-with-non-zero-exit-code for more info on that)
Note: this will exit the bash script for any kind of python failure, not just ctrl+c, e.g. IndexError, AssertionError, etc
An easy way to end an infinite for loop in python is by using Ctrl-C, e.g.:
try:
while True:
print '-',
except KeyboardInterrupt:
print 'done'
However, I want to use some other than Ctrl-C key because on other colleagues machines that might be interpreted by their gui environment (e.g. pycharm) as copy. Is there a way to do this? I need a solution that works both in windows and linux...
Ctrl+C on Linux sends a kill signal to the Python process. I don't think the Python script will be able to override the default behavior of the operating system.
I have a Python program that uses multiple daemon threads. I want to stop the program from outside, preferably from another Python script.
I've tried with kill <pid> from shell, just for a test, but it doesn't work with multi-threaded scripts.
One way would be to make the program check some file every n-seconds as a flag for termination. I'm sure there's some better way I can do this.
Note that I'd like to stop the program cleanly, so some message from outside in a form of an exception would be ideal, I think.
EDIT:
Here's an example of how I did it at the moment:
try:
open('myprog.lck', 'w').close()
while True:
time.sleep(1)
try:
open('myprog.lck').close()
except IOError:
raise KeyboardInterrupt
except KeyboardInterrupt:
print 'MyProgram terminated.'
Deleting file myprog.lck will cause the script to stop. Is the example above bad way to do this?
Use the poison pill technique. Upon receipt of a pill (a special message) your program must handle it and die.. The way you're doing it its ok, but for something more elegant, you should implement a kind of communication between your "killing script" and your main program. For a start, have a look in the standard library for Interprocess Communication and Networking.
I would install a signal handler as described in http://www.doughellmann.com/PyMOTW/signal/index.html#signals-and-threads
You can enter kill -l in your shell to get a list of available signals.
You should be able to kill it from the shell with kill -9 <pid>.
We are running a very large framework of python scripts for test automation and I do really miss the opportunity to kill a running python script with ctrl + c in some situations on Windows.
When the script might be doing some socket communications with long time-outs the only options sometimes is to kill the DOS window.. Is there any options I have missed?
Rather than using blocking calls with long timeouts, use event-driven networking. This will allow you never to have long periods of time doing uninterruptable operations.
Look for instances of:
try:
some code
except:
# catches all exceptions, including ^C
Change to:
try:
some code
except Exception:
# catches most exceptions, but not KeyboardInterrupt or SystemExit