I have two python scripts as below -
parent.py
import os
import signal
shutdown = False
def sigterm_handler(signum, frame):
global shutdown
shutdown = True
if __name__ == '__main__':
signal.signal(signal.SIGTERM, sigterm_handler)
signal.signal(signal.SIGINT, sigterm_handler)
os.chdir(os.path.dirname(os.path.abspath(__file__)))
cmd = 'python child.py'
while True:
if shutdown == True:
break
print 'executing %s' % cmd
exit_code = os.system(cmd)
print 'Exit Code from %s > %s' % (cmd, exit_code)
print 'Exiting Parent'
child.py
import signal
import time
shutdown = False
def sigterm_handler(signum, frame):
global shutdown
shutdown = True
if __name__ == '__main__':
signal.signal(signal.SIGTERM, sigterm_handler)
signal.signal(signal.SIGINT, sigterm_handler)
while True:
if shutdown == True:
break
print 'Child Process Running !!'
time.sleep(1)
If I run parent.py and press ctrl + c on the terminal the child process exits and gets restarted by the parent as the parent is not processing the SIGINT is not being processed by the parent. I want to terminate both parent and the child if ctrl + c is pressed on the terminal. But for cases where the child exits because of some error instead of ctrl + c event, I want the parent to continue executing. I could have handled SIGCHLD in the parent but that doesn't indicate if the child was exited because of a ctrl + c event or something else. How would I achieve this behavior ?
below is the output I get if I run the parent -
executing python child.py
Child Process Running !!
Child Process Running !!
Child Process Running !!
Child Process Running !!
^CExit Code from python child.py > 2
executing python child.py
Child Process Running !!
Child Process Running !!
Child Process Running !!
Child Process Running !!
Child Process Running !!
^CExit Code from python child.py > 2
executing python child.py
Child Process Running !!
Child Process Running !!
Child Process Running !!
Child Process Running !!
^CExit Code from python child.py > 2
............................
............................
I think you'll have better luck with subprocess than os.system. In particular, I think you'll want to use subprocess with shell=False so that your child command is executed without a subshell (which might interfere with your ability to handle these kinds of signal-handling scenarios).
The code below does what you want, if I understand you correctly: CTRL-C causes both child and parent to stop; but if child dies for some other reason, parent will run the child again.
Here's a parent program similar to yours:
import signal
import subprocess
shutdown = False
def sigterm_handler(signum, frame):
print 'parent got shutdown'
global shutdown
shutdown = True
if __name__ == '__main__':
signal.signal(signal.SIGTERM, sigterm_handler)
signal.signal(signal.SIGINT, sigterm_handler)
cmd_args = ['python', 'child.py']
while not shutdown:
print 'executing', cmd_args
try:
subprocess.check_call(cmd_args)
except subprocess.CalledProcessError:
print 'child died'
pass
print 'Exiting Parent'
And here is a child program that runs for a while and then dies with a ZeroDivisionError.
import signal
import sys
import time
def sigterm_handler(signum, frame):
print 'child got shutdown'
sys.exit(0)
if __name__ == '__main__':
signal.signal(signal.SIGTERM, sigterm_handler)
signal.signal(signal.SIGINT, sigterm_handler)
for i in range(3, -1, -1):
print 'Child Process Running', i, i/i
time.sleep(3)
I use the signal function to kill all sub-processes in the mul-process program, the code is shown blow, save as a file named mul_process.py:
import time
import os
import signal
from multiprocessing import Process
processes = []
def fun(x):
print 'current sub-process pid is %s' % os.getpid()
while True:
print 'args is %s' % x
time.sleep(100)
def term(sig_num, frame):
print 'terminate process %d' % os.getpid()
for p in processes:
print p.pid
try:
for p in processes:
print 'process %d terminate' % p.pid
p.terminate()
p.join()
except Exception as e:
print str(e)
if __name__ == '__main__':
print 'current main-process pid is %s' % os.getpid()
for i in range(3):
t = Process(target=fun, args=(str(i),))
t.start()
processes.append(t)
signal.signal(signal.SIGTERM, term)
try:
for p in processes:
p.join()
except Exception as e:
print str(e)
Using 'python mul_process.py' to launch the program on Ubuntu 10.04.4 and Python 2.6, when it start running, in another tab, I use kill -15 with the main process pid to send signal SIGTERM to kill all processes, when the main process receive the signal SIGTERM, it exit after terminate all sub processes, but when I use kill -15 with the sub process pid, it does not work, the program still alive and running as before, and does not print the sentence defined in the function term, seems that the subprocess doesn't receive the SIGTERM.As I know, the sub process will inherit the signal handler, but it doesn`t work, here is the first question.
And then I move the line 'signal.signal(signal.SIGTERM, term)' to position after line 'if name == 'main':', like this:
if __name__ == '__main__':
signal.signal(signal.SIGTERM, term)
print 'current main-process pid is %s' % os.getpid()
for i in range(3):
t = Process(target=fun, args=(str(i),))
t.start()
processes.append(t)
try:
for p in processes:
p.join()
except Exception as e:
print str(e)
Launch the program, and use kill -15 with the main process pid to send the signal SIGTERM, the program receive the signal and call the function term but also doesn't kill any subprocessed and exit itself, this is the second question.
Few problems in your program- Agree that subprocess will inherit signal handler in your 2nd code snippet, But global variable "processes" list won't be shared. So list of process would be available with main process only. "process" would be empty list for other sub process.
You can use queue or pipe kind of mechanism to pass list of process to sub processes. But it will bring another problem
You terminate process1 and handler of process1 try to terminate process2 to process4.
Now process 2 also has same handler,
So Process 2 handler again try to terminate all other process
which will push your program into infinite loop.
I have a main python(testmain.py) script that executes another python script(test.py) using subprocess.Popen command. When I press Ctrl-C , I want the child to exit with exit code 2 and then the parent to display that exit code and then terminate .
I have signal handlers in both parent and child scripts.
testmain.py
def signal_handler(signal, frame):
print "outer signal handler"
exit(2)
signal.signal(signal.SIGINT, signal_handler)
def execute()
proc=subprocess.Popen("python test.py",shell=True)
streamdata=proc.communicate()[0]
rc=proc.returncode
print "return code:",rc
execute()
test.py
def signal_handler(signal, frame):
print "exiting: inner function"
exit(2)
signal.signal(signal.SIGINT, signal_handler)
I checked Delegate signal handling to a child process in python that is kind of similar to my question but in that case, the parent is continuing it's execution, which I don't want.
I want to: 1.exit test.py with exit(2) 2.print that exit code in testmain.py 3.exit test.py with exit(2)
could someone please provide suggestions to do this?
Thanks.
UPDATE : Handling the signal only in the child (test.py) and checking the return code in parent(testmain.py) will do what I want .
if rc==2:
print "child was terminated"
exit(2)
but I was wondering if there is a clean way to do this using signal handling.
Your child process shouldn't care what the parent does i.e., if you want the child to exit with specific status on Ctrl+C then just do that:
import sys
try:
main()
except KeyboardInterrupt: # use default SIGINT handler
sys.exit(2)
Or you could define the signal handler explicitly:
import os
import signal
def signal_handler(signal, frame):
os.write(1, b"outer signal handler\n")
os._exit(2)
signal.signal(signal.SIGINT, signal_handler)
main()
There might be a difference in behavior if there are atexit handlers and/or multiple threads.
Unrelated: depending on what your main() function does, there could be a significant delay before a signal is handled in Python. Some blocking methods on Python 2 may ignore the signal completely: use Python 3 or apply a custom workaround for a specific case e.g., using a timeout parameter for some calls.
You could handle SIGINT in a similar way in the parent:
for cmd in commands:
process = Popen(cmd)
try:
process.wait()
except KeyboardInterrupt:
# child process may still be alive here
for _ in range(5): # wait a while
if process.poll() is not None:
break # the process is dead
time.sleep(.1)
else: # no break, kill the process explicitly
try:
process.kill()
except OSError:
pass
sys.exit("Child exited with %d" % process.wait())
Python 2 doesn't restore signals for child processes e.g., if you SIG_IGN the SIGINT signal in the parent, you could reset the necessary hanlders explicitly using preexec_fn parameter.
I'm struggling with some processes I started with Popen and which start subprocesses. When I start these processes manually in a terminal every process terminates as expected if I send CTRL+C. But running inside a python program using subprocess.Popen any attempt to terminate the process only gets rid of the parent but not of its children.
I tried .terminate() ..kill() as well as ..send_signal() with signal.SIGBREAK, signal.SIGTERM, but in every case I just terminate the parent process.
With this parent process I can reproduce the misbehavior:
#!/usr/bin/python
import time
import sys
import os
import subprocess
import signal
if __name__ == "__main__":
print os.getpid(), "MAIN: start a process.."
p = subprocess.Popen([sys.executable, 'process_to_shutdown.py'])
print os.getpid(), "MAIN: started process", p.pid
time.sleep(2)
print os.getpid(), "MAIN: kill the process"
# these just terminate the parent:
#p.terminate()
#p.kill()
#os.kill(p.pid, signal.SIGINT)
#os.kill(p.pid, signal.SIGTERM)
os.kill(p.pid, signal.SIGABRT)
p.wait()
print os.getpid(), "MAIN: job done - ciao"
The real life child process is manage.py from Django which spawns a few subprocesses and waits for CRTL-C. But the following example seems to work, too:
#!/usr/bin/python
import time
import sys
import os
import subprocess
if __name__ == "__main__":
timeout = int(sys.argv[1]) if len(sys.argv) >= 2 else 0
if timeout == 0:
p = subprocess.Popen([sys.executable, '-u', __file__, '13'])
print os.getpid(), "just waiting..."
p.wait()
else:
for i in range(timeout):
time.sleep(1)
print os.getpid(), i, "alive!"
sys.stdout.flush()
print os.getpid(), "ciao"
So my question in short: how do I kill the process in the first example and get rid of the child processes as well? On windows os.kill(p.pid, signal.CTRL_C_EVENT) seems to work in some cases, but what's the right way to do it? And how does a Terminal do it?
Like Henri Korhonen mentioned in a comment, grouping processes should help. Additionally, if you are on Windows and this is Cygwin Python that starts Windows applications, it appears Cygwin Python can not kill the children. For those cases you would need to run TASKKILL. TASKKILL also takes a group parameter.
Is there a way to ensure all created subprocess are dead at exit time of a Python program? By subprocess I mean those created with subprocess.Popen().
If not, should I iterate over all of the issuing kills and then kills -9? anything cleaner?
You can use atexit for this, and register any clean up tasks to be run when your program exits.
atexit.register(func[, *args[, **kargs]])
In your cleanup process, you can also implement your own wait, and kill it when a your desired timeout occurs.
>>> import atexit
>>> import sys
>>> import time
>>>
>>>
>>>
>>> def cleanup():
... timeout_sec = 5
... for p in all_processes: # list of your processes
... p_sec = 0
... for second in range(timeout_sec):
... if p.poll() == None:
... time.sleep(1)
... p_sec += 1
... if p_sec >= timeout_sec:
... p.kill() # supported from python 2.6
... print 'cleaned up!'
...
>>>
>>> atexit.register(cleanup)
>>>
>>> sys.exit()
cleaned up!
Note -- Registered functions won't be run if this process (parent process) is killed.
The following windows method is no longer needed for python >= 2.6
Here's a way to kill a process in windows. Your Popen object has a pid attribute, so you can just call it by success = win_kill(p.pid) (Needs pywin32 installed):
def win_kill(pid):
'''kill a process by specified PID in windows'''
import win32api
import win32con
hProc = None
try:
hProc = win32api.OpenProcess(win32con.PROCESS_TERMINATE, 0, pid)
win32api.TerminateProcess(hProc, 0)
except Exception:
return False
finally:
if hProc != None:
hProc.Close()
return True
On *nix's, maybe using process groups can help you out - you can catch subprocesses spawned by your subprocesses as well.
if __name__ == "__main__":
os.setpgrp() # create new process group, become its leader
try:
# some code
finally:
os.killpg(0, signal.SIGKILL) # kill all processes in my group
Another consideration is to escalate the signals: from SIGTERM (default signal for kill) to SIGKILL (a.k.a kill -9). Wait a short while between the signals to give the process a chance to exit cleanly before you kill -9 it.
The subprocess.Popen.wait() is the only way to assure that they're dead. Indeed, POSIX OS's require that you wait on your children. Many *nix's will create a "zombie" process: a dead child for which the parent didn't wait.
If the child is reasonably well-written, it terminates. Often, children read from PIPE's. Closing the input is a big hint to the child that it should close up shop and exit.
If the child has bugs and doesn't terminate, you may have to kill it. You should fix this bug.
If the child is a "serve-forever" loop, and is not designed to terminate, you should either kill it or provide some input or message which will force it to terminate.
Edit.
In standard OS's, you have os.kill( PID, 9 ). Kill -9 is harsh, BTW. If you can kill them with SIGABRT (6?) or SIGTERM (15) that's more polite.
In Windows OS, you don't have an os.kill that works. Look at this ActiveState Recipe for terminating a process in Windows.
We have child processes that are WSGI servers. To terminate them we do a GET on a special URL; this causes the child to clean up and exit.
Find out a solution for linux (without installing prctl):
def _set_pdeathsig(sig=signal.SIGTERM):
"""help function to ensure once parent process exits, its childrent processes will automatically die
"""
def callable():
libc = ctypes.CDLL("libc.so.6")
return libc.prctl(1, sig)
return callable
subprocess.Popen(your_command, preexec_fn=_set_pdeathsig(signal.SIGTERM))
Warning: Linux-only! You can make your child receive a signal when its parent dies.
First install python-prctl==1.5.0 then change your parent code to launch your child processes as follows
subprocess.Popen(["sleep", "100"], preexec_fn=lambda: prctl.set_pdeathsig(signal.SIGKILL))
What this says is:
launch subprocess: sleep 100
after forking and before exec of the subprocess, the child registers for "send me a SIGKILL
when my parent terminates".
orip's answer is helpful but has the downside that it kills your process and returns an error code your parent. I avoided that like this:
class CleanChildProcesses:
def __enter__(self):
os.setpgrp() # create new process group, become its leader
def __exit__(self, type, value, traceback):
try:
os.killpg(0, signal.SIGINT) # kill all processes in my group
except KeyboardInterrupt:
# SIGINT is delievered to this process as well as the child processes.
# Ignore it so that the existing exception, if any, is returned. This
# leaves us with a clean exit code if there was no exception.
pass
And then:
with CleanChildProcesses():
# Do your work here
Of course you can do this with try/except/finally but you have to handle the exceptional and non-exceptional cases separately.
I needed a small variation of this problem (cleaning up subprocesses, but without exiting the Python program itself), and since it's not mentioned here among the other answers:
p=subprocess.Popen(your_command, preexec_fn=os.setsid)
os.killpg(os.getpgid(p.pid), 15)
setsid will run the program in a new session, thus assigning a new process group to it and its children. calling os.killpg on it thus won't bring down your own python process also.
poll( )
Check if child process has terminated.
Returns returncode attribute.
A solution for windows may be to use the win32 job api e.g. How do I automatically destroy child processes in Windows?
Here's an existing python implementation
https://gist.github.com/ubershmekel/119697afba2eaecc6330
Is there a way to ensure all created subprocess are dead at exit time of a Python program? By subprocess I mean those created with subprocess.Popen().
You could violate encapsulation and test that all Popen processes have terminated by doing
subprocess._cleanup()
print subprocess._active == []
If not, should I iterate over all of the issuing kills and then kills -9? anything cleaner?
You cannot ensure that all subprocesses are dead without going out and killing every survivor. But if you have this problem, it is probably because you have a deeper design problem.
I actually needed to do this, but it involved running remote commands. We wanted to be able to stop the processes by closing the connection to the server. Also, if, for example, you are running in the python repl, you can select to run as foreground if you want to be able to use Ctrl-C to exit.
import os, signal, time
class CleanChildProcesses:
"""
with CleanChildProcesses():
Do work here
"""
def __init__(self, time_to_die=5, foreground=False):
self.time_to_die = time_to_die # how long to give children to die before SIGKILL
self.foreground = foreground # If user wants to receive Ctrl-C
self.is_foreground = False
self.SIGNALS = (signal.SIGHUP, signal.SIGTERM, signal.SIGABRT, signal.SIGALRM, signal.SIGPIPE)
self.is_stopped = True # only call stop once (catch signal xor exiting 'with')
def _run_as_foreground(self):
if not self.foreground:
return False
try:
fd = os.open(os.ctermid(), os.O_RDWR)
except OSError:
# Happens if process not run from terminal (tty, pty)
return False
os.close(fd)
return True
def _signal_hdlr(self, sig, framte):
self.__exit__(None, None, None)
def start(self):
self.is_stopped = False
"""
When running out of remote shell, SIGHUP is only sent to the session
leader normally, the remote shell, so we need to make sure we are sent
SIGHUP. This also allows us not to kill ourselves with SIGKILL.
- A process group is called orphaned when the parent of every member is
either in the process group or outside the session. In particular,
the process group of the session leader is always orphaned.
- If termination of a process causes a process group to become orphaned,
and some member is stopped, then all are sent first SIGHUP and then
SIGCONT.
consider: prctl.set_pdeathsig(signal.SIGTERM)
"""
self.childpid = os.fork() # return 0 in the child branch, and the childpid in the parent branch
if self.childpid == 0:
try:
os.setpgrp() # create new process group, become its leader
os.kill(os.getpid(), signal.SIGSTOP) # child fork stops itself
finally:
os._exit(0) # shut down without going to __exit__
os.waitpid(self.childpid, os.WUNTRACED) # wait until child stopped after it created the process group
os.setpgid(0, self.childpid) # join child's group
if self._run_as_foreground():
hdlr = signal.signal(signal.SIGTTOU, signal.SIG_IGN) # ignore since would cause this process to stop
self.controlling_terminal = os.open(os.ctermid(), os.O_RDWR)
self.orig_fore_pg = os.tcgetpgrp(self.controlling_terminal) # sends SIGTTOU to this process
os.tcsetpgrp(self.controlling_terminal, self.childpid)
signal.signal(signal.SIGTTOU, hdlr)
self.is_foreground = True
self.exit_signals = dict((s, signal.signal(s, self._signal_hdlr))
for s in self.SIGNALS)
def stop(self):
try:
for s in self.SIGNALS:
#don't get interrupted while cleaning everything up
signal.signal(s, signal.SIG_IGN)
self.is_stopped = True
if self.is_foreground:
os.tcsetpgrp(self.controlling_terminal, self.orig_fore_pg)
os.close(self.controlling_terminal)
self.is_foreground = False
try:
os.kill(self.childpid, signal.SIGCONT)
except OSError:
"""
can occur if process finished and one of:
- was reaped by another process
- if parent explicitly ignored SIGCHLD
signal.signal(signal.SIGCHLD, signal.SIG_IGN)
- parent has the SA_NOCLDWAIT flag set
"""
pass
os.setpgrp() # leave the child's process group so I won't get signals
try:
os.killpg(self.childpid, signal.SIGINT)
time.sleep(self.time_to_die) # let processes end gracefully
os.killpg(self.childpid, signal.SIGKILL) # In case process gets stuck while dying
os.waitpid(self.childpid, 0) # reap Zombie child process
except OSError as e:
pass
finally:
for s, hdlr in self.exit_signals.iteritems():
signal.signal(s, hdlr) # reset default handlers
def __enter__(self):
if self.is_stopped:
self.start()
def __exit__(self, exit_type, value, traceback):
if not self.is_stopped:
self.stop()
Thanks to Malcolm Handley for the initial design. Done with python2.7 on linux.
You can try subalive, a package I wrote for similar problem. It uses periodic alive ping via RPC, and the slave process automatically terminates when the master stops alive pings for some reason.
https://github.com/waszil/subalive
Example for master:
from subalive import SubAliveMaster
# start subprocess with alive keeping
SubAliveMaster(<path to your slave script>)
# do your stuff
# ...
Example for slave subprocess:
from subalive import SubAliveSlave
# start alive checking
SubAliveSlave()
# do your stuff
# ...
It's possible to get some more guarantees on windows by spawning a separate process to oversee the destruction.
import subprocess
import sys
import os
def terminate_process_on_exit(process):
if sys.platform == "win32":
try:
# Or provide this script normally.
# Here just to make it somewhat self-contained.
# see https://stackoverflow.com/a/22559493/3763139
# see https://superuser.com/a/1299350/388191
with open('.process_watchdog_helper.bat', 'x') as file:
file.write(""":waitforpid
tasklist /nh /fi "pid eq %1" 2>nul | find "%1" >nul
if %ERRORLEVEL%==0 (
timeout /t 5 /nobreak >nul
goto :waitforpid
) else (
wmic process where processid="%2" call terminate >nul
)""")
except:
pass
# After this spawns we're pretty safe. There is a race, but we do what we can.
subprocess.Popen(
['.process_watchdog_helper.bat', str(os.getpid()), str(process.pid)],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL
)
# example
class DummyProcess:
def __init__(self, pid):
self.pid = pid
set_terminate_when_this_process_dies(DummyProcess(7516))
This is what I did for my posix app:
When your app exists call the kill() method of this class:
http://www.pixelbeat.org/libs/subProcess.py
Example use here:
http://code.google.com/p/fslint/source/browse/trunk/fslint-gui#608
help for python code:
http://docs.python.org/dev/library/subprocess.html#subprocess.Popen.wait