I use Tornado as a web server, user can submit a task through the front end page, after auditing they can start the submitted task. In this situation, i want to start an asynchronous sub process to handle the task, so i write the following code in an request handler:
def task_handler():
// handle task here
def start_a_process_for_task():
p = multiprocessing.Process(target=task_handler,args=())
p.start()
return 0
I don't care about the sub process and just start a process for it and return to the front end page and tell user the task is started. The task itself will run in the background and will record it's status or results to database so user
can view on the web page later. So here i don't want to use p.join() which is blocking, but without p.join() after the task finished,the sub process becomes a defunct process and as Tornado runs as a daemon and never exits, the defunct process will never disappear.
Anyone knows how to fix this problem, thanks.
The proper way to avoid defunct children is for the parent to gracefully clean up and close all resources of the exited child. This is normally done by join(), but if you want to avoid that, another approach could be to set up a global handler for the SIGCHLD signal on the parent.
SIGCHLD will be emitted whenever a child exits, and in the handler function you should either call Process.join() if you still have access to the process object, or even use os.wait() to "wait" for any child process to terminate and properly reap it. The wait time here should be 0 as you know for sure a child process has just exited. You will also be able to get the process' exit code / termination signal so it can also be a useful method to handle / log child process crashes.
Here's a quick example of doing this:
from __future__ import print_function
import os
import signal
import time
from multiprocessing import Process
def child_exited(sig, frame):
pid, exitcode = os.wait()
print("Child process {pid} exited with code {exitcode}".format(
pid=pid, exitcode=exitcode
))
def worker():
time.sleep(5)
print("Process {pid} has completed it's work".format(pid=os.getpid()))
def parent():
children = []
# Comment out the following line to see zombie children
signal.signal(signal.SIGCHLD, child_exited)
for i in range(5):
c = Process(target=worker)
c.start()
print("Parent forked out worker process {pid}".format(pid=c.pid))
children.append(c)
time.sleep(1)
print("Forked out {c} workers, hit Ctrl+C to end...".format(c=len(children)))
while True:
time.sleep(5)
if __name__ == '__main__':
parent()
One caveat is that I am not sure if this process works on non-Unix operating systems. It should work on Linux, Mac and other Unixes.
You need to join your subprocesses if you do not want to create zombies. You can do it in threads.
This is a dummy example. After 10 seconds, all your subprocesses are gone instead of being zombies. This launches a thread for every subprocess. Threads do not need to be joined or waited. A thread executes subprocess, joins it and then exits the thread as soon as the subprocess is completed.
import multiprocessing
import threading
from time import sleep
def task_processor():
sleep(10)
class TaskProxy(threading.Thread):
def __init__(self):
super(TaskProxy, self).__init__()
def run(self):
p = multiprocessing.Process(target=task_processor,args=())
p.start()
p.join()
def task_handler():
t = TaskProxy()
t.daemon = True
t.start()
return
for _ in xrange(0,20):
task_handler()
sleep(60)
This code:
import multiprocessing as mp
from threading import Thread
import subprocess
import time
class WorkerProcess(mp.Process):
def run(self):
# Simulate long running task
self.subprocess = subprocess.Popen(['python', '-c', 'import time; time.sleep(1000)'])
self.code = self.subprocess.wait()
class ControlThread(Thread):
def run():
jobs = []
for _ in range(2):
job = WorkerProcess()
jobs.append(job)
job.start()
# wait for a while and then kill jobs
time.sleep(2)
for job in jobs:
job.terminate()
if __name__ == "__main__":
controller = ControlThread()
controller.start()
When I terminate the spawned WorkerProcess instances. They die just fine, however the subprocesses python -c 'import time; time.sleep(1000) runs until completition. This is well documented in the official docs, but how do I kill the child processes of a killed process?
A possbile soultion might be:
Wrap WorkerProcess.run() method inside try/except block catching SIGTERM, and terminating the subprocess.call call. But I am not sure how to catch the SIGTERM in the WorkerProcess
I also tried setting signal.signal(signal.SIGINT, handler) in the WorkerProcess, but I am getting ValuError, because it is allowed to be set only in the main thread.
What do I do now?
EDIT: As #svalorzen pointed out in comments this doesn't really work since the reference to self.subprocess is lost.
Finally came to a clean, acceptable solution. Since mp.Process.terminate is a method, we can override it.
class WorkerProcess(mp.Process):
def run(self):
# Simulate long running task
self.subprocess = subprocess.Popen(['python', '-c', 'import time; time.sleep(1000)'])
self.code = self.subprocess.wait()
# HERE
def terminate(self):
self.subprocess.terminate()
super(WorkerProcess, self).terminate()
You can use queues to message to your subprocesses and ask them nicely to terminate their children before exiting themselves. You can't use signals in anywhere else but your main thread, so signals are not suitable for this.
Curiously, when I modify the code like this, even if I interrupt it with control+C, subprocesses will die as well. This may be OS related thing, though.
import multiprocessing as mp
from threading import Thread
import subprocess
import time
from Queue import Empty
class WorkerProcess(mp.Process):
def __init__(self,que):
super(WorkerProcess,self).__init__()
self.queue = que
def run(self):
# Simulate long running task
self.subprocess = subprocess.Popen(['python', '-c', 'import time; time.sleep(1000)'])
while True:
a = self.subprocess.poll()
if a is None:
time.sleep(1)
try:
if self.queue.get(0) == "exit":
print "kill"
self.subprocess.kill()
self.subprocess.wait()
break
else:
pass
except Empty:
pass
print "run"
else:
print "exiting"
class ControlThread(Thread):
def run(self):
jobs = []
queues = []
for _ in range(2):
q = mp.Queue()
job = WorkerProcess(q)
queues.append(q)
jobs.append(job)
job.start()
# wait for a while and then kill jobs
time.sleep(5)
for q in queues:
q.put("exit")
time.sleep(30)
if __name__ == "__main__":
controller = ControlThread()
controller.start()
Hope this helps.
Hannu
Does
import multiprocessing
import schedule
def worker():
#do some stuff
def sched(argv):
schedule.every(0.01).minutes.do(worker)
while True:
schedule.run_pending()
processs = []
..
..
p = multiprocessing.Process(target=sched,args)
..
..
processs.append(p)
for p in processs:
p.terminate()
kills gracefully a list of processes ?
If not what is the simplest way to do it ?
The goal is to reload the configuration file into memory, so I would like to kill all children processes and create others instead, those latter will read the new config file.
Edit : Added more code to explain that I am running a while True loop
Edit : This is the new code after #dano suggestion
def get_config(self):
from ConfigParser import SafeConfigParser
..
return argv
def sched(self, args, event):
#schedule instruction:
schedule.every(0.01).minutes.do(self.worker,args)
while not event.is_set():
schedule.run_pending()
def dispatch_processs(self, conf):
processs = []
event = multiprocessing.Event()
for conf in self.get_config():
process = multiprocessing.Process(target=self.sched,args=( i for i in conf), kwargs={'event' : event})
processs.append((process, event)
return processs
def start_process(self, process):
process.start()
def gracefull_process(self, process):
process.join()
def main(self):
while True:
processs = self.dispatch_processs(self.get_config())
print ("%s processes running " % len(processs) )
for process, event in processs:
self.start_process(process)
time.sleep(1)
event.set()
self.gracefull_process(process)
The good thing about the code, is that I can edit config file and the process will reload its config also.
The problem is that only the first process runs and the others are ignored.
Edit : This saved my life , working with while True in schedule() is not a good idea, so I set up refresh_time instead
def sched(self, args, event):
schedule.every(0.01).minutes.do(self.worker,args)
for i in range(refresh_time):
schedule.run_pending()
time.sleep(1)
def start_processs(self, processs):
for p,event in processs:
if not p.is_alive():
p.start()
time.sleep(1)
event.set()
self.gracefull_processs(processs)
def gracefull_processs(self, processs):
for p,event in processs:
p.join()
processs = self.dispatch_processs(self.get_config())
self.start_processs(processs)
def main(self):
while True:
processs = self.dispatch_processs(self.get_config())
self.start_processs(processs)
break
print ("Reloading function main")
self.main()
If you don't mind only aborting after worker has completed all of its work, its very simple to add a multiprocessing.Event to handle exiting gracefully:
import multiprocessing
import schedule
def worker():
#do some stuff
def sched(argv, event=None):
schedule.every(0.01).minutes.do(worker)
while not event.is_set(): # Run until we're told to shut down.
schedule.run_pending()
processes = []
..
..
event = multiprocessing.Event()
p = multiprocessing.Process(target=sched,args, kwargs={'event' : event})
..
..
processes.append((p, event))
# Tell all processes to shut down
for _, event in processes:
event.set()
# Now actually wait for them to shut down
for p, _ in processes:
p.join()
A: No, both .terminate() & SIG_* methods are rather brutal
In a need to arrange a gracefull end of any process, as described in your post, there rather shall be some "soft-signalling" layer, that allows, on both ends, to send/receive smart-signalls without being dependent on the O/S interpretations ( O/S knows nothing about your application-level context and state of the respective work-unit, that is currently being processed ).
You may want to read about such soft-signalling approach in links referred from >>> https://stackoverflow.com/a/25373416/3666197
No, it doesn't kill a process according to your own definition of gracefully - unless you take some additional steps.
Assuming you're using a unix system (since you mentioned scp), terminate sends a SIGTERM signal to the child process. You can catch this signal in the child process, and act accordingly (wait for scp to finish):
import signal
def on_terminate(signum, stack):
wait_for_current_scp_operation()
signal.signal(signal.SIGTERM, on_terminate)
Here's a tutorial about handling and sending signals
I have have a main process that forks a number of subprocesses. I want to be able to kill these child processes off when my main process gets the kill signal. Ideally I would want to do something along the lines of:
def handler(signum, frame, pid_list):
log('Killing Process')
for pid in pid_list:
os.kill(pid, signal.SIGTERM)
os.waitpid(pid, 0) # need
sys.exit()
if __name__ == "__main__":
<code that creates child processes, pids>
signal.signal(signal.SIGTERM, handler(pid_list))
But of course, that doesn't work... any suggestions?
As #tony suggested you could set daemon=True flag on a child process created using multiprocessing module. To install it on python2.4, type: pip install multiprocessing.
The child processes won't be terminated if the main process is killed by a signal so you need to provide an appropriate signal handler:
#!/usr/bin/env python
import logging, signal, sys, time
import multiprocessing as mp # `pip install multiprocessing` on Python <2.6
class AddProcessNameFilter(logging.Filter):
"""Add missing on Python 2.4 `record.processName` attribute."""
def filter(self, r):
r.processName = getattr(r, 'processName', mp.current_process().name)
return logging.Filter.filter(self, r)
def print_dot():
while True:
mp.get_logger().info(".")
time.sleep(1)
def main():
logger = mp.log_to_stderr()
logger.setLevel(logging.INFO)
logger.addFilter(AddProcessNameFilter()) # fix logging records
# catch TERM signal to allow finalizers to run and reap daemonic children
signal.signal(signal.SIGTERM, lambda *args: sys.exit(-signal.SIGTERM))
# create daemonic child processes
processes = [mp.Process(target=print_dot) for _ in range(2)]
for p in processes:
p.daemon = True
p.start()
print_dot()
if __name__=="__main__":
mp.freeze_support()
main()
What about use this flag when you create a subprocess?
Is there a way to ensure all created subprocess are dead at exit time of a Python program? By subprocess I mean those created with subprocess.Popen().
If not, should I iterate over all of the issuing kills and then kills -9? anything cleaner?
You can use atexit for this, and register any clean up tasks to be run when your program exits.
atexit.register(func[, *args[, **kargs]])
In your cleanup process, you can also implement your own wait, and kill it when a your desired timeout occurs.
>>> import atexit
>>> import sys
>>> import time
>>>
>>>
>>>
>>> def cleanup():
... timeout_sec = 5
... for p in all_processes: # list of your processes
... p_sec = 0
... for second in range(timeout_sec):
... if p.poll() == None:
... time.sleep(1)
... p_sec += 1
... if p_sec >= timeout_sec:
... p.kill() # supported from python 2.6
... print 'cleaned up!'
...
>>>
>>> atexit.register(cleanup)
>>>
>>> sys.exit()
cleaned up!
Note -- Registered functions won't be run if this process (parent process) is killed.
The following windows method is no longer needed for python >= 2.6
Here's a way to kill a process in windows. Your Popen object has a pid attribute, so you can just call it by success = win_kill(p.pid) (Needs pywin32 installed):
def win_kill(pid):
'''kill a process by specified PID in windows'''
import win32api
import win32con
hProc = None
try:
hProc = win32api.OpenProcess(win32con.PROCESS_TERMINATE, 0, pid)
win32api.TerminateProcess(hProc, 0)
except Exception:
return False
finally:
if hProc != None:
hProc.Close()
return True
On *nix's, maybe using process groups can help you out - you can catch subprocesses spawned by your subprocesses as well.
if __name__ == "__main__":
os.setpgrp() # create new process group, become its leader
try:
# some code
finally:
os.killpg(0, signal.SIGKILL) # kill all processes in my group
Another consideration is to escalate the signals: from SIGTERM (default signal for kill) to SIGKILL (a.k.a kill -9). Wait a short while between the signals to give the process a chance to exit cleanly before you kill -9 it.
The subprocess.Popen.wait() is the only way to assure that they're dead. Indeed, POSIX OS's require that you wait on your children. Many *nix's will create a "zombie" process: a dead child for which the parent didn't wait.
If the child is reasonably well-written, it terminates. Often, children read from PIPE's. Closing the input is a big hint to the child that it should close up shop and exit.
If the child has bugs and doesn't terminate, you may have to kill it. You should fix this bug.
If the child is a "serve-forever" loop, and is not designed to terminate, you should either kill it or provide some input or message which will force it to terminate.
Edit.
In standard OS's, you have os.kill( PID, 9 ). Kill -9 is harsh, BTW. If you can kill them with SIGABRT (6?) or SIGTERM (15) that's more polite.
In Windows OS, you don't have an os.kill that works. Look at this ActiveState Recipe for terminating a process in Windows.
We have child processes that are WSGI servers. To terminate them we do a GET on a special URL; this causes the child to clean up and exit.
Find out a solution for linux (without installing prctl):
def _set_pdeathsig(sig=signal.SIGTERM):
"""help function to ensure once parent process exits, its childrent processes will automatically die
"""
def callable():
libc = ctypes.CDLL("libc.so.6")
return libc.prctl(1, sig)
return callable
subprocess.Popen(your_command, preexec_fn=_set_pdeathsig(signal.SIGTERM))
Warning: Linux-only! You can make your child receive a signal when its parent dies.
First install python-prctl==1.5.0 then change your parent code to launch your child processes as follows
subprocess.Popen(["sleep", "100"], preexec_fn=lambda: prctl.set_pdeathsig(signal.SIGKILL))
What this says is:
launch subprocess: sleep 100
after forking and before exec of the subprocess, the child registers for "send me a SIGKILL
when my parent terminates".
orip's answer is helpful but has the downside that it kills your process and returns an error code your parent. I avoided that like this:
class CleanChildProcesses:
def __enter__(self):
os.setpgrp() # create new process group, become its leader
def __exit__(self, type, value, traceback):
try:
os.killpg(0, signal.SIGINT) # kill all processes in my group
except KeyboardInterrupt:
# SIGINT is delievered to this process as well as the child processes.
# Ignore it so that the existing exception, if any, is returned. This
# leaves us with a clean exit code if there was no exception.
pass
And then:
with CleanChildProcesses():
# Do your work here
Of course you can do this with try/except/finally but you have to handle the exceptional and non-exceptional cases separately.
I needed a small variation of this problem (cleaning up subprocesses, but without exiting the Python program itself), and since it's not mentioned here among the other answers:
p=subprocess.Popen(your_command, preexec_fn=os.setsid)
os.killpg(os.getpgid(p.pid), 15)
setsid will run the program in a new session, thus assigning a new process group to it and its children. calling os.killpg on it thus won't bring down your own python process also.
poll( )
Check if child process has terminated.
Returns returncode attribute.
A solution for windows may be to use the win32 job api e.g. How do I automatically destroy child processes in Windows?
Here's an existing python implementation
https://gist.github.com/ubershmekel/119697afba2eaecc6330
Is there a way to ensure all created subprocess are dead at exit time of a Python program? By subprocess I mean those created with subprocess.Popen().
You could violate encapsulation and test that all Popen processes have terminated by doing
subprocess._cleanup()
print subprocess._active == []
If not, should I iterate over all of the issuing kills and then kills -9? anything cleaner?
You cannot ensure that all subprocesses are dead without going out and killing every survivor. But if you have this problem, it is probably because you have a deeper design problem.
I actually needed to do this, but it involved running remote commands. We wanted to be able to stop the processes by closing the connection to the server. Also, if, for example, you are running in the python repl, you can select to run as foreground if you want to be able to use Ctrl-C to exit.
import os, signal, time
class CleanChildProcesses:
"""
with CleanChildProcesses():
Do work here
"""
def __init__(self, time_to_die=5, foreground=False):
self.time_to_die = time_to_die # how long to give children to die before SIGKILL
self.foreground = foreground # If user wants to receive Ctrl-C
self.is_foreground = False
self.SIGNALS = (signal.SIGHUP, signal.SIGTERM, signal.SIGABRT, signal.SIGALRM, signal.SIGPIPE)
self.is_stopped = True # only call stop once (catch signal xor exiting 'with')
def _run_as_foreground(self):
if not self.foreground:
return False
try:
fd = os.open(os.ctermid(), os.O_RDWR)
except OSError:
# Happens if process not run from terminal (tty, pty)
return False
os.close(fd)
return True
def _signal_hdlr(self, sig, framte):
self.__exit__(None, None, None)
def start(self):
self.is_stopped = False
"""
When running out of remote shell, SIGHUP is only sent to the session
leader normally, the remote shell, so we need to make sure we are sent
SIGHUP. This also allows us not to kill ourselves with SIGKILL.
- A process group is called orphaned when the parent of every member is
either in the process group or outside the session. In particular,
the process group of the session leader is always orphaned.
- If termination of a process causes a process group to become orphaned,
and some member is stopped, then all are sent first SIGHUP and then
SIGCONT.
consider: prctl.set_pdeathsig(signal.SIGTERM)
"""
self.childpid = os.fork() # return 0 in the child branch, and the childpid in the parent branch
if self.childpid == 0:
try:
os.setpgrp() # create new process group, become its leader
os.kill(os.getpid(), signal.SIGSTOP) # child fork stops itself
finally:
os._exit(0) # shut down without going to __exit__
os.waitpid(self.childpid, os.WUNTRACED) # wait until child stopped after it created the process group
os.setpgid(0, self.childpid) # join child's group
if self._run_as_foreground():
hdlr = signal.signal(signal.SIGTTOU, signal.SIG_IGN) # ignore since would cause this process to stop
self.controlling_terminal = os.open(os.ctermid(), os.O_RDWR)
self.orig_fore_pg = os.tcgetpgrp(self.controlling_terminal) # sends SIGTTOU to this process
os.tcsetpgrp(self.controlling_terminal, self.childpid)
signal.signal(signal.SIGTTOU, hdlr)
self.is_foreground = True
self.exit_signals = dict((s, signal.signal(s, self._signal_hdlr))
for s in self.SIGNALS)
def stop(self):
try:
for s in self.SIGNALS:
#don't get interrupted while cleaning everything up
signal.signal(s, signal.SIG_IGN)
self.is_stopped = True
if self.is_foreground:
os.tcsetpgrp(self.controlling_terminal, self.orig_fore_pg)
os.close(self.controlling_terminal)
self.is_foreground = False
try:
os.kill(self.childpid, signal.SIGCONT)
except OSError:
"""
can occur if process finished and one of:
- was reaped by another process
- if parent explicitly ignored SIGCHLD
signal.signal(signal.SIGCHLD, signal.SIG_IGN)
- parent has the SA_NOCLDWAIT flag set
"""
pass
os.setpgrp() # leave the child's process group so I won't get signals
try:
os.killpg(self.childpid, signal.SIGINT)
time.sleep(self.time_to_die) # let processes end gracefully
os.killpg(self.childpid, signal.SIGKILL) # In case process gets stuck while dying
os.waitpid(self.childpid, 0) # reap Zombie child process
except OSError as e:
pass
finally:
for s, hdlr in self.exit_signals.iteritems():
signal.signal(s, hdlr) # reset default handlers
def __enter__(self):
if self.is_stopped:
self.start()
def __exit__(self, exit_type, value, traceback):
if not self.is_stopped:
self.stop()
Thanks to Malcolm Handley for the initial design. Done with python2.7 on linux.
You can try subalive, a package I wrote for similar problem. It uses periodic alive ping via RPC, and the slave process automatically terminates when the master stops alive pings for some reason.
https://github.com/waszil/subalive
Example for master:
from subalive import SubAliveMaster
# start subprocess with alive keeping
SubAliveMaster(<path to your slave script>)
# do your stuff
# ...
Example for slave subprocess:
from subalive import SubAliveSlave
# start alive checking
SubAliveSlave()
# do your stuff
# ...
It's possible to get some more guarantees on windows by spawning a separate process to oversee the destruction.
import subprocess
import sys
import os
def terminate_process_on_exit(process):
if sys.platform == "win32":
try:
# Or provide this script normally.
# Here just to make it somewhat self-contained.
# see https://stackoverflow.com/a/22559493/3763139
# see https://superuser.com/a/1299350/388191
with open('.process_watchdog_helper.bat', 'x') as file:
file.write(""":waitforpid
tasklist /nh /fi "pid eq %1" 2>nul | find "%1" >nul
if %ERRORLEVEL%==0 (
timeout /t 5 /nobreak >nul
goto :waitforpid
) else (
wmic process where processid="%2" call terminate >nul
)""")
except:
pass
# After this spawns we're pretty safe. There is a race, but we do what we can.
subprocess.Popen(
['.process_watchdog_helper.bat', str(os.getpid()), str(process.pid)],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL
)
# example
class DummyProcess:
def __init__(self, pid):
self.pid = pid
set_terminate_when_this_process_dies(DummyProcess(7516))
This is what I did for my posix app:
When your app exists call the kill() method of this class:
http://www.pixelbeat.org/libs/subProcess.py
Example use here:
http://code.google.com/p/fslint/source/browse/trunk/fslint-gui#608
help for python code:
http://docs.python.org/dev/library/subprocess.html#subprocess.Popen.wait