How we can hook up a code inside a python process so that it should send an alert in case of shutdown of process/abrupt termination ?
Use Supervisor Daemon
It's not clear what exactly you mean. Shutdown/abort of the process itself? Or of a child process?
Shutdown/abort of a process itself: Have a look at Pythons atexit module; here you can register a callback for when your program cleanly exits. But there is absolutely no way for you to catch all circumstances, if your program fails b/o a serious issue (e.g. segfault) your atexit handlers will never get called. You need a supervising process to catch absolutely all aborts.
Shutdown/abort of a child process: If you e.g. use the subprocess module you can simply call poll() or wait() on popen objects to see if the spawned process is dead / wait for them to die. For a more advanced implementation use Pythons signal module to set a handler for SIGCHLD - this signal is sent to your process whenever one of the child processes terminates.
Related
How can I return a process id of a lengthy process started using Thread in Python before the thread completes its execution?
I'm using Tkinter GUI so I can't start a lengthy process on the main thread so instead I start one on a separate thread.
The thread in turn calls subprocess.popen. This process should run for like 5 -6 hours.
But When I press stopbutton I need this process to stop but I am unable to return the process id of the process created using subprocess.popen.
Is there any solution to this?
If you are using subprocess.Popen simply to spin off another process, there is no reason you need to do so from another thread. A sub-process created this way does not block your main thread. You can continue to do other things while the sub-process is running. You simply keep a reference to the Popen object returned.
The Popen object has all the facilities you need for monitoring / interacting with the sub-process. You can read and write to its standard input and output (via stdin and stdout members, if created with PIPE); you can monitor readability / writability of stdin and stdout (with select module); you can check whether the sub-process is still in existence with poll, reap its exit status with wait; you can stop it with terminate (or kill depending on how emphatic you wish to be).
There are certainly times when it might be advantageous to do this from another thread -- for example, if you need significant interaction with the sub-process and implementing that in the main thread would over-complicate your logic. In that case, it would be best to arrange a mechanism whereby you signal to your other "monitoring" thread that it's time to shutdown and allow the monitoring thread to execute terminate or kill on the sub-process.
I have a Python 2.7 multiprocessing Process which will not exit on parent process exit. I've set the daemon flag which should force it to exit on parent death. The docs state that:
"When a process exits, it attempts to terminate all of its daemonic child processes."
p = Process(target=_serverLaunchHelper, args=args)
p.daemon = True
print p.daemon # prints True
p.start()
When I terminate the parent process via a kill command the daemon is left alive and running (which blocks the port on the next run). The child process is starting a SimpleHttpServer and calling serve_forever without doing anything else. My guess is that the "attempts" part of the docs means that the blocking server process is stopping process death and it's letting the process get orphaned as a result. I could have the child push the serving to another Thread and have the main thread check for parent process id changes, but this seems like a lot of code to just replicate the daemon functionality.
Does someone have insight into why the daemon flag isn't working as described? This is repeatable on windows8 64 bit and ubuntu12 32 bit vm.
A boiled down version of the process function is below:
def _serverLaunchHelper(port)
httpd = SocketServer.TCPServer(("", port), Handler)
httpd.serve_forever()
When a process exits, it attempts to terminate all of its daemonic child processes.
The key word here is "attempts". Also, "exits".
Depending on your platform and implementation, it may be that the only way to get daemonic child processes terminated is to do so explicitly. If the parent process exits normally, it gets a chance to do so explicitly, so everything is fine. But if the parent process is terminated abruptly, it doesn't.
For CPython in particular, if you look at the source, terminating daemonic processes is handled the same way as joining non-daemonic processes: by walking active_children() in an atexit function. So, your daemons will be killed if and only if your atexit handlers get to run. And, as that module's docs say:
Note: the functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called.
Depending on how you're killing the parent, you might be able to work around this by adding a signal handler to intercept abrupt termination. But you might not—e.g., on POSIX, SIGKILL is not intercept able, so if you kill -9 $PARENTPID, this isn't an option.
Another option is to kill the process group, instead of just the parent process. For example, if your parent has PID 12345, kill -- -12345 on linux will kill it and all of its children (assuming you haven't done anything fancy).
I have a socket server that used threading to open a thread for each client that connects.
I also have two other threads that run constantly that are doing maintenance operations.
Basically there is the main thread plus two children running constantly, plus one child for each client that connects.
I want to be able to terminate or restart safely.
I would like to be able to trigger a termination function somehow that would instruct all child processes to terminate safely and then the parent could exit.
Any ideas?
Please do not suggest to connect as a client and send a command that would trigger that.
Already thought of it.
I am looking for a way to do this by executing something in the console.
The python socket server runs as a system service and would like to implement the termination in the init script.
The best way to do this is setup a signal handler in your main thread. This can be done using the signal module. See: http://docs.python.org/library/signal.html. A good way would be to trap the CTRL-C signal (SIGINT).
Please note that the signal handler can also be a class method, so you do not have to use a global method (it took me a while to discover that).
def __init__(self):
signal.signal(signal.SIGINT, self.just_kill_me)
def just_kill_me(self, sig, frame):
self.stopped = True
for t in self.threads:
t.join()
It is not possible to send the equivalent of a kill signal to a thread. Instead you should set a flag that will signal the children to stop.
Your child threads should run in a loop, periodically checking if the parent requests them to stop.
while not parent.stopped:
do_some_maintenance_work
When using mpirun, is it possible to catch signals (for example, the SIGINT generated by ^C) in the code being run?
For example, I'm running a parallelized python code. I can except KeyboardInterrupt to catch those errors when running python blah.py by itself, but I can't when doing mpirun -np 1 python blah.py.
Does anyone have a suggestion? Even finding how to catch signals in a C or C++ compiled program would be a helpful start.
If I send a signal to the spawned Python processes, they can handle the signals properly; however, signals sent to the parent orterun process (i.e. from exceeding wall time on a cluster, or pressing control-C in a terminal) will kill everything immediately.
I think it is really implementation dependent.
In SLURM, I tried to use sbatch --signal USR1#30 to send SIGUSR1 (whose signum is 30,10 or 16) to the program launched by srun commands. And the process received signal SIGUSR1 = 10.
For platform MPI of IBM, according to https://www.ibm.com/support/knowledgecenter/en/SSF4ZA_9.1.4/pmpi_guide/signal_propagation.html
SIGINT, SIGUSR1, SIGUSR2 will be bypassed to processes.
In MPICH, SIGUSR1 is used by the process manager for internal notification of abnormal failures.
ref: http://lists.mpich.org/pipermail/discuss/2014-October/003242.html>
Open MPI on the other had will forward SIGUSR1 and SIGUSR2 from mpiexec to the other processes.
ref: http://www.open-mpi.org/doc/v1.6/man1/mpirun.1.php#sect14>
For IntelMPI, according to https://software.intel.com/en-us/mpi-developer-reference-linux-hydra-environment-variables
I_MPI_JOB_SIGNAL_PROPAGATION and I_MPI_JOB_TIMEOUT_SIGNAL can be set to send signal.
Another thing worth notice: For many python scripts, they will invoke other library or codes through cython, and if the SIGUSR1 is caught by the sub-process, something unwanted might happen.
If you use mpirun --nw, then mpirun itself should terminate as soon as it's started the subprocesses, instead of waiting for their termination; if that's acceptable then I believe your processes would be able to catch their own signals.
The signal module supports setting signal handlers using signal.signal:
Set the handler for signal signalnum to the function handler. handler can be a callable Python object taking two arguments (see below), or one of the special values signal.SIG_IGN or signal.SIG_DFL. The previous signal handler will be returned ...
import signal
def ignore(sig, stack):
print "I'm ignoring signal %d" % (sig, )
signal.signal(signal.SIGINT, ignore)
while True: pass
If you send a SIGINT to a Python interpreter running this script (via kill -INT <pid>), it will print a message and simply continue to run.
I have a process id in Python. I know I can kill it with os.kill(), but how do I check if it is alive ? Is there a built-in function or do I have to go to the shell?
Use subprocess module to spawn process.
There is proc.poll() function - it returns None if process is still alive, otherwise it returns process returncode.
http://docs.python.org/library/subprocess.html
os.kill does not kill processes, it sends them signals (it's poorly named).
If you send signal 0, you can determine whether you are allowed to send other signals. An error code will indicate whether it's a permission problem or a missing process.
See man 2 kill for more info.
Also, if the process is your child, you can get a SIGCHLD when it dies, and you can use one of the wait calls to deal with it.
With psutil you can check if a process id exists:
import psutil
print(psutil.pid_exists(1234))