I have a python program that uses subprocess.Popen to launch another process (python process or whatever), and after launching it I save the child's PID to a file. Let's suppose that suddenly the parent process dies (because of an exception or whatever). Is there any way to access again to the object returned by Popen?
I mean, the basic idea is to read the file at first, and if it exists and it has a PID written on it, then access to that process someway, in order to know the return code or whatever. If there isn't a PID, then launch the process with Popen.
Thanks a lot!!
The Popen object is effectively just a wrapper for the child processes PID, stdin, stdout, and stderr, plus some convenience functions for using those.
So the question is why do you need access to the Popen object? Do you want to communicate with the child, terminate it, or check whether it's still running?
In any case there is no way reacquire a Popen object for an already running process.
The proper way to approach this is to launch the child as a daemon, like Tobu suggested. Part of the procedure for daemonising a process is to close stdin and stdout, so you cannot use those to talk to the child process. Instead most daemons use either pipes or sockets to allow clients to connect to them and to send them messages.
The easiest way to talk to the child is to open a named pipe from the child process at e.g. /etc/my_pipe, open that named pipe from the parent / controlling process, and write / read to / from it.
After a quick look at python-daemon it seems to me that python-daemon will help you daemonise your child process, which is tricky to get right, but it doesn't help you with the messaging side of things.
But like I said, I think you need to tell us why you need a Popen object for the child process before we can help you any further.
If a process dies, all its open file handles are closed. This includes any unnamed pipes created by popen(). So, no, there's no way to recover a Popen object from just a PID. The OS won't even consider your new process the parent, so you won't even get SIGCHLD signals (though waitpid() might still work).
I'm not sure if the child is guaranteed to survive, either, since a write to a pipe with no reader (namely, the redirected stdout of the child) should kill the child with a SIGPIPE.
If you want your parent process to pick up where the child left off, you need to spawn the child to write to a file, usually in /tmp or /var/log, and have it record its PID like you are now (the usual location is /var/run). (Having it write to a named pipe risks getting it killed with SIGPIPE as above.) If you suffix your filename with the PID, then it becomes easy for the manager process to figure out which file belongs to which daemon.
Looks like you're trying to write a daemon, and it needs pidfile support. You can't go wrong with python-daemon.
For example:
import daemon
import lockfile
import os
with daemon.DaemonContext(pidfile=lockfile.FileLock('/var/run/spam.pid')):
os.execl('/path/to/prog', args…)
Related
I'm trying to use a custom job pool (using multiprocessing.Process), and it works nicely, except SIGINT gets passed on from the parent process all the way to the children of the pool workers, Chrome instances in this case. I used signal.signal(signal.SIGINT, signal.SIG_IGN) in the pool workers, which seems to keep them from getting (or at least responding to) the SIGINT, but I'm using third-party code (Selenium) which creates subprocesses that get the SIGINT and stop because of it. I want only the parent process to receive and handle the SIGINT. How do I do this?
My inclination would be to give the pool workers /dev/null as STDIN, so it would give that to its children rather than the terminal, which, if I understand correctly, will keep it from receiving SIGINT from Ctrl+C. However, it seems that multiprocessing.Process already does that, though presumably too late (after the process is created; using a double-fork should get past this). Is there a good way to do that (or anything else that blocks SIGINT) through multiprocessing, or do I need a different solution to work around that? Maybe there's an easy way to use multiprocessing.Queue with Popen(), so I can use that instead?
To be clear, I still need to be able to log to the console (indirectly through the parent process is fine, maybe better even).
I have a python script which attempts to communicate with a python daemon. When the original script is invoked, it checks to see if the daemon exists. If the daemon exists, the original script writes to a named pipe to communicate with the daemon. If the daemon does not exists, the original script attempts to create a daemon using DaemonContext and then writes to the named pipe.
Pseudo-code of the original script:
from daemon import DaemonContext
if daemon_exists():
pass
else:
with DaemonContext():
create_daemon()
communicate_with_daemon()
The problem is that when the daemon is created, the parent process is killed (i.e. communicate_with_daemon will never be executed). This prevents the original script from creating a daemon and communicating with it.
According to this answer, this problem is a limitation of the python-daemon library. How would I get around this?
Thanks.
You're describing not a limitation, but a definition of how a daemon process works.
[…] the parent process is killed (i.e. communicate_with_daemon will never be executed).
Yes, that's right; the daemon process detaches from what started it. That's what makes the process a daemon.
However, this statement is not true:
This prevents the original script from creating a daemon and communicating with it.
There are numerous other ways to communicate between processes. The general name for this is Inter-Process Communication. The solutions are many, and which you choose depends on the constraints of your application.
For example, you could open a socket at a known path and preserve that open file; you could open a network port and communicate through the loopback interface; you could make a "drop-box" communication at a file on the local filesystem store, a database, or otherwise; etc.
i think i have a problem with a ttyUSB device that caused from having 2 open ttyUSB fd's at the same time from different processes.
it goes like this:
i have a main python process, which opens several ttyUSB fd's, read, write, close, and open processes (with popen) to handle each ttyUSB (of course after the fd was closed).
when i do 'lsof | grep ttyUSB' it looks as if all the fd's that were opened in the main process when the child process started, associated to the child process even though they were already closed by the main process. (btw, the fd's are not associated to the main process)
is that behavior normal? (tinycore, kernal 2.6.33.3), do i have a way to prevent it?
thanks.
By default, any file descriptors that a process has open when it forks/execs (which happens during a popen()) are inherited by the child process. If this isn't what you want to happen, you will need to either manually close the file descriptors after forking, or set the fds as close-on-exec using fcntl(fd, F_SETFD, FD_CLOEXEC). (This makes the kernel automatically close the file descriptor when it execs the new process.)
I'm using the Jython 2.51 implementation of Python to write a script that repeatedly invokes another process via subprocess.Popen and uses PIPE to pipe stdout and stderr to the parent process and stdin to the child process. After several hundred loop iterations, I seem to run out of file descriptors.
The Python subprocess documentation mentions very little about freeing file descriptors, other than the close_fds option, which isn't described very clearly (Why should there be any file descriptors besides 0, 1 and 2 open in the first place?). I'm assuming that in CPython, reference counting takes care of the resource freeing issue. What's the proper way to make sure all descriptors get freed when one is done with a Popen object in Jython?
Edit: Just in case it makes a difference, this is a multithreaded program, so there are several Popen processes running simultaneously.
This only answers part of your question, but my understanding is that, when you spawn a new process, it normally inherits all the handles of the parent process. That includes such things as open files and sockets that you're listening on.
On UNIX, that's a side-effect of using 'fork', which duplicates the current process and all of its handles before loading the new executable. On Windows it's more explicit, but Python does it anyway, to try to match the behavior across platforms as much as possible.
The close_fds option, when True, closes all these inherited handles after spawning the subprocess, so the new executable starts with a clean slate. But if your subprocesses are run one at a time, and terminating when they're done, then this shouldn't be the problem.
Is there a way for a child process in Python to detect if the parent process has died?
If your Python process is running under Linux, and the prctl() system call is exposed, you can use the answer here.
This can cause a signal to be sent to the child when the parent process dies.
Assuming the parent is alive when you start to do this, you can check whether it is still alive in a busy loop as such, by using psutil:
import psutil, os, time
me = psutil.Process(os.getpid())
while 1:
if me.parent is not None:
# still alive
time.sleep(0.1)
continue
else:
print "my parent is gone"
Not very nice but...
The only reliable way I know of is to create a pipe specifically for this purpose. The child will have to repeatedly attempt to read from the pipe, preferably in a non-blocking fashion, or using select. It will get an error when the pipe does not exist anymore (presumably because of the parent's death).
You might get away with reading your parent process' ID very early in your process, and then checking, but of course that is prone to race conditions. The parent that did the spawn might have died immediately, and even before your process got to execute its first instruction.
Unless you have a way of verifying if a given PID refers to the "expected" parent, I think it's hard to do reliably.