Develop Reference

Create process in python, only if it doesn't exist

I have a python server that eventually needs a background process to perform an action.
It creates a child process that should be able to last more than its parent. But it shouldn't create such a child process if it is already running (it can happen if a previous parent process created it).
I can think of a couple of different aproaches to solve this problem:
Check all current running processes before creating the new one: Cross-platform way to get PIDs by process name in python
Write a file when the child process starts, delete it when it's done. Check the file before creating a child process.
But none of them seem to perfectly fit my needs. Solution (1) doesn't work well if child process is a fork of its parent. Solution (2) is ugly, it looks prone to failure.
It would be great for me to provide a fixed pid or name at process creation, so I could always look for the process in system in a fixed way and be certain if it is running or not. But I haven't found a way to do this.

"It creates a child process that should be able to last more than its parent." Don't.
Have a longer lived service process create the child for you. Talk to this service over a Unix domain socket. It then can be used to pass file descriptors to the child. The service can also trivially ensure that it only ever has a single child.
This is the pattern that can be used to eliminate the need for children that outlive their parents.
Using command names makes it trivial to do a DoS by just creating a process with the same name that does nothing. Using PID files is ambiguous due to PID reuse. Only having a supervisor that waits on its children can it properly restart them when they exit or ensure that they are running.

How to avoid python-daemon limitations?

I have a python script which attempts to communicate with a python daemon. When the original script is invoked, it checks to see if the daemon exists. If the daemon exists, the original script writes to a named pipe to communicate with the daemon. If the daemon does not exists, the original script attempts to create a daemon using DaemonContext and then writes to the named pipe.
Pseudo-code of the original script:
from daemon import DaemonContext
if daemon_exists():
pass
else:
with DaemonContext():
create_daemon()
communicate_with_daemon()
The problem is that when the daemon is created, the parent process is killed (i.e. communicate_with_daemon will never be executed). This prevents the original script from creating a daemon and communicating with it.
According to this answer, this problem is a limitation of the python-daemon library. How would I get around this?
Thanks.

You're describing not a limitation, but a definition of how a daemon process works.
[…] the parent process is killed (i.e. communicate_with_daemon will never be executed).
Yes, that's right; the daemon process detaches from what started it. That's what makes the process a daemon.
However, this statement is not true:
This prevents the original script from creating a daemon and communicating with it.
There are numerous other ways to communicate between processes. The general name for this is Inter-Process Communication. The solutions are many, and which you choose depends on the constraints of your application.
For example, you could open a socket at a known path and preserve that open file; you could open a network port and communicate through the loopback interface; you could make a "drop-box" communication at a file on the local filesystem store, a database, or otherwise; etc.

Reserve file descriptors to guard against FD exhaustion

I have a python application with multiple threads, with threads 2 to n potentially opening an arbitrary number of files. I want to make sure that when thread 1 tries to open a file, it will definitely not fail because of file descriptor exhaustion. In short, I want to reserve file descriptors without opening them.
I can only control the code being run from thread 1, which is spawned at a time when there's still plenty of file descriptors left.
(As an example, I imagine this could be done by 'reserving' fds by opening /dev/null a couple of times from thread 1, and closing it when thread 1 needs to open a file, thus making sure that there's at least one unused fd, but this ends up introducing a race condition.)
Is there a way to make sure thread 1 will have fds available when it needs them without modifying what threads 2-n do?

You need to use a mutex. For python 2.x that's the mutex or thread module.
In your "thread 1" you will access (obtain/lock) the mutex first, then close the reserved fd and open the real one, then release the mutex.
In your other threads, you simply wait till you can get the mutex, then open the file, then release the mutex.
For python3 it's the Lock from the threading module.
(Note: I'm not commenting on whether or not opening /dev/null achieves what you want in terms of reserving an fd, because I'm not confident about that. I'm just providing you a solution for how to avoid the race that you askeded about :) ).

Jython: subprocess.Popen runs out of file descriptors

I'm using the Jython 2.51 implementation of Python to write a script that repeatedly invokes another process via subprocess.Popen and uses PIPE to pipe stdout and stderr to the parent process and stdin to the child process. After several hundred loop iterations, I seem to run out of file descriptors.
The Python subprocess documentation mentions very little about freeing file descriptors, other than the close_fds option, which isn't described very clearly (Why should there be any file descriptors besides 0, 1 and 2 open in the first place?). I'm assuming that in CPython, reference counting takes care of the resource freeing issue. What's the proper way to make sure all descriptors get freed when one is done with a Popen object in Jython?
Edit: Just in case it makes a difference, this is a multithreaded program, so there are several Popen processes running simultaneously.

This only answers part of your question, but my understanding is that, when you spawn a new process, it normally inherits all the handles of the parent process. That includes such things as open files and sockets that you're listening on.
On UNIX, that's a side-effect of using 'fork', which duplicates the current process and all of its handles before loading the new executable. On Windows it's more explicit, but Python does it anyway, to try to match the behavior across platforms as much as possible.
The close_fds option, when True, closes all these inherited handles after spawning the subprocess, so the new executable starts with a clean slate. But if your subprocesses are run one at a time, and terminating when they're done, then this shouldn't be the problem.

Recover process with subprocess.Popen?

I have a python program that uses subprocess.Popen to launch another process (python process or whatever), and after launching it I save the child's PID to a file. Let's suppose that suddenly the parent process dies (because of an exception or whatever). Is there any way to access again to the object returned by Popen?
I mean, the basic idea is to read the file at first, and if it exists and it has a PID written on it, then access to that process someway, in order to know the return code or whatever. If there isn't a PID, then launch the process with Popen.
Thanks a lot!!

The Popen object is effectively just a wrapper for the child processes PID, stdin, stdout, and stderr, plus some convenience functions for using those.
So the question is why do you need access to the Popen object? Do you want to communicate with the child, terminate it, or check whether it's still running?
In any case there is no way reacquire a Popen object for an already running process.
The proper way to approach this is to launch the child as a daemon, like Tobu suggested. Part of the procedure for daemonising a process is to close stdin and stdout, so you cannot use those to talk to the child process. Instead most daemons use either pipes or sockets to allow clients to connect to them and to send them messages.
The easiest way to talk to the child is to open a named pipe from the child process at e.g. /etc/my_pipe, open that named pipe from the parent / controlling process, and write / read to / from it.
After a quick look at python-daemon it seems to me that python-daemon will help you daemonise your child process, which is tricky to get right, but it doesn't help you with the messaging side of things.
But like I said, I think you need to tell us why you need a Popen object for the child process before we can help you any further.

If a process dies, all its open file handles are closed. This includes any unnamed pipes created by popen(). So, no, there's no way to recover a Popen object from just a PID. The OS won't even consider your new process the parent, so you won't even get SIGCHLD signals (though waitpid() might still work).
I'm not sure if the child is guaranteed to survive, either, since a write to a pipe with no reader (namely, the redirected stdout of the child) should kill the child with a SIGPIPE.
If you want your parent process to pick up where the child left off, you need to spawn the child to write to a file, usually in /tmp or /var/log, and have it record its PID like you are now (the usual location is /var/run). (Having it write to a named pipe risks getting it killed with SIGPIPE as above.) If you suffix your filename with the PID, then it becomes easy for the manager process to figure out which file belongs to which daemon.

Looks like you're trying to write a daemon, and it needs pidfile support. You can't go wrong with python-daemon.
For example:
import daemon
import lockfile
import os
with daemon.DaemonContext(pidfile=lockfile.FileLock('/var/run/spam.pid')):
os.execl('/path/to/prog', args…)