Develop Reference

How to get notified when a child process exits and closes a pipe?

This is a follow-up to my previous question:
Suppose I have a Python program, which forks and execs another Python program with subprocess.Popen.
I would like to create a pipe before the fork and get notification when the child closes the pipe when it exits. I don't want to wait for the pipe synchronously, of course. Is it doable in Python ?

You could use subprocess poll()
Although you will have to test every now and then to see if it has finished
Calling program:
import subprocess
import time
c=subprocess.Popen(['python', 'called.py'], stdout=subprocess.PIPE)
finished = False
while finished == False:
if c.poll() != None:
finished = True
break
time.sleep(1)
print "Still working"
print "finished"
Called program:
import time
time.sleep(15)

Managing a subprocess with gevent and stdin

I have written a small program to start and stop a subprocess on command from read from stdin, using gevent to, hopefully, efficiently wait for both the command input, and for the process to finish.
It has one command R - run, which it read of stdin as a line.
R simply starts sh with the script sleep 2; echo "Hello, world!" if it is not already running
There are two greenlets, one reading command input and one waiting for a new process or waiting for the current process to finish. I use an gevent Event to toggle between the two.
My problem: The greenlet for the process is never allow to finish. The command greenlet appears to always be running and never relinquishes control. Why does the process greenlet never wake from its wait, even when the process is clearly finished?
The source is as follows:
import sys
from gevent import spawn, joinall, sleep
from gevent.subprocess import Popen
from gevent.fileobject import FileObjectPosix
from gevent.event import Event
process = None
trigger = Event()
def start_process():
global process
process = Popen(['/bin/sh', '-c', 'sleep 2; echo Hello, World!'])
def wait_process():
global process
while True:
trigger.wait()
print('INFO - Awaiting new process')
trigger.clear()
process.wait()
process = None
print('INFO - Process done')
def get_input():
global process
stdin_wrapped = FileObjectPosix(sys.stdin)
while True:
line = stdin_wrapped.readline().decode().strip()
if line == 'R':
if not process:
start_process()
trigger.set()
print('OK - Running process')
else:
print('FAIL - Process already running')
sleep(0)
def main():
joinall([
spawn(get_input),
spawn(wait_process)
])
if __name__ == '__main__':
main()
A session looks like this, with a >2s gap between the Rs:
R
OK - Running process
INFO - Awaiting new process
Hello, World!
R
FAIL - Process already running
I expect to see:
R
OK - Running process
INFO - Awaiting new process
Hello, World!
INFO - Process done
R
OK - Running process
My initial ideas are one of two things are going wrong:
This is not the correct way to read a file with gevent
the subprocess wait event is not being used correctly, and it never wakes up. I have not seen an example where it is used like this, but Popen objects can be used with gevent.wait so I assumed this was ok.
If I break the debugger the stack trace shows that is is waiting for the read from stdin to finish, I expected it to have some select like behaviour, and when two greenlets were waiting on something it would execution would resume in the greenlet that finished first.

I have two solutions to my own problem. Either of these will make my example program function as expected.
Pipe for stdin
Open the subprocess with Popen(..., stdin=PIPE). It makes sense that gevent won't work without that since it must wait on something.
Use FileObjectThread
This seems to work regardless of how the subprocess is created, not sure why 😐

What is the best way to see if a python process is already running?

I'm trying to write a script that monitors my python program to see if it is already running.
I started by trying to assign this script to an object:
processWatch = os.system("sudo ps afxj | grep quickConversion.py")
if processWatch > 2: #if more than one is running it won't run.
while 1:
"rest of loop"
I then tried to monitor the object for more than one instance.

You may want to pay a look at psutils to handle processes.
import psutil
processWatch = [p.cmdline() for p in psutil.process_iter()].count(['python', 'quickConversion.py'])
if processWatch > 0:
`while 1: `
`"rest of loop"`

There are Linux commands that return a list of processes:
if 'your_process' in commands.getoutput('ps -A'):
print 'Your process in running.'

Subprocess's Popen is very useful for this. https://docs.python.org/3.4/library/subprocess.html
import subprocess
process = subprocess.Popen(cmd)
if process.poll() is None:
print("The process hasn't terminated yet.")

Lock process until subprocess reaches a certain point

I am running multiple subprocesses in parallel, but I need to lock each process until the subprocess gives an output (via print function). The subprocesses are running a python script that has been packaged to an executable.
The code looks like this:
import multiprocessing as mp
import subprocess
import os
def main(args):
l,inpath = args
l.acquire()
print "Running KNN.exe for files in %s" % os.path.normpath(inpath).split('\\')[-1]
#Run KNN executable as a subprocess
subprocess.call(os.path.join(os.getcwd(), "KNN.exe"))
#This is where I want to wait for any output from the subprocess before releasing the lock
l.release()
#Here I would like to wait until subprocess is done then print that it is done
l.acquire()
print "Done %s" % os.path.normpath(inpath).split('\\')[-1]
l.release()
if __name__ == "__main__":
#Set working directory path containing input text file
os.chdir("C:\Users\Patrick\Google Drive\KNN")
#Get folder names in directory containing GCM input
manager = mp.Manager()
l = manager.Lock()
gcm_dir = "F:\FIDS_GCM_Data_CMIP5\UTRB\UTRB KNN-CAD\Input"
paths = [(l, os.path.join(gcm_dir, folder)) for folder in os.listdir(gcm_dir)]
#Set up multiprocessing pool
p = mp.Pool(mp.cpu_count())
#Map function through input paths
p.map(main, paths)
So the goal is to lock the process so that a subprocess can be run until receiving an output. After which the lock can be released and the subprocess can continue, until it is complete, then I'd like to print that it is complete.
My question is how can I wait for the single (and only) output from the subprocess before releasing the lock on the process (out of multiple)?
Additionally how can I wait for the process to terminate then print that it is complete?

Your code makes use of the call method, which already waits for the subprocess to finish (which means all output has already been generated). I'm inferring from your question you'd like to be able to differentiate between when output is first written and when the subprocess is finished. Below is your code with my recommended modifications inline:
def main(args):
l,inpath = args
l.acquire()
print "Running KNN.exe for files in %s" % os.path.normpath(inpath).split('\\')[-1]
#Run KNN executable as a subprocess
#Use the Popen constructor
proc = subprocess.Popen(os.path.join(os.getcwd(), "KNN.exe"), stdout=subprocess.PIPE)
#This is where I want to wait for any output from the subprocess before releasing the lock
# Wait until the subprocess has written at least 1 byte to STDOUT (modify if you need different logic)
proc.stdout.read(1)
l.release()
#Here I would like to wait until subprocess is done then print that it is done
#proc.wait()
(proc_output, proc_error) = proc.communicate()
l.acquire()
print "Done %s" % os.path.normpath(inpath).split('\\')[-1]
l.release()
Note that the above doesn't assume you want to do anything with the subprocess's output other than check that it has been generated. If you want to do anything with that output that is less trivial than the above (consume 1 byte then drop it on the floor), the proc.stdout (which is a file object) should represent everything that the subprocess generates while running.

Python: subprocess.popen()

I have a question regarding subprocess.popen():
If supposedly the command executed is in a while loop - is there any way for subprocess.popen() to detect it and exit after printing the first output?
Or is there any other way to do this and print the result?
As you see the following program executed on a linux machine just keeps on executing:
>>> import os
>>> import subprocess as sp
>>> p = sp.Popen("yes", stdout=sp.PIPE)
>>> result = p.communicate()[0]

The communicate method is only useful if the program being called is expected to terminate soon with relatively little output. In the case of your example, the "yes" program never terminates, so communicate never completes. In order to deal with subprocesses which execute indefinitely, and may produce a lot of output, you will need to include a loop which repeatedly calls p.poll() until p.poll() returns a value other than None, which would indicate that the process has terminated. While in this loop you should read from p.stdout and p.stderr to consume any output from the program. If you don't consume the output, the buffers may fill up and cause the program to block waiting to be able to write more output.
import subprocess
import time
p = subprocess.Popen("yes", stdout=subprocess.PIPE)
result = ""
start_time = time.time()
while (p.poll() is None):
result += p.stdout.read(8192)
time.sleep(1)
if (time.time() - start_time) > 5:
print "Timeout"
break
print result
Note that the above example will run indefinitely until you kill the "yes" subprocess it is reading input from. If you want to detect that the process doesn't terminate, you can add a time check to the while loop, and jump out at some point once enough time has passed.
If you are certain that your subprocess will terminate of it's own accord, you can simply call communicate() and get back the output, but this does not work in your example for the reasons I explain above.