I read the question/answer/comments on A non-blocking read on a subprocess.PIPE in Python, but I felt a bit lacking.
When I implemented the solution provided, I noticed that this approach works best when the sub-process ends on it own. But if the subprocess is providing a stream of information and we are looking for a single match of output, then that approach doesn't work for my needs (specifically for Windows, if that matters).
Here is my sample:
File ping.py
import time
def main():
for x in range(100):
print x
time.sleep(1)
if __name__ == '__main__':
print("Starting")
time.sleep(2)
main()
File runner.py
import subprocess
import time
import sys
from Queue import Queue, Empty
from threading import Thread
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
# Start process we want to listen to
pPing = subprocess.Popen('ping.py',
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
q = Queue()
t = Thread(target=enqueue_output, args=(pPing.stdout, q))
t.daemon = True
t.start()
# Make sure it's started
print ("get the first line")
try:
line = q.get()
except Empty:
pass
else:
print line.strip()
#look for the 'magic' output
print("empty the queue")
while not q.empty():
line = q.get_nowait().strip()
if (line == "3"):
print("got it!")
sys.exit()
else:
print("not yet")
My expectation is that the runner will make sure the process is started and then wait for the magic output and then stop, which it does. However, the longer the sub-process runs, the longer the runner runs. But since the 'magic' output comes relatively quickly, I have to wait until the subprocess ends before I get anything processed.
What am I missing?
OK, if I understand correctly what you are trying to do, the problem is with ping still being a child process to runner. While you can make read calls non-blocking, the parent process will not actually exit while the child is still running. If you want runner not to wait for the child to finish, read the first line and the first magic output and then exit; you need ping to disassociate itself from the parent process.
Look at this code sample to see how that is done A simple Unix/Linux daemon in Python. Of course you might skip the part where they close and re-open all the I/O streams.
On the same note, I am not sure leaving an open I/O stream connected to the parent will allow the parent to exit, so if that happens to be a problem you might have to figure out another way to exchange data.
Related
I'm having a problem with subprocess poll not returning the return code when the process has finished.
I found out how to set a timeout on subprocess.Popen and used that as the basis for my code. However, I have a call that uses Java that doesn't correctly report the return code so each call "times out" even though it is actually finished. I know the process has finished because when removing the poll timeout check, the call runs without issue returning a good exit code and within the time limit.
Here is the code I am testing with.
import subprocess
import time
def execute(command):
print('start command: {}'.format(command))
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print('wait')
wait = 10
while process.poll() is None and wait > 0:
time.sleep(1)
wait -= 1
print('done')
if wait == 0:
print('terminate')
process.terminate()
print('communicate')
stdout, stderr = process.communicate()
print('rc')
exit_code = process.returncode
if exit_code != 0:
print('got bad rc')
if __name__ == '__main__':
execute(['ping','-n','15','127.0.0.1']) # correctly times out
execute(['ping','-n','5','127.0.0.1']) # correctly runs within the time limit
# incorrectly times out
execute(['C:\\dev\\jdk8\\bin\\java.exe', '-jar', 'JMXQuery-0.1.8.jar', '-url', 'service:jmx:rmi:///jndi/rmi://localhost:18080/jmxrmi', '-json', '-q', 'java.lang:type=Runtime;java.lang:type=OperatingSystem'])
You can see that two examples are designed to time out and two are not to time out and they all work correctly. However, the final one (using jmxquery to get tomcat metrics) doesn't return the exit code and therefore "times out" and has to be terminated, which then causes it to return an error code of 1.
Is there something I am missing in the way subprocess poll is interacting with this Java process that is causing it to not return an exit code? Is there a way to get a timeout option to work with this?
This has the same cause as a number of existing questions, but the desire to impose a timeout requires a different answer.
The OS deliberately gives only a small amount of buffer space to each pipe. When a process writes to one that is full (because the reader has not yet consumed the previous output), it blocks. (The reason is that a producer that is faster than its consumer would otherwise be able to quickly use a great deal of memory for no gain.) Therefore, if you want to do more than one of the following with a subprocess, you have to interleave them rather than doing each in turn:
Read from standard output
Read from standard error (unless it’s merged via subprocess.STDOUT)
Wait for the process to exit, or for a timeout to elapse
Of course, the subprocess might close its streams before it exits, write useful output after you notice the timeout and before you kill it, and/or start additional processes that keep the pipe open indefinitely, so you might want to have multiple timeouts. Probably what’s most informative is the EOF on the pipe, so repeatedly use something like select to wait for (however much is left of) the timeout, issue single reads on the streams that are ready, and wait (with another timeout if you’re concerned about hangs after an early stream closure) on EOF. If the timeout occurs instead, (try to) kill the subprocess, and consider issuing non-blocking reads (or another timeout loop) to get any last available output before closing the pipes.
Using the other answer by #DavisHerring as the basis for more research, I came across a concept that worked for my original case. Here is the code that came out of that.
import subprocess
import threading
import time
def execute(command):
print('start command: {}'.format(command))
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
timer = threading.Timer(10, terminate_process, [process])
timer.start()
print('communicate')
stdout, stderr = process.communicate()
print('rc')
exit_code = process.returncode
timer.cancel()
if exit_code != 0:
print('got bad rc')
def terminate_process(p):
try:
p.terminate()
except OSError:
pass # ignore error
It uses the threading.Timer to make sure that the process doesn't go over the time limit and terminates the process if it does. It otherwise waits for a response back and cancels the timer once it finishes.
I am writing code to run experiments in parallel. I don't have control over what the experiments do, they might open use subprocess.Popen or check_output to run one or multiple additional child processes.
I have two conditions: I want to be able to kill experiments that exceed a time out and I want to kill experiments upon KeyboardInterrupt.
Most ways to terminate processes don't make sure that all subprocesses etc are killed. This is obviously a problem if 100s of experiments are run one after the other but they all spawn child processes that stay alive after the timeout occurred and the experiment was supposedly killed.
The way I am dealing with this now it to include code to store experiment configurations in a database, generating code that loads and runs experiments from command line and then calling these commands via subprocess.Popen(cmd, shell=True, start_new_session=True) and killing them using os.killpg on timeout.
My main question then is: Calling these experiments via command line feels cumbersome, so is there a way to call code directly via multiprocessing.Process(target=fn) and achieving the same effect of start_new_session=True + os.killpg upon timeout and KeyboardInterrupt?
<file1>
def run_exp(config):
do work
return result
if __name__ == "__main__":
save_exp(run_exp(load_config(sys.args)))
<file2>
def monitor(queue):
active = set() # active process ids
while True:
msg = queue.get()
if msg == "sentinel":
<loop over active ids and kill them with os.killpg>
else:
<add or remove id from active set>
def worker(args):
id, queue = args
command = f"python <file1> {id}"
with subprocess.Popen(command, shell=True, ..., start_new_session=True) as process:
try:
queue.put(f"start {process.pid}")
process.communicate(timeout=timeout)
except TimeoutExpired:
os.killpg(process.pid, signal.SIGINT) # send signal to the process group
process.communicate()
finally:
queue.put(f"done {process.pid}")
def main():
<save configs => c_ids>
queue = manager.Queue()
process = Process(target=monitor, args=(queue,))
process.start()
def clean_exit():
queue.put("sentinel")
<terminate pool and monitor process>
r = pool.map_async(worker, [(c_id, queue) for c_id in c_ids])
atexit.register(clean_exit)
r.wait()
<terminate pool and monitor process>
I posted a skeleton of the code that details the approach of starting processes via command line and killing them. An additional complication of that version of my approach is that when the KeyboardInterrupt arrives, the queue already gets terminated (for a lack of a better word) and communicating with the monitor process is impossible (the sentinel message never arrives). Instead I have to resort to writing process ids to file and reading the file back to in the main process to kill the still running processes. If you know a way to work around this queue-issue I'd be eager to learn about it.
I think the problem is you are storing Subprocess pid to kill it you need host process pid, and you used signal.SIGINT which I think should be signal.SIGTERM. try this, instead of this line:
os.killpg(process.pid, signal.SIGINT)
use this line:
os.killpg(os.getpgid(process.pid), signal.SIGTERM)
I guess there is one way to avoid this is using Try catch block.
Say if the KeyboardInterrupt arrives in main() then you could try this:
def main():
try:
<save configs => c_ids>
queue = manager.Queue()
process = Process(target=monitor, args=(queue,))
process.start()
def clean_exit():
queue.put("sentinel")
<terminate pool and monitor process>
r = pool.map_async(worker, [(c_id, queue) for c_id in c_ids])
atexit.register(clean_exit)
r.wait()
<terminate pool and monitor process>
except KeyboardInterrupt as e:
pass
#write the process you want to keep continuing.
Guess this will be helpful.
I have 2 python (2.7) processes.
The parent process needs to send rows of text to a child process, and the child process should process them as they come in (not wait for the parent process to finish).
I have this code which doesn't work:
# Sender
import subprocess
process = subprocess.Popen(['python', 'child.py'], bufsize=1, stdin=subprocess.PIPE)
try:
while True:
process.stdin.write(msg + '\n') # 'msg' is a changing string
# process.stdin.flush() <-- commented out since it doesn't help
except KeyboardInterrupt:
process.stdin.close()
process.wait()
And the child process:
# Receiver
import sys
for line in sys.stdin:
print line.strip()
The problem is that the child process waits until the parent process exits before it prints out the messages.
What I'm trying to achieve is a child process that processes the messages as soon as they are written to the pipe.
Try adding a process.stdin.flush() after your process.stdin.write(). That way you actually send the string to the other process. What you're suffering from here is your kernel caching everything you write. It does this to be more efficient when actually sending the data to the other process. flush force the kernel to send your data regardless of how full the kernel's buffer is.
I tried your code as such:
# Sender
import subprocess
process = subprocess.Popen(['python', 'child.py'], bufsize=1, stdin=subprocess.PIPE)
msg = "This is my message"
try:
while True:
process.stdin.write(msg + '\n') # 'msg' is a changing string
process.stdin.flush() # This code works well for me regardless of the presence of this line
except KeyboardInterrupt:
process.stdin.close()
process.wait()
# Receiver
import sys
for line in sys.stdin:
print line.strip()
With "works well" here i mean that i get "This is my message" printed as fast as the computer can perform. I'm trying this in Python 2.7.12 for the record.
The story of how buffering works for sys.stdin and sys.stdout has made me cry more than once. A similar problem is discussed in Setting smaller buffer size for sys.stdin?.
As to your specific problem, I suggest you change your child to use sys.stdin.readline() instead of iterating over sys.stdin. The former somewhat "buffers less" :)
while True:
line = sys.stdin.readline()
if not line: break
print (line.strip())
In the parent, you'll likely either need to set bufsize=0 in your call to Popen (making your pipe completely unbuffered), or you'll need the process.stdin.flush() line, as Patrik suggests. I'd opt for the latter.
Tested on Python 2.7.14 on Windows 10 64bit.
In my python program I have two subprocesses interconnected by a pipe and with one connected to stdin and the other connected to stdout. My problem is that when the data flow ends the subprocesses hang until I press ctrl+c. It looks to me like the subprocesses are being held open my the pipe. If I could tell when the data flowing through the pipe I could close it manually.
def write(tag_name):
p_r, p_w = os.pipe()
pv = subprocess.Popen('pv', stdin=None, stdout=p_w)
dd = subprocess.Popen('dd bs=64k of=/dev/nst0'.split(), stdin=p_r, stdout=None)
dd.wait()
Just don't use os.pipe(), you can pass subprocess stdout directly to the other process stdin, like this:
def write(tag_name):
pv = subprocess.Popen('pv', stdin=None, stdout=subprocess.PIPE)
dd = subprocess.Popen('dd bs=64k of=/dev/nst0'.split(), stdin=pv.stdout, stdout=None)
dd.wait()
When first command ends, the pipe is broken (as opposed to os.pipe() which need to be closed manually), so it ends the second command as well and the script can continue/end.
I have tested a simple pipe command and with os.pipe() it blocks at the end as you described, but exited when first process ended with my modifications.
You need a non-blocking solution here.
Take a look at my solution: https://github.com/vesellov/bitdust.devel/blob/master/system/nonblocking.py
And you can call it this way (did not tested the code):
import nonblocking, time
p = nonblocking.Popen('pv'.split(), shell=True, )
p.make_nonblocking()
while 1:
if p.state() == nonblocking.PIPE_CLOSED:
# pipe closed, stop
return break
if p.state() == nonblocking.PIPE_READY2READ:
newchunk = p.recv(1024)
if newchunk == '':
# EOF reached, stop
break
# do something with the data here
# you can send it to second stream
try:
time.sleep(0.01)
except KeyboardInterrup:
break
So when you call dd.wait() it will block, that is why your Ctrl-C not working. You need to deal with this manually.... non-blocking streaming is not a trivial story in Python. Check-out Twisted project, you can find a lot of cool stuff :-)
So I'm trying to effectively create a "branch" in a pipe from subprocess. The idea is to load a file with Popen into a pipe's stdout. Then, I can send that stdout to two (or more) stdin's. This works, more or less. The problem comes when the process needs to see an EOF. As far as I can tell, this happens when you use communicate(None) on a subprocess. However, it also seems to depend on the order I spawned the two processes I'm trying to send data to.
#!/usr/bin/env python
from subprocess import *
import shutil
import os
import shlex
inSub=Popen(shlex.split('cat in.txt'),stdout=PIPE)
print inSub.poll()
queue=[]
for i in range(0,3):
temp=Popen(['cat'],stdin=PIPE)
queue=queue+[temp]
while True:
# print 'hi'
buf=os.read(inSub.stdout.fileno(),10000)
if buf == '': break
for proc in queue:
proc.stdin.write(buf)
queue[1].communicate()
print queue[1].poll()
As long as I use queue[1], things hang at the communicate() line. But if I use queue[2], things don't hang. What's going on? It shouldn't depend on the order the subprocesses were created, should it?
(The in.txt file can really be anything, it doesn't matter.)
I can't see any reason why it would be different for any one of the processes. In any case, closing the stdin pipes will cause Python to send the EOF, ending the processes:
...
while True:
# print 'hi'
buf = os.read(inSub.stdout.fileno(),10000)
if buf == '': break
for proc in queue:
proc.stdin.write(buf)
for proc in queue:
proc.stdin.close()
queue[1].communicate()
...