I have 2 python (2.7) processes.
The parent process needs to send rows of text to a child process, and the child process should process them as they come in (not wait for the parent process to finish).
I have this code which doesn't work:
# Sender
import subprocess
process = subprocess.Popen(['python', 'child.py'], bufsize=1, stdin=subprocess.PIPE)
try:
while True:
process.stdin.write(msg + '\n') # 'msg' is a changing string
# process.stdin.flush() <-- commented out since it doesn't help
except KeyboardInterrupt:
process.stdin.close()
process.wait()
And the child process:
# Receiver
import sys
for line in sys.stdin:
print line.strip()
The problem is that the child process waits until the parent process exits before it prints out the messages.
What I'm trying to achieve is a child process that processes the messages as soon as they are written to the pipe.
Try adding a process.stdin.flush() after your process.stdin.write(). That way you actually send the string to the other process. What you're suffering from here is your kernel caching everything you write. It does this to be more efficient when actually sending the data to the other process. flush force the kernel to send your data regardless of how full the kernel's buffer is.
I tried your code as such:
# Sender
import subprocess
process = subprocess.Popen(['python', 'child.py'], bufsize=1, stdin=subprocess.PIPE)
msg = "This is my message"
try:
while True:
process.stdin.write(msg + '\n') # 'msg' is a changing string
process.stdin.flush() # This code works well for me regardless of the presence of this line
except KeyboardInterrupt:
process.stdin.close()
process.wait()
# Receiver
import sys
for line in sys.stdin:
print line.strip()
With "works well" here i mean that i get "This is my message" printed as fast as the computer can perform. I'm trying this in Python 2.7.12 for the record.
The story of how buffering works for sys.stdin and sys.stdout has made me cry more than once. A similar problem is discussed in Setting smaller buffer size for sys.stdin?.
As to your specific problem, I suggest you change your child to use sys.stdin.readline() instead of iterating over sys.stdin. The former somewhat "buffers less" :)
while True:
line = sys.stdin.readline()
if not line: break
print (line.strip())
In the parent, you'll likely either need to set bufsize=0 in your call to Popen (making your pipe completely unbuffered), or you'll need the process.stdin.flush() line, as Patrik suggests. I'd opt for the latter.
Tested on Python 2.7.14 on Windows 10 64bit.
Related
I need to realise a python script who read the ouput of an other process in real time line by line. Obviously I've ever try to use "subprocess" to get the output process with stdout. But i can't get that output in real time, indeed every times python script return the output in several chunk few time after the process launch. For exemple i use this python script :
import subprocess as sub
import sys
proc = sub.Popen(["python", "bin/read_test.py"],
stdout = sub.PIPE,
bufsize = 1)
while True:
if proc.poll() is None:
line = proc.stdout.read()
line = line.decode("utf-8")
print(line, flush = True)
sys.stdout.flush()
else:
proc.stdout.close()
break
read_test.py script to be read :
from time import sleep
for i in range(5):
print(i)
sleep(1)
I've try a lot of methods with "readline()" with for loops but the issue still the same.
Moreover I don't want to use "communicate" because it's a blocking method.
Thanks for your help,
Problem is that you're trying to read stdout fully.
Since python sees that process is still running, it waits until process ends so process output is complete.
To do what you want you probably want to read the output line by line using line = proc.stdout.readline()
You probably have to change your loop to read line, and stop when process ends
proc = sub.Popen(["python", "bin/read_test.py"],
stdout = sub.PIPE,
bufsize = 1)
while True:
line = proc.stdout.readline()
if line:
line = line.decode("utf-8")
print(line)
sys.stdout.flush()
if proc.poll() is not None: # process ends
break
proc.wait()
Also that's not enough: you have to flush the output on the read_test.py side to make sure that the emitter actually sends the lines (when output is redirected, it's not sure). Example program:
import time,sys
for i in range(5):
print(i)
sys.stdout.flush() # not flushing: you get all the lines at the end of the process
time.sleep(1)
I've connected both programs, and got 1 sequenced output (1,2,3,4,5) each second
In my python program I have two subprocesses interconnected by a pipe and with one connected to stdin and the other connected to stdout. My problem is that when the data flow ends the subprocesses hang until I press ctrl+c. It looks to me like the subprocesses are being held open my the pipe. If I could tell when the data flowing through the pipe I could close it manually.
def write(tag_name):
p_r, p_w = os.pipe()
pv = subprocess.Popen('pv', stdin=None, stdout=p_w)
dd = subprocess.Popen('dd bs=64k of=/dev/nst0'.split(), stdin=p_r, stdout=None)
dd.wait()
Just don't use os.pipe(), you can pass subprocess stdout directly to the other process stdin, like this:
def write(tag_name):
pv = subprocess.Popen('pv', stdin=None, stdout=subprocess.PIPE)
dd = subprocess.Popen('dd bs=64k of=/dev/nst0'.split(), stdin=pv.stdout, stdout=None)
dd.wait()
When first command ends, the pipe is broken (as opposed to os.pipe() which need to be closed manually), so it ends the second command as well and the script can continue/end.
I have tested a simple pipe command and with os.pipe() it blocks at the end as you described, but exited when first process ended with my modifications.
You need a non-blocking solution here.
Take a look at my solution: https://github.com/vesellov/bitdust.devel/blob/master/system/nonblocking.py
And you can call it this way (did not tested the code):
import nonblocking, time
p = nonblocking.Popen('pv'.split(), shell=True, )
p.make_nonblocking()
while 1:
if p.state() == nonblocking.PIPE_CLOSED:
# pipe closed, stop
return break
if p.state() == nonblocking.PIPE_READY2READ:
newchunk = p.recv(1024)
if newchunk == '':
# EOF reached, stop
break
# do something with the data here
# you can send it to second stream
try:
time.sleep(0.01)
except KeyboardInterrup:
break
So when you call dd.wait() it will block, that is why your Ctrl-C not working. You need to deal with this manually.... non-blocking streaming is not a trivial story in Python. Check-out Twisted project, you can find a lot of cool stuff :-)
I read the question/answer/comments on A non-blocking read on a subprocess.PIPE in Python, but I felt a bit lacking.
When I implemented the solution provided, I noticed that this approach works best when the sub-process ends on it own. But if the subprocess is providing a stream of information and we are looking for a single match of output, then that approach doesn't work for my needs (specifically for Windows, if that matters).
Here is my sample:
File ping.py
import time
def main():
for x in range(100):
print x
time.sleep(1)
if __name__ == '__main__':
print("Starting")
time.sleep(2)
main()
File runner.py
import subprocess
import time
import sys
from Queue import Queue, Empty
from threading import Thread
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
# Start process we want to listen to
pPing = subprocess.Popen('ping.py',
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
q = Queue()
t = Thread(target=enqueue_output, args=(pPing.stdout, q))
t.daemon = True
t.start()
# Make sure it's started
print ("get the first line")
try:
line = q.get()
except Empty:
pass
else:
print line.strip()
#look for the 'magic' output
print("empty the queue")
while not q.empty():
line = q.get_nowait().strip()
if (line == "3"):
print("got it!")
sys.exit()
else:
print("not yet")
My expectation is that the runner will make sure the process is started and then wait for the magic output and then stop, which it does. However, the longer the sub-process runs, the longer the runner runs. But since the 'magic' output comes relatively quickly, I have to wait until the subprocess ends before I get anything processed.
What am I missing?
OK, if I understand correctly what you are trying to do, the problem is with ping still being a child process to runner. While you can make read calls non-blocking, the parent process will not actually exit while the child is still running. If you want runner not to wait for the child to finish, read the first line and the first magic output and then exit; you need ping to disassociate itself from the parent process.
Look at this code sample to see how that is done A simple Unix/Linux daemon in Python. Of course you might skip the part where they close and re-open all the I/O streams.
On the same note, I am not sure leaving an open I/O stream connected to the parent will allow the parent to exit, so if that happens to be a problem you might have to figure out another way to exchange data.
I am writing a script in which in the external system command may sometimes require user input. I am not able to handle that properly. I have tried using os.popen4 and subprocess module but could not achieve the desired behavior.
Below mentioned example would show this problem using "cp" command. ("cp" command is used to show this problem, i am calling some different exe which may similarly prompt for user response in some scenarios). In this example there are two files present on disk and when user tries to copy file1 to file2, an conformer message comes up.
proc = subprocess.Popen("cp -i a.txt b.txt", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,)
stdout_val, stderr_val = proc.communicate()
print stdout_val
b.txt?
proc.communicate("y")
Now in this example if i read only stdout/stderr and prints it, later on if i try to write "y" or "n" based on user's input, i got an error that channel is closed.
Can some one please help me on achieving this behavior in python such that i can print stdout first, then should take user input and write stdin later on.
I found another solution (Threading) from Non-blocking read on a subprocess.PIPE in python , not sure whether it would help. But it appears it is printing question from cp command, i have modified code but not sure on how to write in threading code.
import sys
from subprocess import PIPE, Popen
from threading import Thread
try:
from Queue import Queue, Empty
except ImportError:
from queue import Queue, Empty
ON_POSIX = 'posix' in sys.builtin_module_names
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
p = Popen(['cp', '-i', 'a.txt', 'b.txt'],stdin=PIPE, stdout=PIPE, bufsize=1, close_fds=ON_POSIX)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.start()
try:
line = q.get_nowait()
except Empty:
print('no output yet')
else:
pass
Popen.communicate will run the subprocess to completion, so you can't call it more than once. You could use the stdin and stdout attributes directly, although that's risky as you could deadlock if the process uses block buffering or the buffers fill up:
stdout_val = proc.stdout.readline()
print stdout_val
proc.stdin.write('y\n')
As there is a risk of deadlock and because this may not work if the process uses block buffering, you would do well to consider using the pexpect package instead.
I don't have a technical answer to this question. More of just a solution. It has something to do with the way the process waits for the input, and once you communicate with the process, a None input is enough to close the process.
For your cp example, what you can do is check the return code immediately with proc.poll(). If the return value is None, you might assume it is trying to wait for input and can ask your user a question. You can then pass the response to the process via proc.communicate(response). It will then pass the value and proceed with the process.
Maybe someone else can chime in with a more technical reason why an initial communicate with a None value closes the process.
So I'm trying to effectively create a "branch" in a pipe from subprocess. The idea is to load a file with Popen into a pipe's stdout. Then, I can send that stdout to two (or more) stdin's. This works, more or less. The problem comes when the process needs to see an EOF. As far as I can tell, this happens when you use communicate(None) on a subprocess. However, it also seems to depend on the order I spawned the two processes I'm trying to send data to.
#!/usr/bin/env python
from subprocess import *
import shutil
import os
import shlex
inSub=Popen(shlex.split('cat in.txt'),stdout=PIPE)
print inSub.poll()
queue=[]
for i in range(0,3):
temp=Popen(['cat'],stdin=PIPE)
queue=queue+[temp]
while True:
# print 'hi'
buf=os.read(inSub.stdout.fileno(),10000)
if buf == '': break
for proc in queue:
proc.stdin.write(buf)
queue[1].communicate()
print queue[1].poll()
As long as I use queue[1], things hang at the communicate() line. But if I use queue[2], things don't hang. What's going on? It shouldn't depend on the order the subprocesses were created, should it?
(The in.txt file can really be anything, it doesn't matter.)
I can't see any reason why it would be different for any one of the processes. In any case, closing the stdin pipes will cause Python to send the EOF, ending the processes:
...
while True:
# print 'hi'
buf = os.read(inSub.stdout.fileno(),10000)
if buf == '': break
for proc in queue:
proc.stdin.write(buf)
for proc in queue:
proc.stdin.close()
queue[1].communicate()
...