In my current program, I start a server using subprocess.Popen() and continue reading from the stdout using readline(). However, when it gets stuck on readline until a new line appears. This is bad because I need to be able to execute other code while waiting for the server to output. Is there a way I can stop this from occurring?
import subprocess
server = subprocess.Popen("startup command", stdout= subprocess.PIPE, encoding= "utf-8")
while True:
out = server.stdout.readline()
if out != "":
print(out)
print("checked for line")
i would prefer to avoid having to multi-thread because different parts of my code would no longer be thread-safe.
You're going to want to use threading as #tim Roberts said. What you'll need to do is have your read loop post events to the main thread. Whether that is with global flags or a queue. Take a look at the documentation for queue.
https://docs.python.org/3/library/queue.html
Use poll() and communicate() instead:
import subprocess
import time
with subprocess.Popen(['bash', '-c', 'sleep 1 && echo OK'], stdout=subprocess.PIPE) as proc:
while proc.poll() is None:
print('<doing something useful>')
time.sleep(0.3)
out, err = proc.communicate()
print(out)
<doing something useful>
<doing something useful>
<doing something useful>
<doing something useful>
b'OK\n'
Related
I am using Python and it's subprocess library to check output from calls using strace, something in the matter of:
subprocess.check_output(["strace", str(processname)])
However, this only gives me the output after the called subprocess already finished, which is very limiting for my use-case.
I need a kind of "stream" or live-output from the process, so I need to read the output while the process is still running instead of only after it finished.
Is there a convenient way to achieve this using the subprocess library?
I'm thinking of a kind of poll every x seconds, but did not find any hints regarding on how to implement this in the documentation.
Many thanks in advance.
As of Python 3.2 (when context manager support was added to Popen), I have found this to be the most straightforward way to continuously stream output from a subprocess:
import subprocess
def run(args):
with subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as process:
for line in process.stdout:
print(line.decode('utf8'))
Had some problems referencing the selected answer for streaming output from a test runner. The following worked better for me:
import subprocess
from time import sleep
def stream_process(process):
go = process.poll() is None
for line in process.stdout:
print(line)
return go
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while stream_process(process):
sleep(0.1)
According to the documentation:
Popen.poll()
Check if child process has terminated. Set and return returncode attribute.
So based on this you can:
process = subprocess.Popen('your_command_here',stdout=subprocess.PIPE)
while True:
output = process.stdout.readline()
if process.poll() is not None and output == '':
break
if output:
print (output.strip())
retval = process.poll()
This will loop, reading the stdout, and display the output in real time.
This does not work in current versions of python. (At least) for Python 3.8.5 and newer you should replace output == '' with output == b''
I have 2 python (2.7) processes.
The parent process needs to send rows of text to a child process, and the child process should process them as they come in (not wait for the parent process to finish).
I have this code which doesn't work:
# Sender
import subprocess
process = subprocess.Popen(['python', 'child.py'], bufsize=1, stdin=subprocess.PIPE)
try:
while True:
process.stdin.write(msg + '\n') # 'msg' is a changing string
# process.stdin.flush() <-- commented out since it doesn't help
except KeyboardInterrupt:
process.stdin.close()
process.wait()
And the child process:
# Receiver
import sys
for line in sys.stdin:
print line.strip()
The problem is that the child process waits until the parent process exits before it prints out the messages.
What I'm trying to achieve is a child process that processes the messages as soon as they are written to the pipe.
Try adding a process.stdin.flush() after your process.stdin.write(). That way you actually send the string to the other process. What you're suffering from here is your kernel caching everything you write. It does this to be more efficient when actually sending the data to the other process. flush force the kernel to send your data regardless of how full the kernel's buffer is.
I tried your code as such:
# Sender
import subprocess
process = subprocess.Popen(['python', 'child.py'], bufsize=1, stdin=subprocess.PIPE)
msg = "This is my message"
try:
while True:
process.stdin.write(msg + '\n') # 'msg' is a changing string
process.stdin.flush() # This code works well for me regardless of the presence of this line
except KeyboardInterrupt:
process.stdin.close()
process.wait()
# Receiver
import sys
for line in sys.stdin:
print line.strip()
With "works well" here i mean that i get "This is my message" printed as fast as the computer can perform. I'm trying this in Python 2.7.12 for the record.
The story of how buffering works for sys.stdin and sys.stdout has made me cry more than once. A similar problem is discussed in Setting smaller buffer size for sys.stdin?.
As to your specific problem, I suggest you change your child to use sys.stdin.readline() instead of iterating over sys.stdin. The former somewhat "buffers less" :)
while True:
line = sys.stdin.readline()
if not line: break
print (line.strip())
In the parent, you'll likely either need to set bufsize=0 in your call to Popen (making your pipe completely unbuffered), or you'll need the process.stdin.flush() line, as Patrik suggests. I'd opt for the latter.
Tested on Python 2.7.14 on Windows 10 64bit.
In my python program I have two subprocesses interconnected by a pipe and with one connected to stdin and the other connected to stdout. My problem is that when the data flow ends the subprocesses hang until I press ctrl+c. It looks to me like the subprocesses are being held open my the pipe. If I could tell when the data flowing through the pipe I could close it manually.
def write(tag_name):
p_r, p_w = os.pipe()
pv = subprocess.Popen('pv', stdin=None, stdout=p_w)
dd = subprocess.Popen('dd bs=64k of=/dev/nst0'.split(), stdin=p_r, stdout=None)
dd.wait()
Just don't use os.pipe(), you can pass subprocess stdout directly to the other process stdin, like this:
def write(tag_name):
pv = subprocess.Popen('pv', stdin=None, stdout=subprocess.PIPE)
dd = subprocess.Popen('dd bs=64k of=/dev/nst0'.split(), stdin=pv.stdout, stdout=None)
dd.wait()
When first command ends, the pipe is broken (as opposed to os.pipe() which need to be closed manually), so it ends the second command as well and the script can continue/end.
I have tested a simple pipe command and with os.pipe() it blocks at the end as you described, but exited when first process ended with my modifications.
You need a non-blocking solution here.
Take a look at my solution: https://github.com/vesellov/bitdust.devel/blob/master/system/nonblocking.py
And you can call it this way (did not tested the code):
import nonblocking, time
p = nonblocking.Popen('pv'.split(), shell=True, )
p.make_nonblocking()
while 1:
if p.state() == nonblocking.PIPE_CLOSED:
# pipe closed, stop
return break
if p.state() == nonblocking.PIPE_READY2READ:
newchunk = p.recv(1024)
if newchunk == '':
# EOF reached, stop
break
# do something with the data here
# you can send it to second stream
try:
time.sleep(0.01)
except KeyboardInterrup:
break
So when you call dd.wait() it will block, that is why your Ctrl-C not working. You need to deal with this manually.... non-blocking streaming is not a trivial story in Python. Check-out Twisted project, you can find a lot of cool stuff :-)
I'm using the following command to run a shell command (creating a subprocess):
cmd = "ls"
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
Then, I want to get its return code when it's finished. I should use wait() or poll()? It seems to me wait() is equal to a poll() enclosed in a busy wait. Something like:
while process.poll() == None:
time.sleep(0.5)
I read that wait() could generate a deadlock if stdout/stderr buffer is filled. process.poll() used like above also could generate a deadlock? If this is true,
I should use process.comunicate() to solve the problem? Nowadays, I only use
process.comunicate() when I'm interested in the subprocess stdout/stderr.
Thanks in advance.
Yes. subprocess.poll, when used in a loop like that, will cause a deadlock if the stdout is piped into your process and you aren't reading from it. If you don't pipe stdout or you're consistently reading from it, neither poll nor wait will deadlock. subprocess.communicate will solve the deadlock in the cases it would occur. However, if you just want to run a command, check its return code, and get its output, use subprocess.check_output, which wraps all of that.
I am writing a script in which in the external system command may sometimes require user input. I am not able to handle that properly. I have tried using os.popen4 and subprocess module but could not achieve the desired behavior.
Below mentioned example would show this problem using "cp" command. ("cp" command is used to show this problem, i am calling some different exe which may similarly prompt for user response in some scenarios). In this example there are two files present on disk and when user tries to copy file1 to file2, an conformer message comes up.
proc = subprocess.Popen("cp -i a.txt b.txt", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,)
stdout_val, stderr_val = proc.communicate()
print stdout_val
b.txt?
proc.communicate("y")
Now in this example if i read only stdout/stderr and prints it, later on if i try to write "y" or "n" based on user's input, i got an error that channel is closed.
Can some one please help me on achieving this behavior in python such that i can print stdout first, then should take user input and write stdin later on.
I found another solution (Threading) from Non-blocking read on a subprocess.PIPE in python , not sure whether it would help. But it appears it is printing question from cp command, i have modified code but not sure on how to write in threading code.
import sys
from subprocess import PIPE, Popen
from threading import Thread
try:
from Queue import Queue, Empty
except ImportError:
from queue import Queue, Empty
ON_POSIX = 'posix' in sys.builtin_module_names
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
p = Popen(['cp', '-i', 'a.txt', 'b.txt'],stdin=PIPE, stdout=PIPE, bufsize=1, close_fds=ON_POSIX)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.start()
try:
line = q.get_nowait()
except Empty:
print('no output yet')
else:
pass
Popen.communicate will run the subprocess to completion, so you can't call it more than once. You could use the stdin and stdout attributes directly, although that's risky as you could deadlock if the process uses block buffering or the buffers fill up:
stdout_val = proc.stdout.readline()
print stdout_val
proc.stdin.write('y\n')
As there is a risk of deadlock and because this may not work if the process uses block buffering, you would do well to consider using the pexpect package instead.
I don't have a technical answer to this question. More of just a solution. It has something to do with the way the process waits for the input, and once you communicate with the process, a None input is enough to close the process.
For your cp example, what you can do is check the return code immediately with proc.poll(). If the return value is None, you might assume it is trying to wait for input and can ask your user a question. You can then pass the response to the process via proc.communicate(response). It will then pass the value and proceed with the process.
Maybe someone else can chime in with a more technical reason why an initial communicate with a None value closes the process.