Develop Reference

How to get the output of a Python subprocess and terminate it afterwards?

I want to get the output of my subprocess. As it runs indefinitely I want to terminate it when certain conditions are fulfilled.
When I start the subprocess by using check_output, I get the output but no handle to terminate the process:
output = subprocess.check_output(cmd, shell=True)
When I start the subprocess by using Popen or run, I get a handle to terminate the process, but no output.
p = subprocess.Popen(cmd, shell=True, preexec_fn=os.setsid)
How can I get both?

when can you know that you've got the full process output? when the process terminates. So no need to terminate it manually. Just wait for it to end, and using check_output is the way.
Now if you want to wait for a given pattern to appear, then terminate, now that's something else. Just read line by line and if some pattern matches, break the loop and end the process
p = subprocess.Popen(cmd, shell=True, preexec_fn=os.setsid) # add stderr=subprocess.PIPE) to merge output & error
for line in p.stdout:
if b"some string" in line: # output is binary
break
p.kill() # or p.terminate()

You'll need to tell Popen you want to read the standard output, but it can be a little tricky.
p = subprocess.Popen(cmd, shell=True, preexec_fn=os.setsid, stdout=subprocess.PIPE)
while True:
chunk = p.stdout.read(1024) # this will hang forever if there's nothing to read
p.terminate()

try this :
import subprocess
p = subprocess.Popen("ls -a", shell=True, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
print((p.stdout.read()))
if p.stdout.read() or p.stderr.read():
p.terminate()

Running shell commands from Python and printing the output in real time

I want to write a function that will execute multiple shell commands one at a time and print what the shell returns in real time.
I currently have the following code which does not print the shell (I am using Windows 10 and python 3.6.2):
commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
p.stdin.write((command + "\n").encode("utf-8"))
p.stdin.close()
p.stdout.read()
How can I see what the shell returns in real time ?
Edit : This question is not a duplicate of the two first links in the comments, they do not help printing in real time.

It is possible to handle stdin and stdout in different threads. That way one thread can be handling printing the output from stdout and another one writing new commands on stdin. However, since stdin and stdout are independent streams, I do not think this can guarantee the order between the streams. For the current example it seems to work as intended, though.
import subprocess
import threading
def stdout_printer(p):
for line in p.stdout:
print(line.rstrip())
commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
universal_newlines=True)
t = threading.Thread(target=stdout_printer, args=(p,))
t.start()
for command in commands:
p.stdin.write((command + "\n"))
p.stdin.flush()
p.stdin.close()
t.join()
Also, note that I am writing stdout line by line, which is normally OK, since it tends to be buffered and being generated a line (or more) at a time. I guess it is possible to handle an unbuffered stdout stream (or e.g. stderr) character-by-character instead, if that is preferable.

I believe you need something like this
commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
p.stdin.write((command + "\n").encode("utf-8"))
out, err = p.communicate()
print("{}".format(out))
print("{}".format(err))

Assuming you want control of the output in your python code you might need to do something like this
import subprocess
def run_process(exe):
'Define a function for running commands and capturing stdout line by line'
p = subprocess.Popen(exe.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
if __name__ == '__main__':
commands = ["foo", "foofoo"]
for command in commands:
for line in run_process(command):
print(line)

Subprocess.Popen stdin in new console

I want to execute a python subprocess in a new console. Once started, I want the user to be able to answer questions asked by this new process on stdin.
I tried the following code:
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd, creationflags=subprocess.CREATE_NEW_CONSOLE)
(o, e) = p.communicate()
As soon as the subprocess asks for input on stdin the following error message is displayed:
EOFError: EOF when reading a line
Is it the good way to achieve this ?

As i'm not really interested in the stdout/stderr redirection, i tried this way:
subprocess.Popen(cmd, cwd=cwd, creationflags=subprocess.CREATE_NEW_CONSOLE)
It works fine now. I guess that it's not compatible to redirect standard input/outputs and to create a new console.

Output of subprocess both to PIPE and directly to stdout

I found a number of questions which looks like mine, but which did not produce a solution I can use (closest is: subprocess output to stdout and to PIPE)
The problem: I want to start a process using subprocess which takes a long time. After running the command I need to parse the stdout-output and the stderr-output.
Currently I do it as follows:
p = subprocess.Popen( command_list, stdout=subprocess.PIPE,
stderr=subprocess.PIPE )
out, error_msg = p.communicate()
print out + "\n\n" + error_msg
#next comes code in which I check out and error_msg
But the drawback of this method is that the user does not see the output of the process while it is running. Only at the end the output is printed.
Is there a way that the output is printed while the command is running (as if I gave the command without stdout/stderr=subprocess.PIPE) and still have the output via p.communicate in the end?
Note: I'm currently developing on python 2.5 (old software release which uses this python version).

This snippet has helped me once in a similar situation:
process = subprocess.Popen(cmd, bufsize=1, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(process.stdout.readline, ''):
print line,
sys.stdout.flush() # please see comments regarding the necessity of this line
process.wait()
errcode = process.returncode

Not able to read subprocess output

I am executing the python command,
proc = subprocess.Popen(cmd,
shell=False,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
after executing command i want to read the stderr and stdout
res = proc.stderr.read()
in res i am expecting any error or ' '
but the reading the stderr is taking infinite time is get hang not reading the values what ever the result it.it goes in infinite time.
Some time back same code is working fine but not idea why its not reading stderr now.
Any Hint, thanks.

Instead of explicitly calling stderr.read(), just do a communicate on the proc.
output, error = proc.communicate()
That way you would get the output and error by communicating with the process.