I want to write a function that will execute multiple shell commands one at a time and print what the shell returns in real time.
I currently have the following code which does not print the shell (I am using Windows 10 and python 3.6.2):
commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
p.stdin.write((command + "\n").encode("utf-8"))
p.stdin.close()
p.stdout.read()
How can I see what the shell returns in real time ?
Edit : This question is not a duplicate of the two first links in the comments, they do not help printing in real time.
It is possible to handle stdin and stdout in different threads. That way one thread can be handling printing the output from stdout and another one writing new commands on stdin. However, since stdin and stdout are independent streams, I do not think this can guarantee the order between the streams. For the current example it seems to work as intended, though.
import subprocess
import threading
def stdout_printer(p):
for line in p.stdout:
print(line.rstrip())
commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
universal_newlines=True)
t = threading.Thread(target=stdout_printer, args=(p,))
t.start()
for command in commands:
p.stdin.write((command + "\n"))
p.stdin.flush()
p.stdin.close()
t.join()
Also, note that I am writing stdout line by line, which is normally OK, since it tends to be buffered and being generated a line (or more) at a time. I guess it is possible to handle an unbuffered stdout stream (or e.g. stderr) character-by-character instead, if that is preferable.
I believe you need something like this
commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
p.stdin.write((command + "\n").encode("utf-8"))
out, err = p.communicate()
print("{}".format(out))
print("{}".format(err))
Assuming you want control of the output in your python code you might need to do something like this
import subprocess
def run_process(exe):
'Define a function for running commands and capturing stdout line by line'
p = subprocess.Popen(exe.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
return iter(p.stdout.readline, b'')
if __name__ == '__main__':
commands = ["foo", "foofoo"]
for command in commands:
for line in run_process(command):
print(line)
Related
I want to get the output of my subprocess. As it runs indefinitely I want to terminate it when certain conditions are fulfilled.
When I start the subprocess by using check_output, I get the output but no handle to terminate the process:
output = subprocess.check_output(cmd, shell=True)
When I start the subprocess by using Popen or run, I get a handle to terminate the process, but no output.
p = subprocess.Popen(cmd, shell=True, preexec_fn=os.setsid)
How can I get both?
when can you know that you've got the full process output? when the process terminates. So no need to terminate it manually. Just wait for it to end, and using check_output is the way.
Now if you want to wait for a given pattern to appear, then terminate, now that's something else. Just read line by line and if some pattern matches, break the loop and end the process
p = subprocess.Popen(cmd, shell=True, preexec_fn=os.setsid) # add stderr=subprocess.PIPE) to merge output & error
for line in p.stdout:
if b"some string" in line: # output is binary
break
p.kill() # or p.terminate()
You'll need to tell Popen you want to read the standard output, but it can be a little tricky.
p = subprocess.Popen(cmd, shell=True, preexec_fn=os.setsid, stdout=subprocess.PIPE)
while True:
chunk = p.stdout.read(1024) # this will hang forever if there's nothing to read
p.terminate()
try this :
import subprocess
p = subprocess.Popen("ls -a", shell=True, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
print((p.stdout.read()))
if p.stdout.read() or p.stderr.read():
p.terminate()
I found a number of questions which looks like mine, but which did not produce a solution I can use (closest is: subprocess output to stdout and to PIPE)
The problem: I want to start a process using subprocess which takes a long time. After running the command I need to parse the stdout-output and the stderr-output.
Currently I do it as follows:
p = subprocess.Popen( command_list, stdout=subprocess.PIPE,
stderr=subprocess.PIPE )
out, error_msg = p.communicate()
print out + "\n\n" + error_msg
#next comes code in which I check out and error_msg
But the drawback of this method is that the user does not see the output of the process while it is running. Only at the end the output is printed.
Is there a way that the output is printed while the command is running (as if I gave the command without stdout/stderr=subprocess.PIPE) and still have the output via p.communicate in the end?
Note: I'm currently developing on python 2.5 (old software release which uses this python version).
This snippet has helped me once in a similar situation:
process = subprocess.Popen(cmd, bufsize=1, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(process.stdout.readline, ''):
print line,
sys.stdout.flush() # please see comments regarding the necessity of this line
process.wait()
errcode = process.returncode
I am writing a python script to ssh into a linux server and execute a shell script that is already stored on the linux server.
Here is what my code look like so far
command = ['ssh into the remote server',
'cd into the directory of the shell script,
'./running the shell script',
]
process = subprocess.Popen(command,
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
err, out = process.communicate()
if out:
print "standard output of subprocess is : "
print out
if err:
print "standard error of subprocess is : "
print err
print "returncode of subprocess: "
print process.returncode
1st question: I can obtain the output of my shell scripts through stderr, but I only obtain it after the entire shell script has finished executing. So if the shell script takes 10 minutes to finish, I only get to see the output of the shell script after 10 minutes.
I want to have the output of my shell scripts return line by line to me just as if I was executing the script manually in the remote server. Can this be done?
2nd question: as you can see, I have three commands in my command list (which is only a small portion of all my commands,) if I put all my commands in the list, I only obtain the output of ALL my commands through stdout ONLY when all my commands has finished executing. If my 1st question cannot be done, is there a way to at least obtain the output of each command after each one has been executed instead of receiving them all at once only when all the commands has finished being executed.
To see the output immediately, don't redirect it:
from subprocess import Popen, PIPE
p = Popen(['ssh', 'user#hostname'], stdin=PIPE)
p.communicate(b"""cd ..
echo 1st command
echo 2nd command
echo ...
""")
If you want both to capture the "live" output in a variable and to display it in the terminal then the solution depends on whether you need to handle stdin/stdout/stderr concurrently.
If input is small and you want to combine stdout/stderr then you could pass all commands at once and read the merged output line-by-line:
from subprocess import Popen, PIPE, STDOUT
p = Popen(['ssh', 'user#hostname'], stdin=PIPE,
stdout=PIPE, stderr=STDOUT, bufsize=1)
p.stdin.write(b"""cd ..
echo 1st command
echo 2nd command
echo ...
""")
p.stdin.close() # no more input
lines = [] # store output here
for line in iter(p.stdout.readline, b''): # newline=b'\n'
lines.append(line) # capture for later
print line, # display now
p.stdout.close()
p.wait()
If you want to capture "live" stdout/stderr separately, see:
Displaying subprocess output to stdout and redirecting it
Subprocess.Popen: cloning stdout and stderr both to terminal and variables
I'm not entirely sure, but maybe you get instant output if you pass the other two commands as arguments to ssh:
command = 'ssh user#example.com \'cd some/path/on/your/server; ./run-the-script.sh\''
The way I understand it, Python first reads and processes all the input and only then returns output. I'm not too familiar with Python, so I might be wrong on this, but if I'm right, this should help.
Don't call .communicate() -- that waits for the process to finish.
Instead, keep reading data from .stdout pipe.
Simple example:
In [1]: import subprocess
In [2]: p = subprocess.Popen(["find", "/"], stdout=subprocess.PIPE)
In [3]: p.stdout
Out[3]: <open file '<fdopen>', mode 'rb' at 0x7f590446dc00>
In [4]: p.stdout.readline()
Out[4]: '/\n'
In [5]: p.stdout.readline()
Out[5]: '/var\n'
In [6]: p.stdout.readline()
Out[6]: '/var/games\n'
I'd like to run a subprocess in python until the subprocess has outputted a certain number of bytes or lines. After this point, I'd like to terminate it. Is this possible with subprocess?
Here's what I have so far:
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
#kill after reaching n bytes of output, proc.terminate()
out, errors = proc.communicate()
Thanks!
One fairly straightforward way to do this would be to redirect the output to a program that will do the byte/line counting for you, for example:
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
head = subprocess.Popen(['head', '-n', '20'], stdin=proc.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc.stdout.close()
out, errors = head.communicate()
I'm not sure how portable this method is, I was only able to test on Linux however on Windows you should be able to use the more command for similar behavior.
Closing proc.stdout is necessary so that proc receives SIGPIPE when head exits.
I am calling an external program within Python script using subprocess. The external program produces a lot of output. I need to capture the output of this program. The current code is something like this:
process = subprocess.Popen('cmd.exe', shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=None)
process.stdin.write('gams "indus89.gms"\r\n')
while process.poll() != None:
line = process.stdout.readline()
print line
The error I am getting with this code is
The process tried to write to a nonexistent pipe.
If I use the following code:
process = subprocess.Popen('cmd.exe', shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=None)
process.stdin.write('gams "indus89.gms"\r\n')
o, e = process.communicate()
print o
then the output of the program is not captured.
How should I alter my code so that I can capture the output of the third party program while it runs?
Popen is overkill.
Try:
output = subprocess.check_output('gams "indus89.gms"\r\n', shell=True)
Hopefully that will work in your environment.