Develop Reference

How to get real time log from external process Python 2

In your Document folder create a folder temp:
/My Documents/temp
Save these few lines as worker.py Python scripts:
import time
from datetime import datetime
for i in range(10):
print '%s...working on iteration %s' % (datetime.now(), i)
time.sleep(0.2)
print '\nCompleted!\n'
Save the code below as caller.py:
import subprocess
cmd = ['python', 'worker.py']
stdout = subprocess.check_output(cmd)
print stdout
(Please note that both Python scripts were saved in to the same folder).
Now using the OS X Terminal or Windows CMD window change the current directory to the folder you created:
cd /My Documents/temp
Now run:
python caller.py
The process takes 2 seconds to complete. When completed it prints out the entire progress log all at once:
2018-01-20 07:52:14.399679...working on iteration 0
...
2018-01-20 07:52:16.216237...working on iteration 9
Completed!
Instead of getting the log printed (all at once after the process has been already completed), I would like the have a real-time progress update. I would like to get every printed line from the process at the same moment it occured.
So, when I run python worker.pycommand it will give me line by line update happening in a real time. How to achieve it?

To get a real-time feed from the subprocess you can use this code in the caller.py
import time
import subprocess
# Start worker process
p = subprocess.Popen(['python', '-u', 'worker.py'], stdout=subprocess.PIPE)
# Loop forever
while True:
# Get new line value
l = p.stdout.readline()
# Stop looping if the child process has terminated
if p.poll() is not None:
break
# Print the line
print l
Note the -u in the subprocess.Popen, you need unbuffered stdout.
https://docs.python.org/3/using/cmdline.html#cmdoption-u
With readline() you are reading a single line per time from the subprocess output. Be aware when the subprocess prints '\nCompleted!\n' you will read it in three loops.
https://docs.python.org/3/tutorial/inputoutput.html#methods-of-file-objects
In the example, the loop will run until the subprocess will terminate.
https://docs.python.org/3/library/subprocess.html#subprocess.Popen.poll

Python subprocess output blocking on application prompt

I am running jirashell in a python script using the subprocess library. I am currently having issues having the outputs print in real time. When I run jirashell it outputs information than prompts the user (y/n). The subprocess won't print out information prior to the prompt until I enter 'y' or 'n'.
The code I am using is
_consumer_key = "justin-git"
_cmd = "jirashell -s {0} -od -k {1} -ck {2} -pt".format(JIRA_SERVER,
_rsa_private_key_path, _consumer_key)
p = subprocess.Popen(_cmd.split(" "), stdout=subprocess.PIPE,
stderr=subprocess.PIPE, bufsize=0)
out, err = p.communicate() # Blocks here
print out
print err
The output is like so:
n # I enter a "n" before program will print output.
Output:
Request tokens received.
Request token: asqvavefafegagadggsgewgqqegqgqge
Request token secret: asdbresbdfbrebsaerbbsbdabweabfbb
Please visit this URL to authorize the OAuth request:
http://localhost:8000/plugins/servlet/oauth/authorize?oauth_token=zzzzzzzzzzzzzzzzzzzzzzzzzzzzz
Have you authorized this program to connect on your behalf to http://localhost:8000? (y/n)
Error:
Abandoning OAuth dance. Your partner faceplants. The audience boos. You feel shame.
Does anyone know how I can have it print the output prior to the prompt than wait for an input of y/n? Note I also need to be able to store the output produced by the command so "os.system()" won't work...
EDIT:
It looks like inside jirashell there is a part of the code that is waiting for an input and this is causing the block. Until something is passed into this input nothing is outputted... Still looking into how I can get around this. I'm in the process of trying to move the portion of code I need into my application. This solution doesn't seem elegant but I can't see any other way right now.
approved = input(
'Have you authorized this program to connect on your behalf to {}? (y/n)'.format(server))

Method which prints and caches the standard output:
You can use a thread which reads the standard output of your subprocess, while the main thread is blocked until the subprocess is done. The following example will run the program other.py, which looks like
#!/usr/bin/env python3
print("Hello")
x = input("Type 'yes': ")
Example:
import threading
import subprocess
import sys
class LivePrinter(threading.Thread):
"""
Thread which reads byte-by-byte from the input stream and writes it to the
standard out.
"""
def __init__(self, stream):
self.stream = stream
self.log = bytearray()
super().__init__()
def run(self):
while True:
# read one byte from the stream
buf = self.stream.read(1)
# break if end of file reached
if len(buf) == 0:
break
# save output to internal log
self.log.extend(buf)
# write and flush to main standard output
sys.stdout.buffer.write(buf)
sys.stdout.flush()
# create subprocess
p = subprocess.Popen('./other.py', stdout=subprocess.PIPE)
# Create reader and start the thread
r = LivePrinter(p.stdout)
r.start()
# Wait until subprocess is done
p.wait()
# Print summary
print(" -- The process is done now -- ")
print("The standard output was:")
print(r.log.decode("utf-8"))
The class LivePrinter reads every byte from the subprocess and writes it to the standard output. (I have to admit, this is not the most efficient approach, but a larger buffer size blocks, the LiveReader until the buffer is full, even though the subprocess is awaiting the answer to a prompt.) Since the bytes are written to sys.stdout.buffer, there shouldn't be a problem with multi-byte utf-8 characters.
The LiveReader class also stores the complete output of the subprocess in the variable log for later use.
As this answer summarizes, it is save to start a thread after forking with subprocess.
Original answer which has problems, when the prompt line doesn't end a line:
The output is delayed because communicate() blocks the execution of your script until the sub-process is done (https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate).
You can read and print the standard output of the subprocess, while it is executed using stdout.readline. There are some issues about buffering, which require this rather complicated iter(process.stdout.readline, b'') construct. The following example uses gpg2 --gen-key because this command starts an interactive tool.
import subprocess
process = subprocess.Popen(["gpg2", "--gen-key"], stdout=subprocess.PIPE)
for stdout_line in iter(process.stdout.readline, b''):
print(stdout_line.rstrip())
Alternative answer which uses shell and does not cache the output:
As Sam pointed out, there is a problem with the above solution, when the prompt line does not end the line (which prompts they usually don't). An alternative solution is to use the shell argument to interact with the sub-process.
import subprocess
subprocess.call("gpg2 --gen-key", shell=True)

How read in realtime an output process with python?

I need to realise a python script who read the ouput of an other process in real time line by line. Obviously I've ever try to use "subprocess" to get the output process with stdout. But i can't get that output in real time, indeed every times python script return the output in several chunk few time after the process launch. For exemple i use this python script :
import subprocess as sub
import sys
proc = sub.Popen(["python", "bin/read_test.py"],
stdout = sub.PIPE,
bufsize = 1)
while True:
if proc.poll() is None:
line = proc.stdout.read()
line = line.decode("utf-8")
print(line, flush = True)
sys.stdout.flush()
else:
proc.stdout.close()
break
read_test.py script to be read :
from time import sleep
for i in range(5):
print(i)
sleep(1)
I've try a lot of methods with "readline()" with for loops but the issue still the same.
Moreover I don't want to use "communicate" because it's a blocking method.
Thanks for your help,

Problem is that you're trying to read stdout fully.
Since python sees that process is still running, it waits until process ends so process output is complete.
To do what you want you probably want to read the output line by line using line = proc.stdout.readline()
You probably have to change your loop to read line, and stop when process ends
proc = sub.Popen(["python", "bin/read_test.py"],
stdout = sub.PIPE,
bufsize = 1)
while True:
line = proc.stdout.readline()
if line:
line = line.decode("utf-8")
print(line)
sys.stdout.flush()
if proc.poll() is not None: # process ends
break
proc.wait()
Also that's not enough: you have to flush the output on the read_test.py side to make sure that the emitter actually sends the lines (when output is redirected, it's not sure). Example program:
import time,sys
for i in range(5):
print(i)
sys.stdout.flush() # not flushing: you get all the lines at the end of the process
time.sleep(1)
I've connected both programs, and got 1 sequenced output (1,2,3,4,5) each second

Parsing pexpect output

I'm trying to parse in real time the output of a program block-buffered, which means that output is not available until the process ends. What I need is just to parse line by line, filter and manage data from the output, as it could run for hours.
I've tried to capture the output with subprocess.Popen(), but yes, as you may guess, Popen can't manage this kind of behavior, it keeps buffering until end of process.
from subprocess import Popen, PIPE
p = Popen("my noisy stuff ", shell=True, stdout=PIPE, stderr=PIPE)
for line in p.stdout.readlines():
#parsing text and getting data
So I found pexpect, which prints the output in real time, as it treats the stdout as a file, or I could even do a dirty trick printing out a file and parsing it outside the function. But ok, it is too dirty, even for me ;)
import pexpect
import sys
pexpect.run("my noisy stuff", logfile=sys.stdout)
But I guess it should a better pythonic way to do this, just manage the stdout like subprocess. Popen does. How can I do this?
EDIT:
Running J.F. proposal:
This is a deliberately wrong audit, it takes about 25 secs. to stop.
from subprocess import Popen, PIPE
command = "bully mon0 -e ESSID -c 8 -b aa:bb:cc:dd:ee:00 -v 2"
p = Popen(command, shell=True, stdout=PIPE, stderr=PIPE)
for line in iter(p.stdout.readline, b''):
print "inside loop"
print line
print "outside loop"
p.stdout.close()
p.wait()
#$ sudo python SCRIPT.py
### <= 25 secs later......
# inside loop
#[!] Bully v1.0-21 - WPS vulnerability assessment utility
#inside loop
#[!] Using 'ee:cc:bb:aa:bb:ee' for the source MAC address
#inside loop
#[X] Unable to get a beacon from the AP, possible causes are
#inside loop
#[.] an invalid --bssid or -essid was provided,
#inside loop
#[.] the access point isn't on channel '8',
#inside loop
#[.] you aren't close enough to the access point.
#outside loop
Using this method instead:
EDIT: Due to large delays and timeouts in the output, I had to fix the child, and added some hacks, so final code looks like this
import pexpect
child = pexpect.spawn(command)
child.maxsize = 1 #Turns off buffering
child.timeout = 50 # default is 30, insufficient for me. Crashes were due to this param.
for line in child:
print line,
child.close()
Gives back the same output, but it prints lines in real time. So... SOLVED Thanks #J.F. Sebastian

.readlines() reads all lines. No wonder you don't see any output until the subprocess ends. You could use .readline() instead to read line by line as soon as the subprocess flushes its stdout buffer:
from subprocess import Popen, PIPE
p = Popen("my noisy stuff", stdout=PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
# process line
..
p.stdout.close()
p.wait()
If you are already have pexpect then you could use it to workaround the block-buffering issue:
import pexpect
child = pexpect.spawn("my noisy stuff", timeout=None)
for line in child:
# process line
..
child.close()
See also stdbuf, pty -based solutions from the question I've linked in the comments.

Getting stdout to display called script containing input

I was looking to implement a python script that called another script and captured its stdout. The called script will contain some input and output messages eg
print ("Line 1 of Text")
variable = raw_input("Input 1 :")
print "Line 2 of Text Input: ", vairable
The section of the code I'm running is
import subprocess
cmd='testfile.py'
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
so, se = p.communicate()
print(so)
The problem that is occurring is that the stdout is not printing until after the script has been executed. This leaves a blank prompt waiting for the user input. Is there a way to get stdout to print while the called script is still running?
Thanks,

There are two problems here.
Firstly, python is buffering output to stdout and you need to prevent this. You could insert a call to sys.stdout.flush() in testfile.py as Ilia Frenkel has suggested, or you could use python -u to execute testfile.py with unbuffered I/O. (See the other stack overflow question that Ilia linked to.)
You need a way of asynchronously reading data from the sub-process and then, when it is ready for input, printing the data you've read so that the prompt for the user appears. For this, it would be very helpful to have an asynchronous version of the subprocess module.
I downloaded the asynchronous subprocess and re-wrote your script to use it, along with using python -u to get unbuffered I/O:
import async_subprocess as subprocess
cmd = ['python', '-u', 'testfile.py']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
so = p.asyncread()
print so,
(so, se) = p.communicate()
print so
When I run this script using python -u I get the following results:
$ python -u script.py
Line 1 of Text
Input 1:
and the script pauses, waiting for input. This is the desired result.
If I then type something (e.g. "Hullo") I get the following:
$ python -u script.py
Line 1 of Text
Input 1:Hullo
Line 2 of Text Input: Hullo

You don't need to capture it's stdout really, just have the child program print out its stuff and quit, instead of feeding the output into your parent program and printing it there. If you need variable output, just use a function instead.
But anyways, that's not what you asked.
I actually got this from another stackoverflow question:
import subprocess, sys
cmd='testfile.py'
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
while True:
out = p.stdout.read(20)
if out == '' and p.poll() != None:
break
if out != '':
sys.stdout.write(out)
sys.stdout.flush()
First, it opens up your process: then it continually reads the output from p and prints it onto the screen using sys.stdout.write. The part that makes this all work is sys.stdout.flush(), which will continually "flush out" the output of the program.