My python script uses subprocess to call a linux utility that is very noisy. I want to store all of the output to a log file and show some of it to the user. I thought the following would work, but the output doesn't show up in my application until the utility has produced a significant amount of output.
#fake_utility.py, just generates lots of output over time
import time
i = 0
while True:
print hex(i)*512
i += 1
time.sleep(0.5)
#filters output
import subprocess
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
for line in proc.stdout:
#the real code does filtering here
print "test:", line.rstrip()
The behavior I really want is for the filter script to print each line as it is received from the subprocess. Sorta like what tee does but with python code.
What am I missing? Is this even possible?
Update:
If a sys.stdout.flush() is added to fake_utility.py, the code has the desired behavior in python 3.1. I'm using python 2.6. You would think that using proc.stdout.xreadlines() would work the same as py3k, but it doesn't.
Update 2:
Here is the minimal working code.
#fake_utility.py, just generates lots of output over time
import sys, time
for i in range(10):
print i
sys.stdout.flush()
time.sleep(0.5)
#display out put line by line
import subprocess
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
#works in python 3.0+
#for line in proc.stdout:
for line in iter(proc.stdout.readline,''):
print line.rstrip()
I think the problem is with the statement for line in proc.stdout, which reads the entire input before iterating over it. The solution is to use readline() instead:
#filters output
import subprocess
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if not line:
break
#the real code does filtering here
print "test:", line.rstrip()
Of course you still have to deal with the subprocess' buffering.
Note: according to the documentation the solution with an iterator should be equivalent to using readline(), except for the read-ahead buffer, but (or exactly because of this) the proposed change did produce different results for me (Python 2.5 on Windows XP).
Bit late to the party, but was surprised not to see what I think is the simplest solution here:
import io
import subprocess
proc = subprocess.Popen(["prog", "arg"], stdout=subprocess.PIPE)
for line in io.TextIOWrapper(proc.stdout, encoding="utf-8"): # or another encoding
# do something with line
(This requires Python 3.)
Indeed, if you sorted out the iterator then buffering could now be your problem. You could tell the python in the sub-process not to buffer its output.
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
becomes
proc = subprocess.Popen(['python','-u', 'fake_utility.py'],stdout=subprocess.PIPE)
I have needed this when calling python from within python.
You want to pass these extra parameters to subprocess.Popen:
bufsize=1, universal_newlines=True
Then you can iterate as in your example. (Tested with Python 3.5)
A function that allows iterating over both stdout and stderr concurrently, in realtime, line by line
In case you need to get the output stream for both stdout and stderr at the same time, you can use the following function.
The function uses Queues to merge both Popen pipes into a single iterator.
Here we create the function read_popen_pipes():
from queue import Queue, Empty
from concurrent.futures import ThreadPoolExecutor
def enqueue_output(file, queue):
for line in iter(file.readline, ''):
queue.put(line)
file.close()
def read_popen_pipes(p):
with ThreadPoolExecutor(2) as pool:
q_stdout, q_stderr = Queue(), Queue()
pool.submit(enqueue_output, p.stdout, q_stdout)
pool.submit(enqueue_output, p.stderr, q_stderr)
while True:
if p.poll() is not None and q_stdout.empty() and q_stderr.empty():
break
out_line = err_line = ''
try:
out_line = q_stdout.get_nowait()
except Empty:
pass
try:
err_line = q_stderr.get_nowait()
except Empty:
pass
yield (out_line, err_line)
read_popen_pipes() in use:
import subprocess as sp
with sp.Popen(my_cmd, stdout=sp.PIPE, stderr=sp.PIPE, text=True) as p:
for out_line, err_line in read_popen_pipes(p):
# Do stuff with each line, e.g.:
print(out_line, end='')
print(err_line, end='')
return p.poll() # return status-code
You can also read lines w/o loop. Works in python3.6.
import os
import subprocess
process = subprocess.Popen(command, stdout=subprocess.PIPE)
list_of_byte_strings = process.stdout.readlines()
Pythont 3.5 added the methods run() and call() to the subprocess module, both returning a CompletedProcess object. With this you are fine using proc.stdout.splitlines():
proc = subprocess.run( comman, shell=True, capture_output=True, text=True, check=True )
for line in proc.stdout.splitlines():
print "stdout:", line
See also How to Execute Shell Commands in Python Using the Subprocess Run Method
I tried this with python3 and it worked, source
When you use popen to spawn the new thread, you tell the operating system to PIPE the stdout of the child processes so the parent process can read it and here, stderr is copied to the stderr of the parent process.
in output_reader we read each line of stdout of the child process by wrapping it in an iterator that populates line by line output from the child process whenever a new line is ready.
def output_reader(proc):
for line in iter(proc.stdout.readline, b''):
print('got line: {0}'.format(line.decode('utf-8')), end='')
def main():
proc = subprocess.Popen(['python', 'fake_utility.py'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
t = threading.Thread(target=output_reader, args=(proc,))
t.start()
try:
time.sleep(0.2)
import time
i = 0
while True:
print (hex(i)*512)
i += 1
time.sleep(0.5)
finally:
proc.terminate()
try:
proc.wait(timeout=0.2)
print('== subprocess exited with rc =', proc.returncode)
except subprocess.TimeoutExpired:
print('subprocess did not terminate in time')
t.join()
The following modification of Rômulo's answer works for me on Python 2 and 3 (2.7.12 and 3.6.1):
import os
import subprocess
process = subprocess.Popen(command, stdout=subprocess.PIPE)
while True:
line = process.stdout.readline()
if line != '':
os.write(1, line)
else:
break
I was having a problem with the arg list of Popen to update servers, the following code resolves this a bit.
import getpass
from subprocess import Popen, PIPE
username = 'user1'
ip = '127.0.0.1'
print ('What is the password?')
password = getpass.getpass()
cmd1 = f"""sshpass -p {password} ssh {username}#{ip}"""
cmd2 = f"""echo {password} | sudo -S apt update"""
cmd3 = " && "
cmd4 = f"""echo {password} | sudo -S apt upgrade -y"""
cmd5 = " && "
cmd6 = "exit"
commands = [cmd1, cmd2, cmd3, cmd4, cmd5, cmd6]
command = " ".join(commands)
cmd = command.split()
with Popen(cmd, stdout=PIPE, bufsize=1, universal_newlines=True) as p:
for line in p.stdout:
print(line, end='')
And to run the update on a local computer, the following code example does this.
import getpass
from subprocess import Popen, PIPE
print ('What is the password?')
password = getpass.getpass()
cmd1_local = f"""apt update"""
cmd2_local = f"""apt upgrade -y"""
commands = [cmd1_local, cmd2_local]
with Popen(['echo', password], stdout=PIPE) as auth:
for cmd in commands:
cmd = cmd.split()
with Popen(['sudo','-S'] + cmd, stdin=auth.stdout, stdout=PIPE, bufsize=1, universal_newlines=True) as p:
for line in p.stdout:
print(line, end='')
I am trying to write some simple tests using the the subprocess module in Python. The program being tested is simply:
def main():
x = int(input("Integer? "))
print('Output is', x // 12)
main()
To test it I'm calling this function:
def test_output():
ret = subprocess.Popen(args=['python3', FILENAME],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
output, err = ret.communicate(b"216\n")
print(f"Output: {output}")
However, the output is capturing both the prompt from the target program's input call as well as the stdout that follows:
Output: b'Integer? Output is 18\n'
How can I get only the Output is 18\n' portion? I don't care about the prompt from input()
Worse Approach: Waiting For A Prompt
Here, we read one byte in blocking mode (to delay until the prompt is printed), and then consuming everything else that's ready and waiting to be read in non-blocking mode, before switching back to blocking mode.
#!/usr/bin/env python3
from select import select
import fcntl
import subprocess
import threading
import os
import sys
def test_output(timeout=0.1):
ret = subprocess.Popen(args=['python3', 'testsubject.py'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
bufsize=0)
# you probably want to replace this with a function that _stores_ stderr
stderr_reader = threading.Thread(target = ret.stderr.read)
stderr_reader.start()
# wait until the prompt has been at least partially written
prompt_start = ret.stdout.read(1)
# go into nonblocking mode
orig_flags = fcntl.fcntl(ret.stdout, fcntl.F_GETFL)
fcntl.fcntl(ret.stdout, fcntl.F_SETFL, orig_flags | os.O_NONBLOCK)
# read further output until there's nothing left
prompt_rest = ret.stdout.read()
print(f"Skipping prompt: {prompt_start.decode('utf-8')}{prompt_rest.decode('utf-8')}", file=sys.stderr)
# exit nonblocking mode
fcntl.fcntl(ret.stdout, fcntl.F_SETFL, orig_flags)
# write to stdin
ret.stdin.write(b'216\n')
ret.stdin.close()
output = ret.stdout.read()
print(f"Output: {output.decode('utf-8')}")
test_output()
Better Approach: Fixing The Program Under Test
The component at fault here is the program being called as a subprocess, not the program calling the subprocess. There are two major ways to fix it:
Don't print the prompt when not being run interactively by a user.
def main():
x = int(input("Integer? " if os.isatty(0) else None))
print('Output is', x // 12)
main()
This works because os.isatty(0) is false when a program is run with stdin=subprocess.PIPE.
Print the prompt to stderr, not stdout, so it's segregated with "diagnostic logs" ("ready for input now" being an example of a diagnostic status), not with output.
def main():
sys.stderr.write("Integer? ")
x = int(input())
print('Output is', x // 12)
main()
I am running a script that iterates through a text file. On each line on the text file there is an ip adress. The script grabs the banner, then writes the ip + banner on another file.
The problem is, it just stops around 500 lines, more or less, with no error.
Another weird thing is if i run it with python3 it does what i said above. If i run it with python it iterates through those 500 lines, then starts at the beggining. I noticed this when i saw repetitions in my output file. Anyway here is the code, maybe you guys can tell me what im doing wrong:
import os
import subprocess
import concurrent.futures
#import time, random
import threading
import multiprocessing
with open("ipuri666.txt") as f:
def multiprocessing_func():
try:
line2 = line.rstrip('\r\n')
a = subprocess.Popen(["curl", "-I", line2, "--connect-timeout", "1", "--max-time", "1"], stdout=subprocess.PIPE)
b = subprocess.Popen(["grep", "Server"], stdin=a.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
#a.stdout.close()
out, err = b.communicate()
g = open("IP_BANNER2","a")
print( "out: {0}".format(out))
g.write(line2 + " " + "out: {0}\n".format(out))
print("err: {0}".format(err))
except IOError:
print("Connection timed out")
if __name__ == '__main__':
#starttime = time.time()
processes = []
for line in f:
p = multiprocessing.Process(target=multiprocessing_func, args=())
processes.append(p)
p.start()
for process in processes:
process.join()
If your use case allows I would recommend just rewriting this as a shell script, there is no need to use Python. (This would likely solve your issue indirectly.)
#!/usr/bin/env bash
readarray -t ips < ipuri666.txt
for ip in ${ips[#]}; do
output=$(curl -I "$ip" --connect-timeout 1 --max-time 1 | grep "Server")
echo "$ip $output" >> fisier.txt
done
The script is slightly simpler than what you are trying to do, for instance I do not capture the error. This should be pretty close to what you are trying to accomplish. I will update again if needed.
I have two Python scripts where script A should constantly read a line from script B:
Script A (controller):
import sys, subprocess
r = subprocess.Popen(["python", "script_b.py"],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
while True:
data = r.stdout.readline().strip()
if not data:
break
print("Data: \"{}\"".format(data))
Script B (receiver):
import sys
for i in xrange(3):
sys.stdout.write("%d\n" % i)
sys.stdout.flush()
Strangely, script B writes a newline as well as flushing after every writes, but script A never prints out anything. If I switch script A to use sys.stdin instead, it would work in the terminal. Why is that?
I have written this demo script to ask my question on subprocess.call().
I am trying to run python test scripts one after another. However in this scenario when one of the test aborts due to invalid test condition, I want to terminate subprocess.call(). and move on to next test script. I have read through other queries but couldn't find sufficient explanation. Appreciate any suggestion or help in this matter. Below are demo files.
File1: listscripts.py -> this file list all tests from a folder and runs them using subprocess.call()
import os
from subprocess import *
import sys,os,time
Lib_Path = "C:\\Demo\\question"
sys.path.append(Lib_Path)
import globalsvar # has a global variable
list = os.listdir("C:\\Demo\\question\\scripts") # this has 3 example basic script
for testscripts in list:
aborttest = globalsvar.aborttestcall # when it encounters invalid condition from testscript1thru3 call() should terminate and go to next test
while not aborttest:
Desttestresultpath = os.path.join('C:/Demo/question/scripts',pyscripts)
call(["python",Desttestresultpath]) #calls individual scripts
aborttest = True
exit(1)
File2: globalsvar.py ( aborttestcall = False )
testscript1.py, testscript2.py and testscript3.py -> has some print statments placed in C:/Demo/question/scripts
testscript1.py and testscript3.py:
import sys,os,time
Lib_Path = "C:\\Demo\\question"
sys.path.append(Lib_Path)
import globalsvar
print "Start of Test\n"
print "testing stdout prints --1"
time.sleep(1)
globalsvar.aborttestcall = False
print "testing stdout prints --2"
time.sleep(1)
print "testing stdout prints --3"
time.sleep(1)
testscript2.py:
import sys,os,time
Lib_Path = "C:\\Demo\\question"
sys.path.append(Lib_Path)
import globalsvar
print "Start of Test\n"
print "testing stdout prints --1"
time.sleep(1)
globalsvar.aborttestcall = True
print "testing stdout prints --2"
time.sleep(1)
print "testing stdout prints --3"
time.sleep(1)
You can run your scripts (among different possibilities) like this:
import subprocess
import os
for file_item in os.listdir('scripts'):
script = os.sep.join(['scripts', file_item])
return_value = subprocess.call(['python', script])
print "OUTPUT: " + str(return_value)
while your inner scripts can exit their process with an exit code that you can evaluate on your calling process.
import time
import sys
print "Doing subprocess testing stuff"
time.sleep(2)
print "Doing more subprocess testing stuff"
# simulate error
time.sleep(2)
print "Error, jump out of the process"
sys.exit(1)
# finish
time.sleep(2)
print "done"
# this can be left out since it is called implicitely
# on successful step out of a process
# sys.exit(0)