I'm trying to get what is currently in the console window in a python script
For example;
print('Bar')
print('Foo')
print('\n')
print([variable I'm asking for])
expected outcome:
Bar
Foo
Bar
Foo
but also:
os.system('dir')
print([the variable I'm asking for])
with the expected outcome:
[directories]
[directories]
I've tried the following (from here):
import subprocess
print('test1')
print('test2\n')
proc = subprocess.Popen('ls', stdout=subprocess.PIPE, shell=True)
cmdstr = proc.stdout.read()
print(cmdstr)
and expected something like this:
test1
test2
test1
test2
but i got this:
test1
test2
'ls' is not recognized as an internal or external command,
operable program or batch file.
b''
EDIT:
I've found this somewhere:
import sys
from io import StringIO
sys.stdout = temporarystd = StringIO() # redirect stdout
# stuff here
sys.stdout = sys.__stdout__ # return stdout to its old state
print(temporarystd.getvalue())
This worked fine for print() but i try to use os commands and those still go to the terminal.
Anyone knows why?
(Also, as mentioned by #dump-eldor; 'ls' in proc = subprocess.Popen('ls', stdout=subprocess.PIPE, shell=True) should've been 'dir')
it looks like you run the program from windows os?if so try "dir" instead of "ls"
Related
I know that it is easy to run an external command from Python and display the output on the screen. For example: (It will display the output only on the screen)
import subprocess
subprocess.Popen(['dir'])
I also know that we can run an external command from Python and get the output. For example: (It will store the output as a string in variable out and will not display the output on the screen anymore)
import subprocess
result = subprocess.Popen(['dir'], stdout=subprocess.PIPE)
out = result.stdout.read()
My question is: Is there a way to display the output on the screen and store and output in a variable simultaneously? In other words, when I run the code, the external command output will display on the screen. At the same time, a variable in the code can get the output displayed on the screen.
Update: I found out if I include stdout=subprocess.PIPE in subprocess.Popen, it will display nothing on the screen (windows command prompt).
Related questions:
live output from subprocess command
Output of subprocess both to PIPE and directly to stdout
If you need real-time output from Popen , you could use something like this (Windows platform according to your last comment):
Byte stream:
import sys, subprocess
out=b''
result = subprocess.Popen(['cmd', '/c', 'dir /B 1*.txt'], stdout=subprocess.PIPE,
universal_newlines=False)
for s_line in result.stdout:
#Parse it the way you want
out += s_line
print( s_line.decode(sys.stdout.encoding).replace('\r\n',''))
# debugging output
print(out.decode(sys.stdout.encoding))
Text stream:
import subprocess
out=''
result = subprocess.Popen(['cmd', '/c', 'dir /B 1*.txt'], stdout=subprocess.PIPE,
universal_newlines=True)
for s_line in result.stdout:
#Parse it the way you want
out += s_line
print( s_line.rstrip())
# debugging output
print(out)
The latter code snippet output:
>>> import subprocess
>>> out=''
>>> result = subprocess.Popen(['cmd', '/c', 'dir /B 1*.txt'], stdout=subprocess.PIPE,
... universal_newlines=True)
>>> for s_line in result.stdout:
... #Parse it the way you want
... out += s_line
... print( s_line.rstrip())
...
1.325.txt
1049363a.txt
1049363b.txt
1051416.txt
1207235log.txt
>>> print(out)
1.325.txt
1049363a.txt
1049363b.txt
1051416.txt
1207235log.txt
>>>
Use | tee (On windows platform, use Powershell):
import subprocess
# Run command in powershell and redirect it by | tee to a file named out.txt
result = subprocess.Popen(['powershell', command, '|', 'tee', 'out.txt'])
# Get standard output
f = open('out.txt')
out = f.read()
f.close()
By this way, the screen will display the output on PowerShell terminal and the output will also be stored in out.
I have a small Python program that executes a terminal command from a pip package called commandwrapper (which is a wrapper for subprocess.popen: https://pypi.python.org/pypi/commandwrapper/0.7). I am also trying to capture the real-time output to the console and to a file.
I have the code:
class Tee(object):
def __init__(self, *files):
self.files = files
def write(self, obj):
for f in self.files:
f.write(obj)
f.flush()
def flush(self) :
for f in self.files:
f.flush()
# Set the stdout/stderr to Tee()
out = open(stdout_log, 'w')
stdout_reset = sys.stdout
sys.stdout = Tee(sys.stdout, out)
process = commandwrapper.WrapCommand(command, shell=True) # Also tried False with same behaivor.
cmd = process.prepareToRun()
# Stream the output to the console and stdoutfiles
while cmd.poll() is None:
msg_out = cmd.stdout.readline()
sys.stdout.write(msg_out)
sys.stdout.flush()
out.close()
This works perfect when I run it in PyCharm. The output of command is written to the file AND displayed on the terminal console in real-time.
However, when I run the same code in a terminal, no output is displayed on the console. How can this be? The stdout is correctly captured in a file, but nothing is written to the console.
Can anyone see any reason that this code would work well and as expected in PyCharm, but not display any stdout to the terminal? I'm at a loss here. If anything, I could deal with it if the behavior was reversed.
Using OSX Yosemite and running bash.
You need to change the logic where you have poll, I used Popen but you can use the wrapper if you prefer:
out = open(stdout_log, 'w')
stdout_reset = sys.stdout
sys.stdout = Tee(sys.stdout, out)
from subprocess import Popen,PIPE,STDOUT
process = Popen([list_of_cmds],stdout=PIPE,stderr=STDOUT)
# Stream the output to the console and stdoutfiles
for line in iter(process.stdout.readline,""):
sys.stdout.write(line)
out.close()
Applying the same logic works with the commandwrapper lib:
process = commandwrapper.WrapCommand(command, shell=True) # Also tried False with same behaivor.
cmd = process.prepareToRun()
# Stream the output to the console and stdoutfiles
for line in iter(cmd.stdout.readline,""):
sys.stdout.write(line)
Even if the child process has exited i.e., cmd.poll() is not None there could be a buffered output left in the pipe. You could see it if you call cmd.stdout.read() after the while cmd.poll() is not None loop has ended.
To reproduce the error without Tee, commandwrapper:
#!/usr/bin/env python
import sys
import time
from subprocess import Popen, PIPE
#XXX DO NOT USE THE CODE -- ITS PURPOSE IS TO DEMONSTRATE THE ISSUE
p = Popen([sys.executable, '-c', "import os; os.write(1, b'abc')"],
stdout=PIPE, bufsize=0) # no buffering in the parent
time.sleep(0.3) # allow the child to exit
while p.poll() is None: # poll() should return non-None value
assert 0, "it is never run unless the computer is slow"
line = p.stdout.readline()
print(line)
output = p.stdout.read() #
print("the output left after the child has died: %r" % (output,))
assert len(output) >= 3
See these posts on how to read subprocess' output in real time and how to redirect it to a file and terminal at the same time:
Python: read streaming input from subprocess.communicate()
How do I push a subprocess.call() output to terminal and file?
Subprocess.Popen: cloning stdout and stderr both to terminal and variables
Python subprocess get children's output to file and terminal?
I am trying to assign the output of a command to a variable without the command thinking that it is being piped. The reason for this is that the command in question gives unformatted text as output if it is being piped, but it gives color formatted text if it is being run from the terminal. I need to get this color formatted text.
So far I've tried a few things. I've tried Popen like so:
output = subprocess.Popen(command, stdout=subprocess.PIPE)
output = output.communicate()[0]
output = output.decode()
print(output)
This will let me print the output, but it gives me the unformatted output that I get when the command is piped. That makes sense, as I'm piping it here in the Python code. But I am curious if there is a way to assign the output of this command, directly to a variable, without the command running the piped version of itself.
I have also tried the following version that relies on check_output instead:
output = subprocess.check_output(command)
output = output.decode()
print(output)
And again I get the same unformatted output that the command returns when the command is piped.
Is there a way to get the formatted output, the output the command would normally give from the terminal, when it is not being piped?
Using pexpect:
2.py:
import sys
if sys.stdout.isatty():
print('hello')
else:
print('goodbye')
subprocess:
import subprocess
p = subprocess.Popen(
['python3.4', '2.py'],
stdout=subprocess.PIPE
)
print(p.stdout.read())
--output:--
goodbye
pexpect:
import pexpect
child = pexpect.spawn('python3.4 2.py')
child.expect(pexpect.EOF)
print(child.before) #Print all the output before the expectation.
--output:--
hello
Here it is with grep --colour=auto:
import subprocess
p = subprocess.Popen(
['grep', '--colour=auto', 'hello', 'data.txt'],
stdout=subprocess.PIPE
)
print(p.stdout.read())
import pexpect
child = pexpect.spawn('grep --colour=auto hello data.txt')
child.expect(pexpect.EOF)
print(child.before)
--output:--
b'hello world\n'
b'\x1b[01;31mhello\x1b[00m world\r\n'
Yes, you can use the pty module.
>>> import subprocess
>>> p = subprocess.Popen(["ls", "--color=auto"], stdout=subprocess.PIPE)
>>> p.communicate()[0]
# Output does not appear in colour
With pty:
import subprocess
import pty
import os
master, slave = pty.openpty()
p = subprocess.Popen(["ls", "--color=auto"], stdout=slave)
p.communicate()
print(os.read(master, 100)) # Print 100 bytes
# Prints with colour formatting info
Note from the docs:
Because pseudo-terminal handling is highly platform dependent, there
is code to do it only for Linux. (The Linux code is supposed to work
on other platforms, but hasn’t been tested yet.)
A less than beautiful way of reading the whole output to the end in one go:
def num_bytes_readable(fd):
import array
import fcntl
import termios
buf = array.array('i', [0])
if fcntl.ioctl(fd, termios.FIONREAD, buf, 1) == -1:
raise Exception("We really should have had data")
return buf[0]
print(os.read(master, num_bytes_readable(master)))
Edit: nicer way of getting the content at once thanks to #Antti Haapala:
os.close(slave)
f = os.fdopen(master)
print(f.read())
Edit: people are right to point out that this will deadlock if the process generates a large output, so #Antti Haapala's answer is better.
A working polyglot example (works the same for Python 2 and Python 3), using pty.
import subprocess
import pty
import os
import sys
master, slave = pty.openpty()
# direct stderr also to the pty!
process = subprocess.Popen(
['ls', '-al', '--color=auto'],
stdout=slave,
stderr=subprocess.STDOUT
)
# close the slave descriptor! otherwise we will
# hang forever waiting for input
os.close(slave)
def reader(fd):
try:
while True:
buffer = os.read(fd, 1024)
if not buffer:
return
yield buffer
# Unfortunately with a pty, an
# IOError will be thrown at EOF
# On Python 2, OSError will be thrown instead.
except (IOError, OSError) as e:
pass
# read chunks (yields bytes)
for i in reader(master):
# and write them to stdout file descriptor
os.write(1, b'<chunk>' + i + b'</chunk>')
Many programs automatically turn off colour printing codes when they detect they are not connected directly to a terminal. Many programs will have a flag so you can force colour output. You could add this flag to your process call. For example:
grep "search term" inputfile.txt
# prints colour to the terminal in most OSes
grep "search term" inputfile.txt | less
# output goes to less rather than terminal, so colour is turned off
grep "search term" inputfile.txt --color | less
# forces colour output even when not connected to terminal
Be warned though. The actual colour output is done by the terminal. The terminal interprets special character espace codes and changes the text colour and background color accordingly. Without the terminal to interpret the colour codes you will just see the text in black with these escape codes interspersed throughout.
I am using the following code to redirect my print statements into a text file.
old_stdout = sys.stdout
log_file = open("message.log","w")
sys.stdout = log_file
print "this will be written to message.log"
subprocess.call("iPython.exe", "script.py") #subprocesses here take this form
sys.stdout = old_stdout
log_file.close()
My problem is that this does not seem to apply to subprocesses. The print statements in "script.py" will not appear in "message.log". How can I make it so that they do?
Use subprocess.Popen instead of subprocess.call, which allows you to redirect stdout and stderr.
import subprocess
with (open('message_stdout.log', 'w'), open('message_stderr.log', 'w')) as (stdout_file, stderr_file):
my_process = subprocess.Popen(["iPython.exe", "script.py"],
stdout=stdout_file,
stderr=stderr_file)
You can also redirect stderr to stdout like this, so that all output from script.py is sent to a single file.
import subprocess
with open('message.log', 'w') as stdout_file:
my_process = subprocess.Popen(["iPython.exe", "script.py"],
stdout=stdout_file,
stderr=subprocess.STDOUT)
However, subprocessing something which just calls iPython to load another script is an awful way to work. Instead, you should just call the script.py modules directly.
I have the following script:
import subprocess
arguments = ["d:\\simulator","2332.txt","2332.log", "-c"]
output=subprocess.Popen(arguments, stdout=subprocess.PIPE).communicate()[0]
print(output)
which gives me b'' as output.
I also tried this script:
import subprocess
arguments = ["d:\\simulator","2332.txt","atp2332.log", "-c"]
process = subprocess.Popen(arguments,stdout=subprocess.PIPE)
process.wait()
print(process.stdout.read())
print("ERROR:" + str(process.stderr))
which gives me the output: b'', ERROR:None
However when I run this at the cmd prompt I get a 5 lines of text.
d:\simulator atp2332.txt atp2332.log -c
I have added to simulator a message box which pops up when it launches. This is presented for all three cases. So I know that I sucessfully launch the simulator. However the python scripts are not caturing the stdout.
What am I doing wrong?
Barry.
If possible (not endless stream of data) you should use communicate() as noted on the page.
Try this:
import subprocess
arguments = ["d:\\simulator","2332.txt","atp2332.log", "-c"]
process = subprocess.Popen(arguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
sout, serr = process.communicate()
print(sout)
print(serr)
The following code gives me text output on stdout.
Perhaps you could try it, and then substitute your command for help
import subprocess
arguments = ["help","2332.txt","atp2332.log", "-c"]
process = subprocess.Popen(arguments,stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process.wait()
print 'Return code', process.returncode
print('stdout:', process.stdout.read())
print("stderr:" + process.stderr.read())