I am trying to shell out to a non-python subprocess and allow it to inherit the stdin and stdout from python. - i am using subprocess.Popen
This would probably work if I am calling from a console, but it definitely doesn't work when I am using the python shell
(I am using IDLE by the way)
Is there any way to convince python to allow a non python subprocess to print it's stdout to the python shell?
This works both from a script and from the interactive interpreter, but not from IDLE:
subprocess.Popen(whatever, stdin=sys.stdout, stdout=sys.stdin)
You can't use the objects which IDLE assigns to sys.stdin and sys.stdout as arguments to subprocess.Popen. These objects (the interfaces to the IDLE shell window) are file-like, but they're not real file handles with fileno attributes, and Unix-like operating systems require a fileno to be specified as the stdin or stdout for a subprocess. I cannot speak for Windows, but I imagine it has similar requirements.
Taymon's answer addresses your question directly in that IDLE's stdin/stdout are actually file-like objects and not the standard file streams associated with a console/terminal. Moreover, in Windows IDLE runs with pythonw.exe, which doesn't even have an attached win32 console.
That said, if you just need the output from a program to be printed to the user in real time, then in many cases (but not all) you can read the output line by line and echo it accordingly. The following works for me in Windows IDLE. It demonstrates reading from a piped stdout line by line. It also shows what happens if the process buffers the pipe, in which case readline will block until either the buffer is full or the pipe closes. This buffering can be manually disabled with some programs (such as the Python interpreter's -u option), and there are workarounds for Unix such as stdbuf.
test1.py
import sys
import subprocess
def test(cmd):
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
it = iter(p.stdout.readline, b'')
for line in it:
print(line.rstrip().decode('ascii'))
print('Testing buffered subprocess...')
test([sys.executable, 'test2.py'])
print('\nTesting unbuffered subprocess...')
#-u: unbuffered binary stdout and stderr
test([sys.executable, '-u', 'test2.py'])
test2.py:
import time
for i in range(5):
print(i)
time.sleep(1)
The output in IDLE should be the following, with the first set of digits printed all at once after a delay and the second set printed line by line.
Testing buffered subprocess...
0
1
2
3
4
Testing unbuffered subprocess...
0
1
2
3
4
Related
I am using Python to call a Shell script with
def run_command(cmd):
print "Start to run: " + cmd
run = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
line = run.stdout.readline().decode()[:-1]
if line == '' and run.poll() is not None:
break
print line # print the log from shell
recode = run.returncode
if recode != 0:
raise Exception("Error occurs!")
print "End to run: " + cmd
Then I run
run_command("sh /home/tome/a.sh")
I notice the console output from a.sh is not in real time, looks like that there is a buffer for the stdout and when the stdout buffer is full, then the output is printed out.
I would ask how to disable the shell stdout buffer in my script a.sh
Thanks!
The buffering in question would largely be a problem on the script's side, not the Python side; while Python would buffer the reads, it wouldn't block unless the buffer was emptied and there was nothing available to read.
So really, you need to disable buffering in the script itself. Adding stdbuf -oL (or -o0 for completely unbuffered, but line buffering should cover you since you read by line as well) to your commands should help in some cases (where the programs don't adjust their own buffering internally).
If you're seeing this behavior only by looking at Python's output, be aware that Python itself can buffer output as well. You can disable this by passing -u when running Python, or setting the environment variable PYTHONUNBUFFERED=1 before running it, or from within a script, you can manually call sys.stdout.flush() after any writes (direct, or implicit via print) to stdout. On modern Python, print takes an argument to force a flush after printing, but since you're on Python 2.x, that's not an option.
I have a... very specific problem. Really tried to find a broader question but couldn't.
I am trying to use mplayer as a subprocess to play music (on windows and also linux), and retain the ability to pass commands to it. I have accomplished this just fine in python 2.7 with subprocess.Popen and p.stdin.write('pause\n').
However this doesn't seem to have survived the trip to Python 3. I have to either use 'pause\n'.encode() or b'pause\n' to convert to bytes, and the mplayer process does not pause. It does seem to work however if i use p.communicate, but I have ruled that out as a possiblity due to this question which claims it can only be called once per process.
Here's is my code:
p = subprocess.Popen('mplayer -slave -quiet "C:\\users\\me\\music\\Nickel Creek\\Nickel Creek\\07 Sweet Afton.mp3"', stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
time.sleep(1)
mplayer.stdin.write(b'pause\n')
time.sleep(1)
mplayer.stdin.write(b'pause\n')
time.sleep(1)
mplayer.stdin.write(b'quit\n')
seeing as this code worked (without the bs) in 2.7, i can only assume encoding the string as a bytes is somehow changing the byte values so that mplayer can't understand it any more? however when i try to see exactly what bytes are sent through the pipeline it looks correct. it could also be windows pipeline acting strange. i've tried this with both cmd.exe and powershell, since i know powershell interprets the pipeline as xml. i used this code to test what comes in through the pipeline:
# test.py
if __name__ == "__main__":
x = ''
with open('test.out','w') as f:
while (len(x) == 0 or x[-1] != 'q'):
x += sys.stdin.read(1)
print(x)
f.write(x)
and
p = subprocess.Popen('python test.py', stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
p.stdin.write(b'hello there\ntest2\nq\n')
seeing as this code worked (without the bs) in 2.7, i can only assume encoding the string as a bytes is somehow changing the byte values so that mplayer can't understand it any more?
'pause\n' in Python 2 is exactly the same value as b'pause\n' -- moreover you could use b'pause\n' on Python 2 too (to communicate the intent of the code).
The difference is that bufsize=0 on Python 2 and therefore .write() pushes the content to the subprocess immediately while .write() on Python 3 puts it in some internal buffer instead. Add .flush() call, to empty the buffer.
Pass universal_newlines=True, to enable the text mode on Python 3 (then you could use 'pause\n' instead of b'pause\n'). You might also need it if mplayer expects os.newline instead of b'\n' as the end of line.
#!/usr/bin/env python3
import time
from subprocess import Popen, PIPE
LINE_BUFFERED = 1
filename = r"C:\Users\me\...Afton.mp3"
with Popen('mplayer -slave -quiet'.split() + [filename],
stdin=PIPE, universal_newlines=True, bufsize=LINE_BUFFERED) as process:
send_command = lambda command: print(command, flush=True, file=process.stdin)
time.sleep(1)
for _ in range(2):
send_command('pause')
time.sleep(1)
send_command('quit')
Unrelated: do not use stdout=PIPE unless you read from the pipe otherwise you may hang the child process. To discard the output, use stdout=subprocess.DEVNULL instead. See How to hide output of subprocess in Python 2.7
I'm new to python and would like to open a windows cmd prompt, start a process, leave the process running and then issue commands to the same running process.
The commands will change so i cant just include these commands in the cmdline variable below. Also, the process takes 10-15 seconds to start so i dont want to waste time waiting for the process to start and run commands each time. just want to start process once. and run quick commands as needed in the same process
I was hoping to use subprocess.Popen to make this work, though i am open to better methods. Note that my process to run is not cmd, but im just using this as example
import subprocess
cmdline = ['cmd', '/k']
cmd = subprocess.Popen(cmdline, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
cmd.stdin.write("echo hi") #would like this to be written to the cmd prompt
print cmd.stdout.readline() #would like to see 'hi' readback
cmd.stdin.write("echo hi again") #would like this to be written to the cmd prompt
print cmd.stdout.readline() #would like to see 'hi again' readback
The results arent what i expect. Seems as though the stdin.write commands arent actually getting in and the readline freezes up with nothing to read.
I have tried the popen.communicate() instead of write/readline, but it kills the process. I have tried setting bufsize in the Popen line, but that didn't make too much difference
Your comments suggest that you are confusing command-line arguments with input via stdin. Namely, the fact that system-console.exe program accepts script=filename parameter does not imply that you can send it the same string as a command via stdin e.g., python executable accepts -c "print(1)" command-line arguments but it is a SyntaxError if you pass it as a command to Python shell.
Therefore, the first step is to use the correct syntax. Suppose the system-console.exe accepts a filename by itself:
#!/usr/bin/env python3
import time
from subprocess import Popen, PIPE
with Popen(r'C:\full\path\to\system-console.exe -cli -',
stdin=PIPE, bufsize=1, universal_newlines=True) as shell:
for _ in range(10):
print('capture.tcl', file=shell.stdin, flush=True)
time.sleep(5)
Note: if you've redirected more than one stream e.g., stdin, stdout then you should read/write both streams concurrently (e.g., using multiple threads) otherwise it is very easy to deadlock your program.
Related:
Q: Why not just use a pipe (popen())? -- mandatory reading for Unix environment but it might also be applicable for some programs on Windows
subprocess readline hangs waiting for EOF -- code example on how to pass multiple inputs, read multiple outputs using subprocess, pexpect modules.
The second and the following steps might have to deal with buffering issues on the side of the child process (out of your hands on Windows), whether system-console allows to redirect its stdin/stdout or whether it works with a console directly, and character encoding issues (how various commands in the pipeline encode text).
Here is some code that I tested and is working on Windows 10, Quartus Prime 15.1 and Python 3.5
import subprocess
class altera_system_console:
def __init__(self):
sc_path = r'C:\altera_lite\15.1\quartus\sopc_builder\bin\system-console.exe --cli --disable_readline'
self.console = subprocess.Popen(sc_path, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
def read_output(self):
rtn = ""
loop = True
i = 0
match = '% '
while loop:
out = self.console.stdout.read1(1)
if bytes(match[i],'utf-8') == out:
i = i+1
if i==len(match):
loop=False
else:
rtn = rtn + out.decode('utf-8')
return rtn
def cmd(self,cmd_string):
self.console.stdin.write(bytes(cmd_string+'\n','utf-8'))
self.console.stdin.flush()
c = altera_system_console()
print(c.read_output())
c.cmd('set jtag_master [lindex [get_service_paths master] 0]')
print(c.read_output())
c.cmd('open_service master $jtag_master')
print(c.read_output())
c.cmd('master_write_8 $jtag_master 0x00 0xFF')
print(c.read_output())
You need to use iter if you want to see the output in real time:
import subprocess
cmdline = ['cmd', '/k']
cmd = subprocess.Popen(cmdline, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
cmd.stdin.write("echo hi\n")#would like this to be written to the cmd prompt
for line in iter(cmd.stdout.readline,""):
print line
cmd.stdin.write("echo hi again\n")#would like this to be written to the cmd prompt
Not sure exactly what you are trying to do but if you want to input certain data when you get certain output then I would recommend using pexpect
I've got a script parent.py trying to to read stdout from a subprocess sub.py in Python.
The parent parent.py:
#!/usr/bin/python
import subprocess
p = subprocess.Popen("sub.py", stdout=subprocess.PIPE)
print p.stdout.read(1)
And the subprocess, sub.py:
#!/usr/bin/python
print raw_input( "hello world!" )
I would expect running parent.py to print the 'h' from "hello world!". Actually, it hangs. I can only get my expected behaviour by adding -u to sub.py's she-bang line.
This confuses me because the -u switch makes no difference when sub.py is run directly from a shell; the shell is somehow privy to the un-flushed output stream, unlike parent.py.
My goal is to run a C program as the subprocess, so I won't be able to control whether or not it flushes stdout. How is it that a shell has better access to a process's stdout than Python running the same thing from subprocess.Popen? Am I going to be able to read such a stdout stream from a C program that doesn't flush its buffers?
EDIT:
Here is an updated example based on korylprince's comment...
## capitalize.sh ##
#!/bin/sh
while [ 1 ]; do
read s
echo $s | tr '[:lower:]' '[:upper:]'
done
########################################
## parent.py ##
#!/usr/bin/python
from subprocess import Popen, PIPE
# cmd = [ 'capitalize.sh' ] # This would work
cmd = [ 'script', '-q', '-f', '-c', 'capitalize.sh', '/dev/null']
p = Popen(cmd, stdin=PIPE)
p.stdin.write("some string\n")
p.wait()
When running through script, I get steady printing of newlines (and if this were a Python, subprocess, it'd raise an EOFerror).
An alternative is
p = subprocess.Popen(["python", "-u", "sub.py"], stdout=subprocess.PIPE)
or the suggestions here.
My experience is that yes, you will be able to read from most C programs without any extra effort.
The Python interpreter takes extra steps to buffer its output which is why it needs the -u switch to disable output buffering. Your typical C program won't do this.
I haven't run into any program (C or otherwise) other than the Python interpreter that I expected to work and didn't within a subshell.
The reason the shell can read output immediately, regardless of "-u" is because the program you're launching from the shell has its output connected to a TTY. When the stdout is connected to a TTY, it is unbuffered (because it is up to the TTY to buffer). When you launch the python subprocess from within python, you're connecting stdout to a pipe, which means you're at the mercy of the subprocess to flush its output when it feels like it.
If you're looking to do complicated interactions with a subprocess, look into this tutorial.
I'm currently trying to write a python script that, among many things, calls an executable and stores what that executable sends to stdout in a variable. Here is what I have:
1 #!/usr/bin/python
2 import subprocess
3
4 subprocess.call("./pmm", shell=True)
How would I get the output of pmm to be stored in a variable?
In Python 2.7 (and 3.1 or above), you can use subprocess.check_output(). Example from the documentation:
>>> subprocess.check_output(["ls", "-l", "/dev/null"])
'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n'
p = subprocess.Popen(["./pmm"], shell=False, stdout=subprocess.PIPE)
output = p.stdout.read()
I wrote a post about this some time ago:
http://trifoliummedium.blogspot.com/2010/12/running-command-line-with-python-and.html
Use p.communicate() to get both stdout and stderr
First you have to save a reference to the subprocess (bind it to a name ... which, in other languages and more informally is referred to as "assigning it to a variable"). So you should use something like proc = subprocess.Popen(...)
From there I recommend that you call proc.poll() to test if the program has completed, and either sleep (using the time.sleep() function, for example) or perform other work (using select.select() for example) and then checking again, later. Or you can call proc.wait() so that you're sure the this ./pmm command has completed it's work before your program continues. The poll() method on an subprocess instance will return "None" if the subprocess it still running; otherwise it'll return the exit value of the command that was running on that subprocess. The wait() method for a subprocess will cause your program to block and then return the exit value.
After that you can call (output, errormsgs) = proc.communicate() to capture any output or error messages from your subprocess. If the output is too large it could cause problems; using the process instance's .stdout (PIPE file descriptor) is tricky and, if you were going to attempt this then you should use features in the fcntl (file descriptor control) module to switch it into a non-blocking mode and be prepared to handle he exceptions raised when attempting read() calls on the buffer when it's empty.