I want to get user input from a subprocess in an new terminal.
import subprocess
additionalBuildArguments = "defaultarg1"
proc = subprocess.Popen(["python", "user_input.py", additionalBuildArguments],
creationflags=subprocess.CREATE_NEW_CONSOLE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
try:
outs, errs = proc.communicate(timeout=15)
except subprocess.TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
additionalBuildArguments = outs or additionalBuildArguments
user_input.py:
import sys
arg = sys.argv[1]
user_input = input(f"Additional build arguments [{arg}] (Push <ENTER> to use these settings):\n")
print(user_input)
as long as I don't set the stdout=subprocess.PIPE and/or the stderr=subprocess.PIPE options I can enter input. But with these options I can't write any input to the console.
Indeed I need these options to redirect the stdout, to have access to the printed user_input in the parent process.
Does anyone know what's the problem here?
Please note that I do not understand why you want to do this, and feel instinctively that you should not. However, it's perfectly possible: just catpure only stdout:
import sys
from subprocess import run
print("Type away: ", end="")
sys.stdout.flush()
r = run(["python", "-c", "print(input())"], capture_output=True, encoding="utf8")
print(f"You entered {r.stdout}")
EDIT Apparently you are using windows. Per the docs your flag is set when shell=True. With shell=True this works for me, but I have no idea whether it will for you:
import sys
from subprocess import run
print("Type away: ", end="")
sys.stdout.flush()
r = run("python -c 'print(input())'", capture_output=True, shell=True, encoding="utf8")
print(f"You entered {r.stdout}")
This can be chained to run in yet a third process, which would be needed to print whilst also capturing stdout, from a subprocess. But at this point we are in the realm of very horrible hacks.
A better, but still hacky, solution, is to re-phrase the problem a bit. You want to spawn a terminal, which apparently you can do, and the user can interact with it correctly, and then you want to get output from that terminal in the spawning code. STDOUT is not the proper channel for this communication. Personally I would structure my code like this:
in spawning code:
generate parametrised script to run in spawned terminal and save it as a temp file
spawn subterminal running the generated script
wait for completion
read temp out file and get data
delete both temp script and temp out file
in generated code:
do as much as possible (you have a full python, after all)
dump output as json to a temporary file
This is still hacky, but it only involves spawning one terminal. Note that I still don't understand why you want to do this, but this should at least work.
Related
I'm working on a script to automate tests of a certain software, and as part of it I need to chech if it runs commands correctly.
I'm currently launching an executeable using subprocess and passing the initial parameters.
My code is: subprocess.run("program.exe get -n WiiVNC", shell=True, check=True)
As far as I understand, this runs the executeable, and is supposed to return an exception if the exit code is 1.
Now, the program launches, but at some point waits for user input like so:
My question is, how do I go about submitting the user input "y" using subprocess once either, the text "Continue with download of "WiiVNC"? (y/n) >" shows up, or once the program waits for user input.
You should use the pexpect module for all complicated subprocessing. In particular, the module is designed to handle the complicated case of either passing through the input to the current process for the user to answer and/or allowing your script to answer the input for the user and continue the subprocess.
Added some code for an example:
### File Temp ###
# #!/bin/env python
# x = input('Type something:')
# print(x)
import pexpect
x = pexpect.spawn('python temp') #Start subprocess.
x.interact() #Imbed subprocess in current process.
# or
x = pexpect.spawn('python temp') #Start subprocess.
find_this_output = x.expect(['Type something:'])
if find_this_output is 0:
x.send('I type this in for subprocess because I found the 0th string.')
Try this:
import subprocess
process = subprocess.Popen("program.exe get -n WiiVNC", stdin=subprocess.PIPE, shell=True)
process.stdin.write(b"y\n")
process.stdin.flush()
stdout, stderr = process.communicate()
I am running jirashell in a python script using the subprocess library. I am currently having issues having the outputs print in real time. When I run jirashell it outputs information than prompts the user (y/n). The subprocess won't print out information prior to the prompt until I enter 'y' or 'n'.
The code I am using is
_consumer_key = "justin-git"
_cmd = "jirashell -s {0} -od -k {1} -ck {2} -pt".format(JIRA_SERVER,
_rsa_private_key_path, _consumer_key)
p = subprocess.Popen(_cmd.split(" "), stdout=subprocess.PIPE,
stderr=subprocess.PIPE, bufsize=0)
out, err = p.communicate() # Blocks here
print out
print err
The output is like so:
n # I enter a "n" before program will print output.
Output:
Request tokens received.
Request token: asqvavefafegagadggsgewgqqegqgqge
Request token secret: asdbresbdfbrebsaerbbsbdabweabfbb
Please visit this URL to authorize the OAuth request:
http://localhost:8000/plugins/servlet/oauth/authorize?oauth_token=zzzzzzzzzzzzzzzzzzzzzzzzzzzzz
Have you authorized this program to connect on your behalf to http://localhost:8000? (y/n)
Error:
Abandoning OAuth dance. Your partner faceplants. The audience boos. You feel shame.
Does anyone know how I can have it print the output prior to the prompt than wait for an input of y/n? Note I also need to be able to store the output produced by the command so "os.system()" won't work...
EDIT:
It looks like inside jirashell there is a part of the code that is waiting for an input and this is causing the block. Until something is passed into this input nothing is outputted... Still looking into how I can get around this. I'm in the process of trying to move the portion of code I need into my application. This solution doesn't seem elegant but I can't see any other way right now.
approved = input(
'Have you authorized this program to connect on your behalf to {}? (y/n)'.format(server))
Method which prints and caches the standard output:
You can use a thread which reads the standard output of your subprocess, while the main thread is blocked until the subprocess is done. The following example will run the program other.py, which looks like
#!/usr/bin/env python3
print("Hello")
x = input("Type 'yes': ")
Example:
import threading
import subprocess
import sys
class LivePrinter(threading.Thread):
"""
Thread which reads byte-by-byte from the input stream and writes it to the
standard out.
"""
def __init__(self, stream):
self.stream = stream
self.log = bytearray()
super().__init__()
def run(self):
while True:
# read one byte from the stream
buf = self.stream.read(1)
# break if end of file reached
if len(buf) == 0:
break
# save output to internal log
self.log.extend(buf)
# write and flush to main standard output
sys.stdout.buffer.write(buf)
sys.stdout.flush()
# create subprocess
p = subprocess.Popen('./other.py', stdout=subprocess.PIPE)
# Create reader and start the thread
r = LivePrinter(p.stdout)
r.start()
# Wait until subprocess is done
p.wait()
# Print summary
print(" -- The process is done now -- ")
print("The standard output was:")
print(r.log.decode("utf-8"))
The class LivePrinter reads every byte from the subprocess and writes it to the standard output. (I have to admit, this is not the most efficient approach, but a larger buffer size blocks, the LiveReader until the buffer is full, even though the subprocess is awaiting the answer to a prompt.) Since the bytes are written to sys.stdout.buffer, there shouldn't be a problem with multi-byte utf-8 characters.
The LiveReader class also stores the complete output of the subprocess in the variable log for later use.
As this answer summarizes, it is save to start a thread after forking with subprocess.
Original answer which has problems, when the prompt line doesn't end a line:
The output is delayed because communicate() blocks the execution of your script until the sub-process is done (https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate).
You can read and print the standard output of the subprocess, while it is executed using stdout.readline. There are some issues about buffering, which require this rather complicated iter(process.stdout.readline, b'') construct. The following example uses gpg2 --gen-key because this command starts an interactive tool.
import subprocess
process = subprocess.Popen(["gpg2", "--gen-key"], stdout=subprocess.PIPE)
for stdout_line in iter(process.stdout.readline, b''):
print(stdout_line.rstrip())
Alternative answer which uses shell and does not cache the output:
As Sam pointed out, there is a problem with the above solution, when the prompt line does not end the line (which prompts they usually don't). An alternative solution is to use the shell argument to interact with the sub-process.
import subprocess
subprocess.call("gpg2 --gen-key", shell=True)
I am trying to run one python file from python windows application.For that I have used subprocess.For getting live streaming output on app console I have tried the below statements.
With PIPE
p = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, shell=True)
for line in iter(p.stdout.readline, ''):
print line
(or)
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
out = process.stdout.read(1)
if out == '' and process.poll() != None:
break
if out != '':
sys.stdout.write(out)
sys.stdout.flush()
Not only above code ,tried so many methods.Getting same results like below:
1.Python windows app taking so much of time to run
2.Then the app window went to "not responding" state for long time
3.Then whole output is printed on the console
I know that the buffer overflow is happening in python app thats why i am not getting live output.
I posted so many queries for this still not getting solution.
Just now found and tried tempfile for this.But i am not sure this will give live streaming output.
Shall I try this way?
import tempfile
import subprocess
w = tempfile.NamedTemporaryFile()
p = subprocess.Popen(cmd, shell=True, stdout=w,
stderr=subprocess.STDOUT, bufsize=0)
with open(w.name, 'r') as r:
for line in r:
print line
w.close()
Or any other best solutions for non blocking,unbuffering live output on windows app.
Any help would be appreciated.
Note:1.The python file which I want to run has more print statements(ie more content)
2.Windows server 2012,python 2.7
I understand you're frustration. It looks like you've almost come to the answer yourself.
I'm building on the answer from this SO post. But that answer doesn't use TemporaryFile and also I used the tail follow method from here which I have found to offer the fastest output to the terminal with very large volumes of output. This eliminates extraneous calls to print.
Side note: If you've got other asynchronous stuff to do then you can wrap up the code below the imports in a function and use the gevent package and import sleep from gevent and Popen, STDOUT from gevent.subprocess. It is what I'm doing and may help you avoid leftover slowdown (only reason I mention it).
import sys
from tempfile import TemporaryFile
from time import sleep
from subprocess import Popen, STDOUT
# the temp file will be automatically cleaned up using context manager
with TemporaryFile() as output:
sub = Popen(cmd, stdout=output, stderr=STDOUT, shell=True)
# sub.poll returns None until the subprocess ends,
# it will then return the exit code, hopefully 0 ;)
while sub.poll() is None:
where = output.tell()
lines = output.read()
if not lines:
# Adjust the sleep interval to your needs
sleep(0.1)
# make sure pointing to the last place we read
output.seek(where)
else:
sys.__stdout__.write(lines)
sys.__stdout__.flush()
# A last write needed after subprocess ends
sys.__stdout__.write(output.read())
sys.__stdout__.flush()
I'm new to python and would like to open a windows cmd prompt, start a process, leave the process running and then issue commands to the same running process.
The commands will change so i cant just include these commands in the cmdline variable below. Also, the process takes 10-15 seconds to start so i dont want to waste time waiting for the process to start and run commands each time. just want to start process once. and run quick commands as needed in the same process
I was hoping to use subprocess.Popen to make this work, though i am open to better methods. Note that my process to run is not cmd, but im just using this as example
import subprocess
cmdline = ['cmd', '/k']
cmd = subprocess.Popen(cmdline, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
cmd.stdin.write("echo hi") #would like this to be written to the cmd prompt
print cmd.stdout.readline() #would like to see 'hi' readback
cmd.stdin.write("echo hi again") #would like this to be written to the cmd prompt
print cmd.stdout.readline() #would like to see 'hi again' readback
The results arent what i expect. Seems as though the stdin.write commands arent actually getting in and the readline freezes up with nothing to read.
I have tried the popen.communicate() instead of write/readline, but it kills the process. I have tried setting bufsize in the Popen line, but that didn't make too much difference
Your comments suggest that you are confusing command-line arguments with input via stdin. Namely, the fact that system-console.exe program accepts script=filename parameter does not imply that you can send it the same string as a command via stdin e.g., python executable accepts -c "print(1)" command-line arguments but it is a SyntaxError if you pass it as a command to Python shell.
Therefore, the first step is to use the correct syntax. Suppose the system-console.exe accepts a filename by itself:
#!/usr/bin/env python3
import time
from subprocess import Popen, PIPE
with Popen(r'C:\full\path\to\system-console.exe -cli -',
stdin=PIPE, bufsize=1, universal_newlines=True) as shell:
for _ in range(10):
print('capture.tcl', file=shell.stdin, flush=True)
time.sleep(5)
Note: if you've redirected more than one stream e.g., stdin, stdout then you should read/write both streams concurrently (e.g., using multiple threads) otherwise it is very easy to deadlock your program.
Related:
Q: Why not just use a pipe (popen())? -- mandatory reading for Unix environment but it might also be applicable for some programs on Windows
subprocess readline hangs waiting for EOF -- code example on how to pass multiple inputs, read multiple outputs using subprocess, pexpect modules.
The second and the following steps might have to deal with buffering issues on the side of the child process (out of your hands on Windows), whether system-console allows to redirect its stdin/stdout or whether it works with a console directly, and character encoding issues (how various commands in the pipeline encode text).
Here is some code that I tested and is working on Windows 10, Quartus Prime 15.1 and Python 3.5
import subprocess
class altera_system_console:
def __init__(self):
sc_path = r'C:\altera_lite\15.1\quartus\sopc_builder\bin\system-console.exe --cli --disable_readline'
self.console = subprocess.Popen(sc_path, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
def read_output(self):
rtn = ""
loop = True
i = 0
match = '% '
while loop:
out = self.console.stdout.read1(1)
if bytes(match[i],'utf-8') == out:
i = i+1
if i==len(match):
loop=False
else:
rtn = rtn + out.decode('utf-8')
return rtn
def cmd(self,cmd_string):
self.console.stdin.write(bytes(cmd_string+'\n','utf-8'))
self.console.stdin.flush()
c = altera_system_console()
print(c.read_output())
c.cmd('set jtag_master [lindex [get_service_paths master] 0]')
print(c.read_output())
c.cmd('open_service master $jtag_master')
print(c.read_output())
c.cmd('master_write_8 $jtag_master 0x00 0xFF')
print(c.read_output())
You need to use iter if you want to see the output in real time:
import subprocess
cmdline = ['cmd', '/k']
cmd = subprocess.Popen(cmdline, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
cmd.stdin.write("echo hi\n")#would like this to be written to the cmd prompt
for line in iter(cmd.stdout.readline,""):
print line
cmd.stdin.write("echo hi again\n")#would like this to be written to the cmd prompt
Not sure exactly what you are trying to do but if you want to input certain data when you get certain output then I would recommend using pexpect
I am writing a script in which in the external system command may sometimes require user input. I am not able to handle that properly. I have tried using os.popen4 and subprocess module but could not achieve the desired behavior.
Below mentioned example would show this problem using "cp" command. ("cp" command is used to show this problem, i am calling some different exe which may similarly prompt for user response in some scenarios). In this example there are two files present on disk and when user tries to copy file1 to file2, an conformer message comes up.
proc = subprocess.Popen("cp -i a.txt b.txt", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,)
stdout_val, stderr_val = proc.communicate()
print stdout_val
b.txt?
proc.communicate("y")
Now in this example if i read only stdout/stderr and prints it, later on if i try to write "y" or "n" based on user's input, i got an error that channel is closed.
Can some one please help me on achieving this behavior in python such that i can print stdout first, then should take user input and write stdin later on.
I found another solution (Threading) from Non-blocking read on a subprocess.PIPE in python , not sure whether it would help. But it appears it is printing question from cp command, i have modified code but not sure on how to write in threading code.
import sys
from subprocess import PIPE, Popen
from threading import Thread
try:
from Queue import Queue, Empty
except ImportError:
from queue import Queue, Empty
ON_POSIX = 'posix' in sys.builtin_module_names
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
p = Popen(['cp', '-i', 'a.txt', 'b.txt'],stdin=PIPE, stdout=PIPE, bufsize=1, close_fds=ON_POSIX)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.start()
try:
line = q.get_nowait()
except Empty:
print('no output yet')
else:
pass
Popen.communicate will run the subprocess to completion, so you can't call it more than once. You could use the stdin and stdout attributes directly, although that's risky as you could deadlock if the process uses block buffering or the buffers fill up:
stdout_val = proc.stdout.readline()
print stdout_val
proc.stdin.write('y\n')
As there is a risk of deadlock and because this may not work if the process uses block buffering, you would do well to consider using the pexpect package instead.
I don't have a technical answer to this question. More of just a solution. It has something to do with the way the process waits for the input, and once you communicate with the process, a None input is enough to close the process.
For your cp example, what you can do is check the return code immediately with proc.poll(). If the return value is None, you might assume it is trying to wait for input and can ask your user a question. You can then pass the response to the process via proc.communicate(response). It will then pass the value and proceed with the process.
Maybe someone else can chime in with a more technical reason why an initial communicate with a None value closes the process.