I am using Python to script running an exe program.
If we open the exe program in the shell, we could enter different command such as "a", "b", "c" in the program. These commands can not be passed as flags into the exe program. I want to use Python to script running this exe program for many times, with custom exe-program specific input.
But if I run the "program.exe" with
p = subprocess.call(['program.exe'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
Python won't terminate. Can I achieve this purpose with subprocess in Python?
Beware: subprocess.call will not return before the child process has terminated. So you have no possibility to write anything to the standard input of the child.
If you can prepare the bunch of commands in advance, and if output has no risk to fill the system buffer, you can still use call that way:
cmds = "a\nb\nc\n"
p = subprocess.call(['program.exe'],
stdin=io.StringIO(cmds),
stdout=subprocess.PIPE,
)
But the more robust way is to directly use the Popen constructor, and then feed the input:
p = subprocess.Popen(['program.exe'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
p.stdin.write("a\n");
p.stdin.write("b\n");
...
p.stdin.close();
p.wait();
If you know that one subcommand will generate very large output you can read it before sending the next one. Beware to avoid being blocked waiting an input that the child has still not sent...
First, you have to use p = subprocess.Popen(…) in order to get the subprocess object. subprocess.call(…) would give you just the return status, and that's not enough.
If p is your connection object, you can send your commands to p.stdin, such as p.stdin.write("a\n"), and then read out p.stdout() until the next indication that the command output is finished. How you detect this depends on said program.
Then you can send the next command and read its output.
At the end, you can do p.stdin.close() in order to signal an EOF ot the other process, and then it should terminate.
Related
I'm trying to port the following c# code into Python. It firstly defines a new process and then runs a windows prompt command (cmd.exe). After that, it executes a command in the prompt and when an external event occurs, it closes the prompt.
//Start the prompt - when an event occured
Process winShell = new Process();
winShell.StartInfo.FileName = "cmd.exe";
winShell.StartInfo.RedirectStandardInput = true;
winShell.Start();
//Execute a command in the prompt
winShell.StandardInput.WriteLine("cd " + projectDirectory);
//Close it - when an event occured
winShell.StandardInput.Flush();
winShell.StandardInput.Close();
winShell.WaitForExit();
I read that for Python 3 (my version 3.7), It is recommended to use subprocess. Unfortunately, I feel a bit confused about which of the function to use. I found call, run and Popen, but I didn't understand how to use them.
I wrote the following lines, but they don't produce any visible result.
import subprocess
subprocess.run(['cmd.exe'])
First of all, I would like that the shell appears and than to write some commands in it. Finally, I want to close it.
Use subprocess.Popen() like this. Each API matches to the corresponding C# API almost 1:1.
p = subprocess.Popen(['cmd.exe'],
stdin=subprocess.PIPE, stdout=subprocess.PIPE,
text=True)
p.stdin.write('dir\n')
p.stdin.close()
print(p.stdout.read())
p.wait()
p.stdout.close()
Other API's such as run(), call(), etc are wrappers for Popen(). For example, the above code is equivalent to this one line.
print(subprocess.run(['cmd.exe'], capture_output=True, text=True, input = 'dir\n').stdout)
I'm new to python and would like to open a windows cmd prompt, start a process, leave the process running and then issue commands to the same running process.
The commands will change so i cant just include these commands in the cmdline variable below. Also, the process takes 10-15 seconds to start so i dont want to waste time waiting for the process to start and run commands each time. just want to start process once. and run quick commands as needed in the same process
I was hoping to use subprocess.Popen to make this work, though i am open to better methods. Note that my process to run is not cmd, but im just using this as example
import subprocess
cmdline = ['cmd', '/k']
cmd = subprocess.Popen(cmdline, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
cmd.stdin.write("echo hi") #would like this to be written to the cmd prompt
print cmd.stdout.readline() #would like to see 'hi' readback
cmd.stdin.write("echo hi again") #would like this to be written to the cmd prompt
print cmd.stdout.readline() #would like to see 'hi again' readback
The results arent what i expect. Seems as though the stdin.write commands arent actually getting in and the readline freezes up with nothing to read.
I have tried the popen.communicate() instead of write/readline, but it kills the process. I have tried setting bufsize in the Popen line, but that didn't make too much difference
Your comments suggest that you are confusing command-line arguments with input via stdin. Namely, the fact that system-console.exe program accepts script=filename parameter does not imply that you can send it the same string as a command via stdin e.g., python executable accepts -c "print(1)" command-line arguments but it is a SyntaxError if you pass it as a command to Python shell.
Therefore, the first step is to use the correct syntax. Suppose the system-console.exe accepts a filename by itself:
#!/usr/bin/env python3
import time
from subprocess import Popen, PIPE
with Popen(r'C:\full\path\to\system-console.exe -cli -',
stdin=PIPE, bufsize=1, universal_newlines=True) as shell:
for _ in range(10):
print('capture.tcl', file=shell.stdin, flush=True)
time.sleep(5)
Note: if you've redirected more than one stream e.g., stdin, stdout then you should read/write both streams concurrently (e.g., using multiple threads) otherwise it is very easy to deadlock your program.
Related:
Q: Why not just use a pipe (popen())? -- mandatory reading for Unix environment but it might also be applicable for some programs on Windows
subprocess readline hangs waiting for EOF -- code example on how to pass multiple inputs, read multiple outputs using subprocess, pexpect modules.
The second and the following steps might have to deal with buffering issues on the side of the child process (out of your hands on Windows), whether system-console allows to redirect its stdin/stdout or whether it works with a console directly, and character encoding issues (how various commands in the pipeline encode text).
Here is some code that I tested and is working on Windows 10, Quartus Prime 15.1 and Python 3.5
import subprocess
class altera_system_console:
def __init__(self):
sc_path = r'C:\altera_lite\15.1\quartus\sopc_builder\bin\system-console.exe --cli --disable_readline'
self.console = subprocess.Popen(sc_path, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
def read_output(self):
rtn = ""
loop = True
i = 0
match = '% '
while loop:
out = self.console.stdout.read1(1)
if bytes(match[i],'utf-8') == out:
i = i+1
if i==len(match):
loop=False
else:
rtn = rtn + out.decode('utf-8')
return rtn
def cmd(self,cmd_string):
self.console.stdin.write(bytes(cmd_string+'\n','utf-8'))
self.console.stdin.flush()
c = altera_system_console()
print(c.read_output())
c.cmd('set jtag_master [lindex [get_service_paths master] 0]')
print(c.read_output())
c.cmd('open_service master $jtag_master')
print(c.read_output())
c.cmd('master_write_8 $jtag_master 0x00 0xFF')
print(c.read_output())
You need to use iter if you want to see the output in real time:
import subprocess
cmdline = ['cmd', '/k']
cmd = subprocess.Popen(cmdline, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
cmd.stdin.write("echo hi\n")#would like this to be written to the cmd prompt
for line in iter(cmd.stdout.readline,""):
print line
cmd.stdin.write("echo hi again\n")#would like this to be written to the cmd prompt
Not sure exactly what you are trying to do but if you want to input certain data when you get certain output then I would recommend using pexpect
I'm trying to run two processes in parallel. Both programs do not "end" without Ctrl+C (by the way, I'm on Linux), and so os.system will not return the output of a command. I want a way to create two processes independently of the main Python thread, and read text from them as it appears. I also want to be able to send characters to the process (not as a command, because the process interprets key presses by itself) I need something like this:
process1 = System("sh process1")
process2 = System("sh process2")
process1.Send("Hello, I'm sending text into process 1.")
text = process1.Read()
process2.Send(text)
Is there a way of doing this? I've looked into the Subprocess module, but I'm not sure it achieves quite what I want - or if it does, I'm not sure how to do it.
many thanks to anyone who answers,
Subprocess does what you want. Here's an example of writing to and reading from an external command:
import subprocess
proc = subprocess.Popen(["sed", "-u", "s/foo/bar/g"],
shell=False, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
proc.stdin.write("foobar\n");
print proc.stdout.readline(); # Writes "barbar"
HERE is a code snippet from my program.
I am trying to open cmd.exe on Windows and pass commands to a separate program and capture the output and parse it WITHOUT having to load cmd.exe every time.
All the examples to do this I found where using Python 2, and there are several changes in Python 3 about PIPES, making me unsure what is going wrong.
#DOScmd is a list of command line parameters to type into command shell.
p = subprocess.Popen('cmd.exe',
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
shell=True,
bufsize=0)
myCall = ' '.join(DOScmd) + '\n'
p.stdin.write( bytes(myCall, 'UTF-8') )
searchLines = p.stdout.readlines()
print(searchLines)
I am calling a program bowtie.exe. Now, bowtie.exe crashes when I do this. I think I might be angering the I/O gods. Any help appreciate.
I am trying to open cmd.exe on Windows and pass commands to a separate program and capture the output and parse it WITHOUT having to load cmd.exe every time.
Unless you want to run commands that are builtin to cmd.exe such as dir then you don't need to start cmd.exe at all:
from subprocess import check_output
for cmd in ["first.exe", "arg1", "arg2"], ["second.exe", ".."]:
output = check_output(cmd)
do_whatever_you_like_with(output)
Is there any way to display the output of a shell command in Python, as the command runs?
I have the following code to send commands to a specific shell (in this case, /bin/tcsh):
import subprocess
import select
cmd = subprocess.Popen(['/bin/tcsh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
poll = select.poll()
poll.register(cmd.stdout.fileno(),select.POLLIN)
# The list "commands" holds a list of shell commands
for command in commands:
cmd.stdin.write(command)
# Must include this to ensure data is passed to child process
cmd.stdin.flush()
ready = poll.poll()
if ready:
result = cmd.stdout.readline()
print result
Also, I got the code above from this thread, but I am not sure I understand how the polling mechanism works.
What exactly is registered above?
Why do I need the variable ready if I don't pass any timeout to poll.poll()?
Yes, it is entirely possible to display the output of a shell comamand as the command runs. There are two requirements:
1) The command must flush its output.
Many programs buffer their output differently according to whether the output is connected to a terminal, a pipe, or a file. If they are connected to a pipe, they might write their output in much bigger chunks much less often. For each program that you execute, consult its documentation. Some versions of /bin/cat', for example, have the -u switch.
2) You must read it piecemeal, and not all at once.
Your program must be structured to one piece at a time from the output stream. This means that you ought not do these, which each read the entire stream at one go:
cmd.stdout.read()
for i in cmd.stdout:
list(cmd.stdout.readline())
But instead, you could do one of these:
while not_dead_yet:
line = cmd.stdout.readline()
for line in iter(cmd.stdout.readline, b''):
pass
Now, for your three specific questions:
Is there any way to display the output of a shell command in Python, as the command runs?
Yes, but only if the command you are running outputs as it runs and doesn't save it up for the end.
What exactly is registered above?
The file descriptor which, when read, makes available the output of the subprocess.
Why do I need the variable ready if I don't pass any timeout to poll.poll()?
You don't. You also don't need the poll(). It is possible, if your commands list is fairly large, that might need to poll() both the stdin and stdout streams to avoid a deadlock. But if your commands list is fairly modest (less than 5Kbytes), then you will be OK just writing them at the beginning.
Here is one possible solution:
#! /usr/bin/python
import subprocess
import select
# Critical: all of this must fit inside ONE pipe() buffer
commands = ['echo Start\n', 'date\n', 'sleep 10\n', 'date\n', 'exit\n']
cmd = subprocess.Popen(['/bin/tcsh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# The list "commands" holds a list of shell commands
for command in commands:
cmd.stdin.write(command)
# Must include this to ensure data is passed to child process
cmd.stdin.flush()
for line in iter(cmd.stdout.readline, b''):
print line