This should be a basic problem, but I'm scratching my head on this one...
I'm trying to build the skeleton of a python script, part of which will use a loop to pipe strings into a camera SDK's console .exe (which opens, waits for user input, captures/saves an image with filename specified by input, waits for next input or to be killed).
I've built some test code which uses another simple .exe program that opens, takes user input, writes it to a .txt, and waits for next input or to be killed:
from subprocess import Popen, PIPE
p = Popen("testinput1.exe", stdin=PIPE, stdout=PIPE, stderr=PIPE, shell=False)
bytes = str.encode("no")
for n in range(1, 5):
p.stdin.write(bytes)
p.stdin.close()
p.terminate()
However, this code on its own doesn't successfully open the process, write to it, and close - mysteriously, it doesn't do anything at all.
If I remove the line
p.terminate()
I can successfully pipe my "nonononono" into the text file, but the subprocess isn't closed, and the .txt file grows by 1KB/sec until I close the "testinput1.exe"
Do I need to put some kind of wait before p.terminate()? Or am I going about the whole process incorrectly?
Related
I am using Python to script running an exe program.
If we open the exe program in the shell, we could enter different command such as "a", "b", "c" in the program. These commands can not be passed as flags into the exe program. I want to use Python to script running this exe program for many times, with custom exe-program specific input.
But if I run the "program.exe" with
p = subprocess.call(['program.exe'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
Python won't terminate. Can I achieve this purpose with subprocess in Python?
Beware: subprocess.call will not return before the child process has terminated. So you have no possibility to write anything to the standard input of the child.
If you can prepare the bunch of commands in advance, and if output has no risk to fill the system buffer, you can still use call that way:
cmds = "a\nb\nc\n"
p = subprocess.call(['program.exe'],
stdin=io.StringIO(cmds),
stdout=subprocess.PIPE,
)
But the more robust way is to directly use the Popen constructor, and then feed the input:
p = subprocess.Popen(['program.exe'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
)
p.stdin.write("a\n");
p.stdin.write("b\n");
...
p.stdin.close();
p.wait();
If you know that one subcommand will generate very large output you can read it before sending the next one. Beware to avoid being blocked waiting an input that the child has still not sent...
First, you have to use p = subprocess.Popen(…) in order to get the subprocess object. subprocess.call(…) would give you just the return status, and that's not enough.
If p is your connection object, you can send your commands to p.stdin, such as p.stdin.write("a\n"), and then read out p.stdout() until the next indication that the command output is finished. How you detect this depends on said program.
Then you can send the next command and read its output.
At the end, you can do p.stdin.close() in order to signal an EOF ot the other process, and then it should terminate.
I'm trying to spawn a subprocess that should still be running after the main process closed. This part works fine, but if I redirect the output of this process to a file, I can't start the script a second time because the process still blocks the log file.
This short example demonstrates the problem:
In this case the second process is "notepad" and is started by "other.cmd". While the main process/script is "test_it.py" which is started by "start_it.cmd".
start_it.cmd
#python test_it.py > test.log
test_it.py
import subprocess
from subprocess import DEVNULL, STDOUT
subprocess.Popen(["other.cmd"], stdin=DEVNULL, stdout=DEVNULL, stderr=STDOUT)
other.cmd
start notepad
When start_it.cmd is executed the second time, it will fail with this error message "The process cannot access the file because it is being used by another process".
How can I start the subprocess so that it doesn't block the log file?
A solution using a pipe.
multiplexer.py
with open('log.txt', 'a') as outputFile:
while True:
data = sys.stdin.read(1024)
if None == data:
break
outputFile.write(data)
start_it.cmd
#python test_it.py | python multiplexer.py
Everything else stays the same.
I found a solution that is close to what I originally intended:
subprocess.Popen("explorer other.cmd", shell=True)
By letting the explorer start the .cmd file this succesfully detaches the called .cmd from the original process. And thus doesn't keep the log file open.
I have a Python helper function to run grunt commands in parallel, using Popen to handle subprocesses. The purpose is communication over CLI. The problem starts when any user input is required for all those processes, e.g. file path, password, 'yes/no' decision:
Enter file path: Enter file path: Enter file path: Enter file path: Enter file path: Enter file path: Enter file path:
Everything up-to-date
Grunt task completed successfully.
User provides input once, one process completes successfully and all others never finish executing.
Code:
from subprocess import check_output, Popen
def run_grunt_parallel(grunt_commands):
return_code = 0
commands = []
for command in grunt_commands:
with tempfile.NamedTemporaryFile(delete=False) as f:
app = get_grunt_application_name(' '.join(command))
commands.append({'app': app, 'process': Popen(command, stdout=f)})
while len(commands):
sleep(5)
next_round = []
for command in commands:
rc = command['process'].poll()
if rc == None:
next_round.append(command)
else:
if rc == 0:
else:
return_code = rc
commands = next_round
return return_code
Is there a way to make sure that user can provide all necessary input for each process?
What you want is almost (if not entirely) impossible. But if you can recognize prompts in a prefix-free fashion (and, if it varies, know from them how many lines of input they expect), you should be able to manage it.
Run each process with two-way unbuffered pipes:
Popen(command, stdin=subprocess.PIPE,
stdout=f, stderr=subprocess.PIPE, bufsize=0)
(Well-behaved programs prompt on standard error. Yours seem to do so, since you showed the prompts despite the stdout=f; if they don’t do so reliably, you get to read that from a pipe as well, search for prompts in it, and copy it to a file yourself.)
Unix
Set all pipes non-blocking. Then use select to monitor the stderr pipes for all processes. (You might try selectors instead.) Buffer what you read separately for each process until you recognize a prompt from one. Then display the prompt (identifying the source process) and accept input from the user—if the output between prompts fits in the pipe buffers, this won’t slow the background work down. Put that user input in a buffer associated with that process, and add its stdin pipe to the select.
When a stdin pipe shows ready, write to it, and remove it from the set if you finish doing so. When a read from a pipe returns EOF, join the corresponding process (or do so in a SIGCHLD handler if you worry that a process might close its end early).
Windows
Unless you have a select emulation available that supports pipes, you’ll have to use threads—one for each process, or one for each pipe if a process might produce output after writing a prompt and before reading the response. Then use a Queue to post prompts as messages to the main thread, which can then use (for example) another per-process Queue to send the user input back to the thread (or its writing buddy).
This works on any threading-supporting platform and has the potential advantage of not relying on pipe buffers to avoid stalling talkative processes.
I have a python script that opens a .exe program using the subprocess module. This .exe program is an infinitely iterative script, in that it will continue to print the results of each iteration until the user closes the window. Every so often, it prints the results of the iteration into a file, replacing the previous data in the file.
My aims here are to:
Run the .exe program, and test for the existence of the file it outputs.
Once the file has been shown to exist, I need to run a test on the file to see if the iteration has converged to within a given tolerance. Once the iteration has converged, I need to kill the .exe subprocess.
This is my current code. It is designed to kill the subprocess once the iterate file has been created:
import subprocess
from subprocess import Popen, PIPE
fileexists = False
iteratecomms = Popen('iterate.exe', stdout=PIPE, stdin=PIPE, stderr=PIPE)
# Begin the iteration. Need to select options 1 and then 1 again at program menu
out, err = iteratecomms.communicate("1\n1\n".encode())
while (fileexists == False):
fileexists = os.path.exists(filelocation)
else:
Popen.kill(iteratecomms)
I know that this is incorrect; the issue is that as soon as I start the out, err = iteratecomms.communicate("1\n1\n".encode()) line, the program begins iterating, and does not move on to the next set of python code. Essentially, I need to start the .exe program, and at the same time test to see if the file has been created. I can't do this, however, because the program runs indefinitely.
How could I get around this? I have assumed that moving on to step 2 (testing the file and killing the subprocess under certain conditions) would not take too much work on top of this; if this is not true, how would I go about completing all of my aims?
Thank you very much for the help!
Edit: Clarified that the external file is overwritten.
I would use the multiprocessing module.
pool = multiprocessing.Pool()
def start_iteration():
return Popen('iterate.exe', stdout=PIPE, stdin=PIPE, stderr=PIPE)
pool.apply_async(start_iteration)
while (fileexists == False):
fileexists = os.path.exists(filelocation)
Popen.kill(???)
The only problem now is that you'll have to somehow find the PID of the process without waiting for Popen to return (because Popen should never return.)
Assuming that you're trying to continuously trying to read this file I would suggest running a tail on the file in question. This can be done from a separate terminal in any *nix family OS, but otherwise I would check out this article for a Python implementation:
http://code.activestate.com/recipes/157035-tail-f-in-python/
After that if you want to kill the program running you should just be able to call terminate on the process running:
import subprocess
sub = subprocess.popen(#Whatever)
#Do something
sub.terminate()
import subprocess
import sys
proc = subprocess.Popen(["program.exe"], stdin=subprocess.PIPE) #the cmd program opens
proc.communicate(input="filename.txt") #here the filename should be entered (runs)
#then the program asks to enter a number:
proc.communicate(input="1") #(the cmd stops here and nothing is passed)
proc.communicate(input="2") # (same not passing anything)
how do i pass and communicate with the cmd using python.
Thanks. (using windows platform)
The docs on communicate() explain this:
Interact with process: Send data to stdin. Read data from stdout and
stderr, until end-of-file is reached. Wait for process to terminate.
communicate() blocks once the input has been sent until the program finishes executing. In your example, the program waits for more input after you send "1", but Python waits for it to exit before it gets to the next line, meaning the whole thing deadlocks.
If you want to read and write a lot interchangeably, make pipes to stdin/stdout and write/read to/from them.