I have an interactive command line exe file on Windows, wrote by someone else. When the program has an exception, it terminates and all my inputs to the program are lost.
So I'm writing a python program that calls a blocking subprocess with subprocess.run() and captures all inputs to stdin for backup, without affecting the interaction to the subprocess. But I didn't find a way to capture the inputs. How do I do this in Python?
An example (but not working) code would be like:
import subprocess
process = subprocess.run(['program.exe'], stderr = subprocess.PIPE,text=True)
# Interact with the program. Wait for it to finish.
if subprocess_object.returncode != 0:
open('input_backup.txt','w').write(process.stdin) # Won't work
open('error_backup.txt','w').write(process.stderr)
One idea I had is to override an StringIO object and give it two stream, one sys.stdin and one file stream to write() to. Then assign sys.stdin to a instance of that overrided class. But I didn't find a way to do it.
Related
I want to figure out a way to programmatically avoid the builtin input() method stopping and waiting for user input.
Here is a snippet showing what I'm trying to do:
import sys
from threading import Thread
def kill_input():
sys.stdout.write('\n')
sys.stdout.flush() # just to make sure the output is really written to stdout and not bufferized
t = Thread(target=kill_input)
t.start()
foo = input('Press some key')
print('input() method has been bypassed')
Expected behavior: the script executes and terminates without waiting for enter key to be pressed.
On the contrary, what's happening is the program stopping to wait for user entering some input.
In my thoughts input() should read the newline character ('\n') printed on stdout by the other thread and terminates by executing the final print statement. That thread should simulate a user pressing the enter key. I do not understand what's going on behind
Maybe one other possible way is to close the stdin file descriptor from the non-main thread and catching the exception on the main one.
def kill_input():
sys.stdin.close()
Possibly I would like to avoid this option and rather understand what's going on behind this logic and find a way to force the main thread to read some mock characters from the stdin.
Edit - using subprocess module
Based on these related posts I've had a look to the subprocess module. I've thought this is the case for the Popen class to come in handy, so I've modified my script to exploit pipes
import sys
from subprocess import Popen, PIPE
def kill_input():
proc = Popen(['python3', '-c', 'pass'], stdin=PIPE)
proc.stdin.write('some text just to force parent proc to read'.encode())
proc.stdin.flush()
proc.stdin.close()
t = Thread(target=kill_input)
t.start()
sys.stdin.read()
print('input() method has been bypassed')
From my understanding, that should create a process with the Popen (the commend python3 -c 'pass' acts like a placeholder) whose (should?) stdin is a unix pipe opened with the parent process.
What I'm expecting is anything written to the child process stdin to go straight to the stdin of the parent in order to be read by the sys.stdin.read(). So the program shouldn't stop to wait for any user input and it should terminates instantly. Unfortunately, it doesn't happen and the script still waits for me pressing enter. I cannot really find out a workaround for this.
[Python version: 3.8.5]
In your first piece of code, you were writing to sys.stdout, which by default won't effect the contents of sys.stdin. Also, by default, you can't directly write to sys.stdin, but you can change it to a different file. To do this, you can use os.pipe(), which will return a tuple of a file descriptor for reading from the new pipe, and a file descriptor for writing to the pipe.
We can then use os.fdopen on these file descriptors, and assign sys.stdin to the read end of the pipe, while in another thread we write to the other end of the pipe.
import sys
import os
from threading import Thread
fake_stdin_read_fd, fake_stdin_write_fd = os.pipe()
fake_stdin_read = os.fdopen(fake_stdin_read_fd, 'r')
fake_stdin_write = os.fdopen(fake_stdin_write_fd, 'w')
sys.stdin = fake_stdin_read
def kill_input():
fake_stdin_write.write('hello\n')
fake_stdin_write.flush()
thread = Thread(target=kill_input)
thread.start()
input()
print('input() method has been bypassed!')
Consider the following Python code:
import io
import time
import subprocess
import sys
from thread import start_new_thread
def ping_function(ip):
filename = 'file.log'
command = ["ping", ip]
with io.open(filename, 'wb') as writer, io.open(filename, 'rb', 1) as reader:
process = subprocess.Popen(command, stdout=writer)
while process.poll() is None:
line = reader.read()
# Do something with line
sys.stdout.write(line)
time.sleep(0.5)
# Read the remaining
sys.stdout.write(reader.read())
ping_function("google.com")
The goal is to run a shell command (in this case ping, but it is not relevant here) and to process the output in real time, which is also saved on a log file.
In other word, ping is running in background and it produces output on the terminal every second. My code will read this output (every 0.5 seconds), parse it and take some action in (almost) real time.
Realtime here means that I don't want to wait the end of the process to read the output. In this case actually ping never completes so an approach like the one I have just described is mandatory.
I have tested the code above and it actually works OK :)
Now I'd like to tun this in a separate thread, so I have replaced the last line with the following:
from thread import start_new_thread
start_new_thread(ping_function, ("google.com", ))
For some reason this does not work anymore, and the reader always return empty strings.
In particular, the string returned by reader.read() is always empty.
Using a Queue or another global variable is not going to help, because I am having problems even to retrieve the data in the first place (i.e. to obtain the output of the shell command)
My questions are:
How can I explain this behavior?
Is it a good idea to run a process inside a separate thread or I should use a different approach? This article suggests that it is not...
How can I fix the code?
Thanks!
You should never fork after starting threads. You can thread after starting a fork, so you can have a thread handle the I/O piping, but...
Let me repeat this: You should never fork after starting threads
That article explains it pretty well. You don't have control over the state of your program once you start threads. Especially in Python with things going on in the background.
To fix your code, just start the subprocess from the main thread, then start threading. It's perfectly OK to process the I/O from the pipes in a thread.
From what I can tell, execv overtakes the current process, and once the called executable finishes, the program terminates. I want to call execv multiple times within the same script, but because of this, that cannot be done.
Is there an alternative to execv that runs within the current process (i.e. prints to same stdout) and won't terminate my program? If so, what is it?
Yes, use subprocess.
os.execv* is not approporiate for your task, from doc:
These functions all execute a new program, replacing the current
process; they do not return. On Unix, the new executable is loaded
into the current process, and will have the same process id as the
caller.
So, as you want the external exe to print to the same output, this is what you might do:
import subprocess
output = subprocess.check_output(['your_exe', 'arg1'])
By default, check_output() only returns output written to standard output. If you want both standard output and error collected, use the stderr argument.
output = subprocess.check_output(['your_exe', 'arg1'], stderr=subprocess.STDOUT)
The subprocess module in the stdlib is the best way to create processes.
import subprocess
import sys
proc = subprocess.Popen(["program.exe"], stdin=subprocess.PIPE) #the cmd program opens
proc.communicate(input="filename.txt") #here the filename should be entered (runs)
#then the program asks to enter a number:
proc.communicate(input="1") #(the cmd stops here and nothing is passed)
proc.communicate(input="2") # (same not passing anything)
how do i pass and communicate with the cmd using python.
Thanks. (using windows platform)
The docs on communicate() explain this:
Interact with process: Send data to stdin. Read data from stdout and
stderr, until end-of-file is reached. Wait for process to terminate.
communicate() blocks once the input has been sent until the program finishes executing. In your example, the program waits for more input after you send "1", but Python waits for it to exit before it gets to the next line, meaning the whole thing deadlocks.
If you want to read and write a lot interchangeably, make pipes to stdin/stdout and write/read to/from them.
Again, the same question.
The reason is - I still can't make it work after reading the following:
Real-time intercepting of stdout from another process in Python
Intercepting stdout of a subprocess while it is running
How do I get 'real-time' information back from a subprocess.Popen in python (2.5)
catching stdout in realtime from subprocess
My case is that I have a console app written in C, lets take for example this code in a loop:
tmp = 0.0;
printf("\ninput>>");
scanf_s("%f",&tmp);
printf ("\ninput was: %f",tmp);
It continuously reads some input and writes some output.
My python code to interact with it is the following:
p=subprocess.Popen([path],stdout=subprocess.PIPE,stdin=subprocess.PIPE)
p.stdin.write('12345\n')
for line in p.stdout:
print(">>> " + str(line.rstrip()))
p.stdout.flush()
So far whenever I read form p.stdout it always waits until the process is terminated and then outputs an empty string. I've tried lots of stuff - but still the same result.
I tried Python 2.6 and 3.1, but the version doesn't matter - I just need to make it work somewhere.
Trying to write to and read from pipes to a sub-process is tricky because of the default buffering going on in both directions. It's extremely easy to get a deadlock where one or the other process (parent or child) is reading from an empty buffer, writing into a full buffer or doing a blocking read on a buffer that's awaiting data before the system libraries flush it.
For more modest amounts of data the Popen.communicate() method might be sufficient. However, for data that exceeds its buffering you'd probably get stalled processes (similar to what you're already seeing?)
You might want to look for details on using the fcntl module and making one or the other (or both) of your file descriptors non-blocking. In that case, of course, you'll have to wrap all reads and/or writes to those file descriptors in the appropriate exception handling to handle the "EWOULDBLOCK" events. (I don't remember the exact Python exception that's raised for these).
A completely different approach would be for your parent to use the select module and os.fork() ... and for the child process to execve() the target program after directly handling any file dup()ing. (Basically you'd be re-implement parts of Popen() but with different parent file descriptor (PIPE) handling.
Incidentally, .communicate, at least in Python's 2.5 and 2.6 standard libraries, will only handle about 64K of remote data (on Linux and FreeBSD). This number may vary based on various factors (possibly including the build options used to compile your Python interpreter, or the version of libc being linked to it). It is NOT simply limited by available memory (despite J.F. Sebastian's assertion to the contrary) but is limited to a much smaller value.
Push reading from the pipe into a separate thread that signals when a chunk of output is available:
How can I read all availably data from subprocess.Popen.stdout (non blocking)?
The bufsize=256 argument prevents 12345\n from being sent to the child process in a chunk smaller than 256 bytes, as it will be when omitting bufsize or inserting p.stdin.flush() after p.stdin.write(). Default behaviour is line-buffering.
In either case you should at least see one empty line before blocking as emitted by the first printf(\n...) in your example.
Your particular example doesn't require "real-time" interaction. The following works:
from subprocess import Popen, PIPE
p = Popen(["./a.out"], stdin=PIPE, stdout=PIPE)
output = p.communicate(b"12345")[0] # send input/read all output
print output,
where a.out is your example C program.
In general, for a dialog-based interaction with a subprocess you could use pexpect module (or its analogs on Windows):
import pexpect
child = pexpect.spawn("./a.out")
child.expect("input>>")
child.sendline("12345.67890") # send a number
child.expect(r"\d+\.\d+") # expect the number at the end
print float(child.after) # assert that we can parse it
child.close()
I had the same problem, and "proc.communicate()" does not solve it because it waits for process terminating.
So here is what is working for me, on Windows with Python 3.5.1 :
import subprocess as sp
myProcess = sp.Popen( cmd, creationflags=sp.CREATE_NEW_PROCESS_GROUP,stdout=sp.PIPE,stderr=sp.STDOUT)
while i<40:
i+=1
time.sleep(.5)
out = myProcess.stdout.readline().decode("utf-8").rstrip()
I guess creationflags and other arguments are not mandatory (but I don't have time to test), so this would be the minimal syntax :
myProcess = sp.Popen( cmd, stdout=sp.PIPE)
for i in range(40)
time.sleep(.5)
out = myProcess.stdout.readline()