Develop Reference

Cannot use python subprocess to send input to console application written in C

I want to automatically test a program written in C using python. I have the C file below and the python code below. I compiled my C code into an executable (a.exe) and running it in the command prompt or normally works as expected. When I try to run it with subprocess, though, I receive no output. I tried using p1.communicate() and p1.read() as well, and both trying to send and receive input and output and only output, but to no avail. Is there a way using python (using subprocess or anything else) to run the C code below and interact with it (sending it both input and output until it closes itself)?
Output and input attempt
from subprocess import Popen, PIPE
p = Popen(['a.exe'], shell=True, stdout=PIPE, stdin=PIPE)
for ii in range(10):
value = str(ii) + '\n'
value = bytes(value, 'UTF-8') # Needed in Python 3.
p.stdin.write(value)
p.stdin.flush()
result = p.stdout.readline().strip()
print(result)
value = '-1\n'
value = bytes(value, 'UTF-8') # Needed in Python 3.
p.stdin.write(value)
p.stdin.flush()
result = p.stdout.readline().strip()
print(result)
Output only attempt
from subprocess import Popen, PIPE
p = Popen(['a.exe'], shell=True, stdout=PIPE, stdin=PIPE)
result = p.stdout.readline().strip()
print(result)
Example C file I'm trying to run for reference
#include <stdio.h>
int main() {
int userInput;
printf("Hello, this is a basic program\n");
do{
printf("Enter user input:\n");
scanf("%d", &userInput);
printf("The user input is: %d\n", userInput);
printf("Entering -1 will exit");
}while(userInput != -1);
printf("Left the loop succesfully\n");
return 0;
}

Non-blocking read on subprocess output without flush() and new line

I have a program that I can't modify. This program print data to stdout WITHOUT flushing it or put \n inside and wait for input and so on.
My question is how can I, from a Python script, print in real-time stdout and write into his stdin? I found some ways to write into his stdin without closing the programs but the problem remains for printing his stdout too.
In fact, there are some issues with thread, Popen and fcntl to print in real time his output but they all assumes that the program flush stdout after each print and include \n in their output.
To be clear, let's say I have a program test.c:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main (void)
{
char name[100], age[100], location[100];
fprintf (stdout, "\nWhat is your name : ");
fgets (name, sizeof (name), stdin);
name[strcspn (name, "\n")] = '\0';
fprintf (stdout, "How old are you : ");
fgets (age, sizeof (age), stdin);
age[strcspn (age, "\n")] = '\0';
fprintf (stdout, "Where do you live : ");
fgets (location, sizeof (location), stdin);
location[strcspn (location, "\n")] = '\0';
fprintf (stdout, "\nHello %s, you have %s years old and live in
%s\n\n", name, age, location);
return 0;
}
How can I, from a Python script, output the first fprintf() then writing answer into his stdin and so on in the same way I could simply launch test.c?
The purpose is to control data sent to this program from a script and still having return on what happens.

You may use the following program which is heavily relying on the subprocess package from the standard Python language:
def run_program(cmd, sinput=''):
"""Run the command line and output (ret, sout, serr)."""
import subprocess
proc = subprocess.Popen(cmd,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
sin = proc.stdin
sout, serr = proc.communicate(sinput.encode("utf-8"))
ret = proc.wait()
return (ret, sout, serr)
So, the program you just wrote in C would give the following:
cmd = ['myprogram']
sinput = 'myname\n6\nWhitechapel\n'
(ret, sout, serr) = run_program(cmd, sinput)
Note that you cannot avoid the \n character simply because it is used by your target program to know that you send the answer to a question.

Python : communication with c++ command line program not working when using <cstdio>

I have the following python code, which is supposed to provide the intial input to a C++ program, then take its output and feed it back into it, until the program finishes execution:
comm.py
p = subprocess.Popen('test__1.exe', bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False)
p.stdin.flush()
p.stdout.flush()
x = b'1\n'
while True:
p.stdin.write(x)
p.stdin.flush()
p.stdout.flush()
x = p.stdout.readline()
print(x)
if p.poll() != None:
break
I am currently testing it with two simple C++ programs:
test__1.cpp:
#include <iostream>
using namespace std;
int main()
{
for( int i = 0; i < 3; ++i )
{
int n;
cin >> n;
cout << n+1 << endl;
}
return 0;
}
test__2.cpp
#include <cstdio>
int main()
{
for( int i = 0; i < 3; ++i )
{
int n;
scanf("%d", &n);
printf("%d\n", n+1);
}
return 0;
}
When comm.py opens test__1.exe everything works fine, but when it opens test__2.exe it hangs on the first call to readline().
Note that this problem does not occur when I feed test__2.exe the whole input before execution (i.e. initially set x = '1\n2\n3\n')
What could be causing this issue?
(Also, comm.py should be able to handle any valid C++ program, so only using iostream would not be an acceptable solution.)
EDIT: I also need the solution to work on Windows.

It is caused by the fact that std::endl flushes the ostream and printf does not flush stdout,
as you can see by amending test__2.cpp as follows:
#include <cstdio>
int main()
{
for( int i = 0; i < 3; ++i )
{
int n;
scanf("%d", &n);
printf("%d\n", n+1);
fflush(stdout); //<-- add this
}
return 0;
}
You say that you want to module to work correctly with any C++ program, so
you can't rely upon it flushing the standard output (or standard error)
after every write.
That means you must cause the program's standard streams to be unbuffered
and do so externally to the program itself. You will need to do that
in comm.py.
In Linux (or other host providing GNU Core Utils), you could so by
executing the program via stdbuf, e.g.
import subprocess
cmd = ['/usr/bin/stdbuf', '-i0','-o0', '-e0', './test__2']
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=False)
p.stdin.flush()
p.stdout.flush()
x = b'1\n'
while True:
p.stdin.write(x)
x = p.stdout.readline()
print(x)
if p.poll() != None:
break
which unbuffers all the standard streams. For Windows, you will need
to research how do the same thing. For the time being I don't know.

Interprocess Communication Python [duplicate]

This question already has answers here:
How do I pass a string into subprocess.Popen (using the stdin argument)?
(12 answers)
Closed 8 years ago.
I'm writing a script in Python that should communicate with a software
"ConsoleApplication.exe"(wrote in C); this last one once started waits for a fixed
lenght command(5 bytes) from his "stdin" and generates (after 2-3 seconds) on
his "stdout" an output that I should read on my Python script.
#
//This is the "ConsoleApplication.c" file
#include <stdio.h>
#include <function.h>
char* command[5];
int main()
{
while(1)
{
scanf("%s\n", &command);
output = function(command);
print("%s\n", output);
}
}
#
#this is Python code
import subprocess
#start the process
p = subprocess.Popen(['ConsoleApplication.exe'], shell=True, stderr=subprocess.PIPE)
#command to send to ConsoleApplication.exe
command_to_send = "000648"
#this seems to work well but I need to send a command stored into a buffer and if Itry
#to use sys.stdout.write(command_to_send)nothing will happen. The problem seem that
#sys.stdout.write expect an object I/O FILE
while True:
out = p.stderr.read(1)
if out == '' and p.poll() != None:
break
if out != '':
sys.stdout.write(out)
sys.stdout.flush()
#
Any suggestions? How can I fix it?
I tried to use
stdout = p.communicate(input='test\n')[0]
but Im getting the following error at runtime:
"TypeError: 'str' does not support the buffer interface"
I also tried this
from subprocess import Popen, PIPE, STDOUT
p = Popen(['ConsoleApplication.exe'], stdout=PIPE, stdin=PIPE, stderr=PIPE)
out, err = p.communicate(input='00056\n'.encode())
print(out)
out, err = p.communicate(input='00043\n'.encode())
print(out)
but I get this error:
"ValueError: Cannot send input after starting communication"

Looks like this question has your solution
Python - How do I pass a string into subprocess.Popen (using the stdin argument)?
Use the Popen.communicate() method

Python subprocess module: parent child communication not working

I'm trying to run the following code as a subprocess
#include<stdio.h>
int main()
{
int a;
printf("Hello\n");
fprintf(stderr, "Hey\n");
scanf("%d", &a);
printf("%d\n", a);
return 0;
}
This script works fine: write to stdin, read from stdout and from stderr.
#!/usr/bin/python
import subprocess
p1=subprocess.Popen("/mnt/test/a.out", stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
p1.stdin.write('1\n')
print p1.stdout.readline()
print p1.stderr.readline()
print p1.stdout.readline()
But this script fails to read any output from stdout and gets blocked there even though the C program does print to stdout before demanding any input. Why is it that I'm unable to read anything from stdout?
#!/usr/bin/python
import subprocess
p1=subprocess.Popen("/mnt/test/a.out", stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
print p1.stdout.readline()
p1.stdin.write('1\n')
print p1.stderr.readline()
print p1.stdout.readline()

You need to flush the stream first. This will make sure all data is actually written to the stream.
#include<stdio.h>
int main()
{
int a;
printf("Hello\n");
fprintf(stderr, "Hey\n");
fflush(stdout); // <--
scanf("%d", &a);
printf("%d\n", a);
return 0;
}
By default, stderr is unbuffered, which is why you don't need to flush it. stdout however is fully buffered, unless it points to a terminal, then it line-buffered (i.e. the \n would automatically trigger flushing.
Have a look here, for setbuf() and setvbuf().

I don't see something like
stdout_data, stderr_data = p1.communicate()
in your code
Popen.communicate(input=None)
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate. The optional input argument should be a string to be sent to the child process, or None, if no data should be sent to the child.
communicate() returns a tuple (stdoutdata, stderrdata).
Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.
Note The data read is buffered in memory, so do not use this method if the data size is large or unlimited.
See docs.python.org
A function I keep in my utility-belt to wrap calling an external program using subprocess is this (modify to suit your needs):
def __run(self, cmd):
"""wrapper, makes it easy to call an external program.
return the result as a newline-separated list
"""
args = shlex.split(cmd)
try:
p = subprocess.Popen(args, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
retdata = p.communicate()[0]
p.wait()
except OSError, e:
print >>sys.stderr, "Execution failed:", e
return (p.returncode, retdata.split('\n'))
Just place your command as you would write it on the cmd-line in a variable an call the function e.g.:
cmd = r'git branch -r'
data = self.__run(cmd)