Develop Reference

Using python subprocess to interact with a command line application that asks data on an unordered way

I’m writing a python program that needs interact with another windows console application. I’m using subprocess for that but I’m having some troubles because this windows app asks for inputs randomly on every call.
For example:
app.exe on first call ask on this order:
Name: input here
Place: input here
Age: input here
app.exe on second call ask on this order:
Place: input here
Name: input here
Age: input here
So when I use:
process = subprocess.Popen("app.exe", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Due this random presentation I need to read every line before use process.stdin.write() flush() and here is the problem, the process.stdout.readline() method is blocked since apparently the line does not contain an end-of-line character due the input it is waiting.
I was reading character by character using process.stdout.readline(1).decode() but I guess there is a better way to face this.
Please if anyone knows how to deal with this, I will appreciate your help
Best Regads!
Update*
Another example
process.stdout.readline().decode() read the line if this line is just text and there is no cursor waiting for an input.
For example:
Hi this is the first line on the console.
Enter your name on this line: input_here
Here the first line is read without issue. But the second it’s not read because is waiting for an input.

I don't really know this will help you to solve your problem but here's the idea :
process.stdout.readline().decode()
This line is working for single line perfectly then check one by one, if "Name" is in it or not, "Place" like
if "Name" in <>:
# Something
elif "Place" in <>:
# Something
else:
# Something

What's behind sys.stdin.readlines()

Question 1:
I have a piece of code like this (Python2.7):
for line in sys.stdin.readlines():
print line
When I run this code, input a string in the terminal and press Enter key, nothing happens. 'print line' doesn't work.
So I imagine there is buffer for sys.stdin.readlines(), but I wonder how does it work? Can I flush it so every time a line is given, 'print line' can be executed imeediatly?
Question2: What's the difference between these two lines:
for line in sys.stdin:
for line in sys.stdin.readline():
I found their behavior is a little different. If I use ctrl+D to terminate the input, I have to press ctrl+D twice in the first case before it's really terminated. While in the second case, only one ctrl+D is enough.

CTRL-D sends the EOF (end of file) control character to stdin in an interactive shell. Usually, you feed a file to the stdin of a process via redirection (e.g. myprogram < myfile), but if you are interactively typing characters into stdin of a process, you need to tell it when to stop reading the "file" you are actively creating.
sys.stdin.readlines waits for stdin to complete (via an EOF control character), then conveniently splits the entire stdin contents (flushed) before the EOF into a list of tokens delimited by newline characters. When you hit ENTER, you send a \n character, which is rendered for you as a new line, but does NOT tell stdin to stop reading.
Regarding the other two lines, I think this might help:
Think of the sys.stdin object as a file. When you EOF, you save that file and then you are not allowed to edit it anymore because it leaves your hands and belongs to stdin. You can perform functions on that file, like readlines, which is a convenient way to say "I want a list, and each element is a line in that file". Or, you can just read one line from it with readline, in which case the for loop would be only iterating over the characters in that line.
What's going on behind the scenes?
Internally, the reference to sys.stdin blocks execution until EOF is received in sys.stdin. Then it becomes a file-like object stored in memory with a read pointer pointing to the beginning.
When you call just readline, the pointer reads until it hits a \n character, returns to you what it just traversed over, and stays put, waiting for you to move it again. Calling readline again will cause the pointer to move until the next \n, if it exists, else EOF.
readlines is really telling the pointer to traverse all the way (\n is functionally meaningless) from its current position (not necessarily beginning of file) until it sees EOF.
Try it out!
Trying it out is the best way to learn.
To see this behavior in action, try making a file with 10 lines, then redirect it to the stdin of a python script that prints sys.stdin.readline 3 times, then print sys.stdin.readlines. You'll see 3 lines printed out then a list containing 7 elements :)

Eclipse/PyDev treats newlines pasted into its console as instructions, but I want it to parse them as part of a long string

I am working on a Python script to automate some repetitive text-fiddling tasks I need to do. I use PyDev as a plugin for Eclipse as my IDE.
I need the script to accept user input pasted from the clipboard. The input will typically be many lines long, with many newline characters included.
I currently have the script asking for input as follows:
oldTableString = raw_input('Paste text of old table here:\n')
The console correctly displays the prompt and waits for user input. However, once I paste text into the console, it appears to interpret any newline characters in the pasted text as presses of the enter button, and executes the code as if the only input it received was the first line of the pasted text (before the first newline character), followed by a press of the enter key (which it interprets as a cue that I'm done giving it input).
I've confirmed that it's only reading the first line of the input via the following line:
print oldTableString
...which, as expected, prints out only the first line of whatever I paste into the console.
How can I get Eclipse to recognize that I want it to parse the entirety of what I paste into the console, newlines included, as a single string?
Thanks!

text = ""
tmp = raw_input("Enter text:\n")
while tmp != "":
text += tmp + "\n"
tmp = raw_input()
print text
This works but you have to press enter one more time.

What about reading directly from the clipboard or looping over every line until it receives a termination symbol or times out. Also, is it important to make it work under Eclipse? Does it work when executed directly?

Call Perl script from Python constantly returning values

I found a question on this site which showed me how to call a Perl script from Python. I'm currently using the following lines of code to achieve this:
pipe = subprocess.Popen(["perl", "./Perl_Script.pl", param], stdout=subprocess.PIPE)
result = pipe.stdout.read()
This works perfectly, but the only issue is that the Perl script takes a few minutes to run. At the end of the Perl script, I use a simple print statement to print my values I need to return back to Python, which gets set to the result variable in Python.
Is there a way I can include more print statements in my Perl script every few seconds that can get returned to Python continuously (instead of waiting a few minutes and returning a long list at the end)?
Ultimately, what I'm doing is using the Perl script to obtain data points that I then send back to Python to plot an eye diagram. Instead of waiting for minutes to plot the eye diagram when the Perl script is finished running, I'd like to return segments of the data to Python continuously, allowing my plot to update every few seconds.

The default UNIX stdio buffer is at least 8k. If you're writing less than 8k, you'll end up waiting until the program ends before the buffer is flushed.
Tell the Perl program to stop buffering output, and probably tell python not to buffer input through the pipe.
$| = 1;
to un-buffer STDOUT in your Perl program.

You need two pieces: To read a line at a time in Python space and to emit a line at a time from Perl. The first can be accomplished with a loop like
while True:
result = pipe.stdout.readline()
if not result:
break
# do something with result
The readline blocks until a line of text (or EOF) is received from the attached process, then gives you the data it read. So long as each chunk of data is on its own line, that should work.
If you run this code without modifying the Perl script, however, you will not get any output for quite a while, possibly until the Perl script is finished executing. This is because Perl block-buffers output to a pipe by default. You can tell it to flush the buffer more often by changing a global variable in the scope in which you are printing:
use English qw(-no_match_vars);
local $OUTPUT_AUTOFLUSH = 1;
print ...;
See http://perl.plover.com/FAQs/Buffering.html and http://perldoc.perl.org/perlvar.html .

pipe.stdout.read() tries to read the whole stream, so it will block until perl is finished.
Try this:
line=' '
while line:
line = pipe.stdout.readline()
print line,

Using textfile as stdin in python under windows 7

I'm a win7-user.
I accidentally read about redirections (like command1 < infile > outfile) in *nix systems, and then I discovered that something similar can be done in Windows (link). And python is also can do something like this with pipes(?) or stdin/stdout(?).
I do not understand how this happens in Windows, so I have a question.
I use some kind of proprietary windows-program (.exe). This program is able to append data to a file.
For simplicity, let's assume that it is the equivalent of something like
while True:
f = open('textfile.txt','a')
f.write(repr(ctime()) + '\n')
f.close()
sleep(100)
The question:
Can I use this file (textfile.txt) as stdin?
I mean that the script (while it runs) should always (not once) handle all new data, ie
In the "never-ending cycle":
The program (.exe) writes something.
Python script captures the data and processes.
Could you please write how to do this in python, or maybe in win cmd/.bat or somehow else.
This is insanely cool thing. I want to learn how to do it! :D

If I am reading your question correctly then you want to pipe output from one command to another.
This is normally done as such:
cmd1 | cmd2
However, you say that your program only writes to files. I would double check the documentation to see if their isn't a way to get the command to write to stdout instead of a file.
If this is not possible then you can create what is known as a named pipe. It appears as a file on your filesystem, but is really just a buffer of data that can be written to and read from (the data is a stream and can only be read once). Meaning your program reading it will not finish until the program writing to the pipe stops writing and closes the "file". I don't have experience with named pipes on windows so you'll need to ask a new question for that. One down side of pipes is that they have a limited buffer size. So if there isn't a program reading data from the pipe then once the buffer is full the writing program won't be able to continue and just wait indefinitely until a program starts reading from the pipe.
An alternative is that on Unix there is a program called tail which can be set up to continuously monitor a file for changes and output any data as it is appended to the file (with a short delay.
tail --follow=textfile.txt --retry | mycmd
# wait for data to be appended to the file and output new data to mycmd
cmd1 >> textfile.txt # append output to file
One thing to note about this is that tail won't stop just because the first command has stopped writing to the file. tail will continue to listen to changes on that file forever or until mycmd stops listening to tail, or until tail is killed (or "sigint-ed").
This question has various answers on how to get a version of tail onto a windows machine.

import sys
sys.stdin = open('textfile.txt', 'r')
for line in sys.stdin:
process(line)

If the program writes to textfile.txt, you can't change that to redirect to stdin of your Python script unless you recompile the program to do so.
If you were to edit the program, you'd need to make it write to stdout, rather than a file on the filesystem. That way you can use the redirection operators to feed it into your Python script (in your case the | operator).
Assuming you can't do that, you could write a program that polls for changes on the text file, and consumes only the newly written data, by keeping track of how much it read the last time it was updated.

When you use < to direct the output of a file to a python script, that script receives that data on it's stdin stream.
Simply read from sys.stdin to get that data:
import sys
for line in sys.stdin:
# do something with line