Develop Reference

How do I wait for a subprocess to finish, store it's output and add a prefix (without a new line) to the output?

I have a Python(3) script that's calling an external command using the subprocess.call([...]) method:
import subprocess
print("Prefix: ")
subprocess.call(["cmd", "--option", "filename.x"])
The command executes without any errors but the problem is the output.
The output isn't "uniform" and sometimes the program will output:
Program output...
Prefix:
And other times the output will be:
Prefix:
Program output....
The result I'm looking for is:
Prefix: Program output...
I know that in order to achieve this result I need to wait for the subprocess to finish, store it's output and then print the prefix (without \n) with the subprocess' output after it, I just can't figure out how to do it.
Thanks.

First you need to import the sys module, so you can use sys.stdout's write and flush methods.
import sys
You'll also need to import the subprocess module so you can use the subprocess.check_output method.
import subprocess
Use sys.stdout.write instead of print:
sys.stdout.write("Prefix: ")
Then you'll need to replace subprocess.call with subprocess.check_output, which runs the given command and waits for the output.
response = subprocess.check_output(["cmd", "--option", "filename.x"])
NOTE: you need to decode the response because it's a bytes object and not a string.
sys.stdout.write(response.decode("UTF-8"))
And finally you need to flush the output:
sys.stdout.flush()
Here is the final result:
import sys, subprocess
sys.stdout.write("Prefix: ")
response = subprocess.check_output(["cmd", "--option", "filename.x"])
sys.stdout.write(response.decode("UTF-8"))
sys.stdout.flush()
Good luck, hopefully no one else will stumble on this question like I did.

Run a subprocess in python and both show the output in "real time" and save it to a variable

I would like to be able to run a subprocess from python code and both see the output in real time and once the process is finished have the output in a variable
Right now I do one of either two things
1) Run subprocess using subprocess.call in that case I get the output in real time but I don't have at the end the output in a variable (I want to parse it and extract values from it)
2) Run subprocess using subprocess.check_output in that case I have the output in a variable but if I want to see it then I have to print it "manually"
Is there a way to get both things "together" ?
Hope it is clear, I can add my code if you need
Thanks !!!
EDIT:
This is my current code
I added a timeout optional parameter (Default value is 1200 and also deal with shell (For some reason same commands that work in Linux do not work in Windows if I don't have the shell=True) the "mode" parameter is the one that I use to differentiate the cases where I want the output in "real time" and I don't have to parse it and the other cases
I was wondering if there is a cleaner and better way to achieve same results

Assuming you are trying to run some command your_command You can use the following:
some_proc = subprocess.Popen(['your_command'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
The stdout=subprocess.PIPE does stdout the result on success. Afterwards, you can access the output as follows:
store_in_var = some_proc.stdout
Now you can parse your store_in_var

import subprocess
from subprocess import PIPE
comd = input('command here : ')
comds = comd.split(' ')
f = subprocess.run(comds, shell= True,stdout=PIPE, stderr=PIPE)
result = f.stdout.decode()
errors = f.stderr.decode()

Passing arguments/strings into already running process - Python 2.7

I have two scripts in Python.
sub.py code:
import time
import subprocess as sub
while 1:
value=input("Input some text or number") # it is example, and I don't care about if it is number-input or text-raw_input, just input something
proces=sub.Popen(['sudo', 'python', '/home/pi/second.py'],stdin=sub.PIPE)
proces.stdin.write(value)
second.py code:
import sys
while 1:
from_sub=sys.stdin()#or sys.stdout() I dont remember...
list_args.append(from_sub) # I dont know if syntax is ok, but it doesn't matter
for i in list_arg:
print i
First I execute sub.py, and I input something, then second.py file will execute and printing everything what I inputed and again and again...
The thing is I don't want to open new process. There should be only one process. Is it possible?
Give me your hand :)

This problem can be solved by using Pexpect. Check my answer over here. It solves a similar problem
https://stackoverflow.com/a/35864170/5134525.
Another way to do that is to use Popen from subprocess module and setting stdin and stdout as pipe. Modifying your code a tad bit can give you the desired results
from subprocess import Popen, PIPE
#part which should be outside loop
args = ['sudo', 'python', '/home/pi/second.py']
process = Popen(args, stdin=PIPE, stdout=PIPE)
while True:
value=input("Input some text or number")
process.stdin.write(value)
You need to open the process outside the loop for this to work. A similar issue is addressed here in case you want to check that Keep a subprocess alive and keep giving it commands? Python
This approach will lead to error if child process quits after first iteration and close all the pipes. You somehow need to block the child process to accept more input. This you can do by either using threads or by using the first option i.e. Pexpect

How to pass two values to stdin using subprocess in python

I am executing a script which prompts for 2 values one after the other. I want to pass the values from the script itself as I want to automate this.
Using the subprocess module, I can easily pass one value:
suppression_output = subprocess.Popen(cmd_suppression, shell=True,
stdin= subprocess.PIPE,
stdout= subprocess.PIPE).communicate('y') [0]
But passing the 2nd value does not seem to work. If I do something like this:
suppression_output = subprocess.Popen(cmd_suppression, shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE).communicate('y/r/npassword')[0]

You should use \n for the new line instead of /r/n -> 'y\npassword'
As your question is not clear, I assumed you have a program which behaves somewhat like this python script, lets call it script1.py:
import getpass
import sys
firstanswer=raw_input("Do you wish to continue?")
if firstanswer!="y":
sys.exit(0) #leave program
secondanswer=raw_input("Enter your secret password:\n")
#secondanswer=getpass.getpass("Enter your secret password:\n")
print "Password was entered successfully"
#do useful stuff here...
print "I should not print it out, but what the heck: "+secondanswer
It asks for confirmation ("y"), then wants you to enter a password. After that it does "something useful", finally prints the password and then exits
Now to get the first program to be run by a second script script2.py it has to look somewhat like this:
import subprocess
cmd_suppression="python ./testscript.py"
process=subprocess.Popen(cmd_suppression,shell=True\
,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
response=process.communicate("y\npassword")
print response[0]
The output of script2.py:
$ python ./script2.py
Do you wish to continue?Enter your secret password:
Password was entered successfully
I should not print it out, but what the heck: password
A problem can most likely appear if the program uses a special method to get the password in a secure way, i.e. if it uses the line I just commented out in script1.py
secondanswer=getpass.getpass("Enter your secret password:\n")
This case tells you that it is probably not a good idea anyway to pass a password via a script.
Also keep in mind that calling subprocess.Popen with the shell=True option is generally a bad idea too. Use shell=False and provide the command as a list of arguments instead:
cmd_suppression=["python","./testscript2.py"]
process=subprocess.Popen(cmd_suppression,shell=False,\
stdin=subprocess.PIPE,stdout=subprocess.PIPE)
It is mentioned a dozen times in the Subprocess Documentation

Try os.linesep:
import os
from subprocess import Popen, PIPE
p = Popen(args, stdin=PIPE, stdout=PIPE)
output = p.communicate(os.linesep.join(['the first input', 'the 2nd']))[0]
rc = p.returncode
In Python 3.4+, you could use check_output():
import os
from subprocess import check_output
input_values = os.linesep.join(['the first input', 'the 2nd']).encode()
output = check_output(args, input=input_values)
Note: the child script might ask for a password directly from the terminal without using subprocess' stdin/stdout. In that case, you might need pexpect, or pty modules. See Q: Why not just use a pipe (popen())?
import os
from pexpect import run # $ pip install pexpect
nl = os.linesep
output, rc = run(command, events={'nodes.*:': 'y'+nl, 'password:': 'test123'+nl},
withexitstatus=1)

Best way to pipe output of Linux sort

I would like process a file line by line. However I need to sort it first which I normally do by piping:
sort --key=1,2 data |./script.py.
What's the best to call sort from within python? Searching online I see subprocess or the sh module might be possibilities? I don't want to read the file into memory and sort in python as the data is very big.

Its easy. Use subprocess.Popen to run sort and read its stdout to get your data.
import subprocess
myfile = 'data'
sort = subprocess.Popen(['sort', '--key=1,2', myfile],
stdout=subprocess.PIPE)
for line in sort.stdout:
your_code_here
sort.wait()
assert sort.returncode == 0, 'sort failed'

I think this page will answer your question
The answer I prefer, from #Eli Courtwright is (all quoted verbatim):
Here's a summary of the ways to call external programs and the advantages and disadvantages of each:
os.system("some_command with args") passes the command and arguments to your system's shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example,
os.system("some_command < input_file | another_command > output_file")
However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc. On the other hand, this also lets you run commands which are simply shell commands and not actually external programs.
http://docs.python.org/lib/os-process.html
stream = os.popen("some_command with args") will do the same thing as os.system except that it gives you a file-like object that you can use to access standard input/output for that process. There are 3 other variants of popen that all handle the i/o slightly differently. If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don't need to worry about escaping anything.
http://docs.python.org/lib/os-newstreams.html
The Popen class of the subprocess module. This is intended as a replacement for os.popen but has the downside of being slightly more complicated by virtue of being so comprehensive. For example, you'd say
print Popen("echo Hello World", stdout=PIPE, shell=True).stdout.read()
instead of
print os.popen("echo Hello World").read()
but it is nice to have all of the options there in one unified class instead of 4 different popen functions.
http://docs.python.org/lib/node528.html
The call function from the subprocess module. This is basically just like the Popen class and takes all of the same arguments, but it simply wait until the command completes and gives you the return code. For example:
return_code = call("echo Hello World", shell=True)
http://docs.python.org/lib/node529.html
The os module also has all of the fork/exec/spawn functions that you'd have in a C program, but I don't recommend using them directly.
The subprocess module should probably be what you use.

I believe sort will read all data in memory, so I'm not sure you will won anything but you can use shell=True in subprocess and use pipeline
>>> subprocess.check_output("ls", shell = True)
'1\na\na.cpp\nA.java\na.php\nerase_no_module.cpp\nerase_no_module.cpp~\nWeatherSTADFork.cpp\n'
>>> subprocess.check_output("ls | grep j", shell = True)
'A.java\n'
Warning
Invoking the system shell with shell=True can be a security hazard if combined with untrusted input. See the warning under Frequently Used Arguments for details.