Develop Reference

How do I properly loop through subprocess.stdout

I'm creating a program where I need to use a powershell session and I found out how I could have a persistent session using the below code. However I want to loop through the new lines of the output of powershell when a command has been run. The for loop below is the only way i've found to do so but it expects an EOF and doesn't get it so it just lingers and the program never exits. How can I get the amount of new lines in stdout so I can properly loop through them?
from subprocess import Popen, PIPE
process = Popen(["powershell"], stdin=PIPE, stdout=PIPE)
def ps(command):
command = bytes("{}\n".format(command), encoding='utf-8')
process.stdin.write(command)
process.stdin.flush()
process.stdout.readline()
return process.stdout.readline().decode("utf-8")
ps("echo hello world")
for line in process.stdout:
print(line.strip().decode("utf-8"))
process.stdin.close()
process.wait()

You need the Powershell command to know when to exit. Typically, the solution is to not just flush, but close the stdin for the child process; when it's done with its work and finds EOF on its input, it should exit on its own. Just change:
process.stdin.flush()
to:
process.stdin.close()
which implies a flush and also ensures the child process knows input is done. If that doesn't work on its own, you might explicitly add a quit or exit (whatever Powershell uses to terminate the session manually) command after the command you're actually running.
If you must run multiple commands in the single subprocess, and each command must be fully consumed before the next one is sent, there are terrible heuristic solutions available, e.g. sending three commands at once, where the second simply echoes a sentinel string and the third explicitly flushes stdout (to ensure block buffering doesn't mean you deadlock waiting for the sentinel when its stuck in subprocess's internal buffers), and your loop can terminate once it sees the sentinel. Without a sentinel, it's worse, because you basically can't tell when the command is done, and just have to use the select/selectors module to poll the process's stdout with a timeout, reading lines whenever there is available data, and assuming the process is done if no new input is available without the expected timeout window.

subprocess Popen stdin will only input and run after the script has finished

Description:
I was trying to make a shell that can be interactive on a chatting software, so I need a cmd.exe as a subprocess and pass strings into the process.
I have this:
from subprocess import Popen
from subprocess import PIPE as p
proc = Popen("cmd",stdout=p,stdin=p,shell=True)
so usually what we do if we need to pass input to the process is by using proc.stdin.write()
but it seems that the string will only pass in and work after the python script is complete
for example, I have
#same thing above
proc.stdin.write("ping 127.0.0.1".encode())
time.sleep(10)
the script will wait for 10 sec then pass and run the ping command.
which means it's impossible to get the result stdout.read() because there is nothing.
I have tried to use subprocess.Popen.communicate() but it closes the pipe after one input.
Is there any way to solve the "only run the command after script finish" thing, or make communicate() not close the pipe?

Writes to pipes are buffered, you need to flush the buffer.
proc.stdin.write("ping 127.0.0.1".encode())
proc.stdin.flush()

Passing arguments/strings into already running process - Python 2.7

I have two scripts in Python.
sub.py code:
import time
import subprocess as sub
while 1:
value=input("Input some text or number") # it is example, and I don't care about if it is number-input or text-raw_input, just input something
proces=sub.Popen(['sudo', 'python', '/home/pi/second.py'],stdin=sub.PIPE)
proces.stdin.write(value)
second.py code:
import sys
while 1:
from_sub=sys.stdin()#or sys.stdout() I dont remember...
list_args.append(from_sub) # I dont know if syntax is ok, but it doesn't matter
for i in list_arg:
print i
First I execute sub.py, and I input something, then second.py file will execute and printing everything what I inputed and again and again...
The thing is I don't want to open new process. There should be only one process. Is it possible?
Give me your hand :)

This problem can be solved by using Pexpect. Check my answer over here. It solves a similar problem
https://stackoverflow.com/a/35864170/5134525.
Another way to do that is to use Popen from subprocess module and setting stdin and stdout as pipe. Modifying your code a tad bit can give you the desired results
from subprocess import Popen, PIPE
#part which should be outside loop
args = ['sudo', 'python', '/home/pi/second.py']
process = Popen(args, stdin=PIPE, stdout=PIPE)
while True:
value=input("Input some text or number")
process.stdin.write(value)
You need to open the process outside the loop for this to work. A similar issue is addressed here in case you want to check that Keep a subprocess alive and keep giving it commands? Python
This approach will lead to error if child process quits after first iteration and close all the pipes. You somehow need to block the child process to accept more input. This you can do by either using threads or by using the first option i.e. Pexpect

Read stdout from subprocess until there is nothing left

I would like to run several commands in the same shell. After some research I found that I could keep a shell open using the return process from Popen. I can then write and read to stdin and stdout. I tried implementing it as such:
process = Popen(['/bin/sh'], stdin=PIPE, stdout=PIPE)
process.stdin.write('ls -al\n')
out = ' '
while not out == '':
out = process.stdout.readline().rstrip('\n')
print out
Not only is my solution ugly, it doesn't work. out is never empty because it hands on the readline(). How can I successfully end the while loop when there is nothing left to read?

Use iter to read data in real time:
for line in iter(process.stdout.readline,""):
print line
If you just want to write to stdin and get the output you can use communicate to make the process end:
process = Popen(['/bin/sh'], stdin=PIPE, stdout=PIPE)
out,err =process.communicate('ls -al\n')
Or simply get the output use check_output:
from subprocess import check_output
out = check_output(["ls", "-al"])

The command you're running in a subprocess is sh, so the output you're reading is sh's output. Since you didn't indicate to the shell it should quit, it is still alive, thus its stdout is still open.
You can perhaps write exit to its stdin to make it quit, but be aware that in any case, you get to read things you don't need from its stdout, e.g. the prompt.
Bottom line, this approach is flawed to start with...

How to communicate with command line program using python?

import subprocess
import sys
proc = subprocess.Popen(["program.exe"], stdin=subprocess.PIPE) #the cmd program opens
proc.communicate(input="filename.txt") #here the filename should be entered (runs)
#then the program asks to enter a number:
proc.communicate(input="1") #(the cmd stops here and nothing is passed)
proc.communicate(input="2") # (same not passing anything)
how do i pass and communicate with the cmd using python.
Thanks. (using windows platform)

The docs on communicate() explain this:
Interact with process: Send data to stdin. Read data from stdout and
stderr, until end-of-file is reached. Wait for process to terminate.
communicate() blocks once the input has been sent until the program finishes executing. In your example, the program waits for more input after you send "1", but Python waits for it to exit before it gets to the next line, meaning the whole thing deadlocks.
If you want to read and write a lot interchangeably, make pipes to stdin/stdout and write/read to/from them.