Develop Reference

How to capture interlaced stdin and stdout from a Python subprocess

I am trying to write a Python script that automatically grades a Python script submitted by a student, where the student's script uses the input() function to get some information from the user.
Suppose the student's script is something simple like this:
name = input('Enter your name: ')
print(f'Hello {name}!')
The portion of the test script that runs the student script is something like this:
import subprocess
run_cmd = 'python student_script.py'
test_input = 'Bob'
p = subprocess.run(run_cmd.split(), input=test_input, capture_output=True, text=True)
After running that portion of the test script, output from the student's script is captured and can be accessed via p.stdout which is a string having this value:
'Enter your name: Hello Bob!\n'
No surprise there, since this is everything output by the student script, but notice that the 'Bob' test input is not included.
In the test report, I want to show the script input and output in the same way that it would appear if the script had been run from a command line, which would look like this:
Enter your name: Bob
Hello Bob!
Given that the scripts are written by students, the prompt message output by the student script could be anything (e.g., What is your name?, Who are you?, Type in name:, etc.) and the student script might also print something other than 'Hello Bob!', so I don't think there is any way to reliably figure out where to correctly insert the 'Bob' test input (and a trailing new line) into p.stdout.
Is there a way to get subprocess.run() to capture interlaced stdin and stdout?
Or is there another way to run a Python script from a Python script that captures interlaced stdin and stdout?
Ideally, for this example, I would be able to get a string having this value:
'Enter your name: Bob\nHello Bob!\n'
I've search SO and read through the subprocess documentation, but thus far I've come up short on finding a solution.

Here's the solution I came up with. I expect there is a more elegant way to do it, but it works on the Ubuntu Linux computer that the automated test scripts run on. I have not tried it on Windows, but I believe it will not work since os.set_blocking() is only supported on Unix per the os module documentation.
import subprocess
import os
import time
run_cmd = 'python student_script.py'
test_input = 'Bob'
# Start the student script running
p = subprocess.Popen(run_cmd.split(), stdin=subprocess.PIPE, stdout=subprocess.PIPE, text = True)
# Give the script some time to run
time.sleep(2)
# String to hold interleaved stdin and stdout text
stdio_text = ''
# Capture everything from stdout
os.set_blocking(p.stdout.fileno(), False) # Prevents readline() blocking
stdout_text = p.stdout.readline()
while stdout_text != '':
stdio_text += stdout_text
stdout_text = p.stdout.readline()
# Append test input to interleaved stdin and stdout text
stdio_text += (test_input + '\n')
try:
# Send test input to stdin and wait for student script to terminate
stdio_text += p.communicate(input=test_input, timeout=5)[0]
except subprocess.TimeoutExpired:
# Something is wrong with student script
pass
p.terminate()
The key to this solution working is os.set_blocking(), which I found out about here. Without it readline() blocks indefinitely.
I don't love the time.sleep(2) since it assumes it will take 2 seconds or less for the student script to reach the point where it calls input(), but there does not seem to be any way to determine when a process is looking for input from stdin. The sleep time could be increased for longer scripts.
If you've got any ideas for improvements, please share.

using python to ssh fully into server

I've been trying to automate ssh'ing into my server however cannot find a way to fully automate the process. To be specific, getting around this input has been the struggle: root#example's password:
My code:
import subprocess
import time
server_ip = 'server'
pwd = b'password'
p = subprocess.Popen(['ssh', 'root#{}'.format(server_ip)],
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
if p == "root#example's password: ":
p.communicate(input= "{}".format(pwd))
else:
time.sleep(2)
if p == "root#example's password: ":
p.communicate(input= "{}".format(pwd))
else:
pass
What it returns:
Pseudo-terminal will not be allocated because stdin is not a terminal.
root#example's password:
user#computer ~ % Permission denied, please try again.
root#example's password:
Permission denied, please try again.
root#example's password:
root#example: Permission denied (publickey,password).
I know my code is very scuffed but it is the furthest I've got to getting in and submitting the password entry request.
Any help is appreciated!

if p == "root#example's password: ": can never be true; p is a subprocess.Popen object, not a string.
You can get a string by reading from the object's standard output, but of course, ssh prints the prompt message on the tty, so you can't easily capture it from Python.
Notice also that without an encoding keyword argument or text=True, you can't send or receive strings; the output you receive will be a b'...' byte string which cannot be compared to a regular string.
... But even if you managed to sort out these problems, taking it from there to a fully working interactive SSH session is still quite some distance to go. I would definitely recommend that you try pexpect or Paramiko instead of rolling your own, especially if you are new to Python.
Tangentially, else: pass is completely unnecessary; there is no reason to add an else: block in the first place if you don't have anything useful to put in it.
And
As "{}".format(pwd) is a really hairy way to write what can be more easily expressed as pwd, or str(pwd) if it's not already a string.

stdin.write echos password during paramiko ssh login

The script provided by TechJS: (https://stackoverflow.com/users/5252192/techjs) in their answer on (How to run sudo with paramiko? (Python)) works perfectly for me.
However, it echos the password in the command line after my additions and i know that's not a good idea. I imagine its from the stdin.write() but i have no idea how to do it differently.
Can anyone suggest a more secure way of storing and inputting the server password? I'm still pretty new and would love a good lesson on proper password security protocol in these situations :)
Thanks so much to any and all help!
import paramiko
import re
import <passwords file> #did chmod 400 for this file
ssh_client= None
server_address='<removed for security>'
server_username='<removed for security>'
server_pass = <password file>.<this server password from passwords file>
command = "<removed for security>"
def main(command, server_address, server_username, server_pass):
try:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname=server_address,
username=server_username,
password=server_pass)
session = ssh.get_transport().open_session()
session.set_combine_stderr(True)
session.get_pty()
session.exec_command("sudo bash -c \"" + command + "\"")
stdin = session.makefile('wb', -1)
stdout = session.makefile('rb', -1)
stdin.write(server_pass + '\n')
stdin.flush()
print(stdout.read().decode("utf-8"))
except Exception as e:
print("The following error has occurred during your requested process")
print(e.message)
finally:
if ssh:
session.close()
ssh.close()
if __name__ == '__main__':
main(command, server_address, server_username, server_pass)

After a lot of research I believe I have an acceptable answer, however please take this with skepticism as I am NOT an expert in this field. You have been advised.
This also did NOT fix the printing of the stdin problem, but i have just removed the print() function all together to remove the issue. This answer is for the password security part ONLY.
tl:dr here is the answer https://alexwlchan.net/2016/11/you-should-use-keyring/
but i will explain in more detail and provide examples below of my code used to store and use passwords while never plain texting them.
LONG ANSWER:
Python has a package that is build for this purpose specifically called keyring(). It allows you to store and call on passwords with relative security. It works off of your login credentials so unfortunately if someone gains access to your account they will have access to this information, but without that you should theoretically be secure (or as secure as one can be i guess)
Keyring() plus a package called getpass() allow for a user to input a password into their system without committing it to plain text and thus preventing accidental leaking through file sharing or the like.
Here is a very simple script I wrote to automatically prompt you through your choices and store the password without ever needing to store it in plain text
import keyring
import getpass
def main():
system = input('System:')
username = input('Please input username:')
keyring.set_password(system,username,getpass.getpass())
print('The password for ' +username+' in '+system+' has been set.\nPlease do not misplace, you will not be able to recover at this point.\nFor misplaced passwords, please resubmit new entry with the same details, it will overwrite the previous entry.')
if __name__=='__main__':
print('Please input the system in which the password will be used,\nand corresponding username.')
main()
(if you're using Python 2 then it needs to be raw_input() )
This is done in an entirely different script so you DO NOT NEED TO HAVE THEM TOGETHER, run one script to set the password, then to call on the corresponding password is very simple in your main script from that point forward.
passwd = keyring.get_password('<system you inputed>','<username you inputed>')
And you're done!
p.s. I personally have placed a bash file on my PATH that runs this script so that if i ever need to create a password it can be done from any directory within the machine, and thus reinforcing good security procedures.

Python - Run a simple command-line program with prompted I/O and "proxy" it, on Windows

I have a simple command-line binary program hello which outputs to STDOUT:
What is your name?
and waits for the user to input it. After receiving their input it outputs:
Hello, [name]!
and terminates.
I want to use Python to run computations on the final output of this program ("Hello, [name]!"), however before the final output I want the Python script to essentially "be" the binary program. In other words I'd like Python to forward all of the prompts to STDOUT and then accept the user's input and give it to the program. However I want to hide the final output so I can process it and show my own results to the user. I do not want to replicate the hello's behavior in the script, as this simple program is a stand-in for a more complex program that I am actually working with.
I was hoping there would be some sort of mechanic in subprocess where I would be able to do something akin to:
while process.is_running():
next_char = process.stdout.read(1)
if next_char == input_prompt_thing: # somehow check if the program is waiting for input
user_input = raw_input(buffer)
process.stdin.write(user_input)
else:
buffer += next_char
I have been playing with subprocess and essentially got as far as realizing I could use process.stdout.read(1) to read from the program before it began blocking, but I can't figure out how to break this loop before the process blocks my Python script. I am not too familiar with console I/O and it is not an area of much expertise for me, so I am starting to feel pretty lost. I appreciate any help!

You could try winpexpect (not tested):
import re
from winpexpect import winspawn
p = winspawn('hello')
prompt = "What is your name?"
p.expect(re.escape(prompt))
name = raw_input(prompt)
p.sendline(name)
p.expect("Hello, .*!")
output = p.after
# ...

How to automate shell input with python?

I am automating some tasks with python, but have hit a bit of a roadblock. One of the tasks I am automating requires user input in the shell.
The requirement is that you to run the command with an email address as a parameter (simple enough), and then you are asked to authenticate with the password for that email address. How can you simulate user input to provide the password?
There are also some menus afterwards which ask options, for which the input need just be to repeatedly hit enter. How is this simulated? Keeping in mind that this window will not always have focus..

I'm not sure what you're asking in the second part, but subprocesses can be controlled with the pexpect module. For example:
#!/usr/bin/env python
import pexpect
import sys
# Get email and password somehow
#email = ...
#password = ...
# Start the subprocess
child = pexpect.spawn('mycommand %s' % email)
# redirect output to stdout
child.logfile_read = sys.stdout
# Assumes the prompt is "password:"
child.expect('password:')
child.sendline(password)
# Wait for the process to close its output
child.expect(pexpect.EOF)

Looks like you are thinking in a wrong way. You just need to send some bytes via pipe to recipient (shell script in your case) and this can be done with subprocess.

I guess you can use expect for this.