Develop Reference

getpass behaves different in pychram IDE and terminal

paaword.py is a script where getpass() asked the user about the password and validates it. but i want to automate the whole process and used subprocess for it (main.py). And i am using python3.10
Problem:
problem is when i run the main.py in pycharm IDE it works normally (it automates the process). but when I run the script python3 main.py in ubuntu terminal it asked for the input.
I dont know why it behaves deifferent in in IDE and terminal?
password.py
import warnings
import getpass
import time
# Suppress warnings
warnings.filterwarnings("ignore", category=getpass.GetPassWarning)
for x in range(10):
print(f"curnt index {x}")
time.sleep(5)
password = getpass.getpass("Enter your password: ")
if password != "test":
print("wrong password")
else:
print("correct password")
main.py
import subprocess
# subprocess
proc = subprocess.Popen(["python", "password.py"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
password = "test"
input_data = f"{password}\n"
# read output from the subprocess in real-time
while True:
if proc.poll() is not None:
break
proc.stdin.write(input_data.encode())
proc.stdin.flush()
output = proc.stdout.readline().decode().strip()
if output:
print(output)
output in pycharm:
output in ubuntu terminal (20.04)

Judging by the screenshots, your OS is Linux.
In Linux, getpass() first tries to read directly from the process' controlling terminal (/dev/tty), or, if that fails, stdin using direct terminal I/O; and only if that fails, it falls back to regular I/O, displaying a warning.
Judging by the warnings in the IDE, the latter is exactly what happens in your first case.
Lib/getpass.py:
def unix_getpass(prompt='Password: ', stream=None):
<...>
try:
# Always try reading and writing directly on the tty first.
fd = os.open('/dev/tty', os.O_RDWR|os.O_NOCTTY)
tty = io.FileIO(fd, 'w+')
<...>
input = io.TextIOWrapper(tty)
<...>
except OSError:
# If that fails, see if stdin can be controlled.
<...>
try:
fd = sys.stdin.fileno()
except (AttributeError, ValueError):
fd = None
passwd = fallback_getpass(prompt, stream) # fallback_getpass is what displays the warnings
input = sys.stdin
<...>
if fd is not None:
try:
old = termios.tcgetattr(fd)
<...>
except termios.error:
<...>
passwd = fallback_getpass(prompt, stream)
<...>
return passwd
As you can see, getpass() is specifically designed to be interactive and resist intercepting its input. So if you need to provide a password automatically, use another way:
store it in a file readable only by you (e.g. SSH does that; you can provide that file as an argument and store other arguments there as well), or
use the system's keyring
and only fall back to getpass if the password was not provided that way and/or if you detect that the program is being run interactively (sys.stdin.isatty())
while it's also possible to provide the password on the command line -- in that case, you have to overwrite it in your process' stored command line to hide it from snooping. I couldn't find a way to do that in Python.
You can check Secure Password Handling in Python | Martin Heinz | Personal Website & Blog for a more detailed rundown of the above. (note: it suggests using envvars and load them from .env which would probably not apply to you. That's designed for .NET projects which due to the rigid structure of MS Visual Studio's build system, have had to rely on envvars for any variable values.)

How to resume a terminal with subprocess python

I am working with the subprocess module in Python. I am trying to run a series of terminals to automate a process.
To break it down:
I am suppose to have 3 terminals open to run a set of commands
like so:
Terminal 1: `cd src` -> `./run_script.sh`
Terminal 2: cd data -> `python prepare_data.py`
Terminal 3: `cd src` -> `./do_something.sh` #runs some docker container
Terminal 4: `cd src` -> `./do_another.sh`
Terminal 3: `./another_bash.sh`
To automate this the following:
class AutomateProcesses:
def run_terminal_1(self):
subprocess.call('./run_script.sh', shell=True, cwd='../src')
def run_terminal_2(self):
subprocess.call('python prepare_data.py', shell=True, cwd='../../data')
def run_terminal_3(self):
subprocess.call('./do_something.sh.sh', shell=True, cwd='../src')
def run_terminal_4(self):
subprocess.call('./do_another.sh', shell=True, cwd='../src')
How do I get back to terminal 3 to run the command?

It looks like you want to run several commands on a "terminal" (actually you don't see any terminal), it is just a sub-process that runs a shell.
I use the tool called pexpect (https://pexpect.readthedocs.io/en/latest/overview.html), it has the Windows-variant wexpect (https://pypi.org/project/wexpect/).
Below is the code sample, using the child variable, you can keep the "terminal" and send commands to it.
import pexpect
# log file to capture all the commands sent to the shell and their responses
output_file = open('log.txt','wb')
# create the bash shell sub-process
child = pexpect.spawn('/bin/bash', logfile=output_file)
child.stdout = output_file
child.expect(bytes('>', 'utf-8'))
# make sure you use the pair (sendline() and expect()) to wait until the command finishes
child.sendline(bytes('ls', 'utf-8'))
child.expect(bytes('>', 'utf-8'))
child.sendline(bytes('echo Hello World', 'utf-8'))
child.expect(bytes('>', 'utf-8'))
output_file.close()

How to keep ssh session open after logging in using subprocess.popen?

I am new to Python.
I am trying to SSH to a server to perform some operations. However, before performing the operations, i need to load a profile, which takes 60-90 seconds. After loading the profile, is there a way to keep the SSH session open so that i can perform the operations later?
p = subprocess.Popen("ssh abc#xyz'./profile'", stdout=subprocess.PIPE, shell=True)
result = p.communicate()[0]
print result
return result
This loads the profile and exits. Is there a way to keep the above ssh session open and run some commands?
Example:
p = subprocess.Popen("ssh abc#xyz'./profile'", stdout=subprocess.PIPE, shell=True)
<More Python Code>
<More Python Code>
<More Python Code>
<Run some scripts/commands on xyz server non-interactively>
After loading the profile, I want to run some scripts/commands on the remote server, which I am able to do by simply doing below:
p = subprocess.Popen("ssh abc#xyz './profile;**<./a.py;etc>**'", stdout=subprocess.PIPE, shell=True)
However, once done, it exists and the next time I want to execute some script on the above server, I need to load the profile again (which takes 60-90 seconds). I am trying to figure out a way where we can create some sort of tunnel (or any other way) where the ssh connection remains open after loading the profile, so that the users don't have to wait 60-90 seconds whenever anything is to be executed.
I don't have access to strip down the profile.

Try an ssh library like asyncssh or spur. Keeping the connection object should keep the session open.
You could send a dummy command like date to prevent the timeout as well.

You have to construct a ssh command like this ['ssh', '-T', 'host_user_name#host_address'] then follow below code.
Code:
from subprocess import Popen, PIPE
ssh_conn = ['ssh', '-T', 'host_user_name#host_address']
# if you have to add port then ssh_conn should be as following
# ssh_conn = ['ssh', '-T', 'host_user_name#host_address', '-p', 'port']
commands = """
cd Documents/
ls -l
cat test.txt
"""
with Popen(ssh_conn, stdin=PIPE, stdout=PIPE, stderr=PIPE, universal_newlines=True) as p:
output, error = p.communicate(commands)
print(output)
print(error)
print(p.returncode)
# or can do following things
p.stdin.write('command_1')
# add as many command as you want
p.stdin.write('command_n')
Terminal Output:
Please let me know if you need further explanations.
N.B: You can add command in commands string as many as you want.

What you want to do is write/read to the process's stdin/stdout.
from subprocess import Popen, PIPE
import shlex
shell_command = "ssh user#address"
proc = Popen(shlex.split(shell_command), stdin=PIPE, universal_newlines=True)
# Do python stuff here
proc.stdin.write("cd Desktop\n")
proc.stdin.write("mkdir Example\n")
# And so on
proc.stdin.write("exit\n")
You must include the trailing newline for each command. If you prefer, print() (as of Python 3.x, where it is a function) takes a keyword argument file, which allows you to forget about that newline (and also gain all the benefits of print()).
print("rm Example", file=proc.stdin)
Additionally, if you need to see the output of your command, you can pass stdout=PIPE and then read via proc.stdout.read() (same for stderr).
You may also want to but the exit command in a try/finally block, to ensure you exit the ssh session gracefully.
Note that a) read is blocking, so if there's no output, it'll block forever and b) it will only return what was available to read from the stdout at that time- so you may need to read repeatedly, sleep for a short time, or poll for additional data. See the fnctl and select stdlib modules for changing blocking -> nonblocking read and polling for events, respectively.

Hello Koshur!
I think that what you are trying to achieve looks like what I've tried in the past when trying to make my terminal accessible from a private website:
I would open a bash instance, keep it open and would listen for commands through a WebSocket connection.
What I did to achieve this was using the O_NONBLOCK flag on STDOUT.
Example
import fcntl
import os
import shlex
import subprocess
current_process = subprocess.Popen(shlex.split("/bin/sh"), stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT) # Open a shell prompt
fcntl.fcntl(current_process.stdout.fileno(), fcntl.F_SETFL,
os.O_NONBLOCK) # Non blocking stdout and stderr reading
What I would have after this is a loop checking for new output in another thread:
from time import sleep
from threading import Thread
def check_output(process):
"""
Checks the output of stdout and stderr to send it to the WebSocket client
"""
while process.poll() is None: # while the process isn't exited
try:
output = process.stdout.read() # Read the stdout PIPE (which contains stdout and stderr)
except Exception:
output = None
if output:
print(output)
sleep(.1)
# from here, we are outside the loop: the process exited
print("Process exited with return code: {code}".format(code=process.returncode))
Thread(target=check_output, args=(current_process,), daemon=True).start() # Start checking for new text in stdout and stderr
So you would need to implement your logic to SSH when starting the process:
current_process = subprocess.Popen(shlex.split("ssh abc#xyz'./profile'"), stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
And send commands like so:
def send_command(process, cmd):
process.stdin.write(str(cmd + "\n").encode("utf-8")) # Write the input to STDIN
process.stdin.flush() # Run the command
send_command(current_process, "echo Hello")
EDIT
I tried to see the minimum Python requirements for the given examples and found out that Thread(daemon) might not work on Python 2.7, which you asked in the tags.
If you are sure to exit the Thread before exiting, you can ignore daemon and use Thread() which works on 2.7. (You could for example use atexit and terminate the process)
References
fcntl(2) man page
https://man7.org/linux/man-pages/man2/fcntl.2.html
fcntl Python 3 Documentation
https://docs.python.org/3/library/fcntl.html
fcntl Python 2.7 Documentation
https://docs.python.org/2.7/library/fcntl.html
O_NONBLOCK Python 3 Documentation
https://docs.python.org/3/library/os.html#os.O_NONBLOCK
O_NONBLOCK Python 2.7 Documentation
https://docs.python.org/2.7/library/os.html#os.O_NONBLOCK

Run program using cmd on remote Windows machine

I want to create a Python script that opens a cmd in remote Windows machine using psexec, and runs my_program.exe from this cmd, and when some event occurs it sends Ctrl+c to my_program.exe which handles this signal somehow.
Here's my code:
from os import chdir, path
from subprocess import Popen, PIPE
psexec_dir = r'C:\Users\amos1\Downloads\PSTools'
chdir(psexec_dir)
path.join(psexec_dir, 'psexec.exe')
command = ['psexec.exe', '\\amos', 'cmd']
p = Popen(command, stdin = PIPE, stdout = PIPE)
p.stdin.write(b'my_program.exe\r\n')
while True:
if some_condition:
ctrl_c = b'\x03'
p.stdin.write(ctrl_c)
break
for line in p.stdout.readlines():
print(line)
p.kill()
The problems:
my_program.exe does not run
p.kill raises WindowsError: [Error 5] Access is denied (even though I used the answers from here and did both chdir and path.join in my code)
Notice that both my computer and the target computer are Windows machines

how to interact with Paramiko's interactive shell session?

I have some Paramiko code where I use the invoke_shell method to request an interactive ssh shell session on a remote server. Method is outlined here: invoke_shell()
Here's a summary of the pertinent code:
sshClient = paramiko.SSHClient()
sshClient.connect('127.0.0.1', username='matt', password='password')
channel = sshClient.get_transport().open_session()
channel.get_pty()
channel.invoke_shell()
while True:
command = raw_input('$ ')
if command == 'exit':
break
channel.send(command + "\n")
while True:
if channel.recv_ready():
output = channel.recv(1024)
print output
else:
time.sleep(0.5)
if not(channel.recv_ready()):
break
sshClient.close()
My question is: is there a better way to interact with the shell? The above works, but it's ugly with the two prompts (the matt#kali:~$ and the $ from raw_input), as shown in the screenshot of a test run with the interactive shell. I guess I need help writing to the stdin for the shell? Sorry, I don't code much. Thanks in advance!

I imported a file, interactive.py, found on Paramiko's GitHub. After importing it, I just had to change my code to this:
try:
import interactive
except ImportError:
from . import interactive
...
...
channel.invoke_shell()
interactive.interactive_shell(channel)
sshClient.close()

You can try disabling echo after invoking the remote shell:
channel.invoke_shell()
channel.send("stty -echo\n")
while True:
command = raw_input() # no need for `$ ' anymore
... ...