Develop Reference

getpass behaves different in pychram IDE and terminal

paaword.py is a script where getpass() asked the user about the password and validates it. but i want to automate the whole process and used subprocess for it (main.py). And i am using python3.10
Problem:
problem is when i run the main.py in pycharm IDE it works normally (it automates the process). but when I run the script python3 main.py in ubuntu terminal it asked for the input.
I dont know why it behaves deifferent in in IDE and terminal?
password.py
import warnings
import getpass
import time
# Suppress warnings
warnings.filterwarnings("ignore", category=getpass.GetPassWarning)
for x in range(10):
print(f"curnt index {x}")
time.sleep(5)
password = getpass.getpass("Enter your password: ")
if password != "test":
print("wrong password")
else:
print("correct password")
main.py
import subprocess
# subprocess
proc = subprocess.Popen(["python", "password.py"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
password = "test"
input_data = f"{password}\n"
# read output from the subprocess in real-time
while True:
if proc.poll() is not None:
break
proc.stdin.write(input_data.encode())
proc.stdin.flush()
output = proc.stdout.readline().decode().strip()
if output:
print(output)
output in pycharm:
output in ubuntu terminal (20.04)

Judging by the screenshots, your OS is Linux.
In Linux, getpass() first tries to read directly from the process' controlling terminal (/dev/tty), or, if that fails, stdin using direct terminal I/O; and only if that fails, it falls back to regular I/O, displaying a warning.
Judging by the warnings in the IDE, the latter is exactly what happens in your first case.
Lib/getpass.py:
def unix_getpass(prompt='Password: ', stream=None):
<...>
try:
# Always try reading and writing directly on the tty first.
fd = os.open('/dev/tty', os.O_RDWR|os.O_NOCTTY)
tty = io.FileIO(fd, 'w+')
<...>
input = io.TextIOWrapper(tty)
<...>
except OSError:
# If that fails, see if stdin can be controlled.
<...>
try:
fd = sys.stdin.fileno()
except (AttributeError, ValueError):
fd = None
passwd = fallback_getpass(prompt, stream) # fallback_getpass is what displays the warnings
input = sys.stdin
<...>
if fd is not None:
try:
old = termios.tcgetattr(fd)
<...>
except termios.error:
<...>
passwd = fallback_getpass(prompt, stream)
<...>
return passwd
As you can see, getpass() is specifically designed to be interactive and resist intercepting its input. So if you need to provide a password automatically, use another way:
store it in a file readable only by you (e.g. SSH does that; you can provide that file as an argument and store other arguments there as well), or
use the system's keyring
and only fall back to getpass if the password was not provided that way and/or if you detect that the program is being run interactively (sys.stdin.isatty())
while it's also possible to provide the password on the command line -- in that case, you have to overwrite it in your process' stored command line to hide it from snooping. I couldn't find a way to do that in Python.
You can check Secure Password Handling in Python | Martin Heinz | Personal Website & Blog for a more detailed rundown of the above. (note: it suggests using envvars and load them from .env which would probably not apply to you. That's designed for .NET projects which due to the rigid structure of MS Visual Studio's build system, have had to rely on envvars for any variable values.)

Why do tmux and vim print garbage in my SSH wrapper script?

I have written an SSH wrapper script that does local line editing. It is invoked similarly to SSH. For example: python3 sshwrapper.py user#example.com -CX. The problem is that when I connect to a remote computer using this script and use vim or tmux there, some garbage is printed. This problem is not specific to SSH, since the problems also appear when I use this script to wrap bash instead of ssh.
Examples:
After starting tmux, some garbage is printed after the bash prompt:
abc#me:~$ ^[[?65;1;9c
When opening a new file in Vim using vim mynewfile.txt, this appears on the first line:
^[[2;2R^[[>65;6003;1c^[]10;rgb:0000/0000/0000^G^[]11;rgb:ffff/ffff/dddd^G
How do I fix the problem?
This is the script in question:
import os
import pty
import select
import signal
import subprocess
import sys
master_fd, slave_fd = pty.openpty()
process = subprocess.Popen(['ssh'] + sys.argv[1:],
stdin=slave_fd,
stdout=slave_fd,
stderr=subprocess.STDOUT,
# Important for Ctrl-c in the remote terminal.
preexec_fn=os.setsid)
def sigint_handler(_signum, _frame):
os.write(master_fd, b'\03') # Send Ctrl-c.
signal.signal(signal.SIGINT, sigint_handler)
def sigtstp_handler(_signum, _frame):
os.write(master_fd, b'\x1A') # Send Ctrl-z.
signal.signal(signal.SIGTSTP, sigtstp_handler)
def sigchld_handler(_signum, _frame):
process.wait()
sys.exit(process.returncode)
signal.signal(signal.SIGCHLD, sigchld_handler)
while process.poll() is None:
# Block until there is something to read or write.
r, w, e = select.select([sys.stdin, master_fd], [], [])
if sys.stdin in r:
# Write to SSH.
user_input = os.read(sys.stdin.fileno(), 4096)
if not user_input:
os.write(master_fd, b'\04') # Send Ctrl-d.
else:
os.write(master_fd, user_input)
if master_fd in r:
# Read from SSH.
data = os.read(master_fd, 4096)
sys.stdout.write(data.decode())
sys.stdout.flush()
I am using Python 3.8.10 on Ubuntu 20.04 on both my local computer and the remote computer. This is a self-education project, so I am writing the program using Python standard libraries only.

There is a bad hack you can try. After ssh into the machine try removing env variable LS_COLORS
export LS_COLORS=none
This change will persist in your session.

In your bashrc:
alias tmux="TERM=screen-256color-bce tmux"
In your .tmux.conf:
set -g default-terminal "xterm-256color"

Try to run your script from a different terminal application (preferably set to the defaults) and see if you still have the problem.

Script to capture everything on screen

So I have this python3 script that does a lot of automated testing for me, it takes roughly 20 minutes to run, and some user interaction is required. It also uses paramiko to ssh to a remote host for a separate test.
Eventually, I would like to hand this script over to the rest of my team however, it has one feature missing: evidence collection!
I need to capture everything that appears on the terminal to a file. I have been experimenting with the Linux command 'script'. However, I cannot find an automated method of starting script, and executing the script.
I have a command in /usr/bin/
script log_name;python3.5 /home/centos/scripts/test.py
When I run my command, it just stalls. Any help would be greatly appreciated!
Thanks :)

Is a redirection of the output to a file what you need ?
python3.5 /home/centos/scripts/test.py > output.log 2>&1
Or if you want to keep the output on the terminal AND save it into a file:
python3.5 /home/centos/scripts/test.py 2>&1 | tee output.log

I needed to do this, and ended up with a solution that combined pexpect and ttyrec.
ttyrec produces output files that can be played back with a few different player applications - I use TermTV and IPBT.
If memory serves, I had to use pexpect to launch ttyrec (as well as my test's other commands) because I was using Jenkins to schedule the execution of my test, and pexpect seemed to be the easiest way to get a working interactive shell in a Jenkins job.
In your situation you might be able to get away with using just ttyrec, and skip the pexpect step - try running ttyrec -e command as mentioned in the ttyrec docs.
Finally, on the topic of interactive shells, there's an alternative to pexpect named "empty" that I've had some success with too - see http://empty.sourceforge.net/. If you're running Ubuntu or Debian you can install empty with apt-get install empty-expect

I actually managed to do it in python3, took a lot of work, but here is the python solution:
def record_log(output):
try:
with open(LOG_RUN_OUTPUT, 'a') as file:
file.write(output)
except:
with open(LOG_RUN_OUTPUT, 'w') as file:
file.write(output)
def execute(cmd, store=True):
proc = Popen(cmd.encode("utf8"), shell=True, stdout=PIPE, stderr=PIPE)
output = "\n".join((out.decode()for out in proc.communicate()))
template = '''Command:\n====================\n%s\nResult:\n====================\n%s'''
output = template % (cmd, output)
print(output)
if store:
record_log(output)
return output
# SSH function
def ssh_connect(start_message, host_id, user_name, key, stage_commands):
print(start_message)
try:
ssh.connect(hostname=host_id, username=user_name, key_filename=key, timeout=120)
except:
print("Failed to connect to " + host_id)
for command in stage_commands:
try:
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(command)
except:
input("Paused, because " + command + " failed to run.\n Please verify and press enter to continue.")
else:
template = '''Command:\n====================\n%s\nResult:\n====================\n%s'''
output = ssh_stderr.read() + ssh_stdout.read()
output = template % (command, output)
record_log(output)
print(output)

Is there a way to make python become interactive in the middle of a script?

I'd like to do something like:
do lots of stuff to prepare a good environement
become_interactive
#wait for Ctrl-D
automatically clean up
Is it possible with python?If not, do you see another way of doing the same thing?

Use the -i flag when you start Python and set an atexit handler to run when cleaning up.
File script.py:
import atexit
def cleanup():
print "Goodbye"
atexit.register(cleanup)
print "Hello"
and then you just start Python with the -i flag:
C:\temp>\python26\python -i script.py
Hello
>>> print "interactive"
interactive
>>> ^Z
Goodbye

The code module will allow you to start a Python REPL.

With IPython v1.0, you can simply use
from IPython import embed
embed()
with more options shown in the docs.

To elaborate on IVA's answer: embedding-a-shell, incoporating code and Ipython.
def prompt(vars=None, message="welcome to the shell" ):
#prompt_message = "Welcome! Useful: G is the graph, DB, C"
prompt_message = message
try:
from IPython.Shell import IPShellEmbed
ipshell = IPShellEmbed(argv=[''],banner=prompt_message,exit_msg="Goodbye")
return ipshell
except ImportError:
if vars is None: vars=globals()
import code
import rlcompleter
import readline
readline.parse_and_bind("tab: complete")
# calling this with globals ensures we can see the environment
print prompt_message
shell = code.InteractiveConsole(vars)
return shell.interact
p = prompt()
p()

Not exactly the thing you want but python -i will start interactive prompt after executing the script.
-i : inspect interactively after running script, (also PYTHONINSPECT=x) and force prompts, even if stdin does not appear to be a terminal
$ python -i your-script.py
Python 2.5.4 (r254:67916, Jan 20 2010, 21:44:03)
...
>>>

You may call python itself:
import subprocess
print "Hola"
subprocess.call(["python"],shell=True)
print "Adios"

how to get console output from a remote computer (ssh + python)

I have googled "python ssh". There is a wonderful module pexpect, which can access a remote computer using ssh (with password).
After the remote computer is connected, I can execute other commands. However I cannot get the result in python again.
p = pexpect.spawn("ssh user#remote_computer")
print "connecting..."
p.waitnoecho()
p.sendline(my_password)
print "connected"
p.sendline("ps -ef")
p.expect(pexpect.EOF) # this will take very long time
print p.before
How to get the result of ps -ef in my case?

Have you tried an even simpler approach?
>>> from subprocess import Popen, PIPE
>>> stdout, stderr = Popen(['ssh', 'user#remote_computer', 'ps -ef'],
... stdout=PIPE).communicate()
>>> print(stdout)
Granted, this only works because I have ssh-agent running preloaded with a private key that the remote host knows about.

child = pexpect.spawn("ssh user#remote_computer ps -ef")
print "connecting..."
i = child.expect(['user#remote_computer\'s password:'])
child.sendline(user_password)
i = child.expect([' .*']) #or use i = child.expect([pexpect.EOF])
if i == 0:
print child.after # uncomment when using [' .*'] pattern
#print child.before # uncomment when using EOF pattern
else:
print "Unable to capture output"
Hope this help..

You might also want to investigate paramiko which is another SSH library for Python.

Try to send
p.sendline("ps -ef\n")
IIRC, the text you send is interpreted verbatim, so the other computer is probably waiting for you to complete the command.