Develop Reference

getpass behaves different in pychram IDE and terminal

paaword.py is a script where getpass() asked the user about the password and validates it. but i want to automate the whole process and used subprocess for it (main.py). And i am using python3.10
Problem:
problem is when i run the main.py in pycharm IDE it works normally (it automates the process). but when I run the script python3 main.py in ubuntu terminal it asked for the input.
I dont know why it behaves deifferent in in IDE and terminal?
password.py
import warnings
import getpass
import time
# Suppress warnings
warnings.filterwarnings("ignore", category=getpass.GetPassWarning)
for x in range(10):
print(f"curnt index {x}")
time.sleep(5)
password = getpass.getpass("Enter your password: ")
if password != "test":
print("wrong password")
else:
print("correct password")
main.py
import subprocess
# subprocess
proc = subprocess.Popen(["python", "password.py"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
password = "test"
input_data = f"{password}\n"
# read output from the subprocess in real-time
while True:
if proc.poll() is not None:
break
proc.stdin.write(input_data.encode())
proc.stdin.flush()
output = proc.stdout.readline().decode().strip()
if output:
print(output)
output in pycharm:
output in ubuntu terminal (20.04)

Judging by the screenshots, your OS is Linux.
In Linux, getpass() first tries to read directly from the process' controlling terminal (/dev/tty), or, if that fails, stdin using direct terminal I/O; and only if that fails, it falls back to regular I/O, displaying a warning.
Judging by the warnings in the IDE, the latter is exactly what happens in your first case.
Lib/getpass.py:
def unix_getpass(prompt='Password: ', stream=None):
<...>
try:
# Always try reading and writing directly on the tty first.
fd = os.open('/dev/tty', os.O_RDWR|os.O_NOCTTY)
tty = io.FileIO(fd, 'w+')
<...>
input = io.TextIOWrapper(tty)
<...>
except OSError:
# If that fails, see if stdin can be controlled.
<...>
try:
fd = sys.stdin.fileno()
except (AttributeError, ValueError):
fd = None
passwd = fallback_getpass(prompt, stream) # fallback_getpass is what displays the warnings
input = sys.stdin
<...>
if fd is not None:
try:
old = termios.tcgetattr(fd)
<...>
except termios.error:
<...>
passwd = fallback_getpass(prompt, stream)
<...>
return passwd
As you can see, getpass() is specifically designed to be interactive and resist intercepting its input. So if you need to provide a password automatically, use another way:
store it in a file readable only by you (e.g. SSH does that; you can provide that file as an argument and store other arguments there as well), or
use the system's keyring
and only fall back to getpass if the password was not provided that way and/or if you detect that the program is being run interactively (sys.stdin.isatty())
while it's also possible to provide the password on the command line -- in that case, you have to overwrite it in your process' stored command line to hide it from snooping. I couldn't find a way to do that in Python.
You can check Secure Password Handling in Python | Martin Heinz | Personal Website & Blog for a more detailed rundown of the above. (note: it suggests using envvars and load them from .env which would probably not apply to you. That's designed for .NET projects which due to the rigid structure of MS Visual Studio's build system, have had to rely on envvars for any variable values.)

How can I execute OS Commands with Python pexpect?

I want to execute a python script, which switches to another user by automatically writing the user password. Both users have no root rights. After the login I want to execute the OS Commands "whoami" to check if the login was successful. Here's the code:
child = pexpect.spawn('su - otheruser)
child.expect_exact('Password:')
child.sendline('password')
print("logged in...")
child.expect('')
child.sendline('whoami')
print(child.before)
I want to print the output from the command to the console (just for debugging) but the output is like "b272' (a combination of random letters) and not the actual whoami user. How can I fix that?
Later I want to create from the switched user some files and so on. So basically, I want to execute OS Commands in a python script which is logged in an other user.

Pexpect searches are not greedy, so it will stop at the first match. When I tested your code with before, match.groups(), after, and buffer, I didn't get an EOF or TIMEOUT, so it must have matched right at the beginning of the read and returned nothing (I'm surprised you got any results at all).
I recommend always following a sendline with an expect, and the end of a prompt (]$) is a good thing to expect, instead of an empty string.
Here is my take on your code, including creating a file:
NOTE - Tested on Centos 7.9, using Python 2.7.
import pexpect
child = pexpect.spawn("su - orcam")
child.expect_exact("Password:")
child.sendline("**********")
child.expect_exact("]$")
print("Logged in...\n")
child.sendline("whoami")
child.expect_exact("]$")
print(child.before + "\n")
child.sendline("echo -e 'Hello, world.' >> hello.txt")
child.expect_exact("]$")
child.sendline("cat hello.txt")
child.expect_exact("]$")
print(child.before + "\n")
child.sendline("exit")
index = child.expect_exact(["logout", pexpect.EOF, ])
print("Logged out: {0}".format(index))
Output:
Logged in...
whoami
orcam
[orcam#localhost ~
cat hello.txt
Hello, world.
[orcam#localhost ~
Logged out: 0

raw_input() prompt disappears when redirecting output to file

I am writing a CLI that accepts an email and password for auth.
The email prompt uses raw_input() and the password prompt uses getpass() for obfuscation.
This setup works fine when outputting directly to console, but falters when redirecting the output to a log file.
Sample code:
user_email = raw_input('Email: ')
user_password = getpass('Password: ')
Sample output without redirection:
$ python script_that_does_stuff.py
Email: me#email.com
Password:
Doing stuff...
Sample output with redirection:
$ python script_that_does_stuff.py > stuff.log
Because I know that it's expecting a user input here, I can type the email, hit enter, and then it will show:
$ python script_that_does_stuff.py > stuff.log
me#email.com
Password:
After inputting a password, it continues as usual, however the log shows the following:
$ cat stuff.log
Email:Doing stuff...
Question:
How can I force the raw_input() prompt to show up in console like the getpass() prompt does when redirecting output to a file?
Environment
This script lives in a legacy Python 2.7 codebase, and is run primarily on Mac OS systems, occasionally Linux.

You can override sys.stdout temporarily to write to the terminal. For example,
import contextlib
import sys
#contextlib.contextmanager
def output_to_terminal():
try:
with open("/dev/tty") as f:
sys.stdout = f
yield
finally:
# Ensure sys.stdout is restored in the event of an error
sys.stdout = sys.__stdout__
with output_to_terminal():
x = raw_input("> ")
print(x)
(This was derived independently; you may want to check source for Python 3's redirect_stdout, also found in the contextlib module, and back port it for your use.)

This answer on another question seems to work for me.
In short, create a custom input function:
def email_input(prompt=None):
if prompt:
sys.stderr.write(str(prompt))
return raw_input()
The calling code then becomes:
user_email = email_input('Email: ')
user_password = getpass('Password: ')
This results in both the Email and Password prompts being sent to stderr (printing to console), and not messing with the redirected log output.

According to official documentation getpass([prompt[, stream]]) has the second optional parameter which indicates output stream to print the prompt to (stderr by default).
When you redirect the output (stdout) the prompt is still printed to stderr for getpass but raw_input does not support setting an output stream so its prompt is redirecting to to the target file.
So to solve your issue, you have to print your prompt to stderr for email as well.

Script to capture everything on screen

So I have this python3 script that does a lot of automated testing for me, it takes roughly 20 minutes to run, and some user interaction is required. It also uses paramiko to ssh to a remote host for a separate test.
Eventually, I would like to hand this script over to the rest of my team however, it has one feature missing: evidence collection!
I need to capture everything that appears on the terminal to a file. I have been experimenting with the Linux command 'script'. However, I cannot find an automated method of starting script, and executing the script.
I have a command in /usr/bin/
script log_name;python3.5 /home/centos/scripts/test.py
When I run my command, it just stalls. Any help would be greatly appreciated!
Thanks :)

Is a redirection of the output to a file what you need ?
python3.5 /home/centos/scripts/test.py > output.log 2>&1
Or if you want to keep the output on the terminal AND save it into a file:
python3.5 /home/centos/scripts/test.py 2>&1 | tee output.log

I needed to do this, and ended up with a solution that combined pexpect and ttyrec.
ttyrec produces output files that can be played back with a few different player applications - I use TermTV and IPBT.
If memory serves, I had to use pexpect to launch ttyrec (as well as my test's other commands) because I was using Jenkins to schedule the execution of my test, and pexpect seemed to be the easiest way to get a working interactive shell in a Jenkins job.
In your situation you might be able to get away with using just ttyrec, and skip the pexpect step - try running ttyrec -e command as mentioned in the ttyrec docs.
Finally, on the topic of interactive shells, there's an alternative to pexpect named "empty" that I've had some success with too - see http://empty.sourceforge.net/. If you're running Ubuntu or Debian you can install empty with apt-get install empty-expect

I actually managed to do it in python3, took a lot of work, but here is the python solution:
def record_log(output):
try:
with open(LOG_RUN_OUTPUT, 'a') as file:
file.write(output)
except:
with open(LOG_RUN_OUTPUT, 'w') as file:
file.write(output)
def execute(cmd, store=True):
proc = Popen(cmd.encode("utf8"), shell=True, stdout=PIPE, stderr=PIPE)
output = "\n".join((out.decode()for out in proc.communicate()))
template = '''Command:\n====================\n%s\nResult:\n====================\n%s'''
output = template % (cmd, output)
print(output)
if store:
record_log(output)
return output
# SSH function
def ssh_connect(start_message, host_id, user_name, key, stage_commands):
print(start_message)
try:
ssh.connect(hostname=host_id, username=user_name, key_filename=key, timeout=120)
except:
print("Failed to connect to " + host_id)
for command in stage_commands:
try:
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(command)
except:
input("Paused, because " + command + " failed to run.\n Please verify and press enter to continue.")
else:
template = '''Command:\n====================\n%s\nResult:\n====================\n%s'''
output = ssh_stderr.read() + ssh_stdout.read()
output = template % (command, output)
record_log(output)
print(output)

How to hide the password in fabric when the command is printed out?

Say I have a fabfile.py that looks like this:
def setup():
pwd = getpass('mysql password: ')
run('mysql -umoo -p%s something' % pwd)
The output of this is:
[host] run: mysql -umoo -pTheActualPassword
Is there a way to make the output look like this?
[host] run: mysql -umoo -p*******
Note: This is not a mysql question!

Rather than modifying / overriding Fabric, you could replace stdout (or any iostream) with a filter.
Here's an example of overriding stdout to censor a specific password. It gets the password from Fabric's env.password variable, set by the -I argument. Note that you could do the same thing with a regular expression, so that you wouldn't have to specify the password in the filter.
I should also mention, this isn't the most efficient code in the world, but if you're using fabric you're likely gluing a couple things together and care more about manageability than speed.
#!/usr/bin/python
import sys
import string
from fabric.api import *
from fabric.tasks import *
from fabric.contrib import *
class StreamFilter(object):
def __init__(self, filter, stream):
self.stream = stream
self.filter = filter
def write(self,data):
data = data.replace(self.filter, '[[TOP SECRET]]')
self.stream.write(data)
self.stream.flush()
def flush(self):
self.stream.flush()
#task
def can_you_see_the_password():
sys.stdout = StreamFilter(env.password, sys.stdout)
print 'Hello there'
print 'My password is %s' % env.password
When run:
fab -I can_you_see_the_password
Initial value for env.password:
this will produce:
Hello there
My password is [[TOP SECRET]]

It may be better to put the password in the user's ~/.my.cnf under the [client] section. This way you don't have to put the password in the python file.
[client]
password=TheActualPassword

When you use the Fabric command run, Fabric isn't aware of whether or not the command you are running contains a plain-text password or not. Without modifying/overriding the Fabric source code, I don't think you can get the output that you want where the command being run is shown but the password is replaced with asterisks.
You could, however, change the Fabric output level, either for the entire Fabric script or a portion, so that the command being run is not displayed. While this will hide the password, the downside is that you wouldn't see the command at all.
Take a look at the Fabric documentation on Managing Output.

Write a shell script that invokes the command in question with the appropriate password, but without echoing that password. You can have the shell script lookup the password from a more secure location than from your .py files.
Then have fabric call the shell script instead.
This solves both the problem of having fabric not display the password and making sure you don't have credentials in your source code.

from fabric.api import run, settings
with settings(prompts={'Enter password: ': mysql_password}):
run("mysql -u {} -p -e {}".format(mysql_user,mysql_query))
or if no prompt available:
from fabric.api import run, hide
with hide('output','running','warnings'):
run("mycommand --password {}".format(my_password))