Develop Reference

Python subprocess, execute shell command which prompts for password

I am using subprocess in python to execute a custom vpn command which excepts password to execute.
below is the command which asks password:
./vpn -u <user_id> -d "description" /var/tmp/1.txt
password: XXXX
below is vpn shell command which works perfectly for above command.
/usr/bin/expect -c 'spawn ./vpn -u <user_id> -d "description" /var/tmp/1.txt; expect "Password"; send "<Password here>\r"; interact'
In python I am trying to achieve the same with below subprocess module, where below script executes below is the output.
"(b"\x1b[31m\xe2\x9c\x97 Can't read 'user id' password from the console.\x1b[0m""
import subprocess
args = ["./vpn", "-u", "<user_id>", "-d", "description", "/var/tmp/1.txt"]
# args = ['sudo','cat', '/var/tmp/1.txt']
proc = subprocess.Popen(args,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
proc.stdin.write('<passowrd>'.encode())
stdout, stderr = proc.communicate()
print(stdout)
Note: if I use time.sleep(2) before writing password, it prompts me for password in console without any error and my goal is to give password in the script.
Is there any better way in python to pass the shell command as this is pretty lengthy OR provide password in subprocess module execute seamlessly.

You are using Expect for Shell/Bash CLI command. When you use Python, just use pexpect to get the same functionality in Python.
Explore pexpect at https://github.com/pexpect/pexpect
Example passing password at executed program is here: https://github.com/pexpect/pexpect/blob/master/examples/passmass.py
Example:
import pexpect
child = pexpect.spawn('ftp ftp.openbsd.org')
child.expect('Name .*: ')
child.sendline('anonymous')
child.expect('Password:')
child.sendline('noah#example.com')

I tried in shell script and it works well: Just for anyone if it will be help full:
/usr/bin/expect<<vpn
spawn ./vpn -u <user_id> -d "description" /var/tmp/1.txt
expect "Password"
send "<password here>\r"
interact
sleep 5
vpn
echo "It's done"

How can I use Python to automate setting a password using the Unix pass command line program

I'm trying to automate setting new passwords using the Unix pass program.
I understand that there is a Python library, pexpect, that might help, but I would like to avoid using third-party libraries.
When using a terminal, the flow looks like this:
$ pass insert --force gmail
>> Enter password for gmail: <type in password using masked prompt>
>> Retype password for gmail: <reenter password>
What I would like my function to do:
Run the command pass insert --force {entry_name}
Capture the output (and echo it for testing)
Check output for the presence of 'password for gmail', and if True
write '{password}\n' to stdin
write '{password}\n' to stdin again
Echo any errors or messages for testing
Issues:
I'm stuck on step 2. The subprocess either hangs indefinitely, times out with an error, or produces no output.
Attempts:
I've tried configurations of Popen(), using both stdin.write() and communicate().
I've set wait() calls at various points.
I've tried both the shell=True and shell=False options (prefer False for security reasons)
Code:
def set_pass_password(entry_name, password):
from subprocess import Popen, PIPE
command = ['pass', 'insert', '--force', entry_name]
sub = Popen(command, stdin=PIPE, stdout=PIPE, stderr=PIPE, universal_newlines=True)
# At this point I assume that the command has run, and that there is an "Enter password..." message
message = sub.stdout.read() # also tried readline() and readlines()
print(message) # never happens, because process hangs on stdout.read()
if 'password for {}'.format(entry_name) in message:
err, msg = sub.communicate(input='{p}\n{p}\n'.format(p=password))
print('errors: {}\nmessage: {}'.format(err, msg))

Edit: the original answer was about passwd, which is what's used to set passwords. I noticed late that you use pass, which is a keystore (doesn't actually change the Unix password). The pass program works differently and will not print a prompt if stdin is not a tty. Therefore the following very simple program works:
def set_pass_password(entry_name, password):
from subprocess import Popen, PIPE
command = ['pass', 'insert', '--force', entry_name]
sub = Popen(command, bufsize=0, stdin=PIPE, stdout=PIPE, stderr=PIPE)
err, msg = sub.communicate(input='{p}\n{p}\n'.format(p=password))
print('errors: {}\nmessage: {}'.format(err, msg))
if __name__ == "__main__":
set_pass_password("ttt", "ttt123asdqwe")
(you will see that both stderr and stdout are empty, if the command succeeded).
For the passwd command:
FYI: the passwd command outputs the prompt to stderr, not stdout.
NOTE: rather than sending the password twice in the same 'write', you might need to wait for the second prompt before sending the password again.
For this simple case, code similar to yours should work, but in general you should use select on all the pipes and send/receive data when the other side is ready, so you don't get deadlocks.

open a putty window and run ssh commands - Python

I am new to python. I need to login to a server daily (Desktop -> 1.32 -> 0.20 -> 3.26). For this I need to open putty and using ssh connection i am logging in. To do all this I want to write a script using python.
By using google I thought subprocess.Popen will do that. But Its not working fine.
1st trail:
import subprocess
pid = subprocess.Popen("putty.exe user#xxx.xx.x.32 -pw password").pid
Its working fine (Opening window logging into .32). But cant able to give input. I came to know that to give input for the same process we need to use pipes.
2nd trail:
from subprocess import Popen, PIPE, STDOUT
p = Popen("putty.exe user#xxx.xx.x.32 -pw password", stdout=PIPE, stdin=PIPE, stderr=STDOUT)
grep_stdout = p.communicate(input=b'ssh xx.xx.x.20\n')[0]
print(grep_stdout.decode())
by using this i cant login for the first server also. After logging in to all servers I need the terminal as alive. how to do this???
Edit
I need to do this in a new putty window. After logging in dont close the window. I have some manual work to do.

use powershell to call putty in order to open a new window
from subprocess import Popen
Popen("powershell putty.exe user#host -pw mypassword")

Use paramiko library python
Establish a SSH connection using -
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname,username, password)
Check the status if connection is alive using -
status = ssh.get_transport().is_active()
#returns True if connection is alive/active
ssh.exec_command() is basically a single session. Use exec_command(command1;command2) to execute multiple commands in one session
Also, you can use this to execute multiple commands in single session
channel = ssh.invoke_shell()
stdin = channel.makefile('wb')
stdout = channel.makefile('rb')
stdin.write('''
Command 1
Command 2
''')
print stdout.read()

There is a SSHv2 protocol implementation for python: http://www.paramiko.org/. You can easily install it with pip:
pip install paramiko
Then you can create ssh client, connect to your host and execute commands:
import paramiko
ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect('hostname', username='login', password='pwd')
stdin, stdout, stderr = ssh_client.exec_command('command')

I created a bat file on windows, which references putty and putty session-specific info. This bat file can run by itself on windows. To call from python, I used the subprocess.run() -- python 3.5+.
Example of bat file named putty.bat:
start c:\app\PuTTy\putty.exe -load 192.168.1.230-node1-logs -l <logon user> -pw <logon user password for putty session>
Breaking down the bat file:
It begins with window's command "start".
c:\app\PuTTy\putty.exe --> is the putty directory on Windows containing putty.exe.
-load --> tells putty to load a putty profile. The profile is the name you see on the putty client, under "Saved Sessions".
192.168.1.230-node1-logs --> my putty session specific profile.
-l for logon --> followed by the putty logon user.
-pw is the logon password --> followed by the putty logon password.
That concludes the contents of "putty.bat".
From within python, is used the subprocess.run() command.
Example:
import subprocess
...
...
try:
process = subprocess.run(["putty.bat"], check=True, stdout=subprocess.PIPE, universal_newlines=True)
print(process.stdout)
except Exception as e:
print("subprocess call error in open putty command")
print(str(e))
I hope you find this helpful

subprocess timeout when exceeds a certain time limit

I'm using the subprocess module of python to run an ssh command over my servers for collecting their disk usage. The one thing on which i'm stuck is if the ssh is not configured in any server then subprocess prompt for the password input which makes my whole script stuck and then i have to voluntarily kill the script itself. I just want it to let go all the servers which asks for password prompt(where ssh is not configured) and continue processing the rest.
def MyFunction(server):
msg=""
ps = subprocess.Popen("ssh -l mygroup %s 'df -k /some/directory'" % server,stdout=subprocess.PIPE,shell=True)
out, err = ps.communicate()
if err != None:
msg += "\n"+err
else:
msg = out
return msg
server_list= ['server A','server B','server C','server D']
for server in server_list:
Final_msg+=MyFunction(server)
Any help would be appreciated! :)

If it is just the thing that you want to avoid ssh ask you for anything, then you can forbid it to do so.
You can use the SSH option
BatchMode
If set to “yes”, passphrase/password querying will be disabled.
This option is useful in scripts and other batch jobs where no user is present to supply the password.
The argument must be “yes” or “no”. The default is “no”.
So just add -o BatchMode=yes:
ps = subprocess.Popen("ssh -o BatchMode=yes -l mygroup %s 'df -k /some/directory'" % server, stdout=subprocess.PIPE, shell=True)
BTW, why do you need shell=True here? Better do
ps = subprocess.Popen(["ssh", "-o", "BatchMode=yes", "-l", "mygroup", server, "df -k /some/directory"], stdout=subprocess.PIPE)
as it is cleaner, safer and internally simpler.

Use subprocess to send a password

I'm attempting to use the python subprocess module to log in to a secure ftp site and then grab a file. However I keep getting hung up on just trying to send the password when it is requested. I so far have the following code:
from subprocess import Popen, PIPE
proc = Popen(['sftp','user#server', 'stop'], stdin=PIPE)
proc.communicate('password')
This still stops at the password prompt. If I enter the password manually it then goes to the ftp site and then enters the password on the command line. I've seen people suggest using pexpect but long story short I need a standard library solution. Is there anyway with subprocess and/or any other stdlib? What am I forgetting above?

Try
proc.stdin.write('yourPassword\n')
proc.stdin.flush()
That should work.
What you describe sounds like stdin=None where the child process inherits the stdin of the parent (your Python program).

Perhaps you should use an expect-like library instead?
For instance Pexpect (example). There are other, similar python libraries as well.

from subprocess import Popen, PIPE
proc = Popen(['sftp','user#server', 'stop'], stdin=PIPE)
proc.communicate(input='password')
Try with input=‘password’ in communicate, that worked for me.

Use Paramiko for SFTP. For anything else, this works:
import subprocess
args = ['command-that-requires-password', '-user', 'me']
proc = subprocess.Popen(args,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
proc.stdin.write('mypassword\n')
proc.stdin.flush()
stdout, stderr = proc.communicate()
print stdout
print stderr

For some reason, I couldn't get any of the standard library answers here to work for me - getting very strange problems with all of them. Someone else here: unable to provide password to a process with subprocess [python] had the same problem, and concluded that ultimately you just have to go with pexpect to be able to send a password.
I wanted to add my final code here to just save the time of anyone having a similar problem, since I wasted so much time on this (Python 3, 2020):
ssh_password = getpass("user's password: ")
ssh_password = (ssh_password + "\n").encode()
scp_command = 'scp xx.xx.xx.xx:/path/to/file.log /local/save/path/'
child = pexpect.spawn(scp_command)
# make output visible for debugging / progress watching
child.logfile = sys.stdout.buffer
i = child.expect([pexpect.TIMEOUT, "password:"])
if i == 0:
print("Got unexpected output: {} {}".format(child.before, child.after))
return
else:
child.sendline(ssh_password)
child.read()
The above code runs an SCP command to pull a file from the remote server onto your local computer - alter the server IP and paths as necessary.
Key things to remember:
Have to have a pexpect.TIMEOUT in the child.expect call
Have to encode to bytes whatever strings you pass in, and have to use the default encode
Write pexpect output to sys.stdout.buffer so that you can actually see what is going on
Have to have a child.read() at the end

I would recommend scrapping the subprocess approach and using the paramiko package for sftp access.

This same problem plagued me for a week. I had to submit a password from user input through subprocess securely because I was trying to avoid introducing a command injection vulnerability. Here is how I solved the problem with a little help from a colleague.
import subprocess
command = ['command', 'option1', '--password']
subprocess.Popen(command, stdin=subprocess.PIPE).wait(timeout=60)
The .wait(timeout=int) was the most important component because it allows the user to feed input to stdin. Otherwise, the timeout is defaulted to 0 and leaves the user no time to enter input, which consequently results in a None or null string. Took me FOREVER to figure this out.
For repeat use-cases where you know you'll have to do this multiple times, you can override the popen function and use it as a private method which I was told by the same programmer is best practice if you anticipate someone else will be interested in maintaining the code later on and you don't want them to mess with it.
def _popen(cmd):
proc_h = subprocess.Popen(cmd, stdin=subprocess.PIPE)
proc_h.wait(timeout=60)
return proc_h.poll() == os.EX_OK
It is important to remove stdout=subprocess.PIPE if the user is going to be prompted for input. Otherwise, the process appears to hang for 60 seconds, and the user doesn't get a prompt, nor do they realize they are expected to give a password. The stdout will naturally go to the shell window and allow the user to pass input to popen().
Also, just to explain why you return proc_h.poll() == os.EX_OK, is that it returns 0 if the command succeeded. This is just c-style best-practice for when you want to return system error codes in the event the function fails, while accounting for the fact that return 0 will be treated as "false" by the interpreter.

This is a pure Python solution using expect - not pexpect.
If on Ubuntu you first need to install expect with:
sudo apt install expect
Python 3.6 or later:
def sftp_rename(from_name, to_name):
sftp_password = 'abigsecret'
sftp_username = 'foo'
destination_hostname = 'some_hostname'
from_name = 'oldfilename.txt'
to_name = 'newfilename.txt'
commands = f"""
spawn sftp -o "StrictHostKeyChecking no"
{sftp_username}#{destination_hostname}
expect "password:"
send "{sftp_password}\r"
expect "sftp>"
send "rename {from_name} {to_name}\r"
expect "sftp>"
send "bye\r"
expect "#"
"""
sp = subprocess.Popen(['expect', '-c', commands], stdin=subprocess.PIPE, stdout=subprocess.PIPE)

since what you want is just grab a file, I am trying to use "sub process" but it is not works for me. So now I am using paramiko, here is my code:
here is one tutorial I found online
Transfer a file from local server to remote server and vice versa using paramiko of python
"https://www.youtube.com/watch?v=dtvV2xKaVjw"
underneath is my code for transfering all the files in one folder from Linux to windows
import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname='11.11.11.1111', username='root', password='********', port=22)
sftp_client = ssh.open_sftp()
source_folder = '/var/ftp/file_pass'
local_folder = 'C:/temp/file_pass'
inbound_files = sftp_client.listdir(source_folder)
print(inbound_files)
for ele in inbound_files:
try:
path_from = source_folder + '/' + ele
path_to = local_folder + '/'+ ele
sftp_client.get(path_from, path_to)
except:
print(ele)
sftp_client.close()
ssh.close()

Python have a built in library called ftplib, that can be used for ftp processes without any hassle. (Assuming the remote server have a ftp service running)
from ftplib import FTP
ftp = FTP('ftp.us.debian.org') # connect to host, default port
ftp.login() # user anonymous, passwd anonymous#
##'230 Login successful.'
ftp.cwd('debian') # change into "debian" directory
##'250 Directory successfully changed.'
ftp.retrlines('LIST')
Otherwise, You can use scp command, which is a command line tool. The problem with the password can be avoided creating password less user for remote host.
import os
os.system('scp remoteuser#remotehost:/remote/location/remotefile.txt /client/location/')
To create a passwordless user in linux systems,
Fallow below Steps. Fallow this SO answer.
> ssh-keyscan remotehost
> known_hosts ssh-keygen -t rsa # ENTER toevery field (One time)
> ssh-copy-id remoteuser#remotehost

The safest way to do this is to prompt for the password beforehand and then pipe it into the command. Prompting for the password will avoid having the password saved anywhere in your code. Here's an example:
from getpass import getpass
from subprocess import Popen, PIPE
password = getpass("Please enter your password: ")
proc = Popen("sftp user#server stop".split(), stdin=PIPE)
# Popen only accepts byte-arrays so you must encode the string
proc.communicate(password.encode())

import subprocess
args = ['command', 'arg1', 'arg2']
proc = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
proc.stdin.write(b'password') ##The b prefix is necessary because it needs a byte type
proc.stdin.flush()
stdout, stderr = proc.communicate()
print(stdout)
print(stderr)

You just forgot the line return (aka user pressing Enter) in your password.
from subprocess import Popen, PIPE
proc = Popen(['sftp','user#server', 'stop'], stdin=PIPE)
proc.communicate('password\n'.encode())
Also .encode() because by default proc.communicate() accept bytes-like object.