I'm using a python script to manage ssh fingerprint problems after a workstation(s) is reimaged.
I attempt to connect with ssh, and if I get a any warnings I deal with them.
However, if there are no errors, then I am asked for a password to connect. At this point I want to terminate the process. However, the script hangs on the password request.
Here's the method:
def ssh_fingerprint_changed(node):
"""
Checks if a node's ssh fingerprint has changed or an old key is found, which can occur when a node is reimaged.
It does this by attempting to connect via ssh and inspecting stdout for an error message.
:param node: the ip or hostname of the node
:return: True if the node's fingerprint doesn't match the client's records. Else False.
"""
changed = False
cmd = ["ssh", "-q", ADMIN_USER + "#" + node, "exit"]
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
print("Checking for fingerprint changes")
for line in proc.stdout: # loop on lines
print("in for loop") # NEVER REACHES HERE IF NO ERRORS, WAITING FOR PASSWORD
if b"Offending key" in line:
print("Offending key found.")
proc.stdin.write(b"no\n") # don't connect
changed = True
elif b"REMOTE HOST IDENTIFICATION HAS CHANGED!" in line:
print("REMOTE HOST IDENTIFICATION HAS CHANGED!")
changed = True
print(changed) # NEVER REACHES HERE IF NO ERRORS, WAITING FOR PASSWORD
if not changed: # then everything's good, but it will be waiting for a password to connect
print("Good to go, terminating ssh test.")
rc = proc.terminate()
else:
rc = proc.wait()
return changed
If I run this from the terminal ./my_python_script.py, I have the problems. Oddly, if I run in PyCharm, it doesn't hang on the password request and terminates shh, continuing with the script as expected.
The easy answer is simply to tell ssh that you don't want to support password authentication at all; you'll still get the messages you want if the host key is changed, but you won't ever have the process hanging waiting for a password to be entered.
cmd = ['ssh',
'-o', 'PasswordAuthentication no', ### <-- THIS LINE HERE
'-o', 'StrictHostKeyChecking yes', ### also, never modify known_hosts
'-q',
'%s#%s' % (ADMIN_USER, + node),
'exit']
If you did not want to process other prompts, I would suggest setting stdin=subprocess.DEVNULL (in Python 3) or passing the -n argument to ssh to prevent stdin from being passed to the process at all.
Related
I have a follow up question that builds off the question I asked here: Run multiple commands in different SSH servers in parallel using Python Paramiko, which was already answered.
Thanks to the answer on the link above, my python script is as follows:
# SSH.py
import paramiko
import argparse
import os
path = "path"
python_script = "worker.py"
# definitions for ssh connection and cluster
ip_list = ['XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX']
port_list = [':XXXX', ':XXXX', ':XXXX']
user_list = ['user', 'user', 'user']
password_list = ['pass', 'pass', 'pass']
node_list = list(map(lambda x: f'-node{x + 1} ', list(range(len(ip_list)))))
cluster = ' '.join([node + ip + port for node, ip, port in zip(node_list, ip_list, port_list)])
# run script on command line of local machine
os.system(f"cd {path} && python {python_script} {cluster} -type worker -index 0 -batch 64 > {path}/logs/'command output'/{ip_list[0]}.log 2>&1")
# loop for IP and password
stdouts = []
clients = []
for i, (ip, user, password) in enumerate(zip(ip_list[1:], user_list[1:], password_list[1:]), 1):
try:
print("Open session in: " + ip + "...")
client = paramiko.SSHClient()
client.connect(ip, user, password)
except paramiko.SSHException:
print("Connection Failed")
quit()
try:
path = f"C:/Users/{user}/Desktop/temp-ines"
stdin, stdout, stderr = ssh.exec_command(
f"cd {path} && python {python_script} {cluster} -type worker -index {i} -batch 64>"
f"C:/Users/{user}/Desktop/{ip}.log 2>&1 &"
)
clients.append(ssh)
stdouts.append(stdout)
except paramiko.SSHException:
print("Cannot run file. Continue with other IPs in list...")
client.close()
continue
# Wait for commands to complete
for i in range(len(stdouts)):
print("hello")
stdouts[i].read()
print("hello1")
clients[i].close()
print('hello2")
print("\n\n***********************End execution***********************\n\n")
This script, which is run locally, is able to SSH into the servers and run the command (i.e., run a python script called worker.py and log the command output to a log file). I.e., it is able to go through the first for loop with no issues.
My issue is related to the second for loop. Please see the print statements I added in the second for loop to be clear. When I run SSH.py locally, this is what I observe:
As you can see, I ssh into each of the servers and then stay at reading the command output of the first server I ssh over to. The worker.py script can take 30 mins or so to complete and the command output is the same on each server -- so it will take 30 mins to read the command output of the first server, then close the SSH connection of the first server, take a couple seconds to read the command output of the second server (as it is the same as the first one and would already be entirely printed), close its SSH connection, and so on. Please see below some of the command line output, if this helps.
Now, my question is, what if I don't want to wait until the worker.py script finishes, i.e., those entire 30 mins? I cannot/do not know how to raise a KeyboardInterrupt exception. What I have tried is quitting the local SSH.py script. However, as you can see from the print statements, this will not close the SSH connections although the training, and thus the log files, will stop logging info. In addition, after I quit the local SSH.py script, if I try to delete any of the log files, I get an error saying "cannot delete file because it is being used in cmd.exe" -- this only happens sometimes and I believe it is because of not closing the SSH connections?
First run in python console:
It hangs: Local python and log file running and saving but no print statements and no python and log file being run/saved in servers.
I run it again so second process starts:
Now, the first process doesn't hang anymore (python running and log files being saved in server). And can close this second run/process. It is like the second run/process helps with the hang of the first run/process.
If I were to run python SSH.py in the terminal it would just hang.
This was not happening before.
If you know that SSHClient.close cleanly close the connection and abort the remote command, call it on response to KeyboardInterrupt.
For this you cannot use the simple solution with stdout.read, as it blocks and prevents handling of the Ctrl+C on Windows.
Use the waiting code from my answer to Run multiple commands in different SSH servers in parallel using Python Paramiko (the while any(x is not None for x in stdouts): snippet).
And wrap it to try:...except (KeyboardInterrupt):.
try:
while any(x is not None for x in stdouts):
for i in range(len(stdouts)):
stdout = stdouts[i]
if stdout is not None:
channel = stdout.channel
# To prevent losing output at the end, first test for exit,
# then for output
exited = channel.exit_status_ready()
while channel.recv_ready():
s = channel.recv(1024).decode('utf8')
print(f"#{i} stdout: {s}")
while channel.recv_stderr_ready():
s = channel.recv_stderr(1024).decode('utf8')
print(f"#{i} stderr: {s}")
if exited:
print(f"#{i} done")
clients[i].close()
stdouts[i] = None
time.sleep(0.1)
except (KeyboardInterrupt):
print("Aborting")
for i in range(len(clients)):
print(f"#{i} closing")
clients[i].close()
If you do not need to separate the stdout and stderr, you can greatly simplify the code by using Channel.set_combine_stderr. See Paramiko ssh die/hang with big output.
I'm trying to spawn an ssh child process using subprocess.
I'm working on Python 2.7.6 on Windows 7
here is my code:
from subprocess import *
r=Popen("ssh sshserver#localhost", stdout=PIPE)
stdout, stderr=r.communicate()
print(stdout)
print(stderr)
The outputs:
None
stdout should contain:
sshserver#localhost's password:
Here's an example of working SSH code that handles the promt for yes/no on the certificate part and also when asked for a password.
#!/usr/bin/python
import pty, sys
from subprocess import Popen, PIPE, STDOUT
from time import sleep
from os import fork, waitpid, execv, read, write
class ssh():
def __init__(self, host, execute='echo "done" > /root/testing.txt', askpass=False, user='root', password=b'SuperSecurePassword'):
self.exec = execute
self.host = host
self.user = user
self.password = password
self.askpass = askpass
self.run()
def run(self):
command = [
'/usr/bin/ssh',
self.user+'#'+self.host,
'-o', 'NumberOfPasswordPrompts=1',
self.exec,
]
# PID = 0 for child, and the PID of the child for the parent
pid, child_fd = pty.fork()
if not pid: # Child process
# Replace child process with our SSH process
execv(command[0], command)
## if we havn't setup pub-key authentication
## we can loop for a password promt and "insert" the password.
while self.askpass:
try:
output = read(child_fd, 1024).strip()
except:
break
lower = output.lower()
# Write the password
if b'password:' in lower:
write(child_fd, self.password + b'\n')
break
elif b'are you sure you want to continue connecting' in lower:
# Adding key to known_hosts
write(child_fd, b'yes\n')
elif b'company privacy warning' in lower:
pass # This is an understood message
else:
print('Error:',output)
waitpid(pid, 0)
The reason (and correct me if i'm wrong here) for you not being able to read the stdin straight away is because SSH runs as a subprocess under a different process ID which you need to read/attach to.
Since you're using windows, pty will not work. there's two solutions that would work better and that's pexpect and as someone pointed out key-based authentication.
In order to achieve a key-based authentication you only need to do the following:
On your client, run: ssh-keygen
Copy your id_rsa.pub content (one line) into /home/user/.ssh/authorized_keys on the server.
And you're done.
If not, go with pexpect.
import pexpect
child = pexpect.spawn('ssh user#host.com')
child.expect('Password:')
child.sendline('SuperSecretPassword')
My goal is to connect to SSH with python and authenticate which i can do with Paramiko or Fabric. But i would like to keep the session open after each execution and read the input/output. With paramiko i can only run 1 command before the session is closed and i am asked to authenticate again and the session hangs. And since fabric is using the paramiko library its giving me the same issue. For example if my directory structure is like this
-home
--myfolder1
--myfolder2
I would like to execute the below commands without having to re-authenticate because the sessions closes.
(make connection)
run cmd: 'pwd'
output: /home
run cmd: 'cd myfolder2'
run cmd: 'pwd'
output: /home/myfolder2
Is this possible with any module that is out there right now? Could it be made from scratch with native python? And also is this just not possible...?
Edit Added code. Without the new open_session it closes and i cannot run any command. After running the first command with this i will be prompted again to authenticate and it creates an infinite loop.
Edit2 If it closes after each command then there is no way this will work at all correct?
edit3 If i run this on a different server and exec_command with the paramikio.SSHClient it wont ask me to reauthenticate but if i 'cd somedir' and then 'pwd' it will output that i am back in the root directory of where i created.
class connect:
newconnection = ''
def __init__(self,username,password):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
ssh.connect('someserver', username=username,password=password,port=22,timeout=5)
except:
print "Count not connect"
sys.exit()
self.newconnection = ssh
def con(self):
return self.newconnection
#This will create the connection
sshconnection = connect('someuser','somepassword').con()
while True:
cmd = raw_input("Command to run: ")
if cmd == "":
break
try:
transport = sshconnection.get_transport()
transport.set_keepalive(999999)
chan = transport.open_session()
chan.settimeout(3)
chan.setblocking(0)
except:
print "Failed to open a channel"
chan.get_exception()
sys.exit()
print "running '%s'" % cmd
stdout_data = []
stderr_data = []
pprint.pprint(chan)
nbytes = 4096
chan.settimeout(5)
chan.get_pty()
chan.exec_command(cmd)
while True:
print "Inside loop " , chan.exit_status_ready()
time.sleep(1.2)
if chan.recv_ready():
print "First if"
stdout_data.append(chan.recv(nbytes))
if chan.recv_stderr_ready():
print "Recv Ready"
stderr_data.append(chan.recv_stderr(nbytes))
if chan.exit_status_ready():
print "Breaking"
break
print 'exit status: ', chan.recv_exit_status()
print ''.join(stdout_data)
This is possible by using the normal modules when you can concatenate the commands into one. Try
pwd ; cd myfolder2 ; pwd
as command. This should work but quickly becomes tedious when you have more complex commands which need arguments and horrible when the arguments contain spaces. The next step then is to copy a script with all the commands to the remote side and tell ssh to execute said script.
Another problem of this approach is that SSH doesn't return until all commands have executed.
Alternatively, you could build a "command server", i.e. a simple TCP server that listens for incoming connections and executes commands sent to it. It's pretty simple to write but also pretty insecure. Again, the solution is to turn the server into a (Python) script which reads commands from stdin and start that script remotely via SSH and then send commands.
I have a class that creates the connection. I can connect and execute 1 command before the channel is closed. On another system i have i can execute multiple commands and the channel does not close. Obviously its a config issue with the systems i am trying to connect to.
class connect:
newconnection = ''
def __init__(self,username,password):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
ssh.connect('somehost', username=username,password=password,port=2222,timeout=5)
except:
print "Count not connect"
sys.exit()
self.newconnection = ssh
def con(self):
return self.newconnection
Then i use 'ls' command just to print some output
sshconnection = connect('someuser','somepassword').con()
stdin, stdout, stderr = sshconnection.exec_command("ls -lsa")
print stdout.readlines()
print stdout
stdin, stdout, stderr = sshconnection.exec_command("ls -lsa")
print stdout.readlines()
print stdout
sshconnection.close()
sys.exit()
After the first exec_command runs it prints the expected output of the dir list. When i print stdout after the first exec_command it looks like the channel is closed
<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x2400f10L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>
Like i said on another system i am able to keep running commands and the connection doesn't close. Is there a way i can keep this open? or a better way i can see the reason why it closes?
edit: So it looks like you can only run 1 command per SSHClient.exec_command... so i decided to get_transport().open_session() and then run a command. The first one always works. The second one always fails and the scripts just hangs
With just paramiko after the exec_command executes the channel is closed and the ssh returns an auth prompt.
Seems its not possible with just paramiko, try fabric or another tool.
** fabric did not work out too.
Please see the following referece as it provides a way to do this in Paramiko:
How do you execute multiple commands in a single session in Paramiko? (Python)
it's possible with netmiko (tested on windows).
this example is written for connecting to cisco devices but the principle is adaptable for others as well.
import netmiko
from netmiko import ConnectHandler
import json
def connect_enable_silent(ip_address,ios_command):
with open ("credentials.txt") as line:
line_1 = json.load(line)
for k,v in line_1.items():
router=(k,v)
try:
ssh = ConnectHandler(**router[1],device_type="cisco_ios",ip=ip_address)
ssh.enable()
except netmiko.ssh_exception.NetMikoAuthenticationException:
#incorrect credentials
continue
except netmiko.ssh_exception.NetMikoTimeoutException:
#oddly enough if it can log in but not able to authenticate to enable mode the ssh.enable() command does not give an authentication error
#but a time-out error instead
try:
ssh = ConnectHandler(username = router[1]['username'],password = router[1]['password'],device_type="cisco_ios", ip=ip_address)
except netmiko.ssh_exception.NetMikoTimeoutException:
# connection timed out (ssh not enabled on device, try telnet)
continue
except Exception:
continue
else:
output = ssh.send_command(ios_command)
ssh.disconnect()
if "at '^' marker." in output:
#trying to run a command that requires enble mode but not authenticated to enable mode
continue
return output
except Exception:
continue
else:
output = ssh.send_command(ios_command)
ssh.disconnect()
return output
output = connect_enable_silent(ip_address,ios_command)
for line in output.split('\n'):
print(line)
Credentials text is meant to store different credentials in case you are planning to call this function to access multiple devices and not all of them using the same credentials. It is in the format:
{"credentials_1":{"username":"username_1","password":"password_1","secret":"secret_1"},
"credentials_2":{"username":"username_2","password":"password_2","secret":"secret_2"},
"credentials_3": {"username": "username_3", "password": "password_3"}
}
The exceptions can be changed to do different things, in my case i just needed it to not return an error and continue trying the next set, which is why most exceptions are silenced.
I am using this code to add server to the known_hosts:
subprocess.Popen(['sshpass', '-p', password, 'ssh', '-o', 'StrictHostKeyChecking=no', add_key], stdout=subprocess.PIPE).communicate()[0]
This adds hostname to the known_hosts but the server hangs as it tries to enter into the host. I just want add hostname to the known_hosts and continue with my other codes. How can I do that? Thanks
This should do the job. This solution use the pexpect library which is a great way to automate commands. You basically call add_known_hosts with the host, user, password that you want added. it will try to ssh to that host and either enters the password or responds to the add to known hosts connection with a yes and then enters the password. Finally it cleanly exits the connection by sending the exit command. You can modify this and not require a username and password and just answer yes to the continue connecting question and then end the ssh process instead of continuing with the password prompt.
import pexpect, time
def add_known_hosts(host, user, password):
child = pexpect.spawn('ssh %s#%s' % (user, host))
i = child.expect(['.* password:', '.* continue connecting (yes/no)?'])
if i == 1:
child.sendline('yes')
child.expect('.* password:')
child.sendline(password)
child.expect('%s#%s' % (user, host))
time.sleep(1)
child.sendline('exit')
add_known_hosts('myhost', 'myusername', 'mypassword')
debugging
from the comments below it seems you are facing issues using pexpect on your system. A good way to just to do a simple test to confirm pexpect is working correctly with ssh is to run the code below. Fill in host, user with your appropriate settings and then run the code, you should be able to interact with the ssh session. At this point you can build up from here and see exactly what text you are expecting to get from ssh and what text you want to send in response.
import pexpect
host, user = 'myhost', 'myusername'
child = pexpect.spawn('ssh %s#%s' % (user, host))
child.interact()
Never mind, Solved it myself. Here's what I did:
test = subprocess.Popen(['sshpass', '-p', password, 'ssh', '-o', 'StrictHostKeyChecking=no', add_key])
time.sleep(5.0)
test.kill()
Thanks!