Develop Reference

How to create a python script to ssh and run commands on multiple linux devices

I've been scalping to find an explained python script to ssh to multiple Linux devices via ssh using paramiko. I found a few tweaked them and actually worked but i still cannot understand how it works.
Can someone please explain in the most appealing way how a python script to ssh to multiple Linux device and run commands works? Just as you would explain to a 6 year old. I mean I saw some scripts and they haver some complexity for no reason. I just wanted a Python script that imports a txt of hostnames and a another txt of commands and runs the same commands to all the Linux devices and returns the output. If someone could explain how such a script works I would be grateful.
ip = input("Please enter the hostname ")
user = ""
password = ""
print ("creating ssh")
ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
print("please wait")
ssh_client.connect(hostname=ip, username=user, password=password)
cmd= "uname -r;pwd"
print("please wait")
sdtin,stdout,stderr = ssh_client.exec_command(cmd)
print("success")
stdout = stdout.readlines()
stdout = "".join(stdout)
print (stdout)
I want this but for multiple devices and getting the hostnames and commands from 2 separate files.

Turns out I've found the answer (the perfect script for what I want).
Here it is:
import paramiko
import time
p = paramiko.SSHClient()
cred = open("hostnames.csv","r")
for i in cred.readlines():
line=i.strip()
ls =line.split(",")
print(ls[0])
p.set_missing_host_key_policy(paramiko.AutoAddPolicy())
p.connect("%s"%ls[0],port =22, username = "%s"%ls[1], password="%s"%ls[2])
stdin, stdout, stderr = p.exec_command('sudo pwd' , get_pty=True)
stdin.write("%s\n"%ls[2])
stdin.flush()
opt = stdout.readlines()
opt ="".join(opt)
print(opt)
cred.close()
So, it basically reads the csv file with is formatted as : hostname,username,password
and runs the script sequentially. Also, it can run sudo commands this way :)

Gracefully abort remote Windows command executed over SSH from Windows Python Paramiko script when Ctrl+C is pressed

I have a follow up question that builds off the question I asked here: Run multiple commands in different SSH servers in parallel using Python Paramiko, which was already answered.
Thanks to the answer on the link above, my python script is as follows:
# SSH.py
import paramiko
import argparse
import os
path = "path"
python_script = "worker.py"
# definitions for ssh connection and cluster
ip_list = ['XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX']
port_list = [':XXXX', ':XXXX', ':XXXX']
user_list = ['user', 'user', 'user']
password_list = ['pass', 'pass', 'pass']
node_list = list(map(lambda x: f'-node{x + 1} ', list(range(len(ip_list)))))
cluster = ' '.join([node + ip + port for node, ip, port in zip(node_list, ip_list, port_list)])
# run script on command line of local machine
os.system(f"cd {path} && python {python_script} {cluster} -type worker -index 0 -batch 64 > {path}/logs/'command output'/{ip_list[0]}.log 2>&1")
# loop for IP and password
stdouts = []

clients = []
for i, (ip, user, password) in enumerate(zip(ip_list[1:], user_list[1:], password_list[1:]), 1):
try:
print("Open session in: " + ip + "...")
client = paramiko.SSHClient()
client.connect(ip, user, password)
except paramiko.SSHException:
print("Connection Failed")
quit()
try:
path = f"C:/Users/{user}/Desktop/temp-ines"
stdin, stdout, stderr = ssh.exec_command(
f"cd {path} && python {python_script} {cluster} -type worker -index {i} -batch 64>"

 f"C:/Users/{user}/Desktop/{ip}.log 2>&1 &"
)

clients.append(ssh)
stdouts.append(stdout)
except paramiko.SSHException:
print("Cannot run file. Continue with other IPs in list...")
client.close()
continue
# Wait for commands to complete
for i in range(len(stdouts)):
print("hello")
stdouts[i].read()
print("hello1")
clients[i].close()
print('hello2")
print("\n\n***********************End execution***********************\n\n")
This script, which is run locally, is able to SSH into the servers and run the command (i.e., run a python script called worker.py and log the command output to a log file). I.e., it is able to go through the first for loop with no issues.
My issue is related to the second for loop. Please see the print statements I added in the second for loop to be clear. When I run SSH.py locally, this is what I observe:
As you can see, I ssh into each of the servers and then stay at reading the command output of the first server I ssh over to. The worker.py script can take 30 mins or so to complete and the command output is the same on each server -- so it will take 30 mins to read the command output of the first server, then close the SSH connection of the first server, take a couple seconds to read the command output of the second server (as it is the same as the first one and would already be entirely printed), close its SSH connection, and so on. Please see below some of the command line output, if this helps.
Now, my question is, what if I don't want to wait until the worker.py script finishes, i.e., those entire 30 mins? I cannot/do not know how to raise a KeyboardInterrupt exception. What I have tried is quitting the local SSH.py script. However, as you can see from the print statements, this will not close the SSH connections although the training, and thus the log files, will stop logging info. In addition, after I quit the local SSH.py script, if I try to delete any of the log files, I get an error saying "cannot delete file because it is being used in cmd.exe" -- this only happens sometimes and I believe it is because of not closing the SSH connections?
First run in python console:
It hangs: Local python and log file running and saving but no print statements and no python and log file being run/saved in servers.
I run it again so second process starts:
Now, the first process doesn't hang anymore (python running and log files being saved in server). And can close this second run/process. It is like the second run/process helps with the hang of the first run/process.
If I were to run python SSH.py in the terminal it would just hang.
This was not happening before.

If you know that SSHClient.close cleanly close the connection and abort the remote command, call it on response to KeyboardInterrupt.
For this you cannot use the simple solution with stdout.read, as it blocks and prevents handling of the Ctrl+C on Windows.
Use the waiting code from my answer to Run multiple commands in different SSH servers in parallel using Python Paramiko (the while any(x is not None for x in stdouts): snippet).
And wrap it to try:...except (KeyboardInterrupt):.
try:
while any(x is not None for x in stdouts):
for i in range(len(stdouts)):
stdout = stdouts[i]
if stdout is not None:
channel = stdout.channel
# To prevent losing output at the end, first test for exit,
# then for output
exited = channel.exit_status_ready()
while channel.recv_ready():
s = channel.recv(1024).decode('utf8')
print(f"#{i} stdout: {s}")
while channel.recv_stderr_ready():
s = channel.recv_stderr(1024).decode('utf8')
print(f"#{i} stderr: {s}")
if exited:
print(f"#{i} done")
clients[i].close()
stdouts[i] = None
time.sleep(0.1)
except (KeyboardInterrupt):
print("Aborting")
for i in range(len(clients)):
print(f"#{i} closing")
clients[i].close()
If you do not need to separate the stdout and stderr, you can greatly simplify the code by using Channel.set_combine_stderr. See Paramiko ssh die/hang with big output.

How can I see contents of gzip file with paramiko?

Once I connect to a remote server as follows,
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
I can do sftp.listdir() and see that there are some gzip files on the remote server, like example.gz.2016. How can I access the text of this file through the sftp connection, without actually downloading the file?

Your question has two parts:
How to view the content of a zip file from the command line
How to execute remote commands and get the output using python&paramiko
First things first: How to list the content of a zip file on the console.
less can look into zip files, so in your case, executing less example.gz.2016 should give you a list of files inside the zip archive
Second: how to execute commands remotely.
import paramiko
ssh = paramiko.SSHClient()
# next is needed if your keys are not yet known on the client.
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(HOST_NAME, username=USER, password=PASSWORD)
stdin, stdout, stderr = ssh.exec_command('less ' + zipfilename)
for line in stdout.readlines():
# here you will have your listing
print (line)
for errline in stderr.readlines():
# Don't forget to check the error output
print ('***', errline)
Good Luck!
EDIT
If you need a sFTP connection to the same server, you need to get it from your ssh connection like this
sftp = ssh.open_sftp()

Python paramiko module using multiple commands

I have a class that creates the connection. I can connect and execute 1 command before the channel is closed. On another system i have i can execute multiple commands and the channel does not close. Obviously its a config issue with the systems i am trying to connect to.
class connect:
newconnection = ''
def __init__(self,username,password):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
ssh.connect('somehost', username=username,password=password,port=2222,timeout=5)
except:
print "Count not connect"
sys.exit()
self.newconnection = ssh
def con(self):
return self.newconnection
Then i use 'ls' command just to print some output
sshconnection = connect('someuser','somepassword').con()
stdin, stdout, stderr = sshconnection.exec_command("ls -lsa")
print stdout.readlines()
print stdout
stdin, stdout, stderr = sshconnection.exec_command("ls -lsa")
print stdout.readlines()
print stdout
sshconnection.close()
sys.exit()
After the first exec_command runs it prints the expected output of the dir list. When i print stdout after the first exec_command it looks like the channel is closed
<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x2400f10L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>
Like i said on another system i am able to keep running commands and the connection doesn't close. Is there a way i can keep this open? or a better way i can see the reason why it closes?
edit: So it looks like you can only run 1 command per SSHClient.exec_command... so i decided to get_transport().open_session() and then run a command. The first one always works. The second one always fails and the scripts just hangs

With just paramiko after the exec_command executes the channel is closed and the ssh returns an auth prompt.
Seems its not possible with just paramiko, try fabric or another tool.
** fabric did not work out too.

Please see the following referece as it provides a way to do this in Paramiko:
How do you execute multiple commands in a single session in Paramiko? (Python)

it's possible with netmiko (tested on windows).
this example is written for connecting to cisco devices but the principle is adaptable for others as well.
import netmiko
from netmiko import ConnectHandler
import json
def connect_enable_silent(ip_address,ios_command):
with open ("credentials.txt") as line:
line_1 = json.load(line)
for k,v in line_1.items():
router=(k,v)
try:
ssh = ConnectHandler(**router[1],device_type="cisco_ios",ip=ip_address)
ssh.enable()
except netmiko.ssh_exception.NetMikoAuthenticationException:
#incorrect credentials
continue
except netmiko.ssh_exception.NetMikoTimeoutException:
#oddly enough if it can log in but not able to authenticate to enable mode the ssh.enable() command does not give an authentication error
#but a time-out error instead
try:
ssh = ConnectHandler(username = router[1]['username'],password = router[1]['password'],device_type="cisco_ios", ip=ip_address)
except netmiko.ssh_exception.NetMikoTimeoutException:
# connection timed out (ssh not enabled on device, try telnet)
continue
except Exception:
continue
else:
output = ssh.send_command(ios_command)
ssh.disconnect()
if "at '^' marker." in output:
#trying to run a command that requires enble mode but not authenticated to enable mode
continue
return output
except Exception:
continue
else:
output = ssh.send_command(ios_command)
ssh.disconnect()
return output
output = connect_enable_silent(ip_address,ios_command)
for line in output.split('\n'):
print(line)
Credentials text is meant to store different credentials in case you are planning to call this function to access multiple devices and not all of them using the same credentials. It is in the format:
{"credentials_1":{"username":"username_1","password":"password_1","secret":"secret_1"},
"credentials_2":{"username":"username_2","password":"password_2","secret":"secret_2"},
"credentials_3": {"username": "username_3", "password": "password_3"}
}
The exceptions can be changed to do different things, in my case i just needed it to not return an error and continue trying the next set, which is why most exceptions are silenced.

Paramiko hangs when running on 150+ servers while running simple commands

I'm trying to run multiple commands on bunch of remote servers using python's Paramiko module.
The commands I'm trying to run are simple commands, such as cat, lspci(with grep) and also small script with only 1 line output.
The thing is, if I provide few machines (~50), it works just fine.
The problem starts when I try run the script on many machines.
try:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, username='root', password='pass')
transport = ssh.get_transport()
channel = transport.open_session()
stdin, stdout, stderr = ssh.exec_command(cmd)
for line in stdout.readlines():
line = line.strip()
sheet1.write(row_line,0,host,style_cell) # writing to xls file
sheet1.write(row_line,1,line,style_cell) # writing to xls file
while channel.recv_ready():
channel.recv(1024)
ssh.close()
expect:
print stdout
print stderr
This is the stdout,stderr I get:
paramiko.ChannelFile from paramiko.Channel 2 (EOF received) (open) window=2097152
paramiko.Transport at 0xce44c9d0L (cipher aes128-ctr, 128 bits) (active; 2 open channel(s))
Please advice,
Thanks!

Maybe it's a problem with specific servers. Try outputting which server it is that causes errors and see if you can recreate the problem if running the script only with that specific server.