Develop Reference

Python: How can I capture the boot logs through SSH on a remote Linux machine?

Objective: I want to write a Python test script to execute a reboot command through SSH on a remote Linux machine, capture the boot logs and check if the boot is successful.
Problem: I am able to send a reboot command using Paramiko and the machine reboots as expected, but I couldn't capture the boot logs and print them out. My code also seems to run without waiting for the boot process to finish.
Here is part of my code:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
ssh.connect(host, username=username, password=password)
except paramiko.SSHException as e:
ssh.get_transport().auth_none(username) # without password
# execute reboot and capture the boot logs
stdin, stdout, stderr = ssh.exec_command("/sbin/reboot")
print(stdout.readlines())
print(stderr.readlines())
# check if reboot is done
exit_status = ''
msg = ''
while True:
if stdout_channel.exit_status_ready():
exit_status = ssh.stdout.channel.recv_exit_status()
print("Exit status: %s" % exit_status)
break
time.sleep(10)
ssh.close()
if exit_status == 0:
print("Reboot successful")
else:
print(Reboot not successful")
Logs are not captured and the following output is printed out before the machine finishing rebooting:
[]
[]
Exit status: 0
Reboot successful
Questions:
a) How can I capture the boot logs?
b) How to properly check for status after boot process is completed? Alternatively, I think I can ssh again and simply run a command after waiting some time for it to reboot.

you need /var/log/boot.log or /var/log/dmesg
You need monitoring. Generally, for LAN you need something like zabbix, but it is to much for 2 hosts. Other way is any script for sending message from remote host to your log server or messenger(using messenger API) or vice versa. For your script you can make systemd Unit service for run it after/before anything.

Is there a limit to the length of the command you can send over an SSH channel/connection using Paramiko SSHClientInteraction?

I'm new to Paramiko and have tried a variety of methods for doing what is a pretty basic task. At this point, all I'm trying to do is execute a command on an APC UPS. In my testing, I discovered I can successfully execute a command, as long as the length of the command is 20 characters or less.
Here is my code -
import paramiko
from paramiko_expect import SSHClientInteraction
host = "xxx.xxx.xxx.xxx" # UPS
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname=host, username='e164468', password='********', banner_timeout=60)
chan = SSHClientInteraction(ssh, timeout=2, display=True)
prompt = 'apc>'
command = 'cipher -rsake disable'
chan.expect(prompt)
chan.send(command)
chan.expect(prompt)
output_raw = chan.current_output_clean
chan.close()
ssh.close()
print ("Here is your output - ")
print(output_raw)
The command I need to remotely execute is 'cipher -rsake disable'. What I get in the terminal when I run this is -
apc>cipher -rsake disablEXCESS TIME RECV_READY TIMEOUT, did you expect() before a send()
If I drop the disable parameter from the command, it works as expected. No matter what I've tried, I get this behavior whenever the command is over 20 characters.
I'm not sure how valid of a test it is but I manually ssh to the UPS and can type in whatever command I need and it works.
Any help would be greatly appreciated!

Gracefully abort remote Windows command executed over SSH from Windows Python Paramiko script when Ctrl+C is pressed

I have a follow up question that builds off the question I asked here: Run multiple commands in different SSH servers in parallel using Python Paramiko, which was already answered.
Thanks to the answer on the link above, my python script is as follows:
# SSH.py
import paramiko
import argparse
import os
path = "path"
python_script = "worker.py"
# definitions for ssh connection and cluster
ip_list = ['XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX']
port_list = [':XXXX', ':XXXX', ':XXXX']
user_list = ['user', 'user', 'user']
password_list = ['pass', 'pass', 'pass']
node_list = list(map(lambda x: f'-node{x + 1} ', list(range(len(ip_list)))))
cluster = ' '.join([node + ip + port for node, ip, port in zip(node_list, ip_list, port_list)])
# run script on command line of local machine
os.system(f"cd {path} && python {python_script} {cluster} -type worker -index 0 -batch 64 > {path}/logs/'command output'/{ip_list[0]}.log 2>&1")
# loop for IP and password
stdouts = []

clients = []
for i, (ip, user, password) in enumerate(zip(ip_list[1:], user_list[1:], password_list[1:]), 1):
try:
print("Open session in: " + ip + "...")
client = paramiko.SSHClient()
client.connect(ip, user, password)
except paramiko.SSHException:
print("Connection Failed")
quit()
try:
path = f"C:/Users/{user}/Desktop/temp-ines"
stdin, stdout, stderr = ssh.exec_command(
f"cd {path} && python {python_script} {cluster} -type worker -index {i} -batch 64>"

 f"C:/Users/{user}/Desktop/{ip}.log 2>&1 &"
)

clients.append(ssh)
stdouts.append(stdout)
except paramiko.SSHException:
print("Cannot run file. Continue with other IPs in list...")
client.close()
continue
# Wait for commands to complete
for i in range(len(stdouts)):
print("hello")
stdouts[i].read()
print("hello1")
clients[i].close()
print('hello2")
print("\n\n***********************End execution***********************\n\n")
This script, which is run locally, is able to SSH into the servers and run the command (i.e., run a python script called worker.py and log the command output to a log file). I.e., it is able to go through the first for loop with no issues.
My issue is related to the second for loop. Please see the print statements I added in the second for loop to be clear. When I run SSH.py locally, this is what I observe:
As you can see, I ssh into each of the servers and then stay at reading the command output of the first server I ssh over to. The worker.py script can take 30 mins or so to complete and the command output is the same on each server -- so it will take 30 mins to read the command output of the first server, then close the SSH connection of the first server, take a couple seconds to read the command output of the second server (as it is the same as the first one and would already be entirely printed), close its SSH connection, and so on. Please see below some of the command line output, if this helps.
Now, my question is, what if I don't want to wait until the worker.py script finishes, i.e., those entire 30 mins? I cannot/do not know how to raise a KeyboardInterrupt exception. What I have tried is quitting the local SSH.py script. However, as you can see from the print statements, this will not close the SSH connections although the training, and thus the log files, will stop logging info. In addition, after I quit the local SSH.py script, if I try to delete any of the log files, I get an error saying "cannot delete file because it is being used in cmd.exe" -- this only happens sometimes and I believe it is because of not closing the SSH connections?
First run in python console:
It hangs: Local python and log file running and saving but no print statements and no python and log file being run/saved in servers.
I run it again so second process starts:
Now, the first process doesn't hang anymore (python running and log files being saved in server). And can close this second run/process. It is like the second run/process helps with the hang of the first run/process.
If I were to run python SSH.py in the terminal it would just hang.
This was not happening before.

If you know that SSHClient.close cleanly close the connection and abort the remote command, call it on response to KeyboardInterrupt.
For this you cannot use the simple solution with stdout.read, as it blocks and prevents handling of the Ctrl+C on Windows.
Use the waiting code from my answer to Run multiple commands in different SSH servers in parallel using Python Paramiko (the while any(x is not None for x in stdouts): snippet).
And wrap it to try:...except (KeyboardInterrupt):.
try:
while any(x is not None for x in stdouts):
for i in range(len(stdouts)):
stdout = stdouts[i]
if stdout is not None:
channel = stdout.channel
# To prevent losing output at the end, first test for exit,
# then for output
exited = channel.exit_status_ready()
while channel.recv_ready():
s = channel.recv(1024).decode('utf8')
print(f"#{i} stdout: {s}")
while channel.recv_stderr_ready():
s = channel.recv_stderr(1024).decode('utf8')
print(f"#{i} stderr: {s}")
if exited:
print(f"#{i} done")
clients[i].close()
stdouts[i] = None
time.sleep(0.1)
except (KeyboardInterrupt):
print("Aborting")
for i in range(len(clients)):
print(f"#{i} closing")
clients[i].close()
If you do not need to separate the stdout and stderr, you can greatly simplify the code by using Channel.set_combine_stderr. See Paramiko ssh die/hang with big output.

Subprocess on remote server

I am using this code for executing command on remote server.
import subprocess
import sys
COMMAND="ls"
ssh = subprocess.Popen(["ssh", "%s" % HOST, COMMAND],
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
result = ssh.stdout.readlines()
if result == []:
error = ssh.stderr.readlines()
print >>sys.stderr, "ERROR: %s" % error
else:
print result
When I try to execute this script, I get prompt for password. Is there any way I could avoid it, for example, can I enter password in script somehow? Also, password should be encrypted somehow so that people who have access to the script cannot see it.

Why make it so complicated? Here's what I suggest:
1) Create a ssh config section in your ~/.ssh/config file:
Host myserver
HostName 50.50.50.12 (fill in with your server's ip)
Port xxxx (optional)
User me (your username for server)
2) If you have generated your ssh keypair do it now (with ssh-keygen). Then upload with:
$ ssh-copy-id myserver
3) Now you can use subprocess with ssh. For example, to capture output, I call:
result = subprocess.check_output(['ssh', 'myserver', 'cat', 'somefile'])
Simple, robust, and the only time a password is needed is when you copy the public key to the server.
BTW, you code will probably work just fine as well using these steps.

One way is to create a public key, put it on the server, and do ssh -i /path/to/pub/key user#host or use paramiko like this:
import paramiko
import getpass
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
p = getpass.getpass()
ssh.connect('hostname', username='user', password=p)
stdin, stdout, stderr = ssh.exec_command('ls')
print stdout.readlines()
ssh.close()

You should use pexpect or paramiko to connect to remote machine,then spawn a child ,and then run subprocess to achieve what you want.

Here's what I did when encountering this issue before:
Set up your ssh keys for access to the server.
Set up an alias for the server you're accessing. Below I'll call it remote_server.
Put the following two lines at the end of ~/.bash_profile.
eval $(ssh-agent -s)
ssh-add
Now every time you start your shell, you will be prompted for a passphrase. By entering it, you will authenticate your ssh keys and put them 'in hand' at the start of your bash session. For the remainder of your session you will be able to run commands like
ssh remote_server ls
without being prompted for a passphrase. Here ls will run on the remote server and return the results to you. Likewise your python script should run without password prompt interruption if you execute it from the shell.
You'll also be able to ssh to the server just by typing ssh remote_server without having to enter your username or password every time.
The upside to doing it this way is that you should be doing this anyway to avoid password annoyances and remembering funky server names :) Also you don't have to worry about having passwords saved anywhere in your script. The only potential downside is that if you want to share the python script with others, they'll have to do this configuring as well (which they should anyway).

You don't really need something like pexpect to handle this. SSH keys already provide a very good and secure solution to this sort of issue.
The simplest way to get the results you want would probably be to generate an ssh key and place it in the .ssh folder of your device. I believe github has a pretty good guide to doing that, if you look into it. Once you set up the keys correctly on both systems, you won't actually have to add a single line to your code. When you don't specify a password it will automatically use the key to authenticate you.

While subprocess.Popen might work for wrapping ssh access, this is not the preferred way to do so.
I recommend using paramiko.
import paramiko
ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect(server, username=user,password=password)
...
ssh_client.close()
And If you want to simulate a terminal, as if a user was typing:
chan=ssh_client.invoke_shell()
def exec_cmd(cmd):
"""Gets ssh command(s), execute them, and returns the output"""
prompt='bash $' # the command line prompt in the ssh terminal
buff=''
chan.send(str(cmd)+'\n')
while not chan.recv_ready():
time.sleep(1)
while not buff.endswith(prompt):
buff+=ssh_client.chan.recv(1024)
return buff[:len(prompt)]
Example usage: exec_cmd('pwd')
If you don't know the prompt in advance, you can set it with:
chan.send('PS1="python-ssh:"\n')

You could use following.
import subprocess
import sys
COMMAND="ls"
ssh = subprocess.Popen("powershell putty.exe user#HOST -pw "password", stdout=PIPE, stdin=PIPE, stderr=STDOUT)
result = ssh.stdout.readlines()
if result == []:
error = ssh.stderr.readlines()
print >>sys.stderr, "ERROR: %s" % error
else:
print result

Executing reboot command over SSH using Paramiko

I use Paramiko for establishing SSH connection with some target device and I want to execute reboot command.
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(zip_hostname, username=username, password=password, timeout=1)
try:
stdin, stdout, stderr = ssh.exec_command("/sbin/reboot -f")
# .........
# some code
# .........
except AuthenticationException, e:
print ''
finally:
ssh.close()
But after executing ssh.exec_command("/sbin/reboot -f") "some code" does not execute because program is stuck in exec_command (the disconnection takes place caused by rebooting). What should I do to solve my problem?

Try this:
ssh.exec_command("/sbin/reboot -f > /dev/null 2>&1 &")
All the output of reboot is redirected to /dev/null to make it produce no output and it is started in the background thanks to the '&' sign in the end. Hopefully the program won't hang on that line this way, because the remote shell gives the prompt back.

Get the transport from the ssh and set the keepalive using:
transport = ssh.get_transport()
transport.set_keepalive(5)
This sets the keepalive to 5 seconds; mind you I would have expected the timeout=1 to have achieved the same thing.

All you need to do is to call channel.exec_command() instead of the high-level interface client.exec_command()
# exec fire and forget
timeout=0.5
transport = ssh.get_transport()
chan = ssh.get_transport().open_session(timeout=timeout)
chan.settimeout(timeout)
try:
chan.exec_command(command)
except socket.timeout:
pass

I was having this issue and managed to avoid it by switching to this command:
/sbin/shutdown -r now
Note this command does not result in any STDOUT or STDERR output

In case you or anyone else gets stuck trying to reboot host with sudo using forwarding agents (ssh keys) or in my case (yubikey)
If you look at this as bash you would reboot a host as non root user like this.
ssh -t -A user#hostname sudo /sbin/reboot
For the -A flag, from ssh man page
Enables forwarding of the authentication agent connection. This can also be specified on a per-host basis in a
configuration file.
Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the
remote host (for the agent’s Unix-domain socket) can access the local agent through the forwarded connection.
An attacker cannot obtain key material from the agent, however they can perform operations on the keys that
enable them to authenticate using the identities loaded into the agent.*
For the -t flag, from ssh man page
Force pseudo-tty allocation. This can be used to execute arbitrary screen-based programs on a remote machine,
which can be very useful, e.g. when implementing menu services. Multiple -t options force tty allocation, even
if ssh has no local tty.*
So lets break this down into how you would do this in paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname=host, username=username)
s = ssh.get_transport().open_session()
paramiko.agent.AgentRequestHandler(s)
ssh.exec_command("sudo /sbin/reboot", get_pty=True)
For authentication forwarding (-A flag in bash ssh command) for paramiko
ssh = paramiko.SSHClient() #'ssh' is client variable
s = ssh.get_transport().open_session() #get 'ssh' transport and open sessions assigned to 's' variable
paramiko.agent.AgentRequestHandler(s) #call in 's' to the forwarding agent for current ssh session
Now for force pseudo-tty allocation (-t flag in bash ssh command) for paramiko
ssh.exec_command("sudo /sbin/reboot", get_pty=True)
Adding 'get_pty=True' to exec_command will allow you execute sudo /sbin/reboot
Hope this helps, everyone's environments are different but this should work as it the exact same thing as if you ran it as bash.