In an automation script, i need to ssh into several servers to gather some data. Unfortunately, few of them seems to have intermittent issues where the connection hangs forever on get_pty().
Here's a snippet of my code. Is there something I can do to kill the connection? I'm using python of course, and unfortunately with python, there is no easy way to kill a thread either :(
client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
connect(hostname, port=port, username=username, password=password, timeout=30)
self.__chan = self.__client.get_transport().open_session()
print self.__chan
print ' after'
print self.__chan.get_pty()
output
<paramiko.Channel 1 (open) window=0 -> <paramiko.Transport at 0xd49410L (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>
after
I believe the issue has to do with a failed attempt to get_transport() or open_session() properly. If so, how can i detect if an issue occurred?
Per the documentation for paramiko regarding get_pty, "It isn’t necessary (or desirable) to call this method if you’re going to exectue a single command with exec_command." http://docs.paramiko.org/en/1.13/api/channel.html#paramiko.channel.Channel.get_pty
Do you have a specific reason for invoking the pseudo-terminal? If you use paramiko.client.SSHClient.exec_command you do not need to invoke one. Depending on your version of paramiko, exec_command also accepts a timeout parameter so that the command will only block for a certain amount of time. Should your version not allow this parameter, you can subclass the client to add it.
from paramiko import SSHClient
class SSHClient_with_Timeout(SSHClient):
## overload the exec_command method
def exec_command(self, command, bufsize=-1, timeout=None):
chan = self._transport.open_session()
chan.settimeout(timeout)
chan.exec_command(command)
stdin = chan.makefile('wb', bufsize)
stdout = chan.makefile('rb', bufsize)
stderr = chan.makefile_stderr('rb', bufsize)
return stdin, stdout, stderr
If that's not the solution you're looking for, you could also set up logging. paramiko logs to the logger "paramiko", so invoking logging.basicConfig(level=logging.DEBUG) will allow you to see what paramiko is up to right before the 'hang.'
Related
I am trying to execute some command on remote server using Paramiko and re-purposed some code from net. From Paramiko documents, it say that you have to close the connection. But I am getting errors while doing that.
class ssh:
paramiko.util.log_to_file("filename_new.log")
client = None
def __init__(self, address, username, password):
self.client=paramiko.SSHClient()
self.client.set_missing_host_key_policy(paramiko.client.AutoAddPolicy())
self.client.connect(hostname=address, username=username, password=password)
def sendCommand(self, command):
if(self.client):
couldNotConnect = False
stdin, stdout, stderr = self.client.exec_command(command, get_pty=True)
stdout = stdout.readlines()
self.client.close()
else:
stdout = "Could Not Connect"
couldNotConnect = True
return stdout
connection = ssh(serverName, userName, passWord)
dfDetails = connection.sendCommand("df -hT")
upTime = connection.sendCommand("uptime")
I am getting the below error:
File "/home/amarc/development/djago/venv/lib/python3.6/site-packages/paramiko/client.py", line 508, in exec_command
chan = self._transport.open_session(timeout=timeout)
AttributeError: 'NoneType' object has no attribute 'open_session'
But when I remove the self.client.close(), it works fine. But I am worried if not closing the connection might cause problem if the program was run multiple times.
And is using __init__ the right way to create a connection since I might be supplying the function with different credentials every time.
I'd recommend you to go for either Fabric (http://www.fabfile.org/) or Ansible and not rely purely on Paramiko since you'll have to reimplement a lot of details. Those tools are already using it but behind the scenes.
The SSHClient.close call has to match with SSHClient.connect.
You call SSHClient.connect once, but SSHClient.close for every command.
Call SSHClient.close only once, after you execute all commands.
I have a class that creates the connection. I can connect and execute 1 command before the channel is closed. On another system i have i can execute multiple commands and the channel does not close. Obviously its a config issue with the systems i am trying to connect to.
class connect:
newconnection = ''
def __init__(self,username,password):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
ssh.connect('somehost', username=username,password=password,port=2222,timeout=5)
except:
print "Count not connect"
sys.exit()
self.newconnection = ssh
def con(self):
return self.newconnection
Then i use 'ls' command just to print some output
sshconnection = connect('someuser','somepassword').con()
stdin, stdout, stderr = sshconnection.exec_command("ls -lsa")
print stdout.readlines()
print stdout
stdin, stdout, stderr = sshconnection.exec_command("ls -lsa")
print stdout.readlines()
print stdout
sshconnection.close()
sys.exit()
After the first exec_command runs it prints the expected output of the dir list. When i print stdout after the first exec_command it looks like the channel is closed
<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x2400f10L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>
Like i said on another system i am able to keep running commands and the connection doesn't close. Is there a way i can keep this open? or a better way i can see the reason why it closes?
edit: So it looks like you can only run 1 command per SSHClient.exec_command... so i decided to get_transport().open_session() and then run a command. The first one always works. The second one always fails and the scripts just hangs
With just paramiko after the exec_command executes the channel is closed and the ssh returns an auth prompt.
Seems its not possible with just paramiko, try fabric or another tool.
** fabric did not work out too.
Please see the following referece as it provides a way to do this in Paramiko:
How do you execute multiple commands in a single session in Paramiko? (Python)
it's possible with netmiko (tested on windows).
this example is written for connecting to cisco devices but the principle is adaptable for others as well.
import netmiko
from netmiko import ConnectHandler
import json
def connect_enable_silent(ip_address,ios_command):
with open ("credentials.txt") as line:
line_1 = json.load(line)
for k,v in line_1.items():
router=(k,v)
try:
ssh = ConnectHandler(**router[1],device_type="cisco_ios",ip=ip_address)
ssh.enable()
except netmiko.ssh_exception.NetMikoAuthenticationException:
#incorrect credentials
continue
except netmiko.ssh_exception.NetMikoTimeoutException:
#oddly enough if it can log in but not able to authenticate to enable mode the ssh.enable() command does not give an authentication error
#but a time-out error instead
try:
ssh = ConnectHandler(username = router[1]['username'],password = router[1]['password'],device_type="cisco_ios", ip=ip_address)
except netmiko.ssh_exception.NetMikoTimeoutException:
# connection timed out (ssh not enabled on device, try telnet)
continue
except Exception:
continue
else:
output = ssh.send_command(ios_command)
ssh.disconnect()
if "at '^' marker." in output:
#trying to run a command that requires enble mode but not authenticated to enable mode
continue
return output
except Exception:
continue
else:
output = ssh.send_command(ios_command)
ssh.disconnect()
return output
output = connect_enable_silent(ip_address,ios_command)
for line in output.split('\n'):
print(line)
Credentials text is meant to store different credentials in case you are planning to call this function to access multiple devices and not all of them using the same credentials. It is in the format:
{"credentials_1":{"username":"username_1","password":"password_1","secret":"secret_1"},
"credentials_2":{"username":"username_2","password":"password_2","secret":"secret_2"},
"credentials_3": {"username": "username_3", "password": "password_3"}
}
The exceptions can be changed to do different things, in my case i just needed it to not return an error and continue trying the next set, which is why most exceptions are silenced.
I'm trying to run multiple commands on bunch of remote servers using python's Paramiko module.
The commands I'm trying to run are simple commands, such as cat, lspci(with grep) and also small script with only 1 line output.
The thing is, if I provide few machines (~50), it works just fine.
The problem starts when I try run the script on many machines.
try:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, username='root', password='pass')
transport = ssh.get_transport()
channel = transport.open_session()
stdin, stdout, stderr = ssh.exec_command(cmd)
for line in stdout.readlines():
line = line.strip()
sheet1.write(row_line,0,host,style_cell) # writing to xls file
sheet1.write(row_line,1,line,style_cell) # writing to xls file
while channel.recv_ready():
channel.recv(1024)
ssh.close()
expect:
print stdout
print stderr
This is the stdout,stderr I get:
paramiko.ChannelFile from paramiko.Channel 2 (EOF received) (open) window=2097152
paramiko.Transport at 0xce44c9d0L (cipher aes128-ctr, 128 bits) (active; 2 open channel(s))
Please advice,
Thanks!
Maybe it's a problem with specific servers. Try outputting which server it is that causes errors and see if you can recreate the problem if running the script only with that specific server.
I'm using Python's Paramiko module to SSH into a remote machine and Tar/ZIP a folder with a LOT of files (over 14K files and 60+gigs of data). The resulting zip is around 10 gigs itself. Now I can run the command to zip/tar directly from the machine with no problem. However, when I try to run the same command through SSHClient.exec_command, it runs for a bit, but eventually the zipping process on the remote machine goes to sleep. And the recv_exit_status just hangs indefinitely. Here is the code I'm using:
stdin, stdout, stderr = ssh.exec_command('cd myDirectory; tar -zcvf output.tgz *')
status = stdout.channel.recv_exit_status()
I also tried using Zip.
stdin, stdout, stderr = ssh.exec_command('cd myDirectory; find -name "*.gz" | zip output.zip -#')
status = stdout.channel.recv_exit_status()
In both cases, if I run the command directly from the remote machine, it finishes zipping/TARing. The result file is is like 9 gigs. But when I try it from Paramiko, it starts, goes more than half way (6ish gigs) and then the process goes to sleep!
I've monitored the processes on the remote machine using top, and the zip/tar WILL start running, but it will eventually go to sleep before finishing. And the python script will hang indefinitely.
Any ideas why this is happening?
It could be a timeout related. Try adding timeout param (in seconds) to the call: exec_command(timeout=20*60). This is 20 min example.
See doc string from that method for more info:
def exec_command(self, command, bufsize=-1, timeout=None, get_pty=False):
"""
Execute a command on the SSH server. A new `.Channel` is opened and
the requested command is executed. The command's input and output
streams are returned as Python ``file``-like objects representing
stdin, stdout, and stderr.
:param str command: the command to execute
:param int bufsize:
interpreted the same way as by the built-in ``file()`` function in
Python
:param int timeout:
set command's channel timeout. See `Channel.settimeout`.settimeout
:return:
the stdin, stdout, and stderr of the executing command, as a
3-tuple
:raises SSHException: if the server fails to execute the command
"""
Also there is another issue that i experience which could also contribute: https://github.com/paramiko/paramiko/issues/109
Try my suggestion in https://github.com/paramiko/paramiko/issues/109#issuecomment-111621658
I also experienced this issue it is due to stdout.channel.eof_received == 0
import paramiko
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect("1.1.1.1", username="root", password="pass")
stdin, stdout, stderr = client.exec_command("service XXX start")
stdin, stdout and stderr are staying open...
>>> print stdin
<paramiko.ChannelFile from <paramiko.Channel 3 (open) window=2097152 in-buffer=50 -> <paramiko.Transport at 0x17eff90L (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>>
>>> print stdout
<paramiko.ChannelFile from <paramiko.Channel 3 (open) window=2097152 in-buffer=50 -> <paramiko.Transport at 0x17eff90L (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>>
>>> print stderr
<paramiko.ChannelFile from <paramiko.Channel 3 (open) window=2097152 in-buffer=50 -> <paramiko.Transport at 0x17eff90L (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>>
So EOF was not received...
>>> print stdin.channel.eof_received
0
Usually I receive True and can just stdout.read(), but to be safe i use this workaround (which works!): Wait for a timeout, force stdout.channel.close() and then stdout.read():
>>> timeout = 30
>>> import time
>>> endtime = time.time() + timeout
>>> while not stdout.channel.eof_received:
... sleep(1)
... if time.time() > endtime:
... stdout.channel.close()
... break
>>> stdout.read()
'Starting XXX: \n[ OK ]\rProgram started . . .\n'
>>>
My solution is client.exec_command('my_cmd', get_pty=True).
get_pty=True can Request a pseudo-terminal from the server.
So If you can run your command in ssh session, then should also work by using exec_command() function.
I just had a similar thing happen to me. I can run a command when logged in via ssh, but running through exec_command eventually puts the command to sleep (S in htop). I found that it might be due to the command producing too much output on either stderr or stdout which I have gathered can overflow buffers, causing the signal that the command has finished to be lost, but I'm definitely no expert. What I did find is that adding > /dev/null 2>&1 to the end of my command (thereby eliminating the need of paramiko to touch stdout and stderr) allows the same command to finish via exec_command.
In summary, my workflow looks like this:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('999.99.99.99', username='me', key_filename='my_key')
stdin, stdout, stderr = ssh.exec_command('my_command > /dev/null 2>&1')
stdout_content = stdout.read()
stderr_content = stderr.read()
ssh.close()
This works for now, but I would still be very interested if someone knows how to fix the original problem without having to change the command to redirect stdout and stderr.
I'm using a simple pexpect script to ssh to a remote machine and grab a value returned by a command.
Is there any way, pexpect or sshwise I can use to ignore the unix greeting?
That is, from
child = pexpect.spawn('/usr/bin/ssh %s#%s' % (rem_user, host))
child.expect('[pP]assword: ', timeout=5)
child.sendline(spass)
child.expect([pexpect.TIMEOUT, prompt])
child.before = '0'
child.sendline ('%s' % cmd2exec)
child.expect([pexpect.EOF, prompt])
# Collected data processing
result = child.before
# logon to the machine returns a lot of garbage, the returned executed command is at the 57th position
print result.split('\r\n') [57]
result = result.split('\r\n') [57]
How can I simply get the returned value, ignoring,
the "Last successful login" and "(c)Copyright" stuff
and without having to concern with the value correct position?
Thanks !
If you have access to the server to which you are logging in, you can try creating a file named .hushlogin in the home directory. The presence of this file silences the standard MOTD greeting and similar stuff.
Alternatively, try ssh -T, which will disable terminal allocation entirely; you won't get a shell prompt, but you may still issue commands and read the response.
There is also a similar thread on ServerFault which may be of some use to you.
If the command isn't interactive, you can just run ssh HOST COMMAND to run the command without all the login excitement happening at all. If the command is interactive, you can frequently use the ssh -t option (ssh -t HOST COMMAND) to force pseudo-tty allocation and trick the remote process to think that it's running attached to a TTY.
I have used paramiko to automate ssh connection and I have found it useful. It can deal with greetings and silent execution.
http://www.lag.net/paramiko/
Hey there you kann kill all that noise by using the sys module and a small class:
class StreamToLogger(object):
"""
Fake file-like stream object that redirects writes to a logger instance.
"""
def __init__(self, logger, log_level=logging.INFO):
self.logger = logger
self.log_level = log_level
self.linebuf = ''
def write(self, buf):
for line in buf.rstrip().splitlines():
self.logger.log(self.log_level, line.rstrip())
#Mak
stdout_logger = logging.getLogger('STDOUT')
sl = StreamToLogger(stdout_logger, logging.INFO)
sys.stdout = sl
stderr_logger = logging.getLogger('STDERR')
sl = StreamToLogger(stderr_logger, logging.ERROR)
sys.stderr = sl
Can't remember where i found that snippet but it works for me :)