The output file is created before the database results are available.
Passing a simple os command works fine:
# command = "whoami > result.txt"
works fine. I would get my user name when I open the result.txt file. The problem is waiting for the database to return the result. It comes out empty even though there is data returned from the actual query
import paramiko
def get_report(command):
# reference: https://stackoverflow.com/questions/5193886/python-paramiko-issue-while-closing-the-connection.
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect('server123', username='username', password='password')
stdin, stdout, stderr = client.exec_command(command)
exit_status = stdout.channel.recv_exit_status()
if exit_status == 0:
print("File processed")
else:
print("Error", exit_status)
client.close()
command = "sql_query.script > result.txt"
get_report(command=command)
I expect to received a data set of first_name, last_name, and location but instead I get Error 108.
If a command does not work, when executed using Paramiko, debug it by reading its error output.
Use stderr.readlines() for that.
If the same command works in regular shell, but not in Paramiko, the problem is usually related to a different environment used by the SSH "exec" channel, which is used by SSHClient.exec_command. See:
Some Unix commands fail with "<command> not found", when executed using Python Paramiko exec_command
Environment variable differences when using Paramiko
Related
I'm a software tester, trying to verify that the log on a remote QNX (a BSD variant) machine will contain the correct entries after specific actions are taken. I am able to list the contents of the directory in which the log resides, and use that information in the command to read (really want to use tail -n XX <file>) the file. So far, I always get a "(No such file or directory)" when trying to read the file.
We are using Froglogic Squish for automated testing, because the Windows UI (that interacts with the server piece on QNX) is built using Qt extensions for standard Windows elements. Squish uses Python 2.7, so I am using Python 2.7.
I am using paramiko for the SSH connection to the QNX server. This has worked great for sending commands to the simulator piece that also runs on the QNX server.
So, here's the code. Some descriptive names have been changed to avoid upsetting my employer.
import sys
import time
import select
sys.path.append(r"C:\Python27\Lib\site-packages")
sys.path.append(r"C:\Python27\Lib\site-packages\pip\_vendor")
import paramiko
# Import SSH configuration variables
ssh_host = 'vvv.xxx.yyy.zzz'
thelog_dir = "/logs/the/"
ssh_user = 'un'
ssh_pw = 'pw'
def execute_Command(fullCmd):
outptLines = []
#
# Try to connect to the host.
# Retry a few times if it fails.
#
i = 1
while True:
try:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(ssh_host, 22, ssh_user, ssh_pw)
break
except paramiko.AuthenticationException:
log ("Authentication failed when connecting to %s" % ssh_host)
return 1
except:
log ("Could not SSH to %s, waiting for it to start" % ssh_host)
i += 1
time.sleep(2)
# If we could not connect within time limit
if i == 30:
log ("Could not connect to %s. Giving up" % ssh_host)
return 1
# Send the command (non-blocking?)
stdin, stdout, stderr = ssh.exec_command(fullCmd, get_pty=True)
for line in iter(stdout.readline, ""):
outptLines.append(line)
#
# Disconnect from the host
#
ssh.close()
return outptLines
def get_Latest_Log():
fullCmd = "ls -1 %s | grep the_2" %thelog_dir
files = execute_Command(fullCmd)
theFile = files[-1]
return theFile
def main():
numLines = 20
theLog = get_Latest_Log()
print("\n\nThe latest log is %s\n\n" %theLog)
fullCmd = "cd /logs/the; tail -n 20 /logs/the/%s" %theLog
#fullCmd = "tail -n 20 /logs/the/%s" %theLog
print fullCmd
logLines = execute_Command(fullCmd)
for line in logLines:
print line
if __name__ == "__main__":
# execute only if run as a script
main()
I have tried to read the file using both tail and cat. I have also tried to get and open the file using Paramiko's SFTP client.
In all cases, the response of trying to read the file fails -- despite the fact that listing the contents of the directory works fine. (?!) And BTW, the log file is supposed to be readable by 'world'. Permissions are -rw-rw-r--.
The output I get is:
"C:\Users\xsat086\Documents\paramikoTest>python SSH_THE_MsgChk.py
The latest log is the_20210628_115455_205.log
cd /logs/the; tail -n 20 /logs/the/the_20210628_115455_205.log
(No such file or directory)the/the_20210628_115455_205.log"
The file name is correct. If I copy and paste the tail command into an interactive SSH session with the QNX server, it works fine.
Is it something to do with the 'non-interactive' nature of this method of sending commands? I read that some implementations of SSH are built upon a command that offers a very limited environment. I don't see how that would impact this tail command.
Or am I doing something stupid in this code?
I cannot really explain completely, why you get the results you get.
But in general a corrupted output is a result of enabling and not handling terminal emulation. You enable the terminal emulation using get_pty=True. Remove it. You should not use the terminal emulation, when automating command execution.
Related question:
Is there a simple way to get rid of junk values that come when you SSH using Python's Paramiko library and fetch output from CLI of a remote machine?
Command i'm trying to run using paramiko in python 3.7:
Windows:
type file.ext4.gz | ssh user#address sudo update.sh
Mac:
cat file.ext4.gz | ssh user#address sudo update.sh
From the cmd / terminals and from .bat / .sh this works, after entering the password. I've been working on a simple python gui (PysimpleGui) to allow the user to fo this, but without the need to enter the password (this is saved from initial connection).
I've tried:
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect(config["IP_ADDRESS"], username=config["USERNAME"], password=config["PASSWORD"], timeout=5)
a = client.open_sftp()
a.put(file_location, "sh update.sh", callback=sent)
While this works to send the file, it doesn't run it and gives the error:
OSError: Failure
I don't want to do this in subprocess, as this tool is to prevent the use of terminal for the "end user"
I've been beating my head against this for 2 days now. Thank you.
EDIT:
Here is the STDIO Code:
def send_ssh(value, input=None):
if input:
transport = client.get_transport()
channel = transport.open_session()
channel.exec_command(value)
with open(input, "rb") as file:
for chunk in iter(functools.partial(file.read, read_size), b''):
if channel.send_ready():
channel.sendall(chunk)
if channel.recv_ready():
print(channel.recv(1024).decode().strip())
if channel.recv_stderr_ready():
print(channel.recv_stderr(1024).decode().strip())
while not channel.exit_status_ready():
if channel.recv_ready():
print(channel.recv(1024).decode().strip())
if channel.recv_stderr_ready():
print(channel.recv_stderr(1024).decode().strip())
else:
w, r, e = client.exec_command(value, get_pty=True)
error = e.read().strip().decode()
if error != "":
return error
else:
return r.read().strip().decode()
Once the file is cat to the script it's the verified by the script. I worked around this by just using SFTP to send the file and running my
cat file | sudo script.sh
this works, but does require that i transfer a 600mb file (thankfully always over a local connection (LAN)) each time. The above code does transfer the file, but it doesn't complete. If i just try sending it via for line in file: i'll corrupt.
Keeping things simpler, below we're using threading to allow synchronous APIs to be used rather than needing to write explicit asynchronous code:
import shutil
client = SSHClient()
client.load_system_host_keys()
client.connect('user#address')
# here's the important part: we're using the file handles returned by exec_command()
update_stdin, update_stdout, update_stderr = client.exec_command('sudo update.sh')
# copy stdout and stderr from the remote thread to our own process's stdout and stderr
t_out = Thread(target=shutil.copyfileobj, args=[update_stdout, sys.stdout]); t_out.start()
t_err = Thread(target=shutil.copyfileobj, args=[update_stderr, sys.stderr]); t_err.start()
# write your local file to the remote stdin, in the foreground: we don't exit until done.
shutil.copyfileobj(open('file.ext4.gz', 'r'), update_stdin)
update_stdin.close()
# optional, but let's be graceful: wait for the threads to exit, and collect exit status
t_out.join(); t_err.join()
result = stdout.channel.recv_exit_status()
print(f"Remote process exited with status {result}")
I have a simple request. I want to connect to an already existing google compute engine instance, run a command, and close the connection.
I have used the great sample code here for instance creation and deletion.
Additionally, I have a startup script running which works perfectly.
Now I am reading this article to use paramiko to connect to my instance. This may or may not be the best thing to do, so please correct me if I am going down the wrong path.
I have the following sample code:
import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(
paramiko.AutoAddPolicy())
ssh.connect('35.***.***.**',username='user',password='pass')
stdin, stdout, stderr = ssh.exec_command("sudo su -")
stdin, stdout, stderr = ssh.exec_command("ls -l")
stdout.readlines()
Now - I am not sure which username or password I am supposed to use.
When I run this code, I do not get the list of files and directories in my root as I want, but I do get a list of files and directories in the default user account's home - so it is connecting.
My goal is to connect to a gce instance, run a command, and that is it! For some reason it is trickier than I anticipated. Am I doing something wrong here?
If you are facing a similar use case you can explore gcloud ssh. It worked for me, but I cannot comment if this is best practice or not.
My solution here was something like the following:
import subprocess
def check_for_completion(instance_name = ""):
cmd = "gcloud compute ssh %s --zone=us-east1-b --command=\"sudo -S -i -u root -p '' ls /root/temp/ \""%(instance_name)
try:
res = subprocess.check_output(cmd, shell=True)
items = str(res).split('\n')
return {'response':items,'complete':False}
except:
return {'response':None,'complete':True}
I have a class that creates the connection. I can connect and execute 1 command before the channel is closed. On another system i have i can execute multiple commands and the channel does not close. Obviously its a config issue with the systems i am trying to connect to.
class connect:
newconnection = ''
def __init__(self,username,password):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
ssh.connect('somehost', username=username,password=password,port=2222,timeout=5)
except:
print "Count not connect"
sys.exit()
self.newconnection = ssh
def con(self):
return self.newconnection
Then i use 'ls' command just to print some output
sshconnection = connect('someuser','somepassword').con()
stdin, stdout, stderr = sshconnection.exec_command("ls -lsa")
print stdout.readlines()
print stdout
stdin, stdout, stderr = sshconnection.exec_command("ls -lsa")
print stdout.readlines()
print stdout
sshconnection.close()
sys.exit()
After the first exec_command runs it prints the expected output of the dir list. When i print stdout after the first exec_command it looks like the channel is closed
<paramiko.ChannelFile from <paramiko.Channel 1 (closed) -> <paramiko.Transport at 0x2400f10L (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>
Like i said on another system i am able to keep running commands and the connection doesn't close. Is there a way i can keep this open? or a better way i can see the reason why it closes?
edit: So it looks like you can only run 1 command per SSHClient.exec_command... so i decided to get_transport().open_session() and then run a command. The first one always works. The second one always fails and the scripts just hangs
With just paramiko after the exec_command executes the channel is closed and the ssh returns an auth prompt.
Seems its not possible with just paramiko, try fabric or another tool.
** fabric did not work out too.
Please see the following referece as it provides a way to do this in Paramiko:
How do you execute multiple commands in a single session in Paramiko? (Python)
it's possible with netmiko (tested on windows).
this example is written for connecting to cisco devices but the principle is adaptable for others as well.
import netmiko
from netmiko import ConnectHandler
import json
def connect_enable_silent(ip_address,ios_command):
with open ("credentials.txt") as line:
line_1 = json.load(line)
for k,v in line_1.items():
router=(k,v)
try:
ssh = ConnectHandler(**router[1],device_type="cisco_ios",ip=ip_address)
ssh.enable()
except netmiko.ssh_exception.NetMikoAuthenticationException:
#incorrect credentials
continue
except netmiko.ssh_exception.NetMikoTimeoutException:
#oddly enough if it can log in but not able to authenticate to enable mode the ssh.enable() command does not give an authentication error
#but a time-out error instead
try:
ssh = ConnectHandler(username = router[1]['username'],password = router[1]['password'],device_type="cisco_ios", ip=ip_address)
except netmiko.ssh_exception.NetMikoTimeoutException:
# connection timed out (ssh not enabled on device, try telnet)
continue
except Exception:
continue
else:
output = ssh.send_command(ios_command)
ssh.disconnect()
if "at '^' marker." in output:
#trying to run a command that requires enble mode but not authenticated to enable mode
continue
return output
except Exception:
continue
else:
output = ssh.send_command(ios_command)
ssh.disconnect()
return output
output = connect_enable_silent(ip_address,ios_command)
for line in output.split('\n'):
print(line)
Credentials text is meant to store different credentials in case you are planning to call this function to access multiple devices and not all of them using the same credentials. It is in the format:
{"credentials_1":{"username":"username_1","password":"password_1","secret":"secret_1"},
"credentials_2":{"username":"username_2","password":"password_2","secret":"secret_2"},
"credentials_3": {"username": "username_3", "password": "password_3"}
}
The exceptions can be changed to do different things, in my case i just needed it to not return an error and continue trying the next set, which is why most exceptions are silenced.
I'm trying to run an interactive command through paramiko. The cmd execution tries to prompt for a password but I do not know how to supply the password through paramiko's exec_command and the execution hangs. Is there a way to send values to the terminal if a cmd execution expects input interactively?
ssh = paramiko.SSHClient()
ssh.connect(server, username=username, password=password)
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command("psql -U factory -d factory -f /tmp/data.sql")
Does anyone know how this can addressed? Thank you.
The full paramiko distribution ships with a lot of good demos.
In the demos subdirectory, demo.py and interactive.py have full interactive TTY examples which would probably be overkill for your situation.
In your example above ssh_stdin acts like a standard Python file object, so ssh_stdin.write should work so long as the channel is still open.
I've never needed to write to stdin, but the docs suggest that a channel is closed as soon as a command exits, so using the standard stdin.write method to send a password up probably won't work. There are lower level paramiko commands on the channel itself that give you more control - see how the SSHClient.exec_command method is implemented for all the gory details.
I had the same problem trying to make an interactive ssh session using ssh, a fork of Paramiko.
I dug around and found this article:
Updated link (last version before the link generated a 404): http://web.archive.org/web/20170912043432/http://jessenoller.com/2009/02/05/ssh-programming-with-paramiko-completely-different/
To continue your example you could do
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command("psql -U factory -d factory -f /tmp/data.sql")
ssh_stdin.write('password\n')
ssh_stdin.flush()
output = ssh_stdout.read()
The article goes more in depth, describing a fully interactive shell around exec_command. I found this a lot easier to use than the examples in the source.
Original link: http://jessenoller.com/2009/02/05/ssh-programming-with-paramiko-completely-different/
You need Pexpect to get the best of both worlds (expect and ssh wrappers).
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(server_IP,22,username, password)
stdin, stdout, stderr = ssh.exec_command('/Users/lteue/Downloads/uecontrol-CXC_173_6456-R32A01/uecontrol.sh -host localhost ')
alldata = ""
while not stdout.channel.exit_status_ready():
solo_line = ""
# Print stdout data when available
if stdout.channel.recv_ready():
# Retrieve the first 1024 bytes
solo_line = stdout.channel.recv(1024)
alldata += solo_line
if(cmp(solo_line,'uec> ') ==0 ): #Change Conditionals to your code here
if num_of_input == 0 :
data_buffer = ""
for cmd in commandList :
#print cmd
stdin.channel.send(cmd) # send input commmand 1
num_of_input += 1
if num_of_input == 1 :
stdin.channel.send('q \n') # send input commmand 2 , in my code is exit the interactive session, the connect will close.
num_of_input += 1
print alldata
ssh.close()
Why the stdout.read() will hang if use dierectly without checking stdout.channel.recv_ready(): in while stdout.channel.exit_status_ready():
For my case ,after run command on remote server , the session is waiting for user input , after input 'q' ,it will close the connection .
But before inputting 'q' , the stdout.read() will waiting for EOF,seems this methord does not works if buffer is larger .
I tried stdout.read(1) in while , it works
I tried stdout.readline() in while , it works also.
stdin, stdout, stderr = ssh.exec_command('/Users/lteue/Downloads/uecontrol')
stdout.read() will hang
I'm not familiar with paramiko, but this may work:
ssh_stdin.write('input value')
ssh_stdin.flush()
For information on stdin:
http://docs.python.org/library/sys.html?highlight=stdin#sys.stdin
Take a look at example and do in similar way
(sorce from http://jessenoller.com/2009/02/05/ssh-programming-with-paramiko-completely-different/):
ssh.connect('127.0.0.1', username='jesse',
password='lol')
stdin, stdout, stderr = ssh.exec_command(
"sudo dmesg")
stdin.write('lol\n')
stdin.flush()
data = stdout.read.splitlines()
for line in data:
if line.split(':')[0] == 'AirPort':
print line
You can use this method to send whatever confirmation message you want like "OK" or the password. This is my solution with an example:
def SpecialConfirmation(command, message, reply):
net_connect.config_mode() # To enter config mode
net_connect.remote_conn.sendall(str(command)+'\n' )
time.sleep(3)
output = net_connect.remote_conn.recv(65535).decode('utf-8')
ReplyAppend=''
if str(message) in output:
for i in range(0,(len(reply))):
ReplyAppend+=str(reply[i])+'\n'
net_connect.remote_conn.sendall(ReplyAppend)
output = net_connect.remote_conn.recv(65535).decode('utf-8')
print (output)
return output
CryptoPkiEnroll=['','','no','no','yes']
output=SpecialConfirmation ('crypto pki enroll TCA','Password' , CryptoPkiEnroll )
print (output)