Develop Reference

Command output is corrupted when executed using Python Paramiko exec_command

I'm a software tester, trying to verify that the log on a remote QNX (a BSD variant) machine will contain the correct entries after specific actions are taken. I am able to list the contents of the directory in which the log resides, and use that information in the command to read (really want to use tail -n XX <file>) the file. So far, I always get a "(No such file or directory)" when trying to read the file.
We are using Froglogic Squish for automated testing, because the Windows UI (that interacts with the server piece on QNX) is built using Qt extensions for standard Windows elements. Squish uses Python 2.7, so I am using Python 2.7.
I am using paramiko for the SSH connection to the QNX server. This has worked great for sending commands to the simulator piece that also runs on the QNX server.
So, here's the code. Some descriptive names have been changed to avoid upsetting my employer.
import sys
import time
import select
sys.path.append(r"C:\Python27\Lib\site-packages")
sys.path.append(r"C:\Python27\Lib\site-packages\pip\_vendor")
import paramiko
# Import SSH configuration variables
ssh_host = 'vvv.xxx.yyy.zzz'
thelog_dir = "/logs/the/"
ssh_user = 'un'
ssh_pw = 'pw'
def execute_Command(fullCmd):
outptLines = []
#
# Try to connect to the host.
# Retry a few times if it fails.
#
i = 1
while True:
try:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(ssh_host, 22, ssh_user, ssh_pw)
break
except paramiko.AuthenticationException:
log ("Authentication failed when connecting to %s" % ssh_host)
return 1
except:
log ("Could not SSH to %s, waiting for it to start" % ssh_host)
i += 1
time.sleep(2)
# If we could not connect within time limit
if i == 30:
log ("Could not connect to %s. Giving up" % ssh_host)
return 1
# Send the command (non-blocking?)
stdin, stdout, stderr = ssh.exec_command(fullCmd, get_pty=True)
for line in iter(stdout.readline, ""):
outptLines.append(line)
#
# Disconnect from the host
#
ssh.close()
return outptLines
def get_Latest_Log():
fullCmd = "ls -1 %s | grep the_2" %thelog_dir
files = execute_Command(fullCmd)
theFile = files[-1]
return theFile
def main():
numLines = 20
theLog = get_Latest_Log()
print("\n\nThe latest log is %s\n\n" %theLog)
fullCmd = "cd /logs/the; tail -n 20 /logs/the/%s" %theLog
#fullCmd = "tail -n 20 /logs/the/%s" %theLog
print fullCmd
logLines = execute_Command(fullCmd)
for line in logLines:
print line
if __name__ == "__main__":
# execute only if run as a script
main()
I have tried to read the file using both tail and cat. I have also tried to get and open the file using Paramiko's SFTP client.
In all cases, the response of trying to read the file fails -- despite the fact that listing the contents of the directory works fine. (?!) And BTW, the log file is supposed to be readable by 'world'. Permissions are -rw-rw-r--.
The output I get is:
"C:\Users\xsat086\Documents\paramikoTest>python SSH_THE_MsgChk.py
The latest log is the_20210628_115455_205.log
cd /logs/the; tail -n 20 /logs/the/the_20210628_115455_205.log
(No such file or directory)the/the_20210628_115455_205.log"
The file name is correct. If I copy and paste the tail command into an interactive SSH session with the QNX server, it works fine.
Is it something to do with the 'non-interactive' nature of this method of sending commands? I read that some implementations of SSH are built upon a command that offers a very limited environment. I don't see how that would impact this tail command.
Or am I doing something stupid in this code?

I cannot really explain completely, why you get the results you get.
But in general a corrupted output is a result of enabling and not handling terminal emulation. You enable the terminal emulation using get_pty=True. Remove it. You should not use the terminal emulation, when automating command execution.
Related question:
Is there a simple way to get rid of junk values that come when you SSH using Python's Paramiko library and fetch output from CLI of a remote machine?

SSHLibrary prompts for password

I have a problem with a test suite. I use robot framework and python.I created a function in python which executes a console command in a remote Linux client.
def purge_default_dns(device_ip):
ssh_val = "usr1#" + device_ip
command = "ctool dns | grep -v \"grep\" | awk \'{print $2}\'"
test = check_output(["ssh", ssh_val, "-p", "6022", command])
The check_output() function connects with device_ip and executes command. If I try to connect with a fully qualified domain name (ex. my.domain.io), then I get a prompt for password (which is empty). I press enter and command executes regular. Is there any parameter that passes for example Enter when password prompt comes up?
I tried ssh -e switch , I don't want to change ssh client , I just need a generic solution.
For example using paramiko library in the code below , I can create an paramiko SSHClient , which has a parameter for password and doesn't prompt anything. While I can't use paramiko right now , I need something else with SSHLirary to go around the problem.
def send_ssh_command(device_ip , command):
hostname = device_ip
password = ""
username = "usr1"
port = 6022
try:
client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.WarningPolicy())
client.connect(hostname, port=port, username=username, password=password)
stdin , stdout , stderr = client.exec_command(command)
command_return_val = stdout.read()
finally:
client.close()
return command_return_val
Thank you.

To get this straight, the only solution you look for is to pass the password on the command line to the default OS ssh client, and do not/cannot install any libraries (paramiko, etc) that can help you achieve the same result through other means?
I'm asking this, because the robot framework's SSHLibrary provides this out of the box; you already have the python's solution with paramiko; and the general linux solution is to install the sshpass package, and use it to pass the value:
sshpass -p "YOUR_PASS" ssh -usr1#my.domain.io:6022
So if all of these are not an option, you are left with two alternatives - either hack something around SSH_ASKPASS - here's an article with a sample, or use expect to pass it - this one is what I'd prefer out of the two.
Here's a very good SO answer with an expect script wrapper around ssh. In your method, you will have to first create a file with its content, set an executable flag on it, and then call that file in check_output(), passing as arguments the password, 'ssh' and all its arguments.

Why You need to go with python , I am using below code in robotframework for the same:
[Arguments] ${host}=${APP_SERVER} ${username}=${APP_USERNAME} ${password}=${APP_PASSWORD}
Open Connection ${host} timeout=2m
Login ${username} ${password}
${out} ${err} ${rc}= Execute Command cd ${PATH};ctool dns | grep -v \"grep\" | awk \'{print $2}\' * return_stdout=True return_stderr=True return_rc=True
Should Be Equal ${rc} ${0}

How to list current directory files through socket API?

So right now I have a simple FTP system to transfer files.
But I am confused about how I would run commands on the server machine from a client machine.
How would I open a terminal on the server machine from my client machine to use commands such as ls or mkdir or cd? Or can I do this straight from Socket Programming

You could use the python module subprocess. (https://pymotw.com/2/subprocess/)
For example, assuming you have a client/server 'dialogue' set up using sockets, you could do something like this:
client.py
# assume 's' is your socket already connected to the server
# prompt the user for a command to send
cmd = raw_input("user > ")
s.send(cmd) # send your command to the server
# let's say you input 'ls -la'
You could put the above code inside a loop that only breaks when you enter 'quit' or something, to continually send and receive commands. You would need a loop or something similar on the server side too, to continually accept and return the output from your commands. You could also use threads.
server.py
# on the server side do this
# s is again your socket bound to a port
# but we're on the server side this time!
from subprocess import Popen, PIPE
cmd = s.recv(1024)
# cmd now has 'ls -la' assigned to it
# parse it a bit
cmd = cmd.split() # to get ['ls', '-la']
# now we execute the command on the server with subprocess
p = Popen(cmd, stdout=PIPE)
result = p.communicate()
# result is, in this case, the listing of files in the current directory
s.send(result[0]) # result[0] should be a str
# you now make sure to receive your result on the client
Note: I think a newer version is subprocess32, but all methods are the same as far as I remember.

Changing Linux username or password with Python script

I'm writing a Python script that changes the username and password of a Linux account user - it's part of a larger internal web-gui system that queues up password change requests from apache2 (which can't run as root), and then changes the passwords itself. The python script itself obviously must run as root in order to change passwords.
The password change function is pretty straightforward:
def chpasswd(user, passwd):
if os.getuid() != 0:
syslog.syslog("Error: chpasswd.py must be run as root")
return
proc = Popen(
['/usr/sbin/chpasswd'],
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
stderr = subprocess.PIPE
)
print "Changing: " + user + ':' + passwd
out, err = proc.communicate(user + ':' + passwd)
proc.wait()
print out
if proc.returncode != 0:
print "Error: Return code", proc.returncode, ", stderr: ", out, err
if out:
syslog.syslog("stdout: " + out)
if err:
syslog.syslog("stderr: " + err)
The print statements are just there for temporary debugging. This runs fine and doesn't report any errors - there's nothing on out or err; but for some reason the actual UNIX password simply isn't changed.
The script which invokes this function is listening on a locally bound TCP socket. When it receives a change password request (in the form of user:password - later to be encrypted but for now plaintext) it adds it to a queue and then invokes the chpasswd function.
So, typical usage would be:
# telnet localhost 7001
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
jsmith:mynewpassword
When the server is running in a bash window (not as a daemon) it prints out:
# python chpasswd.py
Starting Password Server
New connection from: 127.0.0.1
Changing: jsmith:mynewpassword
The last statement, you can see, is the print statement in my chpasswd function.
But after doing the above, when I actually try to login as a user using the new password, I get:
$ su jsmith
Password:
su: Authentication failure
Is there some obvious thing I'm doing wrong here? My suspicion was that somehow the connection with Popen is not actually closing, or perhaps the single line user:password text is not being transmitted. So I tried doing something like:
out, err = proc.communicate(user + ':' + passwd + '\x04')
Notice the extra \x04 character at the end, indicating End Of Transmission. Adding this in still didn't get it to work however - the password remained unchanged.
I'm running this on Debian Wheezy, in case it makes any difference.
Update:
Investigating further, I can see that my chpasswd function actually is changing the password - if I cat the /etc/shadow file before and after connecting to my password server, I see there is a different hash.
It's just that when I try to authenticate using the plaintext password, it doesn't work. Therefore, my suspicion is that somehow, the communication with Popen is either adding additional characters, or losing characters somehow. Of course, since /etc/shadow is a hash, I can't figure out exactly what's going on here.

The problem in this particular instance was that telnet adds "\r\n" after you press return on entering text. Since your server was not stripping the data of whitespace this was preserved when changing the password.
It is possible to get telnet to not send the carriage return and newline characters by ending a line with the end-of-transmission character (EOT). You can do this by pressing Ctrl-D.
eg
$ telnet localhost 7001
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
jsmith:mynewpassword^DChanging: jsmith:mynewpassword
Alternatively you can pipe the line into telnet
echo -n jsmith:mynewpassword | telnet localhost 7001
Obviously, you'll only want to do this for testing or the new password will end up in your shell history. (The -n argument suppresses the printing of newline characters by echo)
Or you might want to do away with telnet altogether and use netcat instead.
echo -n jsmith:mynewpassword | netcat localhost 7001

Python run mutiple ssh commands in the same session

My goal is to connect to SSH with python and authenticate which i can do with Paramiko or Fabric. But i would like to keep the session open after each execution and read the input/output. With paramiko i can only run 1 command before the session is closed and i am asked to authenticate again and the session hangs. And since fabric is using the paramiko library its giving me the same issue. For example if my directory structure is like this
-home
--myfolder1
--myfolder2
I would like to execute the below commands without having to re-authenticate because the sessions closes.
(make connection)
run cmd: 'pwd'
output: /home
run cmd: 'cd myfolder2'
run cmd: 'pwd'
output: /home/myfolder2
Is this possible with any module that is out there right now? Could it be made from scratch with native python? And also is this just not possible...?
Edit Added code. Without the new open_session it closes and i cannot run any command. After running the first command with this i will be prompted again to authenticate and it creates an infinite loop.
Edit2 If it closes after each command then there is no way this will work at all correct?
edit3 If i run this on a different server and exec_command with the paramikio.SSHClient it wont ask me to reauthenticate but if i 'cd somedir' and then 'pwd' it will output that i am back in the root directory of where i created.
class connect:
newconnection = ''
def __init__(self,username,password):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
ssh.connect('someserver', username=username,password=password,port=22,timeout=5)
except:
print "Count not connect"
sys.exit()
self.newconnection = ssh
def con(self):
return self.newconnection
#This will create the connection
sshconnection = connect('someuser','somepassword').con()
while True:
cmd = raw_input("Command to run: ")
if cmd == "":
break
try:
transport = sshconnection.get_transport()
transport.set_keepalive(999999)
chan = transport.open_session()
chan.settimeout(3)
chan.setblocking(0)
except:
print "Failed to open a channel"
chan.get_exception()
sys.exit()
print "running '%s'" % cmd
stdout_data = []
stderr_data = []
pprint.pprint(chan)
nbytes = 4096
chan.settimeout(5)
chan.get_pty()
chan.exec_command(cmd)
while True:
print "Inside loop " , chan.exit_status_ready()
time.sleep(1.2)
if chan.recv_ready():
print "First if"
stdout_data.append(chan.recv(nbytes))
if chan.recv_stderr_ready():
print "Recv Ready"
stderr_data.append(chan.recv_stderr(nbytes))
if chan.exit_status_ready():
print "Breaking"
break
print 'exit status: ', chan.recv_exit_status()
print ''.join(stdout_data)

This is possible by using the normal modules when you can concatenate the commands into one. Try
pwd ; cd myfolder2 ; pwd
as command. This should work but quickly becomes tedious when you have more complex commands which need arguments and horrible when the arguments contain spaces. The next step then is to copy a script with all the commands to the remote side and tell ssh to execute said script.
Another problem of this approach is that SSH doesn't return until all commands have executed.
Alternatively, you could build a "command server", i.e. a simple TCP server that listens for incoming connections and executes commands sent to it. It's pretty simple to write but also pretty insecure. Again, the solution is to turn the server into a (Python) script which reads commands from stdin and start that script remotely via SSH and then send commands.