Python Paramiko get full output

Python Paramiko get full output - python

I am setting up a remote SSH connection to a remote server and running a specific command to dump a DB table. The remote server is a hardened linux OS with it's own shell. I am running a remote sql type of command to dump out a lot of data. My python script is using an interactive SSH session to do this. As you can see below, I am running a command, letting it sleep for 5 seconds, then dumping the buffer. I've tried many different options for the "remote_conn.recv()" function but I cannot get the full output. The output is very large and it is paged (press space for more). I am not even getting the complete output of the first page. If there are 50 lines on the first page, I'm getting the first 4 only.
Are there better ways of doing this? How I can just grab the complete output? Below is the Paramiko portion of my script.
# Create instance of SSHClient object
remote_conn_pre = paramiko.SSHClient()
# Automatically add untrusted hosts
remote_conn_pre.set_missing_host_key_policy(
paramiko.AutoAddPolicy())
# initiate SSH connection
remote_conn_pre.connect(options.ip, username=options.userName, password=options.password)
# Use invoke_shell to establish an 'interactive session'
remote_conn = remote_conn_pre.invoke_shell()
remote_conn.send("<remote sql command here>")
time.sleep(5)
output = remote_conn.recv(65535)
print output
remote_conn.close()

I was able to get the full output by grabbing smaller chunks of the buffer and concat'ing them together until I hit a certain string in the buffer.
while True:
data = remote_conn.recv(500)
if "<string>" in data:
break
else:
finalOutput += data

Related

Cannot Print output for Fortigate Command with Python [duplicate]

This question already has answers here:
Get output from a Paramiko SSH exec_command continuously
(6 answers)
Closed 11 months ago.
I am having an issue with getting my python script to return or print any output for the following command run on my fortigate firewall
'diagnose sniffer packet any "host X.X.X.X and port 53" 4 0 a
def dns_pcap():
device = ConnectHandler(device_type="fortinet", ip="X.X.X.X", username="xxxxxxxx", password="xxxxxxxxx")
lines = []
gi_pcap = device.send_command('diagnose sniffer packet GI "host X.X.X>X and port 53" 4 0 a')
output = device.read_channel()
print(output)
dns_pcap()
The script outputs nothing to my terminal, anyone have any idea how to get the output of this to command to print to my screen?
(Also please note I am using python2.7)
I have many scripts running to both fortinet and cisco devices, and they all print outputs from variable assigned commands to screen when i execute my scripts, but not in this case
I am assuming it is because the output is not static like a 'show system interface' but I am not sure how to handle dynamic output from a command.

The output is the result of the method device.send_command. output = device.read_channel() is not necessary. Replace print(output) with print(gi_pcap) and you'll be on the right track.
If the command requires any sort of interaction, that can also cause it to freeze up because the script is waiting for a prompt that will never come. You can see if this is happening by saving the session log output. You can do this by adding session_log="device.txt" to your session arguments, with device.txt being any filename relevant to you. The contents of that file will be whatever is being sent back and forth in the SSH session to the device.

The answer was indeed found in this post
Get output from a Paramiko SSH exec_command continuously
using paramiko and the get_pty=True argument in the function allowed me to print the pseudu terminal
Thanks very much to Richard Dodson for the help

Can you share what is the interval time of getting an output in your Fortigate device for this command? Depending on the interval time, netmiko should have also worked.
What I have also tested in a different vendor using Netmiko for an dynamic output that I tried to get data in 10 seconds interval (Totally 100 seconds), it worked without an issue, I was able to get the output properly. However, when I have increased time of the interval of getting data as 11 or 12, I get some errors so it did not work.
Can you also try with Netmiko "timing" method to get your data? If the interval is shorter than 1-2 seconds, this method should also help. (For example, I can get ping output with 100 count without an error using timing method. Did you also try to get ping output in your network box if it works?)
I think that the size of the data you expect from your output is also important in your case. If the size is too big to be shown in a CLI screen, this may cause also a problem. In that case, you might need to open a file to save your data and use it from that file might be another option.
Better if you can advise if there is a regular time interval of this command along with the expected size of the data.
In addition, what I have also figure it out that in case the output takes more than 100 seconds, it might cause the problem. Here is the sample of how to this option following:
{
'device_type': 'device_type',
'ip': '135.243.92.119',
'username': 'admin',
'password': 'admin',
'port': port,
"fast_cli": False, # fast_cli
"global_delay_factor": 2 # if the outputs takes more than 100 seconds.
}

Get Network Bandwidth and write the data using subprocess (Python : v3.8.x)

Iam trying to monitor the network bandwidth using any command, run it using subprocess and then load the data in json format for my API.
I currently came across 3 terminal commands.
1- vnstat
2- fast
3- nload
Fast is better in providing the real time network bandwidth. But nload also give real time data in a clear manner. How do i get the data in either of the cases if iam using subprocess so that my API doesnt end up facing a request timeout error.??

I was basically trying to achieve the output of the subprocess for a command and write it in json format.
cmd=['vnstat']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
output = p.stdout.read().decode('utf-8')
p.stdout.close()
status = p.returncode
if status is None:
network_bandwidth = {"network_bandwidth": {}}
for line in output.splitlines():
each_line = line.split()
network_bandwidth["network_bandwidth"] = {"key":"Here i did the processing part where i extracted the received and transmitted data and prepared my json response"}
else:
logger.error("Error in executing the command '{}'".format(cmd))
return network_bandwidth

Python Parallel SSH get only command output

I'm new to Python and i'm looking to run multiple parallel ssh connections and commands to the devices. I'm using pssh link for it.
Issue is the device returns some big header after the connection like 20-30 lines.
When i use the below code what is printed out is the result of the command but at the top there's also the big header that is printed after the login.
hosts = ['XX.XXX.XX.XXX']
client = ParallelSSHClient(hosts, user='XXXX', password='XXXXX')
output = client.run_command('command')
for host in output:
for line in output[host]['stdout']:
print line
Anyway i can get JUST the command output ?

Not sure I understand what you mean.
I'm using pssh as well, and seems like I'm using the same method as you are in order to print my command's output, see below:
client = pssh.ParallelSSHClient(nodes, pool_size=args.batch, timeout=10, num_retries=1)
output = client.run_command(command, sudo=True)
for node in output:
for line in output[node]['stdout']:
print '[{0}] {1}'.format(node, line)
Could you elaborate a bit more? Maybe provide an example of a command you run and the output you get?

checkout pssh.
This tool uses multi-threads and performs quickly.
you can read more about it here.

Python: Save output files on another computer through ssh

I want to save output files (binary, matplotlib, etc.) of a Python program in a different computer over ssh. So far, I have saved files in the same computer that runs the program and to do that I have a line in my Python code filename = '/OutputFiles/myOutput.txt'. How do I change this line so that I can save the output in a different computer through ssh? It can be assumed that the ssh login password for the remote computer is in my keyring.

(New answer because OP specified they wanted to write multiple files)
You want to look into the paramiko module. It might look something like this:
import paramiko
connection = paramiko.SSHClient()
connection.set_missing_host_key_policy(paramiko.AutoAddPolicy())
connection.connect(10.10.10.10, username=myuser, password=mypass)
ftp = connection.open_sftp()
for i in range(10):
results = do_stuff()
f = ftp.open(i+'.txt', 'w+')
f.write(results)
f.close()
ftp.close()
connection.close()

The easiest way to do this would be to write your output to standard out and pipe it to ssh (assuming you're on a Mac, Linux, or other *nix-based machine). For example:
python myProgram | ssh user#host 'cat > outfile.txt'

Python and telnetlib on 2.7. Totally not working for me

Below is the streaming data that i get when I connect from a command line. Yet..quit shocking...I get no data from from python. Why cant I print to screen the output? One would think I should just connect to to server, ready very eager.
tn = telnetlib.Telnet()
tn.open(HOST, TRADE_PORT)
while True:
print '-',tn.read_very_eager()
-
-
-
-
Below us a working telnet session on same machine.
ubuntu#ubuntu:~$ telnet 222.222.222.222 2000
Trying 222.222.222.222...
Connected to 222.222.222.222.
Escape character is '^]'.
S=SXL,P=157.00,Q=3221,T=130621,N=40111759,E=371993|S=SXL,P=157.00,Q=45,T=130621,N=40111760,E=371994|S=SXL,P=157.00,Q=1401,T=130621,N=40111761,E=371995|S=SXL,P=157.00,Q=46,T=130621,N=40111762,E=371996|S=SXL,P=157.00,Q=37,T=130621,N=40111763,E=371997|S=SXL,P=157.00,Q=60,T=130621,N=40111764,E=371998|S=RIO,P=6514.50,Q=57,T=130621,N=10085304,E=371999|S=SXL,P=157.00,Q=39,T=130621,N=40111765,E=372000|S=SXL,P=157.00,Q=701,T=130621,N=40111766,E=372001|S=SXL,P=157.00,Q=119,T=130621,N=40111767,E=372002|S=SXL,P=157.00,Q=120,T=130621,N=40111768,E=372003|S=SXL,P=157.00,Q=47,T=130621,N=40111769,E=372004|S=SXL,P=157.00,Q=22,T=130621,N=40111770,E=372005,C=XT|S=SXL,P=157.00,Q=98,T=130621,N=40111771,E=372006|S=SXL,P=157.00,Q=924,T=130621,N=40111772,E=372007|S=SXL,P=157.00,Q=381,T=130621,N=40111773,E=372008|S=SXL,P=157.00,Q=221,T=130621,N=40111774,E=372009|S=SXL,P=157.00,Q=220,T=130621,N=40111775,E=372010|S=SXL,P=157.00,Q=313,T=130621,N=40111776,E=372011|S=RIO,P=6514.00,Q=61,T=130621,N=10085305,E=372012|S=RIO,P=6514.00,Q=32,T=130621,N=10085306,E=372013|S=PMV,P=762.00,Q=17,T=130621,N=40111777,E=372014|S=RIO,P=6514.00,Q=8,T=130621,N=10085307,E=372015|S=RIO,P=6514.00,Q=18,T=130621,N=10085308,E=372016|S=RIO,P=6514.00,Q=65,T=130621,N=10085309,E=372017|S=AMP,P=456.50,Q=129,T=130621,N=20091811,E=372018|S=AMP,P=456.50,Q=482,T=130621,N=20091812,E=372019|S=GPT,P=353.50,Q=109,T=130621,N=20091813,E=372020|S=RSG,P=47.50,Q=194,T=130621,N=10085310,E=372021|S=AGK,P=1499.00,Q=147,T=130621,N=40111778,E=372022|S=AGK,P=1499.00,Q=183,T=130621,N=40111779,E=372023|S=AGK,P=1499.00,Q=1,T=130621,N=40111780,E=372024|S=AGK,P=1499.00,Q=183,T=130621,N=40111781,E=372025|S=AGK,P=1499.00,Q=29,T=130621,N=40111782,E=372026|S=AGK,P=1499.00,Q=77,T=130621,N=40111783,E=372027|S=AGK,P=1499.00,Q=21,T=130621,N=40111784,E=372028|S=AGK,P=1499.00,Q=492,T=130621,N=40111785,E=372029|S=AGK,P=1498.50,Q=68,T=130621,N=40111786,E=372030|S=AGK,P=1499.00,Q=7,T=130621,N=40111787,E=372031|S=AGK,P=1499.00,Q=25,T=130621,N=40111788,E=372032|S=SYDDA,P=388.00,Q=50,T=130621,N=30083122,E=372033|S=PCL,P=6.10,Q=145,T=130621,N=30083123,E=372034|S=VAS,P=6727.00,Q=744,T=130622,N=30083124,E=372035|S=REA,P=4081.00,Q=2,T=130622,N=40111789,E=372036|S=MMS,P=1269.00,Q=2,T=130622,N=30083125,E=372037|S=DMP,P=1488.00,Q=2,T=130622,N=40111790,E=372038|S=WPL,P=3733.00,Q=24,T=130622,N=30083126,E=372039|S=NWS,P=1928.00,Q=1,T=130622,N=40111791,E=372040|S=WBC,P=3218.00,Q=118,T=130622,N=20091814,E=372041|S=WOR,P=1610.00,Q=9,T=130622,N=30083127,E=372042|S=NWS,P=1928.00,Q=3,T=130622,N=40111792,E=372043|S=SUN,P=1298.00,Q=50,T=130622,N=20091815,E=372044|S=WES,P=4264.00,Q=1,T=130622,N=30083128,E=372045|S=RIO,P=6515.00,Q=1,T=130622,N=10085311,E=372046|S=SUL,P=1360.00,Q=6,T=130622,N=40111793,E=372047|S=WES,P=4264.00,Q=43,T=130622,N=30083129,E=372048|S=QBE,P=1566.00,Q=46,T=130622,N=20091816,E=372049|S=RIO,P=6515.00,Q=16,T=130622,N=10085312,E=372050|S=WOW,P=3347.00,Q=48,T=130622,N=40111794,E=372051|S=ORI,P=2302.00,Q=13,T=130622,N

Use tn.read_some() (blocking) instead of tn.read_very_eager() : it will wait for data to be available before returning

Develop Reference

Python is a programming language that lets you work quickly and integrate systems more effectively.