Develop Reference

Paramiko not receiving all data from Channel

I've done a script to configure a couple of ASA devices. It does the job perfectly BUT I can't get the whole output from the devices I'm configuring, at some point the data gets stucked and there's no more output. I want to have that in order to check for problems or misconfigurations, etc. I'm configuring around 500 IPs, objects, groups, etc. from files on ASA firewalls... I don't know what to do, I haven't found any command to clean or erase Paramiko's buffer :(
Any ideas? This is my code:
import paramiko
import re
import time
from tqdm.auto import tqdm
from io import StringIO
device_ip = 'X.X.X.X'
ssh = paramiko.SSHClient() # Connection
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname = device_ip, username = username, password = password)
connection = ssh.invoke_shell()
output_data = StringIO()
connection.send("conf t\n") # Enter configuration mode
time.sleep(1)
file = open(file_name, 'r') # IP list file
lines = file.readlines()
objects = []
ip_subnet = re.compile(r'([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\/([0-9]+)')
group_name = "GRP-OBJ-EU-Blablabla"
for line in tqdm(lines):
match = ip_subnet.findall(line.strip())
ip = match[0][0]
subnet = match[0][1] # Not used, for future use may be...
object_name = "obj-"+ip
object_configuration = "object network "+object_name
object_network = "host "+ip
object_description = "description whatever"
objects.append(object_name)
connection.send(object_configuration+"\n")
time.sleep(1)
connection.send(object_network+"\n")
time.sleep(1)
connection.send(object_description+"\n")
time.sleep(1)
received_data = connection.recv(5000).decode(encoding='utf-8')
if received_data:
output_data.write(received_data)
group_command = "object-group network "+group_name
connection.send(group_command+"\n")
time.sleep(1)
for object_host in tqdm(objects):
connection.send("network-object object "+object_host+"\n")
time.sleep(1)
received_data = connection.recv(5000).decode(encoding='utf-8')
if received_data:
output_data.write(received_data)
connection.send("end \n")
time.sleep(1)
connection.send("wr \n")
time.sleep(5)
connection.close()
ssh.close()
file.close()
print(output_data)
I've tried a line like this one below but it does not work either:
device_output = connection.recv(1000000000000).decode(encoding='utf-8')

I could not directly test your code, but I had a similar problem in the past and found a practical solution. To get all the data at once, you can open a notepad and print the received_data data in it. You can also use ascii outside of the encoding='utf-8' method.
Example Code:
received_data = remote_connection.recv(99999999)
result = received_data.decode('ascii').strip("\n")
LOGfile = "log.txt"
log = open(LOGfile, 'a')
log.write(result )
log.close()
I recommend Netmiko-TTP library to parse all the data you want except the Paramiko library. I have given a sample code link below. I hope that will be useful.
https://github.com/MertKulac/Nokia--TTP--Template--Command--Set/blob/main/show%20system%20informtaion.py

Well I've found the solution... it's pretty dumb. I was not exiting enable mode... and it seems that was it! lol I don't get the relation but anyways the lines below work... Thanks everyone for your help!
....
connection.send("end \n")
time.sleep(1)
connection.send("wr \n")
time.sleep(5)
device_output = connection.recv(10000000).decode(encoding='utf-8')
connection.close()
ssh.close()
file.close()
print(device_output)

Changing output of speedtest.py and speedtest-cli to include IP address in output .csv file

I added a line in the python code “speedtest.py” that I found at pimylifeup.com. I hoped it would allow me to track the internet provider and IP address along with all the other speed information his code provides. But when I execute it, the code only grabs the next word after the find all call. I would also like it to return the IP address that appears after the provider. I have attached the code below. Can you help me modify it to return what I am looking for.
Here is an example what is returned by speedtest-cli
$ speedtest-cli
Retrieving speedtest.net configuration...
Testing from Biglobe (111.111.111.111)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by GLBB Japan (Naha) [51.24 km]: 118.566 ms
Testing download speed................................................................................
Download: 4.00 Mbit/s
Testing upload speed......................................................................................................
Upload: 13.19 Mbit/s
$
And this is an example of what it is being returned by speediest.py to my .csv file
Date,Time,Ping,Download (Mbit/s),Upload(Mbit/s),myip
05/30/20,12:47,76.391,12.28,19.43,Biglobe
This is what I want it to return.
Date,Time,Ping,Download (Mbit/s),Upload (Mbit/s),myip
05/30/20,12:31,75.158,14.29,19.54,Biglobe 111.111.111.111
Or may be,
05/30/20,12:31,75.158,14.29,19.54,Biglobe,111.111.111.111
Here is the code that I am using. And thank you for any help you can provide.
import os
import re
import subprocess
import time
response = subprocess.Popen(‘/usr/local/bin/speedtest-cli’, shell=True, stdout=subprocess.PIPE).stdout.read().decode(‘utf-8’)
ping = re.findall(‘km]:\s(.*?)\s’, response, re.MULTILINE)
download = re.findall(‘Download:\s(.*?)\s’, response, re.MULTILINE)
upload = re.findall(‘Upload:\s(.*?)\s’, response, re.MULTILINE)
myip = re.findall(‘from\s(.*?)\s’, response, re.MULTILINE)
ping = ping[0].replace(‘,’, ‘.’)
download = download[0].replace(‘,’, ‘.’)
upload = upload[0].replace(‘,’, ‘.’)
myip = myip[0]
try:
f = open(‘/home/pi/speedtest/speedtestz.csv’, ‘a+’)
if os.stat(‘/home/pi/speedtest/speedtestz.csv’).st_size == 0:
f.write(‘Date,Time,Ping,Download (Mbit/s),Upload (Mbit/s),myip\r\n’)
except:
pass
f.write(‘{},{},{},{},{},{}\r\n’.format(time.strftime(‘%m/%d/%y’), time.strftime(‘%H:%M’), ping, download, upload, myip))

Let me know if this works for you, it should do everything you're looking for
#!/usr/local/env python
import os
import csv
import time
import subprocess
from decimal import *
file_path = '/home/pi/speedtest/speedtestz.csv'
def format_speed(bits_string):
""" changes string bit/s to megabits/s and rounds to two decimal places """
return (Decimal(bits_string) / 1000000).quantize(Decimal('.01'), rounding=ROUND_UP)
def write_csv(row):
""" writes a header row if one does not exist and test result row """
# straight from csv man page
# see: https://docs.python.org/3/library/csv.html
with open(file_path, 'a+', newline='') as csvfile:
writer = csv.writer(csvfile, delimiter=',', quotechar='"')
if os.stat(file_path).st_size == 0:
writer.writerow(['Date','Time','Ping','Download (Mbit/s)','Upload (Mbit/s)','myip'])
writer.writerow(row)
response = subprocess.run(['/usr/local/bin/speedtest-cli', '--csv'], capture_output=True, encoding='utf-8')
# if speedtest-cli exited with no errors / ran successfully
if response.returncode == 0:
# from the csv man page
# "And while the module doesn’t directly support parsing strings, it can easily be done"
# this will remove quotes and spaces vs doing a string split on ','
# csv.reader returns an iterator, so we turn that into a list
cols = list(csv.reader([response.stdout]))[0]
# turns 13.45 ping to 13
ping = Decimal(cols[5]).quantize(Decimal('1.'))
# speedtest-cli --csv returns speed in bits/s, convert to bytes
download = format_speed(cols[6])
upload = format_speed(cols[7])
ip = cols[9]
date = time.strftime('%m/%d/%y')
time = time.strftime('%H:%M')
write_csv([date,time,ping,download,upload,ip])
else:
print('speedtest-cli returned error: %s' % response.stderr)

$/usr/local/bin/speedtest-cli --csv-header > speedtestz.csv
$/usr/local/bin/speedtest-cli --csv >> speedtestz.csv
output:
Server ID,Sponsor,Server Name,Timestamp,Distance,Ping,Download,Upload,Share,IP Address
Does that not get you what you're looking for? Run the first command once to create the csv with header row. Then subsequent runs are done with the append '>>` operator, and that'll add a test result row each time you run it
Doing all of those regexs will bite you if they or a library that they depend on decides to change their debugging output format
Plenty of ways to do it though. Hope this helps

pyzmail: use a loop variable to get a single raw message

First, I should say, I don't really know much about computer programming, but I find Python fairly easy to use for automating simple tasks, thanks to Al Sweigart's book, "Automate the boring stuff."
I want to collect email text bodies. I'm trying to move homework to email to save paper. I thought I could do that by getting the numbers of the unseen mails and just looping through that. If I try that, the IDLE3 shell just becomes unresponsive, ctrl c does nothing, I have to restart the shell.
Question: Why can't I just use a loop variable in server.fetch()??
for msgNum in unseenMessages:
rawMessage = server.fetch([msgNum], ['BODY[]', 'FLAGS'])
It seems you need an actual number like 57, not msgNum, in there, or it won't work.
After looking at various questions and answers here on SO, the following works for me. I suppose it collects all the email bodies in one swoop.
import pyzmail
import pprint
from imapclient import IMAPClient
server = IMAPClient('imap.qq.com', use_uid=True, ssl=True)
server.login('myEmail#foxmail.com', 'myIMAPpassword')
select_info = server.select_folder('Inbox')
unseenMessages = server.search(['UNSEEN'])
rawMessage = server.fetch(unseenMessages, ['BODY[]', 'FLAGS'])
for msgNum in unseenMessages:
message = pyzmail.PyzMessage.factory(rawMessage[msgNum][b'BODY[]'])
text = message.text_part.get_payload().decode(message.text_part.charset)
print('Text' + str(msgNum) + ' = ')
print(text)

I've found this gist with nice and clean code and a page with many helping examples
The main difference between API of imaplib and pyzmail is pyzmail is all-in-one package with parsing and all client-server communication. But these packages are splitted into different packages in standard library. Basically, they both provided almost the same functionality and with the same methods.
As an additional important note here, pyzmail looks quite abandoned.
To save a useful code from that gist, I copy it here as it is with very small modifications like extracting main function (note, it's for Python 3):
#!/usr/bin/env python
#
# Very basic example of using Python 3 and IMAP to iterate over emails in a
# gmail folder/label. This code is released into the public domain.
#
# This script is example code from this blog post:
# http://www.voidynullness.net/blog/2013/07/25/gmail-email-with-python-via-imap/
#
# This is an updated version of the original -- modified to work with Python 3.4.
#
import sys
import imaplib
import getpass
import email
import email.header
import datetime
EMAIL_ACCOUNT = "notatallawhistleblowerIswear#gmail.com"
# Use 'INBOX' to read inbox. Note that whatever folder is specified,
# after successfully running this script all emails in that folder
# will be marked as read.
EMAIL_FOLDER = "Top Secret/PRISM Documents"
def process_mailbox(M):
"""
Do something with emails messages in the folder.
For the sake of this example, print some headers.
"""
rv, data = M.search(None, "ALL")
if rv != 'OK':
print("No messages found!")
return
for num in data[0].split():
rv, data = M.fetch(num, '(RFC822)')
if rv != 'OK':
print("ERROR getting message", num)
return
msg = email.message_from_bytes(data[0][1])
hdr = email.header.make_header(email.header.decode_header(msg['Subject']))
subject = str(hdr)
print('Message %s: %s' % (num, subject))
print('Raw Date:', msg['Date'])
# Now convert to local date-time
date_tuple = email.utils.parsedate_tz(msg['Date'])
if date_tuple:
local_date = datetime.datetime.fromtimestamp(
email.utils.mktime_tz(date_tuple))
print ("Local Date:", \
local_date.strftime("%a, %d %b %Y %H:%M:%S"))
def main(host, login, folder):
with imaplib.IMAP4_SSL(host) as M:
rv, data = M.login(login, getpass.getpass())
print(rv, data)
rv, mailboxes = M.list()
if rv == 'OK':
print("Mailboxes:")
print(mailboxes)
rv, data = M.select(folder)
if rv == 'OK':
print("Processing mailbox...\n")
process_mailbox(M)
else:
print("ERROR: Unable to open mailbox ", rv)
if __name__ == '__main__':
try:
main('imap.gmail.com', EMAIL_ACCOUNT, EMAIL_FOLDER)
except imaplib.IMAP4.error as e:
print('Error while processing mailbox:', e)
sys.exit(1)

How do I run python '__main__' program file from bash prompt in Windows10?

I am trying to run a python3 program file and am getting some unexpected behaviors.
I'll start off first with my PATH and env setup configuration. When I run:
which Python
I get:
/c/Program Files/Python36/python
From there, I cd into the directory where my python program is located to prepare to run the program.
Roughly speaking this is how my python program is set up:
import modulesNeeded
print('1st debug statement to show program execution')
# variables declared as needed
def aFunctionNeeded():
print('2nd debug statement to show fxn exe, never prints')
... function logic...
if __name__ == '__main__':
aFunctionNeeded() # Never gets called
Here is a link to the repository with the code I am working with in case you would like more details as to the implementation. Keep in mind that API keys are not published, but API keys are in local file correctly:
https://github.com/lopezdp/API.Mashups
My question revolves around why my 1st debug statements inside the files are printing to the terminal, but not the 2nd debug statements inside the functions?
This is happening in both of the findRestaurant.py file and the geocode.py file.
I know I have written my if __name__ == '__main__': program entry point correctly as this is the same exact way I have done it for other programs, but in this case I may be missing something that I am not noticing.
If this is my output when I run my program in my bash terminal:
$ python findRestaurant.py
inside geo
inside find
then, why does it appear that my aFunctionNeeded() method shown in my pseudo code is not being called from the main?
Why do both programs seem to fail immediately after the first debug statements are printed to the terminal?
findRestaurant.py File that can also be found in link above
from geocode import getGeocodeLocation
import json
import httplib2
import sys
import codecs
print('inside find')
sys.stdout = codecs.getwriter('utf8')(sys.stdout)
sys.stderr = codecs.getwriter('utf8')(sys.stderr)
foursquare_client_id = "..."
foursquare_client_secret = "..."
def findARestaurant(mealType,location):
print('inside findFxn')
#1. Use getGeocodeLocation to get the latitude and longitude coordinates of the location string.
latitude, longitude = getGeocodeLocation(location)
#2. Use foursquare API to find a nearby restaurant with the latitude, longitude, and mealType strings.
#HINT: format for url will be something like https://api.foursquare.com/v2/venues/search?client_id=CLIENT_ID&client_secret=CLIENT_SECRET&v=20130815&ll=40.7,-74&query=sushi
url = ('https://api.foursquare.com/v2/venues/search?client_id=%s&client_secret=%s&v=20130815&ll=%s,%s&query=%s' % (foursquare_client_id, foursquare_client_secret,latitude,longitude,mealType))
h = httplib2.Http()
result = json.loads(h.request(url,'GET')[1])
if result['response']['venues']:
#3. Grab the first restaurant
restaurant = result['response']['venues'][0]
venue_id = restaurant['id']
restaurant_name = restaurant['name']
restaurant_address = restaurant['location']['formattedAddress']
address = ""
for i in restaurant_address:
address += i + " "
restaurant_address = address
#4. Get a 300x300 picture of the restaurant using the venue_id (you can change this by altering the 300x300 value in the URL or replacing it with 'orginal' to get the original picture
url = ('https://api.foursquare.com/v2/venues/%s/photos?client_id=%s&v=20150603&client_secret=%s' % ((venue_id,foursquare_client_id,foursquare_client_secret)))
result = json.loads(h.request(url, 'GET')[1])
#5. Grab the first image
if result['response']['photos']['items']:
firstpic = result['response']['photos']['items'][0]
prefix = firstpic['prefix']
suffix = firstpic['suffix']
imageURL = prefix + "300x300" + suffix
else:
#6. if no image available, insert default image url
imageURL = "http://pixabay.com/get/8926af5eb597ca51ca4c/1433440765/cheeseburger-34314_1280.png?direct"
#7. return a dictionary containing the restaurant name, address, and image url
restaurantInfo = {'name':restaurant_name, 'address':restaurant_address, 'image':imageURL}
print ("Restaurant Name: %s" % restaurantInfo['name'])
print ("Restaurant Address: %s" % restaurantInfo['address'])
print ("Image: %s \n" % restaurantInfo['image'])
return restaurantInfo
else:
print ("No Restaurants Found for %s" % location)
return "No Restaurants Found"
if __name__ == '__main__':
findARestaurant("Pizza", "Tokyo, Japan")
geocode.py File that can also be found in link above
import httplib2
import json
print('inside geo')
def getGeocodeLocation(inputString):
print('inside of geoFxn')
# Use Google Maps to convert a location into Latitute/Longitute coordinates
# FORMAT: https://maps.googleapis.com/maps/api/geocode/json?address=1600+Amphitheatre+Parkway,+Mountain+View,+CA&key=API_KEY
google_api_key = "..."
locationString = inputString.replace(" ", "+")
url = ('https://maps.googleapis.com/maps/api/geocode/json?address=%s&key=%s' % (locationString, google_api_key))
h = httplib2.Http()
result = json.loads(h.request(url,'GET')[1])
latitude = result['results'][0]['geometry']['location']['lat']
longitude = result['results'][0]['geometry']['location']['lng']
return (latitude,longitude)

The reason you're not seeing the output of the later parts of your code is that you've rebound the standard output and error streams with these lines:
sys.stdout = codecs.getwriter('utf8')(sys.stdout)
sys.stderr = codecs.getwriter('utf8')(sys.stderr)
I'm not exactly sure why those lines are breaking things for you, perhaps your console does not expect utf8 encoded output... But because they don't work as intended, you're not seeing anything from the rest of your code, including error messages, since you rebound the stderr stream along with the stdout stream.

SecureCRT python scripting

I'm writing a script that will find out what router model and what IOS version a Cisco router is using. I'm writing it in Python using the SecureCRT api. The script sends a show version command that displays information about the router, including the information I need. I then use the SecureCRT api to pull all of that text from the application screen and then I iterate through the text to and use if statements to match router models to see which one it is. Everytime i run the script it runs and doesnt error out but the "new.txt" file is blank.
# $language = "python"
# $interface = "1.0"
crt.Screen.Synchronous = True
ModelIOSScreen = ""
def Main():
ModelIOS()
def ModelIOS():
crt.Screen.Send("show version" + chr(13))
crt.Screen.WaitForString(">")
Screen = crt.Screen.Get(-1, 1, 50, 70)
ModelIOSScreen = str(Screen.split(" ", -1))
RouterModel = ""
for word in ModelIOSScreen:
if word == "2811":
RouterModel = "2811"
elif word == "2801":
RouterModel = "2801"
elif word == "CISCO2911/K9":
RouterModel = "2911"
file = open("new.txt", "w")
file.write(ModelIOSScreen)

I'm on my phone and could probably write a better answer, but I'm about to go to bed. You never close the file you open. Using the following works better.
with open(file, "w") as fp:
fp.write(variable)