Develop Reference

Paramiko not receiving all data from Channel

I've done a script to configure a couple of ASA devices. It does the job perfectly BUT I can't get the whole output from the devices I'm configuring, at some point the data gets stucked and there's no more output. I want to have that in order to check for problems or misconfigurations, etc. I'm configuring around 500 IPs, objects, groups, etc. from files on ASA firewalls... I don't know what to do, I haven't found any command to clean or erase Paramiko's buffer :(
Any ideas? This is my code:
import paramiko
import re
import time
from tqdm.auto import tqdm
from io import StringIO
device_ip = 'X.X.X.X'
ssh = paramiko.SSHClient() # Connection
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname = device_ip, username = username, password = password)
connection = ssh.invoke_shell()
output_data = StringIO()
connection.send("conf t\n") # Enter configuration mode
time.sleep(1)
file = open(file_name, 'r') # IP list file
lines = file.readlines()
objects = []
ip_subnet = re.compile(r'([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\/([0-9]+)')
group_name = "GRP-OBJ-EU-Blablabla"
for line in tqdm(lines):
match = ip_subnet.findall(line.strip())
ip = match[0][0]
subnet = match[0][1] # Not used, for future use may be...
object_name = "obj-"+ip
object_configuration = "object network "+object_name
object_network = "host "+ip
object_description = "description whatever"
objects.append(object_name)
connection.send(object_configuration+"\n")
time.sleep(1)
connection.send(object_network+"\n")
time.sleep(1)
connection.send(object_description+"\n")
time.sleep(1)
received_data = connection.recv(5000).decode(encoding='utf-8')
if received_data:
output_data.write(received_data)
group_command = "object-group network "+group_name
connection.send(group_command+"\n")
time.sleep(1)
for object_host in tqdm(objects):
connection.send("network-object object "+object_host+"\n")
time.sleep(1)
received_data = connection.recv(5000).decode(encoding='utf-8')
if received_data:
output_data.write(received_data)
connection.send("end \n")
time.sleep(1)
connection.send("wr \n")
time.sleep(5)
connection.close()
ssh.close()
file.close()
print(output_data)
I've tried a line like this one below but it does not work either:
device_output = connection.recv(1000000000000).decode(encoding='utf-8')

I could not directly test your code, but I had a similar problem in the past and found a practical solution. To get all the data at once, you can open a notepad and print the received_data data in it. You can also use ascii outside of the encoding='utf-8' method.
Example Code:
received_data = remote_connection.recv(99999999)
result = received_data.decode('ascii').strip("\n")
LOGfile = "log.txt"
log = open(LOGfile, 'a')
log.write(result )
log.close()
I recommend Netmiko-TTP library to parse all the data you want except the Paramiko library. I have given a sample code link below. I hope that will be useful.
https://github.com/MertKulac/Nokia--TTP--Template--Command--Set/blob/main/show%20system%20informtaion.py

Well I've found the solution... it's pretty dumb. I was not exiting enable mode... and it seems that was it! lol I don't get the relation but anyways the lines below work... Thanks everyone for your help!
....
connection.send("end \n")
time.sleep(1)
connection.send("wr \n")
time.sleep(5)
device_output = connection.recv(10000000).decode(encoding='utf-8')
connection.close()
ssh.close()
file.close()
print(device_output)

regex to grep string from config file in python

I have config file which contains network configurations something like given below.
LISTEN=192.168.180.1 #the network which listen the traffic
NETMASK=255.255.0.0
DOMAIN =test.com
Need to grep the values from the config. the following is my current code.
import re
with open('config.txt') as f:
data = f.read()
listen = re.findall('LISTEN=(.*)',data)
print listen
the variable listen contains
192.168.180.1 #the network which listen the traffic
but I no need the commented information but sometimes comments may not exist like other "NETMASK"

If you really want to this using regular expressions I would suggest changing it to LISTEN=([^#$]+)
Which should match anything up to the pound sign opening the comment or a newline character.

I come up with solution which will have common regex and replace "#".
import re
data = '''
LISTEN=192.168.180.1 #the network which listen the traffic
NETMASK=255.255.0.0
DOMAIN =test.com
'''
#Common regex to get all values
match = re.findall(r'.*=(.*)#*',data)
print "Total match found"
print match
#Remove # part if any
for index,val in enumerate(match):
if "#" in val:
val = (val.split("#")[0]).strip()
match[index] = val
print "Match after removing #"
print match
Output :
Total match found
['192.168.180.1 #the network which listen the traffic', '255.255.0.0', 'test.com']
Match after removing #
['192.168.180.1', '255.255.0.0', 'test.com']

data = """LISTEN=192.168.180.1 #the network which listen the traffic"""
import re
print(re.search(r'\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}', data).group())
>>>192.168.180.1
print(re.search(r'[0-9]+(?:\.[0-9]+){3}', data).group())
>>>192.168.180.1

In my experience regex is slow runtime and not very readable. I would do:
with open('config.txt') as f:
for line in f:
if not line.startswith("LISTEN="):
continue
rest = line.split("=", 1)[1]
nocomment = rest.split("#", 1)[0]
print nocomment

I think the better approach is to read the whole file as the format it is given in. I wrote a couple of tutorials, e.g. for YAML, CSV, JSON.
It looks as if this is an INI file.
Example Code
Example INI file
INI files need a header. I assume it is network:
[network]
LISTEN=192.168.180.1 #the network which listen the traffic
NETMASK=255.255.0.0
DOMAIN =test.com
Python 2
#!/usr/bin/env python
import ConfigParser
import io
# Load the configuration file
with open("config.ini") as f:
sample_config = f.read()
config = ConfigParser.RawConfigParser(allow_no_value=True)
config.readfp(io.BytesIO(sample_config))
# List all contents
print("List all contents")
for section in config.sections():
print("Section: %s" % section)
for options in config.options(section):
print("x %s:::%s:::%s" % (options,
config.get(section, options),
str(type(options))))
# Print some contents
print("\nPrint some contents")
print(config.get('other', 'use_anonymous')) # Just get the value
Python 3
Look at configparser:
#!/usr/bin/env python
import configparser
# Load the configuration file
config = configparser.RawConfigParser(allow_no_value=True)
with open("config.ini") as f:
config.readfp(f)
# Print some contents
print(config.get('network', 'LISTEN'))
gives:
192.168.180.1 #the network which listen the traffic
Hence you need to parse that value as well, as INI seems not to know #-comments.

Converting Serial data format using python

I am new to python,currently working on the project reading serial port from micro controller to capture sensor data. The serial data i received looks like this:
[5;17H 0.029[5;40H 0.736[5;63H 9.557[7;17H 0.038[7;40H 0.001 [7;63H 0.008[9;17H-34.199[9;40H 25.800[9;63H 13.799[14;17H -4.623[14;40H 0.597[14;63H218.920[19;14H
this serial data actually have escape sequence 'x1b' before open bracket. How do i get rid of them, escape sequence and text format(5;17H..) and just print sensor data x,y,z format line by line. Can somebody help me.. Thank you..
I'm using python serial code:
import serial
ser = serial.Serial('COM9', 115200, bytesize=8, timeout=0)
while True:
data = ser.read(size=8).decode("utf-8")
s = str(data)
print(data)
ser.close()

Sensor data record starts with \033, so split at this for instance:
data_list = data.split('\033')
for v in data:
print (v)

Raspberry PI Writing CSV Python

As part of a project, I am trying to use a Raspberry PI to capture WiFi networks and write it to a CSV file. If the quality of the signal is over 30/70 I want to capture all WiFi SSIDs and their relevant MAC address and record it.
The issue seems to be relevant syntax but I cannot seem to figure out what is wrong.
def wifiscan():
ssid = []
scanoutput = check_output(["iwlist", "wlan0", "scan"])
curtime = time.strftime("%I:%M:%S")
ssid.append(curtime)
for line in scanoutput.split():
line=str(line)
if line.startswith("Quality"):
line=line[8:-25]
if(line>30 and line.startswith("ESSID")
line=line[7:-1]
ssid.append(line)
with open('/home/pi/Desktop/Project/Results/'+'test.csv','a') as csvfile:
csvwriter = csv.writer(csvfile,delimiter=',')
csvwriter.writerow(ssid)
print ssid
Update 1:
Update 2:

I made some changes to your code but i don't know exactaly what you want to accomplish nor what errors you are experiencing, I just fixed some logic an sintax.
As it is, the code will append the ssid data to your csv only if quality is better than 30.
Let me know if that is what you want.
def wifiscan():
ssid = []
scanoutput = check_output(["iwlist", "wlan0", "scan"])
curtime = time.strftime("%I:%M:%S")
ssid.append(curtime)
quality = 0
essid = ""
for line in scanoutput.split('\n'):
line=str(line)
if line.startswith("Quality"):
quality=int(line[8:-25])
if quality>30 and line.startswith("ESSID"):
line=line[7:-1]
ssid.append(quality)
ssid.append(line)
with open('/home/pi/Desktop/Project/Results/'+'test.csv','a') as csvfile:
csvwriter = csv.writer(csvfile,delimiter=',')
csvwriter.writerow(ssid)
print ssid
Note that the code is sensible to any change in the sintax of the input you provide.
Also, to avoid syntax change issues in your input, maybe you should read about regular expressions in python: https://docs.python.org/2/howto/regex.html

Reading bytes from ser.read()

I have one XBee router (XB-X) connected to an ultrasonic range sensor and the other XBee coordinator (XB-Y) connected to my PC. XB-X will send the sensor reading to XB-Y. I am trying to extract the bytes from ser.read() on my PC using Python.
I got a weird bytes of string consist of ">>~}3#zi167.89" on output, may I know how to extract only the floating point number (167.89 in this example)? The number of bytes are specified by setting ser.read(size=xx). Is there any alternative way of doing it?
import serial
import zigbee
from xbee import ZigBee
import re
ser = serial.Serial("COM4", 9600)
while True:
try:
input = ser.read(20)
m = re.search(r'(\d+\.\d+)', input)
if m:
num = m.group()
# statements...
It is working now, carelessly typed 'group' as 'groups'.

Just use a regular expression:
import re
input = ser.read()
m = re.search(r'(?P<num>\d+\.\d+)', input)
if m:
num = float(m.group('num'))