Develop Reference

How to parse Log file to object list

I'm working with data tipe Log (ROS).
Multiple objects are saved in Log file like this:
header:
seq: 2
stamp:
secs: 1596526199
nsecs: 140017032
frame_id: ''
level: 2
name: "/replicator_node"
msg: "Replicator node dumping to /tmp/replicator_dumps"
file: "replicator_node.py"
function: "__init__"
line: 218
topics: [/move_mongodb_entries/status, /move_mongodb_entries/goal, /move_mongodb_entries/result,
/move_mongodb_entries/cancel, /rosout, /move_mongodb_entries/feedback]
header:
seq: 2
stamp:
secs: 1596526198
nsecs: 848793029
frame_id: ''
level: 2
name: "/mongo_server"
msg: "2020-08-04T09:29:58.848+0200 [initandlisten] connection accepted from 127.0.0.1:58672\
\ #1 (1 connection now open)"
file: "mongodb_server.py"
function: "_mongo_loop"
line: 139
topics: [/rosout]
As you can see not everything is in same line as it's name.
I want to pars it to have object list - so I could access it like that:
object[1].msg would give me:
"2020-08-04T09:29:58.848+0200 [initandlisten] connection accepted from 127.0.0.1:58672 #1 (1 connection now open)"
Also, sometimes file name is something like: \home\nfoo\foo.py which results in log file as:
file: "\home
foo\foo.py"

It's an interesting exercise... Assuming that the structure is really consistent for all log entries, you can try something like this - pretty convoluted, but it works for the example in the question:
ros = """[your log above]"""
def manage_lists_2(log_ind, list_1, list_2, mystr):
if log_ind == 0:
list_1.append(mystr.split(':')[0].strip())
list_2[-log_ind].append(mystr.split(':')[1].strip())
m_keys2 = []
m_key_vals2 = [[],[]]
header_keys2 = []
header_key_vals2 = [[],[]]
stamp_keys2 = []
stamp_key_vals2 = [[],[]]
for log in logs:
for l in log.splitlines():
if l[0]!=" ":
items = [m_keys2, m_key_vals2]
elif l[0:3] != " ":
items = [header_keys2, header_key_vals2]
else:
items = [stamp_keys2, stamp_key_vals2]
manage_lists_2(logs.index(log), items[0], items[1], l)
for val in m_key_vals2:
for a, b, in zip(m_keys2,val):
print(a, ": ",b)
if a == "header":
for header_key in header_keys2:
print('\t',header_key,':',header_key_vals2[m_keys2.index(a)][header_keys2.index(header_key)])
if header_key == "stamp":
for stamp_key in stamp_keys2:
print('\t\t',stamp_key,':',stamp_key_vals2[m_keys2.index(a)][stamp_keys2.index(stamp_key)])
print('---')
Output:
header :
seq : 2
stamp :
secs : 1596526199
nsecs : 140017032
frame_id : 'one id'
level : 2
name : "/replicator_node"
msg : "Replicator node dumping to /tmp/replicator_dumps"
file : "replicator_node.py"
function : "__init__"
line : 218
topics : [/move_mongodb_entries/status, /move_mongodb_entries/goal, /move_mongodb_entries/result, /move_mongodb_entries/cancel, /rosout, /move_mongodb_entries/feedback]
---
header :
seq : 2
stamp :
secs : 1596526199
nsecs : 140017032
frame_id : 'one id'
level : 3
name : "/mongo_server"
msg : "2020-08-04T09
file : "mongodb_server.py"
function : "_mongo_loop"
line : 139
topics : [/rosout]
Having gone through that, I would recommend that - if you are going to do this on a regular basis - you find a way to store the data in xml format; it's a natural fit for it.

Empty json object when read text file

I'm testing a python script with text data. Able to run the script and return valid json file if the text include in the script but I got empty json object when run the script and with separate text file.
The output only empty json file
{
"ospf": []
}
The code below return empty json object when run it with read text file
import json
result = {}
l = []
with open('data.txt') as myf:
for i in myf:
if i:
p = [parameter for parameter in i.split("*")]
for line, x in enumerate(p[0].split("\n")):
if x and "Ls id" in x:
ls_id, ip = x.split(": ")
ls_id = ls_id.strip()
ip = ip.strip()
for y in p[1:]:
if y and "P-2-P" in y:
temp = {ls_id:ip}
for items in y.split("\n"):
try:
key, value = items.split(": ")
key = key.strip()
value = value.strip()
temp[key] = value
except ValueError:
pass
l.append(temp)
result["ospf"] = l
print (json.dumps(result,indent=2))
with open('data.json', 'w') as json_file:
json.dump(result, json_file)
When executed the code below ok with the text data include as data..no problem
import json
data = '''
Type : Router
Ls id : 1.1.1.2
Adv rtr : 1.1.1.2
Ls age : 201
Len : 84
Link count: 5
* Link ID: 1.1.1.2
Data : 255.255.255.255
Link Type: StubNet
Metric : 1
Priority : Medium
* Link ID: 1.1.1.4
Data : 192.168.100.34
Link Type: P-2-P
Metric : 1
* Link ID: 192.168.100.33
Data : 255.255.255.255
Link Type: StubNet
Metric : 1
Priority : Medium
* Link ID: 1.1.1.1
Data : 192.168.100.53
Link Type: P-2-P
Metric : 1
* Link ID: 192.168.100.54
Data : 255.255.255.255
Link Type: StubNet
Metric : 1
Priority : Medium
Type : Router
Ls id : 1.1.1.1
Adv rtr : 1.1.1.1
Ls age : 1699
Len : 96
Options : ASBR E
seq# : 80008d72
chksum : 0x16fc
Link count: 6
* Link ID: 1.1.1.1
Data : 255.255.255.255
Link Type: StubNet
Metric : 1
Priority : Medium
* Link ID: 1.1.1.1
Data : 255.255.255.255
Link Type: StubNet
Metric : 12
Priority : Medium
* Link ID: 1.1.1.3
Data : 192.168.100.26
Link Type: P-2-P
Metric : 10
* Link ID: 192.168.100.25
Data : 255.255.255.255
Link Type: StubNet
Metric : 10
Priority : Medium
* Link ID: 1.1.1.2
Data : 192.168.100.54
Link Type: P-2-P
Metric : 10
* Link ID: 192.168.100.53
Data : 255.255.255.255
Link Type: StubNet
Metric : 10
Priority : Medium'''
import json
result = {}
l = []
for i in data.split("\n\n"):
if i:
p = [parameter for parameter in i.split("*")]
for line, x in enumerate(p[0].split("\n")):
if x and "Ls id" in x:
ls_id, ip = x.split(": ")
ls_id = ls_id.strip()
ip = ip.strip()
for y in p[1:]:
if y and "P-2-P" in y:
temp = {ls_id:ip}
for items in y.split("\n"):
try:
key, value = items.split(": ")
key = key.strip()
value = value.strip()
temp[key] = value
except ValueError:
pass
l.append(temp)
result["ospf"] = l
print (json.dumps(result,indent=2))
with open('data.json', 'w') as json_file:
json.dump(result, json_file)
I'm not sure where i make wrong. Please advise me further. Thank you.

A simple workaround would be to concat the file to one large string. then your code works as expected. This is definetly no clean answer but you could let the rest of your code unchanged.
import json
result = {}
l = []
with open('data.txt') as myf:
a = ''.join(myf)
for i in a.split("\n\n"):
if i:
p = [parameter for parameter in i.split("*")]
for line, x in enumerate(p[0].split("\n")):
if x and "Ls id" in x:
ls_id, ip = x.split(": ")
ls_id = ls_id.strip()
ip = ip.strip()
for y in p[1:]:
if y and "P-2-P" in y:
temp = {ls_id:ip}
for items in y.split("\n"):
try:
key, value = items.split(": ")
key = key.strip()
value = value.strip()
temp[key] = value
except ValueError:
pass
l.append(temp)
result["ospf"] = l
print (json.dumps(result,indent=2))
with open('data.json', 'w') as json_file:
json.dump(result, json_file)

How to compare dictionaries and see what changed?

I am having 3 dictionaries in my python code :
self.new_port_dict = {} # Dictionary to store the new ports
from curr_host
self.old_port_dict = {} # Dictionary to store the old ports from old_host
self.results_ports_dict = {} # Holds the result of changed/newly added ports
The script needs to compare what port changed, I am almost there just unable to present help me out :
def comp_ports(self,filename):
try:
f = open(filename)
self.prev_report = pickle.load(f) # NmapReport
for s in self.prev_report.hosts:
self.old_port_dict[s.address] = set()
for x in s.get_open_ports():
self.old_port_dict[s.address].add(x)
for s in self.report.hosts:
self.new_port_dict[s.address] = set()
for x in s.get_open_ports():
self.new_port_dict[s.address].add(x)
print "The following Host/ports were available in old scan : !!"
print `self.old_port_dict`
print "--------------------------------------------------------"
print "The following Host/ports have been added in new scan: !!"
print `self.new_port_dict`
for h in self.old_port_dict.keys():
self.results_ports_dict[h] = self.new_port_dict[h]- self.old_port_dict[h]
print "Result Change: for",h ,"->",self.results_ports_dict[h]
except Exception as l:
print l
This gives a output as :
The following Host/ports were available in old scan : !!
{'172.16.0.41': set([(80, 'tcp'), (666, 'tcp')]), '172.16.0.163': set([(80, 'tcp'), (22, 'tcp')])}
--------------------------------------------------------
The following Host/ports have been added in new scan: !!
{'172.16.0.41': set([(80, 'tcp'), (22, 'tcp')]), '172.16.0.163': set([(80, 'tcp'), (22, 'tcp')])}
Result Change: for 172.16.0.41 -> set([(22, 'tcp')]) From set([(80, 'tcp'), (666, 'tcp')])
Result Change: for 172.16.0.163 -> set([]) From set([(80, 'tcp'), (22, 'tcp')])
As you can clearly see , I have the resulting changed dictionary as well. I want to just print :
For "host_name" , Port changed from "port_id" to "new_port_id"
ex: For 172.16.0.41, Port changed from (666, 'tcp') to (22, 'tcp')

Based on Alex Martelli's answer on Is there a better way to compare dictionary values
You can do this :
#eval the differences between the 2 dict:
diff_key=[key for key in old_port_dict if old_port_dict[key]!=new_port_dict[key]]
for key in diff_key:
print "For %s, Port changed from %s to %s" %(key,old_port_dict[key],new_port_dict[key])

I believe your are not comparing the dictionaries but actually the values corresponding to the keys.
The basic ideas here are:
A host must not always be present in past and present scans and hance the usage of collections.defaultdict, to ensure a straightforward comparison of values even in the absence of a host. Because a value for the missing key will be automatically generated (an empty set)
Have 3 operations on the set of ports
& (intersection): to see which ports have remained constant across scans (same ports)
old - new: to see which ports were in the old scan but no longer in the new (deleted ports)
new - old: to see which ports are in the new scan but were not in the old (added ports)
There are several ways to move the code around to optimize, but I guess the point is much more clarity.
Hope it helps
import collections
scan0 = collections.defaultdict(set, {
'172.16.0.41': set([(80, 'tcp'), (666, 'tcp')]),
'172.16.0.163': set([(80, 'tcp'), (22, 'tcp')])
})
scan1 = collections.defaultdict(set, {
'172.16.0.41': set([(80, 'tcp'), (22, 'tcp')]),
'172.16.0.163': set([(80, 'tcp'), (22, 'tcp')])
})
hosts = sorted(set(scan0.keys() + scan1.keys()))
scan_same = dict()
scan_new = dict()
scan_del = dict()
for host in hosts:
scan_same[host] = scan0[host] & scan1[host]
scan_new[host] = scan1[host] - scan0[host]
scan_del[host] = scan0[host] - scan1[host]
print()
print('-' * 10, 'Same')
for host, ports in scan_same.items():
print(host, ':')
for port in ports:
print(':::', port[0], '/', port[1])
print()
print('*' * 10, 'Added')
for host, ports in scan_new.items():
print(host, ':')
for port in ports:
print(':::', port[0], '/', port[1])
print()
print('=' * 10, 'Deleted')
for host, ports in scan_del.items():
print(host, ':')
for port in ports:
print(':::', port[0], '/', port[1])
This will output:
---------- Same
172.16.0.163 :
::: 80 / tcp
::: 22 / tcp
172.16.0.41 :
::: 80 / tcp
*********** Added
172.16.0.163 :
172.16.0.41 :
::: 22 / tcp
========== Deleted
172.16.0.163 :
172.16.0.41 :
::: 666 / tcp

You need to keep another set like old_port_dict_changed = self.old_port_dict[h] - self.new_port_dict[h] now the changed ports from the old_port_dict are in old_port_dict_changed and the new ports which replaced the old ones are in results_ports_dict. Does that help?

Converting multi-line script output to dictionary using regex

I got the following script output:
***************************************************
[g4u2680c]: searching for domains
---------------------------------------------------
host = g4u2680c.houston.example.com
ipaddr = [16.208.16.72]
VLAN = [352]
Gateway= [16.208.16.1]
Subnet = [255.255.248.0]
Subnet = [255.255.248.0]
Cluster= [g4u2679c g4u2680c g9u1484c g9u1485c]
host = g4u2680c.houston.example.com
ipaddr = [16.208.16.72]
VLAN = [352]
Gateway= [16.208.16.1]
Subnet = [255.255.248.0]
Subnet = [255.255.248.0]
Cluster= [g4u2679c g4u2680c g9u1484c g9u1485c]
* script completed Mon Jun 15 06:13:14 UTC 2015 **
* sleeping 30 to avoid DOS on dns via a loop **
I need to extract the 2 host list into a dictionary, with out the brackets.
Here is my code:
#!/bin/env python
import re
text="""***************************************************
[g4u2680c]: searching for domains
---------------------------------------------------
host = g4u2680c.houston.example.com
ipaddr = [16.208.16.72]
VLAN = [352]
Gateway= [16.208.16.1]
Subnet = [255.255.248.0]
Subnet = [255.255.248.0]
Cluster= [g4u2679c g4u2680c g9u1484c g9u1485c]
host = g4u2680c.houston.example.com
ipaddr = [16.208.16.72]
VLAN = [352]
Gateway= [16.208.16.1]
Subnet = [255.255.248.0]
Subnet = [255.255.248.0]
Cluster= [g4u2679c g4u2680c g9u1484c g9u1485c]
* script completed Mon Jun 15 06:13:14 UTC 2015 **
* sleeping 30 to avoid DOS on dns via a loop **
***************************************************
"""
seq = re.compile(r"host.+?\n\n",re.DOTALL)
a=seq.findall(text)
matches = re.findall(r'\w.+=.+', a[0])
matches = [m.split('=', 1) for m in matches]
matches = [ [m[0].strip().lower(), m[1].strip().lower()] for m in matches]
#should have function with regular expression to remove bracket here
d = dict(matches)
print d
What I got so far for the first host:
{'subnet': '[255.255.248.0]', 'vlan': '[352]', 'ipaddr': '[16.208.16.72]', 'cluster': '[g4u2679c g4u2680c g9u1484c g9u1485c]', 'host': 'g4u2680c.houston.example.com', 'gateway': '[16.208.16.1]'}
I need help to find the regex to remove the bracket as the value in the dictionary contain data with and without bracket.
Or if there is a better and simpler way to transform the original script output into dictionary.

You can use: (\w+)\s*=\s*\[?([^\n\]]+)\]?
demo
import re
p = re.compile(ur'(\w+)\s*=\s*\[?([^\n\]]+)\]?', re.MULTILINE)
test_str = u"host = g4u2680c.houston.example.com\n ipaddr = [16.208.16.72]\n VLAN = [352]\n Gateway= [16.208.16.1]\n Subnet = [255.255.248.0]\n Subnet = [255.255.248.0]\n Cluster= [g4u2679c g4u2680c g9u1484c g9u1485c]\n\nhost = g4u2680c.houston.example.com\n ipaddr = [16.208.16.72]\n VLAN = [352]\n Gateway= [16.208.16.1]\n Subnet = [255.255.248.0]\n Subnet = [255.255.248.0]\n Cluster= [g4u2679c g4u2680c g9u1484c g9u1485c]\n"
re.findall(p, test_str)

You can simply use re.findall and dict :
>>> dict([(i,j.strip('[]')) for i,j in re.findall(r'(\w+)\s*=\s*(.+)',text)])
{'Subnet': '255.255.248.0', 'VLAN': '352', 'ipaddr': '16.208.16.72', 'Cluster': 'g4u2679c g4u2680c g9u1484c g9u1485c', 'host': 'g4u2680c.houston.example.com', 'Gateway': '16.208.16.1'}
And about the brackets you can remove them by str.strip method.

You can try out this.
matches = [m.replace('[','').replace(']','').split('=', 1) for m in matches]

python return list from linux command output

I am new to python and I'm learning rapidly, but this is beyond my current level of understanding. I'm trying to to pull the output from the linux command apcaccess into a list in python.
apcaccess is a linux command to get the status of an APC UPS. The output is this:
$ apcaccess
APC : 001,035,0933
DATE : 2014-11-12 13:38:27 -0500
HOSTNAME : doormon
VERSION : 3.14.10 (13 September 2011) debian
UPSNAME : UPS
CABLE : USB Cable
DRIVER : USB UPS Driver
UPSMODE : Stand Alone
STARTTIME: 2014-11-12 12:28:00 -0500
MODEL : Back-UPS ES 550G
STATUS : ONLINE
LINEV : 118.0 Volts
LOADPCT : 15.0 Percent Load Capacity
BCHARGE : 100.0 Percent
TIMELEFT : 46.0 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME : 0 Seconds
SENSE : Medium
LOTRANS : 092.0 Volts
HITRANS : 139.0 Volts
ALARMDEL : 30 seconds
BATTV : 13.6 Volts
LASTXFER : No transfers since turnon
NUMXFERS : 2
XONBATT : 2014-11-12 12:33:35 -0500
TONBATT : 0 seconds
CUMONBATT: 53 seconds
XOFFBATT : 2014-11-12 12:33:43 -0500
STATFLAG : 0x07000008 Status Flag
SERIALNO : 4B1335P17084
BATTDATE : 2013-08-28
NOMINV : 120 Volts
NOMBATTV : 12.0 Volts
FIRMWARE : 904.W1 .D USB FW:W1
END APC : 2014-11-12 13:38:53 -0500
I've tried different iterations of Popen such as:
def check_apc_ups():
output = subprocess.Popen("apcaccess", stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
x1, x2, x3, x4, x5 = output
I would like to be able to pull each line into a list or tuple containing all 32 and then only display/print what I need, such as TIMELEFT and BCHARGE.
Any help would be greatly appreciated.

There are already answers how to get the output of the command into python.
It is not clear what you are going to do with the output. Maybe, a dictionary (dict) is better than a list for you:
# stolen from Hackaholic's answer
import subprocess
child = subprocess.Popen('apcaccess',stdout=subprocess.PIPE)
msg,err = child.communicate()
# now create the dict:
myDict={}
#for i in msg.split("\n"): # loop over lines
for i in msg.splitlines(): # EDIT: See comments
splitted=i.split(":") # list like ["HOSTNAME ", " doormon"]
# remove leading & trailing spaces, add to dict
myDict[splitted[0].strip()]=splitted[1].strip()
#Now, you can easily access the items:
print myDict["SERIALNO"]
print myDict["STATUS"]
print myDict["BATTV"]
for k in myDict.keys():
print k +" = "+ myDict[k]

from subprocess import check_output
out = check_output(["apcaccess"])
spl = [ele.split(":",1)for ele in out.splitlines()]
d = {k.rstrip():v.lstrip() for k,v in spl}
print(d['BCHARGE'])
print(d["TIMELEFT"])
100.0 Percent
46.0 Minutes
from subprocess import check_output
def get_apa():
out = check_output(["apcaccess"])
spl = [ele.split(":", 1) for ele in out.splitlines()]
d = {k.rstrip(): v.lstrip() for k, v in spl}
return d
output = get_apa()
print (output['BCHARGE'])
100.0 Percent
To print all key/values pairings:
for k,v in get_apa().items():
print("{} = {}".format(k,v))

what you need is subprocess module
import subprocess
child = subprocess.Popen('apcaccess',stdout=subprocess.PIPE)
msg,err = child.communicate()
print(msg.split())