Develop Reference

Parsing XML attrib with Python, ' and " character spliting

I'm working with the NVD XML and attempting to parse and split the XML to eventually enter into a DB. The issue I'm running into is the parsed XML attrib is either with " or ' around the values. I'm not able to split the strings of these. I've included the code and the entry that it currently fails on. The expected output is
product,america's_first_federal_credit_union,america's_first_fcu_mobile_banking
Code
#!/usr/bin/env python
import os
import sys
import time
from subprocess import call
import xml.etree.ElementTree
import re
range_from = 2017
range_to = 2017
def process_entry(entry):
cve = entry.attrib.get("name")
print cve
cpes = get_cpes_affected(entry)
def get_cpes_affected(entry):
child = []
for e in entry.iter():
if "}prod" in e.tag:
print e.attrib
print unichr(34)
if unichr(34) in e.attrib:
print "hey yo"
child.append("product," + str(e.attrib).split('"')[1] + "," + str(e.attrib).split('"')[3])
else:
child.append("product," + str(e.attrib).split("'")[3] + "," + str(e.attrib).split("'")[7])
#print e.tag, e.attrib
if "'prev'" in e.attrib:
child.append("version," + str(e.attrib).split("'")[7] + "," + str(e.attrib).split("'")[3])
if "}vers" in e.tag and "'prev'" not in e.attrib:
child.append("version," + str(e.attrib).split("'")[3] + ",")
#print e.tag, e.attrib
for derp in child:
print derp
for i in range(range_from, range_to+1):
os.system("wget -O tmp.zip https://nvd.nist.gov/download/nvdcve-%i.xml.zip" % i)
os.system("unzip -o tmp.zip")
e = xml.etree.ElementTree.parse('nvdcve-%i.xml' % i).getroot()
for entry in e:
process_entry(entry)
Example of an XML entry that is being parsed
<entry type="CVE" name="CVE-2017-5916" seq="2017-5916" published="2017-05-05" modified="2017-05-16" severity="Medium" CVSS_version="2.0" CVSS_score="4.3" CVSS_base_score="4.3" CVSS_impact_subscore="2.9" CVSS_exploit_subscore="8.6" CVSS_vector="(AV:N/AC:M/Au:N/C:P/I:N/A:N)">
<desc>
<descript source="cve">The America's First Federal Credit Union (FCU) Mobile Banking app 3.1.0 for iOS does not verify X.509 certificates from SSL servers, which allows man-in-the-middle attackers to spoof servers and obtain sensitive information via a crafted certificate.</descript>
</desc>
<loss_types>
<conf/>
</loss_types>
<range>
<network/>
</range>
<refs>
<ref source="MISC" url="https://medium.com/#chronic_9612/follow-up-76-popular-apps-confirmed-vulnerable-to-silent-interception-of-tls-protected-data-64185035029f" adv="1">https://medium.com/#chronic_9612/follow-up-76-popular-apps-confirmed-vulnerable-to-silent-interception-of-tls-protected-data-64185035029f</ref>
</refs>
<vuln_soft>
<prod name="america's_first_fcu_mobile_banking" vendor="america's_first_federal_credit_union">
<vers num="3.1.0" prev="1" edition=":~~~iphone_os~~"/>
</prod>
</vuln_soft>
Entry it fails on
{'vendor': "america's_first_federal_credit_union", 'name': "america's_first_fcu_mobile_banking"}
And just to include an example of a string it's able to split without issue
{'vendor': 'emirates_nbd_bank_p.j.s.c', 'name': 'emirates_nbd_ksa'}
Sorry forgot to include the error
Traceback (most recent call last):
File "prev-version-load.py", line 49, in <module>
process_entry(entry)
File "prev-version-load.py", line 18, in process_entry
cpes = get_cpes_affected(entry)
File "prev-version-load.py", line 33, in get_cpes_affected
child.append("product," + str(e.attrib).split("'")[3] + "," + str(e.attrib).split("'")[7])
IndexError: list index out of range

This has nothing to do with parsing xml, but with how you format the output.
Unlike in shell scripting, where most things are just string and you can just do string fiddling to get the output you want, python is an object-oriented languages, objects in Python has types. In particular e.attrib is a dictionary type and you can't do string operations on a dictionary.
I'd suggest using ElementTree's findall() method instead of doing what I think you were trying to do. For example, I think this is what you are really trying to do:
#!/usr/bin/env python
from xml.etree import ElementTree as ET
range_from = 2017
range_to = 2017
def process_entry(entry):
cve = entry.attrib.get("name")
print cve
cpes = get_cpes_affected(entry)
def get_cpes_affected(entry):
prods = entry.findall('nvd:vuln_soft/nvd:prod', namespaces=namespaces)
for prod in prods:
print prod.attrib
print '"'
for prod in prods:
print "product,{},{}".format(prod.attrib['vendor'], prod.attrib['name'])
for vers in prod.findall('nvd:vers', namespaces=namespaces):
if vers.get('edition'):
print "version,{},".format(vers.attrib['edition'])
elif vers.get('prev') == '1':
print "version,{},".format(vers.attrib['prev'])
else:
print "version,{},".format(vers.attrib['num'])
namespaces = {'nvd': 'http://nvd.nist.gov/feeds/cve/1.2'}
# OPTIONAL: registering namespace is useful for outputting XML with ET.tostring()/ET.dump()
#for prefix, ns in namespaces.items():
# ET.register_namespace(prefix, ns)
for i in range(range_from, range_to+1):
e = ET.parse('nvdcve-%i.xml' % i).getroot()
for entry in e:
process_entry(entry)

Consider replacing...
if "}prod" in e.tag:
print unichr(34)
if unichr(34) in e.attrib:
print "hey yo"
child.append("product," + str(e.attrib).split('"')[1] + "," + str(e.attrib).split('"')[3])
else:
child.append("product," + str(e.attrib).split("'")[3] + "," + str(e.attrib).split("'")[7])
#print e.tag, e.attrib
if "'prev'" in e.attrib:
child.append("version," + str(e.attrib).split("'")[7] + "," + str(e.attrib).split("'")[3])
if "}vers" in e.tag and "'prev'" not in e.attrib:
child.append("version," + str(e.attrib).split("'")[3] + ",")
With...
reg=r"\"|'(?=[^\"]*')|'(?=\W*\")"
if "prod" in e.tag:
#print(re.split(reg,str(e.attrib)))
child.append("product," + re.split(reg,str(e.attrib))[3] + "," + re.split(reg,str(e.attrib))[7])
#print e.tag, e.attrib
if "prev" in e.attrib:
child.append("version," + re.split(reg,str(e.attrib))[7] + "," + re.split(reg,str(e.attrib))[3])
if "vers" in e.tag and "prev" not in e.attrib:
child.append("version," + re.split(reg,str(e.attrib))[3] + ",")
Let me know if this works, I will explain.
UPDATE
Even better solution is below:-
if "prod" in e.tag:
#print(e.attrib)
child.append("product," + e.attrib['name'] + "," + e.attrib['vendor'])
if "prev" in e.attrib:
child.append("version," + e.attrib['prev'] + "," + e.attrib['num'])
if "vers" in e.tag and "prev" not in e.attrib:
child.append("version," + e.attrib['num'] + ",")
A working example with your given xml is here for all three cases yours, my original solution and updated solution.

Issue with concatenating and replace in Pandas dataframe

I am stymied with something, and if I fix it one way, it breaks something else.
I have a set of data that lists file status by country. What I want to do is, for each country in the Country column, print all missing files by each status in the VisitStatus column. So for all rows where country=France, then for every visit that is "Complete", list the number of missing files.
There are two dataframes that I am concatenating into one combined set to work with and deliver final output. I am concat'ing df_s and df_ins into df_combined.
When I grab a set of unique values for the Country and VisitStatus columns to loop over, then try to write out the results per country to an Excel file workbook, quirks in the data kick out a 'duplicate sheetname' error. In one of the source dataframes, there is a status of "Do Not Review" in the VisitStatus column, but in the other source dataframe, it's named "Do not review", lowercase for the second two words. When they're concatenated, this kicks out unique values of "Do Not Review" and "Do not review". Then when the xslx writer tries to make the workbooks for the second one, it checks it against the existing workbooks DISREGARDING CASE, finds the first one, decides they are the same since it is ignoring case, and kicks out the error saying that the 'Do not review' worksheet already exists.
If I run replace() and change all the "Do not review" values in the VisitStatus column into "Do Not Review" so they all match and don't give two results for that when I call unique(), it breaks and gives me a KeyError on VisitStatus.
So far I have read thread after thread about this and haven't been able to solve this. I just tried running the replace() on the source dataframe, and then it throws an error saying that "status" is a float and can't be handled like a string.
I'm at a loss. Thanks in advance!
# COMBO
# Merge the screening and in study datasets
df_combined = pd.concat([df_s,df_ins], axis=0, ignore_index=True)
df_combined = df_combined.query('VisitStatus != "Hand Off Information"')
print(df_combined.columns.values)
print("---------------------------------------------------------------------------------")
# Display and save out country and missing file status
statuses = df_combined['VisitStatus'].unique()
countries = df_combined['Country'].unique()
for status in statuses:
print("X" + status + "X")
print('\n')
print (statuses)
for country in countries:
for status in statuses:
print('\n')
print("---> Missing Files for " + country + " all visits with status of: " + str(status))
df_cmb = df_combined[(df_combined.Country==country) & (df_combined.VisitStatus==status)]
print('\n')
numRows=df_cmb.shape[0]
if numRows > 0:
print("----> Number of visits in " + str(status) + " subset: " + str(numRows))
print("DRF Forms Missing: " + str(df_cmb['DRF-Form-Uploaded'].sum()) + " vs. " + str(numRows - df_cmb['DRF-Form-Uploaded'].sum()) + " collected")
print("CSSRS Forms Missing: " + str(df_cmb['CSSRS-Form-Uploaded'].sum()) + " vs. " + str(numRows - df_cmb['CSSRS-Form-Uploaded'].sum()) + " collected")
print("CDR Forms Missing: " + str(df_cmb['CDR-Form-Uploaded'].sum()) + " vs. " + str(numRows - df_cmb['CDR-Form-Uploaded'].sum()) + " collected")
print("CDR Audio Missing: " + str(df_cmb['CDR-Audio-Uploaded'].sum()) + " vs. " + str(numRows - df_cmb['CDR-Audio-Uploaded'].sum()) + " collected")
print("MMSE Forms Missing: " + str(df_cmb['MMSE-Form-Uploaded'].sum()) + " vs. " + str(numRows - df_cmb['MMSE-Form-Uploaded'].sum()) + " collected")
print("MMSE Audio Missing: " + str(df_cmb['MMSE-Audio-Uploaded'].sum()) + " vs. " + str(numRows - df_cmb['MMSE-Audio-Uploaded'].sum()) + " collected")
print("RBANS Forms Missing: " + str(df_cmb['RBANS-Form-Uploaded'].sum()) + " vs. " + str(numRows - df_cmb['RBANS-Form-Uploaded'].sum()) + " collected")
print("RBANS Audio Missing: " + str(df_cmb['RBANS-Audio-Uploaded'].sum()) + " vs. " + str(numRows - df_cmb['RBANS-Audio-Uploaded'].sum()) + " collected")
print("--------------------------------------")
print('\n')
else:
print("No " + status + " files/visits for " + country)
if country =="United States":
country="USA"
# something is borked in the next line - somehow there are two "Do Not Review" status types in the combined file, triggers an "already in use" for sheetname
df_cmb.to_excel(combo_writer, header=True, index=False, sheet_name=str(country)[:3] + "-by-" + str(status))

Oh lord. I'm answering my own question.
So I tinkered some more and nothing else made sense, so I started wondering if I was putting in the arguments for replace() correctly, and I had them backwards. I assumed the "Do not review" needed to be changed to "Do Not Review", but it was the other way around...I assumed incorrectly as to which source file data needed to be modified. Once I flipped them, it works.

Python code to send command through command line

final="cacls " + "E:/" + "\"" + list1[2] + " " + list1[3] + "\"" + " /p " + str
os.system(final)
I am trying to set permission to a folder Using Python but while running this command , User input needs to be provided too i.e
it asks ARE YOU SURE(Y/N) and the user needs to enter "Y" or "N"
Is there any way to use python to send the user input "Y" along with the above code?
pro = subprocess.Popen(final,shell=True, stdin=subprocess.PIPE)
pro.communicate(bytes("Y\r\n",'utf-8'))
I have added the following code . The program exits without setting the permission.
http://jimmyg.org/blog/2009/working-with-python-subprocess.html#writing-to-standard-input

Try using the subprocess module
import subprocess
cmd = ["cacls", "E:/" + list1[2], list1[3], "/p", str]
pro = subprocess.Popen(final, stdin=subprocess.PIPE)
pro.communicate("y\r\n")

As a smart programmer, use PBS
Then, the code is:
from pbs import type as echo# Isn't it echo for Windows? If not, use the correct one
script = Command("/path/to/cacls ")
print script(echo("Y"), ("E:/" + "\"" + list1[2] + " " + list1[3] + "\"" + " /p " + str).split())

Usage of subprocess class in Python

final="cacls " + "E:/" + "\"" + list1[2] + " " + list1[3] + "\"" + " /p " + str
pro = subprocess.Popen(final,shell=True, stdin=subprocess.PIPE)
pro.communicate(bytes("Y\r\n",'utf-8'))
Trying to set permission to a folder Using Python but while running this command , User input needs to be provided too i.e
it asks ARE YOU SURE(Y/N) and the user needs to enter "Y" or "N"
The above code is not setting the permission.
This was the question i had asked before:
Python code to send command through command line

As a smart programmer, use PBS
Then, the code is:
from pbs import type as echo# Isn't it echo for Windows? If not, use the correct one
script = Command("/path/to/cacls ")
print script(echo("Y"), ("E:/" + "\"" + list1[2] + " " + list1[3] + "\"" + " /p " + str).split())

IRC Bot - flood protection (python)

if data.find('PRIVMSG') != -1:
nick = data.split('!')[ 0 ].replace(':','')
text = ''
if data.count(text) >= 200:
sck.send('KICK ' + " " + chan + " :" 'flooding' + '\r\n')
I'm trying to code a flood protection for the bot, I want it to kick a user if he enters more then 200 characters, how can I make it so it can read the other lines instead of just the first line? and the code above doesn't work, it doesnt kick the user but if I change the sck.send() to sck.send('PRIVMSG ' + chan + " :" 'flooding' + '\r\n') it works.
fixed the kicking problem, and the code works now, but it only reads the first line, not sure how to make it read the other lines if the user keeps flooding the channel.
if data.find('PRIVMSG') != -1:
nick = data.split('!')[ 0 ].replace(':','')
text = ''
if data.count(text) >= 200:
sck.send('KICK ' + " " + chan + " " + nick + " :" 'flooding' + '\r\n')

As far as I remember, the colon is a reserved character in the IRC protocol. That is, the first colon in a server message denotes the start of user-supplied data (that's also why ":" is not allowed in nicks/channel names). Hence, it suffices to search for the first colon and calculate the length of the remaining string.
Furthermore, data.find('PRIVMSG') is pretty unreliable. What if a user types the word "PRIVMSG" in regular channel conversation? Go look up the IRC RFC, it specifies the format of PRIVMSGs in detail.
Besides, you should be a little more specific. What exactly is the problem you're facing? Extracting the nick? Calculating the message length? Connecting to IRC?