Related
I use the following bit to make an sql statement using the key:values in the dict
record_number = 627
temp_dict = {
"FOO": 752,
"BAR": "test",
"I": "zzzzz",
"Hate": "tesname",
"SQL": "testsomethingesle",
"SO": "commentlol",
"MCUH": "asadsa",
"FILLING": "zzzzzz",
"NAME": "''",
}
update_query = (
"UPDATE table_name SET {}".format(
", ".join("{} = '?'".format(k) for k in temp_dict)
)
+ " WHERE RECNUM = '"
+ record_number
+ "';"
)
update_values = tuple(temp_dict.values())
cur.execute(update_query, update_values)
the update_query prints out correctly
UPDATE table_name SET FOO = '?', BAR = '?', I = '?', Hate = '?', SQL = '?', SO = '?', MCUH = '?', FILLING = '?', NAME = '?' WHERE RECNUM = '627';
and the update_values also looks right
(752, 'test', 'zzzzz', 'tesname', 'testsomethingesle', 'commentlol', 'asadsa', 'zzzzzz', "''")
but I get back the following error
firebirdsql.OperationalError: conversion error from string "?"
My understanding is that ? is basically a placeholder value and if I put in a tuple or list as the second parameter in the cur.execute() it should replace the ? with the values passed in.
What am I doing wrong?
You're generating a statement that has string literals with a question mark ('?'), not a question mark used as a parameter placeholder (plain ?). This means that when you execute the statement, you're trying to assign the literal value ? to a column, and if that column is not a CHAR, VARCHAR or BLOB, this produces an error, because there is no valid conversion from the string ? to the other data types.
You need to uses "{} = ?" instead (notice the absence of single quotes around the question mark).
I am new to python and am trying to read a file and create a dictionary from it.
The format is as follows:
.1.3.6.1.4.1.14823.1.1.27 {
TYPE = Switch
VENDOR = Aruba
MODEL = ArubaS3500-48T
CERTIFICATION = CERTIFIED
CONT = Aruba-Switch
HEALTH = ARUBA-Controller
VLAN = Dot1q INSTRUMENTATION:
Card-Fault = ArubaController:DeviceID
CPU/Memory = ArubaController:DeviceID
Environment = ArubaSysExt:DeviceID
Interface-Fault = MIB2
Interface-Performance = MIB2
Port-Fault = MIB2
Port-Performance = MIB2
}
The first line OID (.1.3.6.1.4.1.14823.1.1.27 { ) I want this to be the key and the remaining lines are the values until the }
I have tried a few combinations but am not able to get the correct regex to match these
Any help please?
I have tried something like
lines = cache.readlines()
for line in lines:
searchObj = re.search(r'(^.\d.*{)(.*)$', line)
if searchObj:
(oid, cert ) = searchObj.groups()
results[searchObj(oid)] = ", ".join(line[1:])
print("searchObj.group() : ", searchObj.group(1))
print("searchObj.group(1) : ", searchObj.group(2))
You can try this:
import re
data = open('filename.txt').read()
the_key = re.findall("^\n*[\.\d]+", data)
values = [re.split("\s+\=\s+", i) for i in re.findall("[a-zA-Z0-9]+\s*\=\s*[a-zA-Z0-9]+", data)]
final_data = {the_key[0]:dict(values)}
Output:
{'\n.1.3.6.1.4.1.14823.1.1.27': {'VENDOR': 'Aruba', 'CERTIFICATION': 'CERTIFIED', 'Fault': 'MIB2', 'VLAN': 'Dot1q', 'Environment': 'ArubaSysExt', 'HEALTH': 'ARUBA', 'Memory': 'ArubaController', 'Performance': 'MIB2', 'CONT': 'Aruba', 'MODEL': 'ArubaS3500', 'TYPE': 'Switch'}}
You could use a nested dict comprehension along with an outer and inner regex.
Your blocks can be separated by
.numbers...numbers.. {
// values here
}
In terms of regular expression this can be formulated as
^\s* # start of line + whitespaces, eventually
(?P<key>\.[\d.]+)\s* # the key
{(?P<values>[^{}]+)} # everything between { and }
As you see, we split the parts into key/value pairs.
Your "inner" structure can be formulated like
(?P<key>\b[A-Z][-/\w]+\b) # the "inner" key
\s*=\s* # whitespaces, =, whitespaces
(?P<value>.+) # the value
Now let's build the "outer" and "inner" expressions together:
rx_outer = re.compile(r'^\s*(?P<key>\.[\d.]+)\s*{(?P<values>[^{}]+)}', re.MULTILINE)
rx_inner = re.compile(r'(?P<key>\b[A-Z][-/\w]+\b)\s*=\s*(?P<value>.+)')
result = {item.group('key'):
{match.group('key'): match.group('value')
for match in rx_inner.finditer(item.group('values'))}
for item in rx_outer.finditer(string)}
print(result)
A demo can be found on ideone.com.
I am working with Python 2.7 to extract data from a JSON API and push it into a SQL-Server table.
I am having trouble with inserting data into the database where some of the entries returned from the JSON response are missing a section of the dictionary. IE, "CustomFields": 90% of the entries have information, however 10% don't therefore I get an index error
eg
"CustomFields":[
],
vs
"CustomFields":[
{
"Type":"foo",
"Name":"foo",
"Value":"foo"
},
{
"Type":"foo",
"Name":"foo",
"Value":"foo"
},
{
"Type":"foo",
"Name":"foo",
"Value":"foo"
},
What would I change so that if I get a missing index, replace those with 'NULL' entries into the database.
response = '*API URL*'
json_response = json.loads(urllib2.urlopen(response).read())
conn = pypyodbc.connect(r'Driver={SQL Server};Server=*Address*;Database=*DataBase*;Trusted_Connection=yes;')
conn.autocommit = False
c = conn.cursor()
c.executemany("INSERT INTO phil_targetproccess (ResourceType, Id, Name, StartDate, EndDate, TimeSpent, CreateDate, ModifyDate, LastStateChangeDate, ProjectName, EntityStateName, RequestTypeName, AssignedTeamMember#1, Area, SubArea, BusinessTeam) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" ,
[(x['ResourceType'],
x['Id'],
x['Name'],
(parse_date(x['StartDate'])),
(parse_date(x['EndDate'])),
x['TimeSpent'],
(parse_date(x['CreateDate'])),
(parse_date(x['ModifyDate'])),
(parse_date(x['LastStateChangeDate'])),
x['Project']['Name'],
x['EntityState']['Name'],
x['RequestType']['Name'],
y['GeneralUser']['FirstName']+' '+y['GeneralUser']['LastName'],
x['CustomFields'][0]['Value'],
x['CustomFields'][1]['Value'],
x['CustomFields'][2]['Value'])
for x in json_response['Items']
for y in x['Assignments']['Items']])
Many thanks.
I think your issue is here
x['CustomFields'][0]['Value'],
x['CustomFields'][1]['Value'],
x['CustomFields'][2]['Value']
When CustomFields has no elements
Try
x['CustomFields'][0]['Value'] if len(x['CustomFields']) > 0 else '',
x['CustomFields'][1]['Value'] if len(x['CustomFields']) > 1 else '',
x['CustomFields'][2]['Value'] if len(x['CustomFields']) > 2 else '',
You can use get method to check whether that value in CustomFields
is available if so check the length of the list and then get the value of the dictionary in that list using the same get method.
For example:
customfield_value = (x['CustomFields'][0]).get("Value",None) if len(x['CustomFields'])>0 else None
This will return None if the value is not present in the index 0. you can follow the same for getting values from other 2 indices. If you didn't understand please comment it 'll explain further.
Final Script. Thanks for the help!
c.executemany("INSERT INTO phil_targetproccess (ResourceType, Id, Name, StartDate, EndDate, TimeSpent, CreateDate, "
"ModifyDate, LastStateChangeDate, ProjectName, EntityStateName, RequestTypeName, AssignedTeamMember1, "
"AssignedTeamMember2, AssignedTeamMember3, AssignedTeamMember4, Area, SubArea, BusinessTeam) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
[(x['ResourceType'],
x['Id'],
x['Name'],
(parse_date(x['StartDate'])),
(parse_date(x['EndDate'])),
x['TimeSpent'],
(parse_date(x['CreateDate'])),
(parse_date(x['ModifyDate'])),
(parse_date(x['LastStateChangeDate'])),
x['Project']['Name'],
x['EntityState']['Name'],
x['RequestType']['Name'],
x['Assignments']['Items'][0]['GeneralUser']['FirstName'] + ' ' + x['Assignments']['Items'][0]['GeneralUser']['LastName'] if len(x['Assignments']['Items']) > 0 else None,
x['Assignments']['Items'][1]['GeneralUser']['FirstName'] + ' ' + x['Assignments']['Items'][1]['GeneralUser']['LastName'] if len(x['Assignments']['Items']) > 1 else None,
x['Assignments']['Items'][2]['GeneralUser']['FirstName'] + ' ' + x['Assignments']['Items'][2]['GeneralUser']['LastName'] if len(x['Assignments']['Items']) > 2 else None,
x['Assignments']['Items'][3]['GeneralUser']['FirstName'] + ' ' + x['Assignments']['Items'][3]['GeneralUser']['LastName'] if len(x['Assignments']['Items']) > 3 else None,
x['CustomFields'][0]['Value'] if len(x['CustomFields']) > 0 else '',
x['CustomFields'][1]['Value'] if len(x['CustomFields']) > 1 else '',
x['CustomFields'][2]['Value'] if len(x['CustomFields']) > 2 else '')
for x in json_response['Items']])
Need this:
POST&https%3A%2F%2Fsecure.trademe.co.nz%2FOauth%2FRequestToken&oauth_callback%3Dhttp%253A%252F%252Fwww.website-tm-access.co.nz%252Ftrademe-callback%26oauth_consumer_key%3DC74CD73FDBE37D29BDD21BAB54BC70E422%26oauth_nonce%3D7O3kEe%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1285532322%26oauth_version%3D1.0%26scope%3DMyTradeMeRead%252CMyTradeMeWrite
Myattempt:
New_base_string ="POST&https%3A%2F%2Fsecure.trademe.co.nz%2FOauth%2FRequestToken&oauth_callback%3Dhttp%253A%252F%252Fwww.website-tm-access.co.nz%252Ftrademe-callback%26oauth_consumer_key%" + str(consumer_key) +"3DC74CD73FDBE37D29BDD21BAB54BC70E422%26oauth_nonce%3" + str(nonce) + "%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3" + str(time) + "%26oauth_version%3D1.0%26scope%3DMyTradeMeRead%252CMyTradeMeWrite"
I just tried to append it to the end, will this work or will i need to append to a list and then encode?
so like this:
headers = { my_variable + other_variable }
authorization = '5C82CC6BC7C6472154FBC9CAB24A29A2 ' + ', '.join([key + '="' + urllib.parse.quote_plus(str(value)) + '"' for key, value in headers.items()])
General
If you want to URL encode parameters to your POST request the best way is:
import urllib
f = { 'eventName' : 'myEvent',
'eventDescription' : 'cool event',
'url' : 'http://www.google.com'}
print 'POST&%s' % urllib.urlencode(f)
Output:
POST&eventName=myEvent&url=http%3A%2F%2Fwww.google.com&eventDescription=cool+event
with Dictionary its not ordered if you want to order it just use a list
import urllib
f = [ ('eventName', 'myEvent'),
('eventDescription', 'cool event'),
('url', 'http://www.google.com')]
print 'POST&%s' % urllib.urlencode(f)
Output
POST&eventName=myEvent&eventDescription=cool+event&url=http%3A%2F%2Fwww.google.com
How to get your need this string (Python 3.5)
While the general example is tested in python 2.7, I wrote your example with python 3.5 code.
import urllib.parse
method = "POST"
url = "https://secure.trademe.co.nz/Oauth/RequestToken"
params = [('oauth_callback', 'http://www.website-tm-access.co.nz/trademe-callback'),
('oauth_consumer_key', 'C74CD73FDBE37D29BDD21BAB54BC70E422'),
('oauth_nonce', '7O3kEe'),
('oauth_signature_method', 'HMAC-SHA1'),
('oauth_timestamp', 1285532322),
('oauth_version', 1.0),
('scope', "MyTradeMeRead,MyTradeMeWrite")]
print('POST&%(url)s&%(params)s' % { 'url' : urllib.parse.quote_plus(url), 'params' : urllib.parse.quote_plus(urllib.parse.urlencode(params)) })
Output
POST&https%3A%2F%2Fsecure.trademe.co.nz%2FOauth%2FRequestToken&oauth_callback%3Dhttp%253A%252F%252Fwww.website-tm-access.co.nz%252Ftrademe-callback%26oauth_consumer_key%3DC74CD73FDBE37D29BDD21BAB54BC70E422%26oauth_nonce%3D7O3kEe%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1285532322%26oauth_version%3D1.0%26scope%3DMyTradeMeRead%252CMyTradeMeWrite
I am using suds package to query a API from a website, the data returned from their website looks like this,:
(1). Can anyone tell me what kind of format is this?
(2). If so, what will be the easiest way to parse the data looks like this? I have dealt quite a lot with HTML/XML format using BeautifulSoup but before I lift my finger to write regular expressions for this type of format. I am curious is this some type of 'popular format' and there are actually some beautiful parser already written. Thanks.
# Below are the header and tail of the response..
(DetailResult)
{
status = (Status){ message = None code = "0" }
searchArgument = (DetailSearchArgument){ reqPartNumber = "BQ" reqMfg = "T" reqCpn = None }
detailsDto[] = (DetailsDto){
summaryDto = (SummaryDto){ PartNumber = "BQ" seMfg = "T" description = "Fast" }
packageDto[] =
(PackageDto){ fetName = "a" fetValue = "b" },
(PackageDto){ fetName = "c" fetValue = "d" },
(PackageDto){ fetName = "d" fetValue = "z" },
(PackageDto){ fetName = "f" fetValue = "Sq" },
(PackageDto){ fetName = "g" fetValue = "p" },
additionalDetailsDto = (AdditionalDetailsDto){ cr = None pOptions = None inv = None pcns = None }
partImageDto = None
riskDto = (RiskDto){ life= "Low" lStage = "Mature" yteol = "10" Date = "2023"}
partOptionsDto[] = (ReplacementDto){ partNumber = "BQ2" manufacturer = "T" type = "Reel" },
inventoryDto[] =
(InventoryDto){ distributor = "V" quantity = "88" buyNowLink = "https://www..." },
(InventoryDto){ distributor = "R" quantity = "7" buyNowLink = "http://www.r." },
(InventoryDto){ distributor = "RS" quantity = "2" buyNowLink = "http://www.rs.." },
},
}
This looks like some kind of nested repr output, similar to JSON but with
structure or object name information ("a Status contains a message and a code").
If it's nested, regexes alone won't do the job. Here is a rough pass at a pyparsing
parser
sample = """
... given sample text ...
"""
from pyparsing import *
# punctuation
LPAR,RPAR,LBRACE,RBRACE,LBRACK,RBRACK,COMMA,EQ = map(Suppress,"(){}[],=")
identifier = Word(alphas,alphanums+"_")
# define some types that can get converted to Python types
# (parse actions will do conversion at parse time)
NONE = Keyword("None").setParseAction(replaceWith(None))
integer = Word(nums).setParseAction(lambda t:int(t[0]))
quotedString.setParseAction(removeQuotes)
# define a placeholder for a nested object definition (since objDefn
# will be referenced within its own definition)
objDefn = Forward()
objType = Combine(LPAR + identifier + RPAR)
objval = quotedString | NONE | integer | Group(objDefn)
objattr = Group(identifier + EQ + objval)
arrayattr = Group(identifier + LBRACK + RBRACK + EQ + Group(OneOrMore(Group(objDefn)+COMMA)) )
# use '<<' operator to assign content to previously declared Forward
objDefn << objType + LBRACE + ZeroOrMore((arrayattr | objattr) + Optional(COMMA)) + RBRACE
# parse sample text
result = objDefn.parseString(sample)
# use pprint to list out indented parsed data
import pprint
pprint.pprint(result.asList())
Prints:
['DetailResult',
['status', ['Status', ['message', None], ['code', '0']]],
['searchArgument',
['DetailSearchArgument',
['reqPartNumber', 'BQ'],
['reqMfg', 'T'],
['reqCpn', None]]],
['detailsDto',
[['DetailsDto',
['summaryDto',
['SummaryDto',
['PartNumber', 'BQ'],
['seMfg', 'T'],
['description', 'Fast']]],
['packageDto',
[['PackageDto', ['fetName', 'a'], ['fetValue', 'b']],
['PackageDto', ['fetName', 'c'], ['fetValue', 'd']],
['PackageDto', ['fetName', 'd'], ['fetValue', 'z']],
['PackageDto', ['fetName', 'f'], ['fetValue', 'Sq']],
['PackageDto', ['fetName', 'g'], ['fetValue', 'p']]]],
['additionalDetailsDto',
['AdditionalDetailsDto',
['cr', None],
['pOptions', None],
['inv', None],
['pcns', None]]],
['partImageDto', None],
['riskDto',
['RiskDto',
['life', 'Low'],
['lStage', 'Mature'],
['yteol', '10'],
['Date', '2023']]],
['partOptionsDto',
[['ReplacementDto',
['partNumber', 'BQ2'],
['manufacturer', 'T'],
['type', 'Reel']]]],
['inventoryDto',
[['InventoryDto',
['distributor', 'V'],
['quantity', '88'],
['buyNowLink', 'https://www...']],
['InventoryDto',
['distributor', 'R'],
['quantity', '7'],
['buyNowLink', 'http://www.r.']],
['InventoryDto',
['distributor', 'RS'],
['quantity', '2'],
['buyNowLink', 'http://www.rs..']]]]]]]]