I am trying to create text strings by populating from a json object string.
When I iterate through the list of dictionaries, the iterator doubles the string. How do I fix this?
Code so far:
import json
data = '''{
"text": "aaa",
"text2": "bbb",
"data": [
{
"id": "1",
"text": "Red"
}, {
"id": "2",
"text": "Blue"
}
]
}'''
data_decoded = json.loads(data)
data_list = data_decoded['data']
insertQuery = "update "+ data_decoded['text'] +" set "
#print(insertQuery)
for pair in data_list:
for k, v in pair.items():
if k == data_decoded['text2']:
where = ' \"' + k + '\" = \'' + v + '\''
else:
insertQuery = insertQuery + ' where \"' +k+'\" = \''+ v + '\''
query = insertQuery + where
print(query)
Output:
update aaa set where "id" = '1' where "text" = 'Red' "id" = '2'
update aaa set where "id" = '1' where "text" = 'Red' where "id" = '2' where "text" = 'Blue' "id" = '2'
My desired result is for every key value pair the code prints one sentence, like so:
update aaa set where "id" = '1' where "text" = 'Red'
update aaa set where "id" = '2' where "text" = 'Blue'
Not entirely sure, but you can just access your dictionary items rather than looping over them :)
If you use >= python 3.6
query = ''
field = data_decoded['text']
for pair in data_list:
query += f"update {field} set where id = {pair['id']} where text = {pair ['text']}\n"
Otherwise:
query = ''
field = data_decoded['text']
for pair in data_list:
query += "update {field} set where id = {id} where text = {text}\n".format(field=field, id=pair['id'], text=pair['text'])
Related
I have to read a csv file and create a JSON list out of it. Currently , I first read each row, add to list and use JSON dumps to create the JSON output. There are 2 issues I am facing
JSON dumps adds single quotes to the attributes which is not what I want . I wanted each key-value pair to be enclosed in its own double quotes.
It is using the CSV file headers for the keys which are not in camel case but I need them in camel case
This is what my program produces
{
"Reqlist":[
{
'FieldName1' : 'val1'
},
{
'Fieldname2' : 'val2'
}
],
'metaData' : 'metaVal'
}
This is the output I expect
{
"Reqlist":[
{
"fieldName1" : "val1"
},
{
"fieldName2" : "val2"
}
],
"metaData" : "metaVal"
}
Sample code :
reader = csv.DictReader(open(file_data), restkey='INVALID', restval='INVALID')
headers = reader.fieldnames
error_count = 0
success_count =0
dict=[]
header_count = set(headers)
json_error_data = json.dumps({})
csv_list =[]
error_list={}
myDict =[]
print(headers)
if(len(header_count)!=constants.EXPECTED_HEADER_COUNT or (set(headers)!=set(constants.FIELD_NAMES))):
print('error for record')
else:
for row in reader:
if('INVALID' in row.values()):
error_count +=1
else:
success_count +=1
csv_list.append(row)
except Exception as e:
logging.error('error')
if(error_count>0 and success_count == 0 ):
print('save the errors')
elif(success_count>0):
jsonlist = json.dumps({'Reqlist': csv_list })
new = json.loads(jsonlist)
a_dict = {'metaData': 'metaVal'}
new.update(a_dict)
def convert_to_camel(dict1):
new_dict = {}
for key, value in dict1.items():
key = key[0].lower() + key[1:]
value = value if type(value) is not dict else convert_to_camel(value)
new_dict[key] = value
return new_dict
csv_list = [convert_to_camel(i) for i in csv_list]
This should work for camel case
For single quote thing, i have written a json-like library for python. It's not perfect, but here it is.
i've got a dot delimited string which I need to convert to Json. This is an example with different types of strings:
my.dictionary.value -> value
my.dictionary.list[0].value -> value
my.dictionary.list[1].value.list[0].value -> value
I have no problems converting the first type of string using a recursive approach:
def put(d, keys, item):
if "." in keys:
key, rest = keys.split(".", 1)
if key not in d:
d[key] = {}
put(d[key], rest, item)
else:
d[keys] = item
But i'm struggling to find a solution for the lists. Is there a library that provides out of the box string to json conversion? Thank you for your time.
AFAIK, there isn't any modules that would do this
Here is a sample code to converted a series of dotted strings into json format. You just have create a new list when you see the pattern [n] in the string that would be used as a key.
import re
import json
def parse_dotted_strlines(strlines):
res= {}
for line in strlines.splitlines():
parse_dotted_str(line, res)
return res
def parse_dotted_str(s, res):
if '.' in s:
key, rest = s.split('.', 1)
# Check if key represents a list
match = re.search(r'(.*)\[(\d)\]$', key)
if match:
# List
key, index = match.groups()
val = res.get(key, {}) or []
assert type(val) == list, f'Cannot set key {key} as of type list as i
t was earlier marked as {type(val)}'
while len(val) <= int(index):
val.append({})
val[index] = parse_dotted_str(rest, {})
res[key] = val
else:
# Dict
res[key] = parse_dotted_str(rest, res.get(key, {}))
elif '->' in s:
key, val = s.split('->')
res[key.strip()] = val.strip()
return res
Sample input and output
lines = """
my.dictionary.value -> value
my.dictionary.list[0].value -> value
my.dictionary.list[1].value.list[0].value -> value
"""
res = parse_dotted_strlines(lines)
print (json.dumps(res, indent=4))
{
"my": {
"dictionary": {
"value": "value",
"list": [
{
"value": "value"
},
{
"value": {
"list": [
{
"value": "value"
}
]
}
}
]
}
}
}
the json package is what you need
import json
mydict = """{
"str1": "str",
"list1": ["list1_str1", "list1_str2"],
"list2": ["list2_str1", "list2_str2", ["list2_str11", "list_str12"]]
}"""
json.loads(mydict)
>> {'str1': 'str',
'list1': ['list1_str1', 'list1_str2'],
'list2': ['list2_str1', 'list2_str2', ['list2_str11', 'list_str12']]}
I'm trying to convert the following piece of data into a dict so I can have key:value pair and have them nested :
data="Group A\n name : rey\n age : 6\n status : active\n in role : 201\n weight : 25\n interests\n Out Side : 16016\n In Side : 0\n Out : 2804\n\n name : dan\n age : 5\n status : inactive\n in role : 201\n weight : 18\n interests\n Out Side : 16016\n In Side : 0\n Out : 2804\n\n"
Problem is, not all of them have : and some of them are supposed to be grouped together (i.e part of Group A). \n line break definitely helps me in separating them.
Here's the result I'm hoping to get :
[
{
"Group A":
[
{"name": "rey", "age": "6", "status": "active"},
{"name": "dan", "age": "5", "status": "inactive"}
]
}
]
What do I have right now? I've separated some of them into dict with :
result = {}
for row in data.split('\n'):
if ':' in row:
key, value = row.split(':')
result[key] = value.strip()
This outputs me :
{' name ': 'dan', ' weight ': '18', ' In Side ': '0', ' Out ': '2804', ' in role ': '201', ' status ': 'inactive', ' Out Side ': '16016', ' age ': '5'}
But this messes up with the existing order in which the data is shown above - and not all of it came out.
I'm kind of capturing this data from an external program and so limited to only Python version 2.7. Any ideas would be super helpful!
You should find the pattern in your data :
Your data is separated by empty lines
The inner data start with space
Using those observations you can write :
def parse(data):
aa = {}
curRecord = None
curGroup = None
for row in data.split('\n'):
if row.startswith(' '):
# this is a new key in the inner record
if ':' in row :
if curRecord == None:
curRecord = {}
curGroup.append(curRecord)
key, value = row.split(':')
curRecord[key.strip()] = value.strip()
elif row == '':
# this signal new inner record
curRecord = None
else:
# otherwise , it is a new group
curRecord = None
curGroup = []
aa[row.strip()] = curGroup
return aa
>>> import json
>>> print( json.dumps(parse(data)) );
{"Group A": [{"name": "rey", ... }, {"name": "dan", ... }]}
I will use setdefault to create new lists within your Group. This will work for multiple Groups in case you have group B, C, D...
import json
def convert_string(data):
result = {}
current = ""
key_no = -1
for pair in data.split("\n"):
if ':' not in pair:
if "Group" in pair:
result.setdefault(pair,[])
current = pair
key_no = -1
elif "name" in pair:
result[current].append({})
key_no +=1
key, value = pair.split(':')
result[current][key_no][key.strip()] = value.strip()
elif all(s not in pair.lower() for s in ("weight","in role","out","in side")):
key, value = pair.split(':')
result[current][key_no][key.strip()] = value.strip()
return result
print (json.dumps(convert_string(d),indent=4))
{
"Group A": [
{
"name": "rey",
"age": "6",
"status": "active"
},
{
"name": "dan",
"age": "5",
"status": "inactive"
}
]
}
I have a file which contains several json records. I have to parse this file and load each of the jsons to a particular SQL-Server table. However, the table might not exist on the database, in which case I have to also create it first before loading. So, I have to parse the json file and figure out the fields/columns and create the table. Then I will have to de-serialize the jsons into records and insert them into the table created. However, the caveat is that some fields in the json are optional i.e. a field might be absent from one json record but could be present in another record. Below is an example file with 3 records :-
{ id : 1001,
name : "John",
age : 30
} ,
{ id : 1002,
name : "Peter",
age : 25
},
{ id : 1002,
name : "Kevin",
age : 35,
salary : 5000
},
Notice that the field salary appears only in the 3rd record. The results should be :-
CREATE TABLE tab ( id int, name varchar(100), age int, salary int );
INSERT INTO tab (id, name, age, salary) values (1001, 'John', 30, NULL)
INSERT INTO tab (id, name, age, salary) values (1002, 'Peter', 25, NULL)
INSERT INTO tab (id, name, age, salary) values (1003, 'Kevin', 35, 5000)
Can anyone please help me with some pointers as I am new to Python. Thanks.
You could try this:
import json
TABLE_NAME = "tab"
sqlstatement = ''
with open ('data.json','r') as f:
jsondata = json.loads(f.read())
for json in jsondata:
keylist = "("
valuelist = "("
firstPair = True
for key, value in json.items():
if not firstPair:
keylist += ", "
valuelist += ", "
firstPair = False
keylist += key
if type(value) in (str, unicode):
valuelist += "'" + value + "'"
else:
valuelist += str(value)
keylist += ")"
valuelist += ")"
sqlstatement += "INSERT INTO " + TABLE_NAME + " " + keylist + " VALUES " + valuelist + "\n"
print(sqlstatement)
However for this to work, you'll need to change your JSON file to correct the syntax like this:
[{
"id" : 1001,
"name" : "John",
"age" : 30
} ,
{
"id" : 1002,
"name" : "Peter",
"age" : 25
},
{
"id" : 1003,
"name" : "Kevin",
"age" : 35,
"salary" : 5000
}]
Running this gives the following output:
INSERT INTO tab (age, id, name) VALUES (30, 1001, 'John')
INSERT INTO tab (age, id, name) VALUES (25, 1002, 'Peter')
INSERT INTO tab (salary, age, id, name) VALUES (5000, 35, 1003, 'Kevin')
Note that you don't need to specify NULLs. If you don't specify a column in the insert statement, it should automatically insert NULL into any columns you left out.
In Python, you can do something like this using sqlite3 and json, both from the standard library.
import json
import sqlite3
# The string representing the json.
# You will probably want to read this string in from
# a file rather than hardcoding it.
s = """[
{
"id": 1001,
"name": "John",
"age" : 30
},
{
"id" : 1002,
"name" : "Peter",
"age" : 25
},
{
"id" : 1002,
"name" : "Kevin",
"age" : 35,
"salary" : 5000
}
]"""
# Read the string representing json
# Into a python list of dicts.
data = json.loads(s)
# Open the file containing the SQL database.
with sqlite3.connect("filename.db") as conn:
# Create the table if it doesn't exist.
conn.execute(
"""CREATE TABLE IF NOT EXISTS tab(
id int,
name varchar(100),
age int,
salary int
);"""
)
# Insert each entry from json into the table.
keys = ["id", "name", "age", "salary"]
for entry in data:
# This will make sure that each key will default to None
# if the key doesn't exist in the json entry.
values = [entry.get(key, None) for key in keys]
# Execute the command and replace '?' with the each value
# in 'values'. DO NOT build a string and replace manually.
# the sqlite3 library will handle non safe strings by doing this.
cmd = """INSERT INTO tab VALUES(
?,
?,
?,
?
);"""
conn.execute(cmd, values)
conn.commit()
This will create a file named 'filename.db' in the current directory with the entries inserted.
To test the tables:
# Testing the table.
with sqlite3.connect("filename.db") as conn:
cmd = """SELECT * FROM tab WHERE SALARY NOT NULL;"""
cur = conn.execute(cmd)
res = cur.fetchall()
for r in res:
print(r)
{ "contact_entry": [
{"luid":"1",
"first_name":"vuvrpm",
"last_name":"basjrkkjst",
"phone_numbers":["919000000000"],
"emails":["yahoo#bingo.com"]
}
]
}
trying to make json out of following and pass it as body in Rest call
first_name is generated randomly and stored in strFirstName.
I tried this
jsonStr = jsonStr+ "{"+ '"luid"' +":" +'"1"' + "," +"first_name" + ":" + """'firstName"""' + "," + '"last_name"' + ":" + '"basjrkkjst"' + "," + '"phone_numbers"' + ":" + "[" + '"str(xx_msisdn)"' + "]," + '"emails"' + ":" + "[" + '"yahoo#bingo.com"' +"]}";
Output is:
{ "contact_entry": [{"luid":"1",first_name:kxggqo,"last_name":"basjrkkjst","phone_numbers":["str(xx_msisdn)"],"emails":["yahoo#bingo.com"]}
Can somebody help me?
Try the json module.
import json
first_name = "vuvrpm" # random value
entry = {
"contact_entry": [{
"luid":"1",
"first_name": first_name
"last_name":"basjrkkjst",
"phone_numbers":["919000000000"],
"emails":["yahoo#bingo.com"]
}]
}
print json.dumps(entry)
Will print:
{"contact_entry": [{"first_name": "vuvrpm", "last_name": "basjrkkjst", "phone_numbers": ["919000000000"], "luid": "1", "emails": ["yahoo#bingo.com"]}]}
(prettified):
{
"contact_entry": [{
"first_name": "vuvrpm",
"last_name": "basjrkkjst",
"phone_numbers": ["919000000000"],
"luid": "1",
"emails": ["yahoo#bingo.com"]
}]
}
Why u want to make json-like string? all you have to do is to create python dict and thats all.
import requests
my_json = { "contact_entry": [ {"luid":"1", "first_name":"vuvrpm", "last_name":"basjrkkjst", "phone_numbers":["919000000000"], "emails":["yahoo#bingo.com"]}] }
r = requests.post(URL, json=my_json)
every entry can be defined like this:
entry = {}
entry["luid"] = "1"
entry["firtst_name"] = "vuvrpm"
...
entry["phone_numbers"] = ["919000000000"]
...
and the output is 100% valid "json" (in this case dict):
{"luid":"1", "first_name":"vuvrpm", "last_name":"basjrkkjst", "phone_numbers":["919000000000"], "emails":["yahoo#bingo.com"]}