Related
Inputs
I have two lists of lists.
rule_seq =
[['#1', '#2', '#3'],
['#1', '#2', '#3']]
KG_seq =
[['nationality', 'placeOfBirth', 'locatedIn'],
['nationality', 'hasFather', 'nationality']]
I have to map the values in the same index to the dictionary with the value of rule_seq as the key in the list of above.
My desired output is
Output
unify_dict =
{'#1': ['nationality'],
'#2': ['placeOfBirth', 'hasFather'],
'#3': ['locatedIn', 'nationality']}
I made a dictionary as follows by flattening and zipping both lists of lists to check whether keys and values are in the dictionary.
My code is as follows.
def create_unify_dict(rule_seq, KG_seq):
unify_dict = collections.defaultdict(list)
flat_aug_rule_list = list(itertools.chain.from_iterable(rule_seq))
flat_aug_KG_list = list(itertools.chain.from_iterable(KG_seq))
[unify_dict[key].append(val) for key, val in zip(flat_aug_rule_list, flat_aug_KG_list)
if key not in unify_dict.keys() or val not in unify_dict[key]]
return unify_dict
unify_dict = create_unify_dict(rule_seq, KG_seq)
Is there a simpler way to get the result I want?
You can just call append using the same defualtdict with second level of nesting.
from collections import defaultdict
result = defaultdict(list)
for keyList,valueList in zip(rule_seq, KG_seq):
for key,item in zip(keyList, valueList):
if item not in result[key]: result[key].append(item)
OUTPUT:
defaultdict(<class 'list'>,
{'#1': ['nationality'],
'#2': ['placeOfBirth', 'hasFather'],
'#3': ['locatedIn', 'nationality']})
You can use collections:
import collections
# Create a defaultdict with list as value type
result = collections.defaultdict(list)
for s0, s1 in zip(rule_seq, KG_seq):
for v0, v1 in zip(s0, s1):
if v1 not in result[v0]:
result[v0].append(v1)
print({k: v for k, v in result.items()})
# {
# '#1': ['nationality'],
# '#2': ['placeOfBirth', 'hasFather'],
# '#3': ['locatedIn', 'nationality'],
# }
This would be my homemade approach using no modules. Vanilla Python.
combine = [list(set(l)) for l in [[lst[i] for lst in KG_seq] for i in range(len(KG_seq[0]))]]
dct = {place:st for place,st in zip(rule_seq[0],combine)}
output
{'#1': ['nationality'], '#2': ['hasFather', 'placeOfBirth'], '#3': ['nationality', 'locatedIn']}
oversimplified version
combine = []
for i in range(len(KG_seq[0])):
group = []
for lst in KG_seq:
group.append(lst[i])
combine.append(group)
newComb = []
for simp in combine:
newComb.append(list(set(simp)))
dct = {}
for place,st in zip(rule_seq[0],combine):
dct[place] = st
print(dct)
undersimplified
dct = {place:st for place,st in zip(rule_seq[0],[list(set(l)) for l in [[lst[i] for lst in KG_seq] for i in range(len(KG_seq[0]))]])}
Based on the following assumptions there could be several forms to what your method look like
rule_seq and kg_seq are equal in length
rule_seq and kg_seq items are also equal in length
One liner
def one_liner(rule_seq, kg_seq):
ret = {}
[ret.update({idx: ret.get(idx, set()) | {val}}) for arr_idx, arr_val in zip(rule_seq, kg_seq) for idx, val in zip(arr_idx, arr_val)]
return ret
Single loop + one liner
def one_loop(rule_seq, kg_seq):
ret = {}
for arr_idx, arr_val in zip(rule_seq, kg_seq):
[ret.update({idx: ret.get(idx, set()) | {val}}) for idx, val in zip(arr_idx, arr_val)]
return ret
Nested loops
def nested_loop(rule_seq, kg_seq):
ret = {}
for arr_idx, arr_val in zip(rule_seq, kg_seq):
for idx, val in zip(arr_idx, arr_val):
ret[idx] = ret.get(idx, set()) | {val}
return ret
Testing these out
one_liner(rule_seq, KG_seq)
{'#1': {'nationality'},
'#2': {'hasFather', 'placeOfBirth'},
'#3': {'locatedIn', 'nationality'}}
one_loop(rule_seq, KG_seq)
{'#1': {'nationality'},
'#2': {'hasFather', 'placeOfBirth'},
'#3': {'locatedIn', 'nationality'}}
nested_loop(rule_seq, KG_seq)
{'#1': {'nationality'},
'#2': {'hasFather', 'placeOfBirth'},
'#3': {'locatedIn', 'nationality'}}
d = {}
for i in range(len(rule_seq)):
for j in range(len(rule_seq[i])):
rule, kg = rule_seq[i][j], KG_seq[i][j]
if (rule not in d.keys()):
d[rule] = [kg]
elif kg not in d[rule]:
d[rule].append(kg)
result:
{'#1': ['nationality'], '#2': ['placeOfBirth', 'hasFather'], '#3': ['locatedIn', 'nationality']}
I'm looking for any suggestions to resolve an issue I'm facing. It might seem as a simple problem, but after a few days trying to find an answer - I think it is not anymore.
I'm receiving data (StringType) in a following JSON-like format, and there is a requirement to turn it into flat key-value pair dictionary. Here is a payload sample:
s = """{"status": "active", "name": "{\"first\": \"John\", \"last\": \"Smith\"}", "street_address": "100 \"Y\" Street"}"""
and the desired output should look like this:
{'status': 'active', 'name_first': 'John', 'name_last': 'Smith', 'street_address': '100 "Y" Street'}
The issue is I can't find a way to turn original string (s) into a dictionary. If I can achieve that the flattening part is working perfectly fine.
import json
import collections
import ast
#############################################################
# Flatten complex structure into a flat dictionary
#############################################################
def flatten_dictionary(dictionary, parent_key=False, separator='_', value_to_str=True):
"""
Turn a nested complex json into a flattened dictionary
:param dictionary: The dictionary to flatten
:param parent_key: The string to prepend to dictionary's keys
:param separator: The string used to separate flattened keys
:param value_to_str: Force all returned values to string type
:return: A flattened dictionary
"""
items = []
for key, value in dictionary.items():
new_key = str(parent_key) + separator + key if parent_key else key
try:
value = json.loads(value)
except BaseException:
value = value
if isinstance(value, collections.MutableMapping):
if not value.items():
items.append((new_key,None))
else:
items.extend(flatten_dictionary(value, new_key, separator).items())
elif isinstance(value, list):
if len(value):
for k, v in enumerate(value):
items.extend(flatten_dictionary({str(k): (str(v) if value_to_str else v)}, new_key).items())
else:
items.append((new_key,None))
else:
items.append((new_key, (str(value) if value_to_str else value)))
return dict(items)
# Data sample; sting and dictionary
s = """{"status": "active", "name": "{\"first\": \"John\", \"last\": \"Smith\"}", "street_address": "100 \"Y\" Street"}"""
d = {"status": "active", "name": "{\"first\": \"John\", \"last\": \"Smith\"}", "street_address": "100 \"Y\" Street"}
# Works for dictionary type
print(flatten_dictionary(d))
# Doesn't work for string type, for any of the below methods
e = eval(s)
# a = ast.literal_eval(s)
# j = json.loads(s)
Try:
import json
import re
def jsonify(s):
s = s.replace('"{','{').replace('}"','}')
s = re.sub(r'street_address":\s+"(.+)"(.+)"(.+)"', r'street_address": "\1\2\3"',s)
return json.loads(s)
If you must keep the quotes around Y, try:
def jsonify(s):
s = s.replace('"{','{').replace('}"','}')
search = re.search(r'street_address":\s+"(.+)"(.+)"(.+)"',s)
if search:
s = re.sub(r'street_address":\s+"(.+)"(.+)"(.+)"', r'street_address": "\1\2\3"',s)
dict_version = json.loads(s)
dict_version['street_address'] = dict_version['street_address'].replace(search.group(2),'"'+search.group(2)+'"')
return dict_version
A more generalized attempt:
def jsonify(s):
pattern = r'(?<=[,}])\s*"(.[^\{\}:,]+?)":\s+"([^\{\}:,]+?)"([^\{\}:,]+?)"([^\{\}:,]+?)"([,\}])'
s = s.replace('"{','{').replace('}"','}')
search = re.search(pattern,s)
matches = []
if search:
matches = re.findall(pattern,s)
s = re.sub(pattern, r'"\1": "\2\3\4"\5',s)
dict_version = json.loads(s)
for match in matches:
dict_version[match[0]] = dict_version[match[0]].replace(match[2],'"'+match[2]+'"')
return dict_version
I have a json in below format:
{"MainName":[{"col1":"12345","col2":"False","col3":"190809","SubName1":{"col4":30.00,"SubName2":{"col5":"19703","col6":"USD"}},"col7":"7372267","SubName3":[{"col8":"345337","col9":"PC"}],"col10":"10265","col11":"29889004","col12":"calculated","col13":"9218","SubName4":{"col14":1,"SubName5":{"col15":"1970324","col16":"integer"}},"col17":"434628","col18":"2020-02-06T13:47:40.000-0800","col19":"754878037","SubName6":{"col20":30.00,"SubName7":{"col21":"19703248","col22":"USD"}}},{"col1":"12345","col2":"False","col3":"190809","SubName1":{"col4":30.00,"SubName2":{"col5":"19703","col6":"USD"}},"col7":"7372267","SubName3":[{"col8":"345337","col9":"PC"}],"col10":"10265","col11":"29889004","col12":"calculated","col13":"9218","SubName4":{"col14":1,"SubName5":{"col15":"1970324","col16":"integer"}},"col17":"434628","col18":"2020-02-06T13:47:40.000-0800","col19":"754878037","SubName6":{"col20":30.00,"SubName7":{"col21":"19703248","col22":"USD"}}}],"skip":0,"top":2,"next":"/v1/APIName?skip=2&top=2"}
I want to convert it into csv with below format:
MainName_col1,MainName_col2,MainName_col3,MainName_SubName1_col4,MainName_SubName1_SubName2_col5,MainName_SubName1_SubName2_col6,MainName_col7,MainName_SubName3_col8,MainName_SubName3_col9,MainName_col10,MainName_col11,MainName_col12,MainName_col13,MainName_SubName4_col14,MainName_SubName4_SubName5_col15,MainName_SubName4_SubName5_col16,MainName_col17,MainName_col18,MainName_col19,MainName_SubName6_col20,MainName_SubName6_SubName7_col21,MainName_SubName6_SubName7_col22
12345,False,190809,30.0,19703,USD,7372267,345337,PC,10265,29889004,calculated,9218,1,1970324,integer,434628,2020-02-06T13:47:40.000-0800,754878037,30.0,19703248,USD
12345,False,190809,30.0,19703,USD,7372267,345337,PC,10265,29889004,calculated,9218,2,123453,integer,434628,2020-02-06T13:47:40.000-0800,754878037,30.0,19703248,USD
Kindly help me out in this.
Use below function to flatten your JSON data.
dc = {"MainName":[{"col1":"12345","col2":False,"col3":"190809","SubName1":{"col4":30.00,"SubName2":{"col5":"19703","col6":"USD"}},"col7":"7372267","SubName3":[{"col8":"345337","col9":"PC"}],"col10":"10265","col11":"29889004","col12":"calculated","col13":"9218","SubName4":{"col14":1,"SubName5":{"col15":"1970324","col16":"integer"}},"col17":"434628","col18":"2020-02-06T13:47:40.000-0800","col19":"754878037","SubName6":{"col20":30.00,"SubName7":{"col21":"19703248","col22":"USD"}}}],"skip":0,"top":1,"next":"/v1/APIName?skip=1&top=1"}
def flatten(root: str, dict_obj: dict):
flat = {}
for i in dict_obj.keys():
val = dict_obj[i]
if not isinstance(val, dict) and not isinstance(val, list):
flat[f'{root}_{i}'] = val
else:
if isinstance(val, list):
val = val[-1]
flat.update(flatten(f'{root}_{i}', val))
return flat
flatten('MainName', dc['MainName'][0])
It will give you expected output. Then use it the way you want.
{'MainName_col1': '12345',
'MainName_col2': False,
'MainName_col3': '190809',
'MainName_SubName1_col4': 30.0,
'MainName_SubName1_SubName2_col5': '19703',
'MainName_SubName1_SubName2_col6': 'USD',
'MainName_col7': '7372267',
'MainName_SubName3_col8': '345337',
'MainName_SubName3_col9': 'PC',
'MainName_col10': '10265',
'MainName_col11': '29889004',
'MainName_col12': 'calculated',
'MainName_col13': '9218',
'MainName_SubName4_col14': 1,
'MainName_SubName4_SubName5_col15': '1970324',
'MainName_SubName4_SubName5_col16': 'integer',
'MainName_col17': '434628',
'MainName_col18': '2020-02-06T13:47:40.000-0800',
'MainName_col19': '754878037',
'MainName_SubName6_col20': 30.0,
'MainName_SubName6_SubName7_col21': '19703248',
'MainName_SubName6_SubName7_col22': 'USD'}
As of my understanding, your dc will look like below
dc = {"MainName":[{"col1":"12345","col2":"False","col3":"190809","SubName1":{"col4":30.00,"SubName2":{"col5":"19703","col6":"USD"}},"col7":"7372267","SubName3":[{"col8":"345337","col9":"PC"}],"col10":"10265","col11":"29889004","col12":"calculated","col13":"9218","SubName4":{"col14":1,"SubName5":{"col15":"1970324","col16":"integer"}},"col17":"434628","col18":"2020-02-06T13:47:40.000-0800","col19":"754878037","SubName6":{"col20":30.00,"SubName7":{"col21":"19703248","col22":"USD"}}},{"col1_a":"12345XX","col2_b":"False","col3_c":"190809","SubName1":{"col4_d":30.00,"SubName2":{"col5_e":"19703","col6_f":"USD"}},"col7_g":"7372267","SubName3":[{"col8_h":"345337","col9":"PC"}],"col10_i":"10265","col11_j":"29889004","col12_k":"calculated","col13_l":"9218","SubName4":{"col14_m":1,"SubName5":{"col15_n":"1970324","col16_o":"integer"}},"col17_p":"434628","col18_q":"2020-02-06T13:47:40.000-0800","col19_r":"754878037","SubName6":{"col20_s":30.00,"SubName7":{"col21_t":"19703248","col22_u":"USDZZ"}}}],"skip":0,"top":2,"next":"/v1/APIName?skip=2&top=2"}
I used the above answer to flatten everything into single object
def flatten(root: str, dict_obj: dict):
flat = {}
for i in dict_obj.keys():
val = dict_obj[i]
if not isinstance(val, dict) and not isinstance(val, list):
flat[f'{root}_{i}'] = val
else:
if isinstance(val, list):
val = val[-1]
flat.update(flatten(f'{root}_{i}', val))
return flat
keys_list = []
values_list = []
for i in range(len(dc['MainName'])):
result = flatten('MainName', dc['MainName'][i])
keys_list.append(list(result.keys()))
values_list.append(list(result.values()))
for k in keys_list:
for res in k:
guestFile = open("sample.csv","a")
guestFile.write(res)
guestFile.write(",")
guestFile.close()
for v in values_list:
for res in v:
guestFile = open("sample.csv","a")
guestFile.write(str(res))
guestFile.write(",")
guestFile.close()
Checkout my code at https://repl.it/#TamilselvanLaks/jsontocsvmul
Note: Use the 'run' button to run the program, left side you can see sample.csv
there you can see all keys as like you want
Please let me know my answer meets your expectation
I am having this text
/** Goodmorning
Alex
Dog
House
Red
*/
/** Goodnight
Maria
Cat
Office
Green
*/
I would like to have Alex , Dog , House and red in one list and Maria,Cat,office,green in an other list.
I am having this code
with open(filename) as f :
for i in f:
if i.startswith("/** Goodmorning"):
#add files to list
elif i.startswith("/** Goodnight"):
#add files to other list
So, is there any way to write the script so it can understands that Alex belongs in the part of the text that has Goodmorning?
I'd recommend you to use dict, where "section name" will be a key:
with open(filename) as f:
result = {}
current_list = None
for line in f:
if line.startswith("/**"):
current_list = []
result[line[3:].strip()] = current_list
elif line != "*/":
current_list.append(line.strip())
Result:
{'Goodmorning': ['Alex', 'Dog', 'House', 'Red'], 'Goodnight': ['Maria', 'Cat', 'Office', 'Green']}
To search which key one of values belongs you can use next code:
search_value = "Alex"
for key, values in result.items():
if search_value in values:
print(search_value, "belongs to", key)
break
I would recommend to use Regular expressions. In python there is a module for this called re
import re
s = """/** Goodmorning
Alex
Dog
House
Red
*/
/** Goodnight
Maria
Cat
Office
Green
*/"""
pattern = r'/\*\*([\w \n]+)\*/'
word_groups = re.findall(pattern, s, re.MULTILINE)
d = {}
for word_group in word_groups:
words = word_group.strip().split('\n\n')
d[words[0]] = words[1:]
print(d)
Output:
{'Goodmorning': ['Alex', 'Dog', 'House', 'Red'], 'Goodnight':
['Maria', 'Cat', 'Office', 'Green']}
expanding on Olvin Roght (sorry can't comment - not enough reputation) I would keep a second dictionary for the reverse lookup
with open(filename) as f:
key_to_list = {}
name_to_key = {}
current_list = None
current_key = None
for line in f:
if line.startswith("/**"):
current_list = []
current_key = line[3:].strip()
key_to_list[current_key] = current_list
elif line != "*/":
current_name=line.strip()
name_to_key[current_name]=current_key
current_list.append(current_name)
print key_to_list
print name_to_key['Alex']
alternative is to convert the dictionary afterwards:
name_to_key = {n : k for k in key_to_list for n in key_to_list[k]}
(i.e if you want to go with the regex version from ashwani)
Limitation is that this only permits one membership per name.
I am novice with Python and currently I am trying to use it to parse some custom output formated string. In fact format contains named lists of float and lists of tuples of float. I wrote a function but it looks excessive. How can it be done in more suitable way for Python?
import re
def extract_line(line):
line = line.lstrip('0123456789# ')
measurement_list = list(filter(None, re.split(r'\s*;\s*', line)))
measurement = {}
for elem in measurement_list:
elem_list = list(filter(None, re.split(r'\s*=\s*', elem)))
name = elem_list[0]
if name == 'points':
points = list(filter(None, re.split(r'\s*\(\s*|\s*\)\s*',elem_list[1].strip(' {}'))))
for point in points:
p = re.match(r'\s*(\d+(?:\.\d+)?)\s*,\s*(\d+(?:\.\d+)?)\s*', point).groups()
if 'points' not in measurement.keys():
measurement['points'] = []
measurement['points'].append(tuple(map(float,p)))
else:
values = list(filter(None, elem_list[1].strip(' {}').split(' ')))
for value in values:
if name not in measurement.keys():
measurement[name] = []
measurement[name].append(float(value))
return measurement
to_parse = '#10 points = { ( 2.96296 , 0.822213 ) ( 3.7037 , 0.902167 ) } ; L = { 5.20086 } ; P = { 3.14815 3.51852 } ;'
print(extract_line(to_parse))
You can do it using re.findall:
import re
to_parse = '#10 points = { ( 2.96296 , 0.822213 ) ( 3.7037 , 0.902167 ) } ; L = { 5.20086 } ; P = { 3.14815 3.51852 } ;'
m_list = re.findall(r'(\w+)\s*=\s*{([^}]*)}', to_parse)
measurements = {}
for k,v in m_list:
if k == 'points':
elts = re.findall(r'([0-9.]+)\s*,\s*([0-9.]+)', v)
measurements[k] = [tuple(map(float, elt)) for elt in elts]
else:
measurements[k] = [float(x) for x in v.split()]
print(measurements)
Feel free to put it in a function and to check if keys don't already exists.
This:
import re
a=re.findall(r' ([\d\.eE-]*) ',to_parse)
map(float, a)
>> [2.96296, 0.822213, 3.7037, 0.902167, 5.20086, 3.14815]
Will give you your list of numbers, is that what you look for?