Problem
I am dealing with a response object and I am trying to massage it into something which could be easily consumed by a Typescript frontend.
The object:
r_obj = [
{
'custom_fields': [
{'id': 360018501198, 'value': '5678'},
{'id': 360023508598, 'value': 'Jim'},
{'id': 1900000084913, 'value': '2021-09-03'}
]
},
{
'custom_fields': [
{'id': 360018501198, 'value': '1234'},
{'id': 360023508598, 'value': 'Bob'},
{'id': 1900000084913, 'value': '2021-09-03'}
]
}
]
Desired Structure
[
{
"order_id": '5678',
"name": "Jim",
"date": '2021-09-03'
},
{
"order_id": '1234',
"name": "Bob",
"date": '2021-09-03'
},
]
So id values in the custom_field payload are known and I want to map them to enum representation and with the hope that the output payload contain saner k,v pairs.
The Code
I have only managed to do the mapping of the enum values.
if __name__ == "__main__":
class FieldEnum(enum.IntEnum):
ORDER_ID = 360018501198
DATE = 1900000084913
NAME = 360023508598
r_obj = [
{
'custom_fields': [
{'id': 360018501198, 'value': '5678'},
{'id': 360023508598, 'value': 'Jim'},
{'id': 1900000084913, 'value': '2021-09-03'}
]
},
{
'custom_fields': [
{'id': 360018501198, 'value': '1234'},
{'id': 360023508598, 'value': 'Bob'},
{'id': 1900000084913, 'value': '2021-09-03'}
]
}
]
for row in r_obj:
for field in row['custom_fields']:
if field['id']:
field['id'] = FieldEnum(field['id']).name.lower()
print(field)
Code Output
{'id': 'order_id', 'value': '5678'}
{'id': 'name', 'value': 'Jim'}
{'id': 'date', 'value': '2021-09-03'}
{'id': 'order_id', 'value': '1234'}
{'id': 'name', 'value': 'Bob'}
{'id': 'date', 'value': '2021-09-03'}
I am struggling with how best to group(?) each set of 3 dicts into its own list structure.
EDIT
As per #balderman, I have altered to this:
if __name__ == "__main__":
class FieldEnum(enum.IntEnum):
ORDER_ID = 360018501198
DATE = 1900000084913
NAME = 360023508598
r_obj = [
{
'custom_fields': [
{'id': 360018501198, 'value': '5678'},
{'id': 360023508598, 'value': 'Jim'},
{'id': 1900000084913, 'value': '2021-09-03'}
]
},
{
'custom_fields': [
{'id': 360018501198, 'value': '1234'},
{'id': 360023508598, 'value': 'Bob'},
{'id': 1900000084913, 'value': '2021-09-03'}
]
}
]
out = list()
for entry in r_obj:
out.append({})
for f in entry['custom_fields']:
out[-1][FieldEnum(f['id']).name.lower()] = f['value']
print(out)
Code Output
[
{
'order_id': '5678',
'name': 'Jim',
'date': '2021-09-03'
},
{
'order_id': '1234',
'name': 'Bob',
'date': '2021-09-03'
}
]
Which is the desired output.
I was wondering if there is another way syntactically to achieve the same output?
I would do it with a list comprehension to make it one-liner
code:
import enum
class FieldEnum(enum.IntEnum):
ORDER_ID = 360018501198
DATE = 1900000084913
NAME = 360023508598
r_obj = [
{
'custom_fields': [
{'id': 360018501198, 'value': '5678'},
{'id': 360023508598, 'value': 'Jim'},
{'id': 1900000084913, 'value': '2021-09-03'}
]
},
{
'custom_fields': [
{'id': 360018501198, 'value': '1234'},
{'id': 360023508598, 'value': 'Bob'},
{'id': 1900000084913, 'value': '2021-09-03'}
]
}
]
print([{FieldEnum(f['id']).name.lower():f['value'] for f in entry['custom_fields'] } for entry in r_obj])
result:
[
{"order_id": "5678", "name": "Jim", "date": "2021-09-03"},
{"order_id": "1234", "name": "Bob", "date": "2021-09-03"},
]
try the below
lookup = {360018501198: 'order_id', 360023508598: 'name', 1900000084913: 'date'}
data = [{
'custom_fields': [
{'id': 360018501198, 'value': '5678'},
{'id': 360023508598, 'value': 'Jim'},
{'id': 1900000084913, 'value': '2021-09-03'}
]
},
{
'custom_fields': [
{'id': 360018501198, 'value': '1234'},
{'id': 360023508598, 'value': 'Bob'},
{'id': 1900000084913, 'value': '2021-09-03'}
]
}
]
out = list()
for entry in data:
out.append({})
for f in entry['custom_fields']:
out[-1][lookup[f['id']]] = f['value']
print(out)
output
[{'order_id': '5678', 'name': 'Jim', 'date': '2021-09-03'}, {'order_id': '1234', 'name': 'Bob', 'date': '2021-09-03'}]
Related
I have a json array on python which I want to filter based on the value of : my_json_array['models']['variants']['condition']['type']
my json array looks like the following :
my_json_array = [
{'id': 1,
'models': [
{'color': {'code': '#626262', 'name': 'Gray'},
'variants': [{'id': 1,
'condition': [{'id': 1,
'type': 'type1',
'value': 14900},
{'id': 2,
'type': 'type2',
'value': 14000}]]
]
}]
I'm looking for a method to remove condition items with type = type2. The output should look like this :
my_json_array = [{
'id': 1,
'models': [
{'color': {'code': '#626262', 'name': 'Gray'},
'variants': [{'id': 1,
'condition': [{'id': 1,
'type': 'type1',
'value': 14900}]]
]
}]
Do you mean this?
my_json_array = [
{
'id': 1,
'models': [
{
'color': {'code': '#626262', 'name': 'Gray'},
'variants': [
{
'id': 1,
'condition': [
{
'id': 1,
'type': 'type1',
'value': 14900
},
{
'id': 2,
'type': 'type2',
'value': 14000
}
]
}
]
}
]
}
]
for mydict in my_json_array:
for model in mydict['models']:
for variant in model['variants']:
for condition in variant['condition']:
if condition['type']=="type2":
variant['condition'].remove(condition)
print(my_json_array) # [{'id': 1, 'models': [{'color': {'code': '#626262', 'name': 'Gray'}, 'variants': [{'id': 1, 'condition': [{'id': 1, 'type': 'type1', 'value': 14900}]}]}]}]
1.I need to get the value for business' name and append it to a list.
2.I need to get the value policies and append to a list after checking parent.
3.if parent is Marketing name has to added to level1.
4.if parent is Advertising name has to added to level2.
5.if some place Business is [] I need to pass None instead of Null List
Also need to check key exists or not for some keys there is a chance of missing policies, business
Sample dictionary is below
searchtest = [{'_index': 'newtest',
'_type': '_doc',
'_id': '100',
'_score': 1.0,
'_source': {'id': '100',
'name': 'A',
'Business': [{'id': '7', 'name': 'Enterprise'},
{'id': '8', 'name': 'Customer'}],
'policies': [{'id': '332',
'name': 'Second division',
'parent': 'Marketing'},
{'id': '3323', 'name': 'First division', 'parent': 'Marketing'}]}},
{'_index': 'newtest',
'_type': '_doc',
'_id': '101',
'_score': 1.0,
'_source': {'id': '101',
'name': 'B',
'Business': [{'id': '7'},
{'id': '8', 'name': 'Customer'}],
'policies': [{'id': '332',
'name': 'Second division',
'parent': 'Marketing'},
{'id': '3323', 'name': 'First division', 'parent': 'Marketing'}]}}]
Expected out
[
{
"id": "100",
"name": "A",
"Business": ["Enterprise", "Customer"],
"level1": ['Second division', 'First division'],
"level2": [ ]
},
{
"id": "101",
"name": "B",
"Business": ["Enterprise", "Customer"],
"level1": ['Second division', 'First division'],
"level2": [ ]
}
]
COde is below
def do_the_thing(lst):
resp = []
parents_mapper = {
'Marketing': 'level1',
'Advertising': 'level2'
}
for el in lst:
d = {
'id': el['_source']['id'],
'name': el['_source']['name'],
'Business': [],
'level1': [],
'level2': []
}
for business in el.get("_source", {}).get("business", {}).get("name", ""):
business_name = business.get('name')
if business_name:
d['Business'].append(business_name)
for policy in el.get('policies', []):
policy_parent = policy.get('parent')
parent_found = parents_mapper.get(policy_parent)
policy_name = policy.get('name')
if parent_found and policy_name:
d[parent_found].append(policy_name)
resp.append(d)
return resp
if __name__ == "__main__":
import pprint
pp = pprint.PrettyPrinter(4)
pp.pprint(do_the_thing(searchtest))
My output
[ {'Business': [], 'id': '100', 'level1': [], 'level2': [], 'name': 'A'},
{'Business': [], 'id': '101', 'level1': [], 'level2': [], 'name': 'B'}]
The problem in my output you can see:
'Business', 'level1' is [] is null list.
Adding one more dictionary for testing
searchtest = [{'_index': 'newtest',
'_type': '_doc',
'_id': '100',
'_score': 1.0,
'_source': {'id': '100',
'name': 'A',
'policies': [{'id': '332',
'name': 'Second division',
'parent': 'Marketing'},
{'id': '3323', 'name': 'First division', 'parent': 'Marketing'}]}},
{'_index': 'newtest',
'_type': '_doc',
'_id': '101',
'_score': 1.0,
'_source': {'id': '101',
'name': 'B',
'Business': [{'id': '9'}, {'id': '10', 'name': 'Customer'}],
'policies': [{'id': '332',
'name': 'Second division',
'parent': 'Marketing'},
{'id': '3323', 'name': 'First division', 'parent': 'Advertising'}]}}]
In the above dictionary you can see that there is no Business in 100 key and for 101 there is no name inside the Business key. So there will be key error will be coming. Need to handle that
You are not collecting data from dict, You have to select particular key to get it's value.
replace this :
for el in lst:
d = {
'id': el['_source']['id'],
'name': el['_source']['name'],
'Business': [],
'level1': [],
'level2': []
}
with this:
for el in data:
d = {
'id' : el['_source']['id'],
'name' : el['_source']['name'],
'Business' : [name['name'] for name in el['_source']['Business']],
'level1' : [name['name'] for name in el['_source']['policies']],
'level2' : []
}
output:
[ { 'Business': ['Enterprise', 'Customer'],
'id': '100',
'level1': ['Second division', 'First division'],
'level2': [],
'name': 'A'},
{ 'Business': ['Enterprise', 'Customer'],
'id': '101',
'level1': ['Second division', 'First division'],
'level2': [],
'name': 'B'}]
I have seen Python: remove dictionary from list and Splitting a list of dictionaries into several lists of dictionaries - but this question is slightly different.
Consider this working example (same in Python 2 or 3):
#!/usr/bin/env python
from __future__ import print_function
origarr = [
{ 'name': 'test01', 'type': 0, 'value': 42 },
{ 'name': 'test02', 'type': 0, 'value': 142 },
{ 'name': 'test03', 'type': 2, 'value': 242 },
{ 'name': 'test04', 'type': 2, 'value': 342 },
{ 'name': 'test05', 'type': 3, 'value': 42 },
]
print("origarr: {}".format(origarr))
lastdictelem = origarr.pop()
print("\nlastdictelem: {}".format(lastdictelem))
print("after pop, origarr: {}".format(origarr))
namestofilter = [ 'test01', 'test02' ]
newarr = []
for iname in namestofilter:
# find the object having the name iname
foundidx = -1
for ix, idict in enumerate(origarr):
if idict.get('name') == iname:
foundidx = ix
break
if foundidx > -1:
# remove dict object via pop at index, save removed object
remdict = origarr.pop(foundidx)
# add removed object to newarr:
newarr.append(remdict)
print("\nafter namestofilter:")
print("newarr: {}".format(newarr))
print("origarr: {}".format(origarr))
Basically, mylist.pop() removes the last element from mylist as an object (here a dict), and returns it - then I can trivially insert it in a new array/list; this is illustrated by the first printout of this script:
$ python2 test.py
origarr: [{'name': 'test01', 'type': 0, 'value': 42}, {'name': 'test02', 'type': 0, 'value': 142}, {'name': 'test03', 'type': 2, 'value': 242}, {'name': 'test04', 'type': 2, 'value': 342}, {'name': 'test05', 'type': 3, 'value': 42}]
lastdictelem: {'name': 'test05', 'type': 3, 'value': 42}
after pop, origarr: [{'name': 'test01', 'type': 0, 'value': 42}, {'name': 'test02', 'type': 0, 'value': 142}, {'name': 'test03', 'type': 2, 'value': 242}, {'name': 'test04', 'type': 2, 'value': 342}]
Now, what I would like to do, is define an array with values for the name key in a dict (say, namestofilter = [ 'test01', 'test02' ]), and have those dicts removed from the orriginal array/list, and put into a new array/list (as .pop() would do with a single element and an object reference).
Since pop removes the item at a specific index and returns it, the above code does exactly that - and works:
...
after namestofilter:
newarr: [{'name': 'test01', 'type': 0, 'value': 42}, {'name': 'test02', 'type': 0, 'value': 142}]
origarr: [{'name': 'test03', 'type': 2, 'value': 242}, {'name': 'test04', 'type': 2, 'value': 342}]
... but I was wondering - is there a more compact way of doing that, other than "manually" for-looping through the two arrays, and calling .pop()/.append() individually (as done in the example)?
I'm not sure is there a way to it compact - probaly not.
But you can simpify code a little bit and also don't spend O(n) for each .pop:
origarr = [
{ 'name': 'test01', 'type': 0, 'value': 42 },
{ 'name': 'test02', 'type': 0, 'value': 142 },
{ 'name': 'test03', 'type': 2, 'value': 242 },
{ 'name': 'test04', 'type': 2, 'value': 342 },
{ 'name': 'test05', 'type': 3, 'value': 42 },
]
namestofilter = set([ 'test01', 'test02' ]). # could be a list as in question
print("origarr: {}".format(origarr))
lastdictelem = origarr.pop()
print("\nlastdictelem: {}".format(lastdictelem))
print("after pop, origarr: {}".format(origarr))
shift = 0
newarr = []
for ix, idict in enumerate(origarr):
if idict['name'] in namestofilter:
shift += 1
newarr.append(idict)
continue
origarr[ix-shift] = origarr[ix]
origarr = origarr[:-shift] # perhaps it is a slicing O(n) copy overhead
print("\nafter namestofilter:")
print("newarr: {}".format(newarr))
print("origarr: {}".format(origarr))
Output:
origarr: [{'name': 'test01', 'type': 0, 'value': 42}, {'name': 'test02', 'type': 0, 'value': 142}, {'name': 'test03', 'type': 2, 'value': 242}, {'name': 'test04', 'type': 2, 'value': 342}, {'name': 'test05', 'type': 3, 'value': 42}]
lastdictelem: {'name': 'test05', 'type': 3, 'value': 42}
after pop, origarr: [{'name': 'test01', 'type': 0, 'value': 42}, {'name': 'test02', 'type': 0, 'value': 142}, {'name': 'test03', 'type': 2, 'value': 242}, {'name': 'test04', 'type': 2, 'value': 342}]
after namestofilter:
newarr: [{'name': 'test01', 'type': 0, 'value': 42}, {'name': 'test02', 'type': 0, 'value': 142}]
origarr: [{'name': 'test03', 'type': 2, 'value': 242}, {'name': 'test04', 'type': 2, 'value': 342}]
I am trying to convert the nested complex json into the dataframe this is my json format
There are multiple json rows like this i just have used here only one which is the id above now when i am trying to convert the dataframe i am getting the output like this
I further want to display the innings batsmen name stats runs
ballsFaced . I want to show the complete data into the dataframe.
{
'1164223': {
'fullSummaryLink': '/series/19014/scorecard/1164223/new-zealand-a-vs-india-a-1st-unofficial-odi-india-a-tour-of-nz-2018-19',
'innings': {
'1': {
'batsmen': [
{
'name': 'HD Rutherford',
'href': 'http://www.espncricinfo.com/ci/content/player/331375.html',
'stats': {
'runs': {
'name': 'runs',
'text': 'RUNS',
'value': '70'
},
'ballsFaced': {
'name': 'ballsFaced',
'text': 'BF',
'value': '66'
},
'notouts': {
'name': 'notouts',
'text': 'Not Out',
'value': '0'
}
}
},
{
'name': 'JDS Neesham',
'href': 'http://www.espncricinfo.com/ci/content/player/355269.html',
'stats': {
'runs': {
'name': 'runs',
'text': 'RUNS',
'value': '79'
},
'ballsFaced': {
'name': 'ballsFaced',
'text': 'BF',
'value': '48'
},
'notouts': {
'name': 'notouts',
'text': 'Not Out',
'value': '1'
}
}
}
],
'team': {
'teamDisplayName': 'NEW ZEALAND A',
'innDisplayName': 'INNINGS',
'runs': 308,
'overs': 50,
'wickets': 6,
'description': 'complete',
'inningsRunWicket': '308/6',
'inningStatus': ''
},
'bowlers': [
{
'name': 'S Kaul',
'href': 'http://www.espncricinfo.com/ci/content/player/326017.html',
'stats': {
'overs': {
'name': 'overs',
'text': 'O',
'value': '10'
},
'conceded': {
'name': 'conceded',
'text': 'R',
'value': '74'
},
'wickets': {
'name': 'wickets',
'text': 'E',
'value': '2'
}
}
},
{
'name': 'K Gowtham',
'href': 'http://www.espncricinfo.com/ci/content/player/424377.html',
'stats': {
'overs': {
'name': 'overs',
'text': 'O',
'value': '9'
},
'conceded': {
'name': 'conceded',
'text': 'R',
'value': '46'
},
'wickets': {
'name': 'wickets',
'text': 'E',
'value': '1'
}
}
}
]
},
'2': {
'bowlers': [
{
'name': 'HK Bennett',
'href': 'http://www.espncricinfo.com/ci/content/player/226493.html',
'stats': {
'overs': {
'name': 'overs',
'text': 'O',
'value': '10'
},
'conceded': {
'name': 'conceded',
'text': 'R',
'value': '65'
},
'wickets': {
'name': 'wickets',
'text': 'E',
'value': '2'
}
}
},
{
'name': 'LH Ferguson',
'href': 'http://www.espncricinfo.com/ci/content/player/493773.html',
'stats': {
'overs': {
'name': 'overs',
'text': 'O',
'value': '10'
},
'conceded': {
'name': 'conceded',
'text': 'R',
'value': '75'
},
'wickets': {
'name': 'wickets',
'text': 'E',
'value': '2'
}
}
}
],
'batsmen': [
{
'name': 'V Shankar',
'href': 'http://www.espncricinfo.com/ci/content/player/477021.html',
'stats': {
'runs': {
'name': 'runs',
'text': 'RUNS',
'value': '87'
},
'ballsFaced': {
'name': 'ballsFaced',
'text': 'BF',
'value': '80'
},
'notouts': {
'name': 'notouts',
'text': 'Not Out',
'value': '1'
}
}
},
{
'name': 'SS Iyer',
'href': 'http://www.espncricinfo.com/ci/content/player/642519.html',
'stats': {
'runs': {
'name': 'runs',
'text': 'RUNS',
'value': '54'
},
'ballsFaced': {
'name': 'ballsFaced',
'text': 'BF',
'value': '54'
},
'notouts': {
'name': 'notouts',
'text': 'Not Out',
'value': '0'
}
}
}
],
'team': {
'teamDisplayName': 'INDIA A',
'innDisplayName': 'INNINGS',
'runs': 311,
'overs': 49,
'wickets': 6,
'description': 'target reached',
'inningsRunWicket': '311/6',
'inningStatus': ''
}
}
},
'isAvailable': True
}
}
This is the output of my code which i have tried and i have given the image sample thats the output.. I am trying to make into simple format which i have not been able to do that .
this is my python code which converts the dictionary of json values into the pandas dataframe
df = pd.concat({k: pd.DataFrame(v) for k, v in scorecard_summary.items()})
df
I am trying to convert the complex json into the pandas dataframe but i am having problem when i am converting it. I am not possibly sure how to convert the complex json into the pandas dataframe.
I want to show all the data in the dataframe this is how i am getting it and i am sharing the format of my json data so that it can be understandable
{
'1164223': {
'fullSummaryLink': '/series/19014/scorecard/1164223/new-zealand-a-vs-india-a-1st-unofficial-odi-india-a-tour-of-nz-2018-19',
'innings': {
'1': {
'batsmen': [
{
'name': 'HD Rutherford',
'href': 'http://www.espncricinfo.com/ci/content/player/331375.html',
'stats': {
'runs': {
'name': 'runs',
'text': 'RUNS',
'value': '70'
},
'ballsFaced': {
'name': 'ballsFaced',
'text': 'BF',
'value': '66'
},
'notouts': {
'name': 'notouts',
'text': 'Not Out',
'value': '0'
}
}
},
{
'name': 'JDS Neesham',
'href': 'http://www.espncricinfo.com/ci/content/player/355269.html',
'stats': {
'runs': {
'name': 'runs',
'text': 'RUNS',
'value': '79'
},
'ballsFaced': {
'name': 'ballsFaced',
'text': 'BF',
'value': '48'
},
'notouts': {
'name': 'notouts',
'text': 'Not Out',
'value': '1'
}
}
}
],
'team': {
'teamDisplayName': 'NEW ZEALAND A',
'innDisplayName': 'INNINGS',
'runs': 308,
'overs': 50,
'wickets': 6,
'description': 'complete',
'inningsRunWicket': '308/6',
'inningStatus': ''
},
'bowlers': [
{
'name': 'S Kaul',
'href': 'http://www.espncricinfo.com/ci/content/player/326017.html',
'stats': {
'overs': {
'name': 'overs',
'text': 'O',
'value': '10'
},
'conceded': {
'name': 'conceded',
'text': 'R',
'value': '74'
},
'wickets': {
'name': 'wickets',
'text': 'E',
'value': '2'
}
}
},
{
'name': 'K Gowtham',
'href': 'http://www.espncricinfo.com/ci/content/player/424377.html',
'stats': {
'overs': {
'name': 'overs',
'text': 'O',
'value': '9'
},
'conceded': {
'name': 'conceded',
'text': 'R',
'value': '46'
},
'wickets': {
'name': 'wickets',
'text': 'E',
'value': '1'
}
}
}
]
},
'2': {
'bowlers': [
{
'name': 'HK Bennett',
'href': 'http://www.espncricinfo.com/ci/content/player/226493.html',
'stats': {
'overs': {
'name': 'overs',
'text': 'O',
'value': '10'
},
'conceded': {
'name': 'conceded',
'text': 'R',
'value': '65'
},
'wickets': {
'name': 'wickets',
'text': 'E',
'value': '2'
}
}
},
{
'name': 'LH Ferguson',
'href': 'http://www.espncricinfo.com/ci/content/player/493773.html',
'stats': {
'overs': {
'name': 'overs',
'text': 'O',
'value': '10'
},
'conceded': {
'name': 'conceded',
'text': 'R',
'value': '75'
},
'wickets': {
'name': 'wickets',
'text': 'E',
'value': '2'
}
}
}
],
'batsmen': [
{
'name': 'V Shankar',
'href': 'http://www.espncricinfo.com/ci/content/player/477021.html',
'stats': {
'runs': {
'name': 'runs',
'text': 'RUNS',
'value': '87'
},
'ballsFaced': {
'name': 'ballsFaced',
'text': 'BF',
'value': '80'
},
'notouts': {
'name': 'notouts',
'text': 'Not Out',
'value': '1'
}
}
},
{
'name': 'SS Iyer',
'href': 'http://www.espncricinfo.com/ci/content/player/642519.html',
'stats': {
'runs': {
'name': 'runs',
'text': 'RUNS',
'value': '54'
},
'ballsFaced': {
'name': 'ballsFaced',
'text': 'BF',
'value': '54'
},
'notouts': {
'name': 'notouts',
'text': 'Not Out',
'value': '0'
}
}
}
],
'team': {
'teamDisplayName': 'INDIA A',
'innDisplayName': 'INNINGS',
'runs': 311,
'overs': 49,
'wickets': 6,
'description': 'target reached',
'inningsRunWicket': '311/6',
'inningStatus': ''
}
}
},
'isAvailable': True
}
}
this is my python code which converts the dictionary of json values into the pandas dataframe
df = pd.concat({k: pd.DataFrame(v) for k, v in scorecard_summary.items()})
df