I want to merge list of dictionary provided below with unique channel and zrepcode.
sample input:
[
{
"channel": 1,
"zrepcode": "123456",
"turn": 7833.9
},
{
"channel": 1,
"zrepcode": "123456",
"pipeline": 324
},
{
"channel": 1,
"zrepcode": "123456",
"inv_bal": 941.16
},
{
"channel": 1,
"zrepcode": "123456",
"display": 341
},
{
"channel": 3,
"zrepcode": "123456",
"display": 941.16
},
{
"channel": 3,
"zrepcode": "123456",
"turn": 7935.01
},
{
"channel": 3,
"zrepcode": "123456",
"pipeline": 0
},
{
"channel": 3,
"zrepcode": "123456",
"inv_bal": 341
},
{
"channel": 3,
"zrepcode": "789789",
"display": 941.16
},
{
"channel": 3,
"zrepcode": "789789",
"turn": 7935.01
},
{
"channel": 3,
"zrepcode": "789789",
"pipeline": 0
},
{
"channel": 3,
"zrepcode": "789789",
"inv_bal": 341
}
]
Sample output:
[
{'channel': 1, 'zrepcode': '123456', 'turn': 7833.9, 'pipeline': 324.0,'display': 341,'inv_bal': 941.16},
{'channel': 3, 'zrepcode': '123456', 'turn': 7935.01, 'pipeline': 0.0, 'display': 941.16, 'inv_bal': 341.0},
{'channel': 3, 'zrepcode': '789789', 'turn': 7935.01, 'pipeline': 0.0, 'display': 941.16, 'inv_bal': 341.0}
]
Easily solved with our good friend collections.defaultdict:
import collections
by_key = collections.defaultdict(dict)
for datum in data: # data is the list of dicts from the post
key = (datum.get("channel"), datum.get("zrepcode")) # form the key tuple
by_key[key].update(datum) # update the defaultdict by the key tuple
print(list(by_key.values()))
This outputs
[
{'channel': 1, 'zrepcode': '123456', 'turn': 7833.9, 'pipeline': 324, 'inv_bal': 941.16, 'display': 341},
{'channel': 3, 'zrepcode': '123456', 'display': 941.16, 'turn': 7935.01, 'pipeline': 0, 'inv_bal': 341},
{'channel': 3, 'zrepcode': '789789', 'display': 941.16, 'turn': 7935.01, 'pipeline': 0, 'inv_bal': 341},
]
Related
From a SQL stored procedure that performs a join on 3 tables I get the data below.
data = [
{"service_order_number": "ABC", "item_id": 0, "ticket_id": 10},
{"service_order_number": "ABC", "item_id": 0, "ticket_id": 11},
{"service_order_number": "ABC", "item_id": 1, "ticket_id": 12},
{"service_order_number": "DEF", "item_id": 3, "ticket_id": 13},
{"service_order_number": "DEF", "item_id": 3, "ticket_id": 14},
{"service_order_number": "DEF", "item_id": 3, "ticket_id": 15}]
I would like to group the data on service_order_number and item_id to return a list of dicts like below.
[
{
"service_order_number": "ABC",
"line_items": [
{
"item_id": 0,
"tickets": [
{
"ticket_id": 10
},
{
"ticket_id": 11
}
]
},
{
"item_id": 1,
"tickets": [
{
"ticket_id": 12
}
]
}
]
},
{
"service_order_number": "DEF",
"line_items": [
{
"item_id": 3,
"tickets": [
{
"ticket_id": 13
},
{
"ticket_id": 14
},
{
"ticket_id": 15
}
]
}
]
}
]
The hierarchy would be service_order_number > item_id > ticket_id
Is there an easy way to convert this data into my desired structure?
Here is a possible solution using defaultdict()
import json
from collections import defaultdict
my_dict = defaultdict(lambda: defaultdict(list))
for item in data:
(my_dict[item['service_order_number']][item['item_id']]
.append({'ticket_id': item['ticket_id']}))
res = [{'service_order_number': service_order_number,
'line_items': [{'item_id': item_id, 'tickets': tickets}
for item_id, tickets in item_group.items()]}
for service_order_number, item_group in my_dict.items()
]
print(json.dumps(res, indent=1, default=int))
[
{
"service_order_number": "ABC",
"line_items": [
{
"item_id": 0,
"tickets": [
{
"ticket_id": 10
},
{
"ticket_id": 11
}
]
},
{
"item_id": 1,
"tickets": [
{
"ticket_id": 12
}
]
}
]
},
{
"service_order_number": "DEF",
"line_items": [
{
"item_id": 3,
"tickets": [
{
"ticket_id": 13
},
{
"ticket_id": 14
},
{
"ticket_id": 15
}
]
}
]
}
]
I have dictionary which is below
{
"aggregations": {
"A": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{ "key": "ADL", "doc_count": 1 },
{ "key": "SDD", "doc_count": 1 },
{ "key": "JJD", "doc_count": 1 }
]
},
"B": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{ "key": "ABC", "doc_count": 1 },
{ "key": "CDE", "doc_count": 1 },
{ "key": "FGH", "doc_count": 1 }
]
},
"C": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{ "key": "XYX", "doc_count": 1 },
{ "key": "NXS", "doc_count": 1 }
]
}
}
}
aggregations.keys will be aggregationfilters.fieldName
aggregations.buckets.key will be aggregationfilters.values.title
aggregationfilters.values.paragraph is null everytime
aggregations.buckets.doc_count will be aggregationfilters.values.count
Basically I need to extract aggregations.keys and aggregations.bucket values and put into different dictionary.
Need to write a general code structure to do that.
I cannot do with .pop(rename) the dictioanry
My expected out
{
"aggregationfilters": [
{
"name": "ABC",
"fieldName": "A",
"values": [
{ "title": "ADL", "paragraph": null, "count": 1 },
{ "title": "SDD", "paragraph": null, "count": 1 },
{ "title": "JJD", "paragraph": null, "count": 1 }
]
}, {
"name": "CDE",
"fieldName": "B",
"values": [
{ "title": "ABC", "paragraph": null, "count": 1 },
{ "title": "CDE", "paragraph": null, "count": 1 },
{ "title": "FGH", "paragraph": null, "count": 1 }
]
}, {
"name": "FGH",
"fieldName": "C",
"values": [
{ "title": "XYX", "paragraph": null, "count": 1 },
{ "title": "NXS", "paragraph": null, "count": 1 }
]
}
]
}
Well, this works, but even with my best effort this still doesn't look that clean.
import json
source = {
"aggregations": {
"A": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{"key": "ADL", "doc_count": 1},
{"key": "SDD", "doc_count": 1},
{"key": "JJD", "doc_count": 1},
],
},
"B": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{"key": "ABC", "doc_count": 1},
{"key": "CDE", "doc_count": 1},
{"key": "FGH", "doc_count": 1},
],
},
"C": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [{"key": "XYX", "doc_count": 1}, {"key": "NXS", "doc_count": 1}],
},
}
}
convert_map = {
"buckets": "values",
"doc_count": "count",
"key": "title",
}
remove_map = {"sum_other_doc_count", "doc_count_error_upper_bound"}
add_map = {"name": "Changed VAL_", "fieldName": "VAL_"}
def converting_generator(
source_: dict, convert_map_: dict, remove_map_: set, add_map_: dict
):
working_dict = {k: v for k, v in source_.items()}
variable_identifier = "VAL_"
for key, inner_dic in working_dict.items():
inner_dic: dict
for rm_key in remove_map_:
try:
inner_dic.pop(rm_key)
except KeyError:
pass
for add_key, add_val in add_map_.items():
inner_dic[add_key] = add_val.replace(variable_identifier, key)
dumped = json.dumps(inner_dic, indent=2)
for original, target in convert_map_.items():
dumped = dumped.replace(original, target)
yield json.loads(dumped)
converted = {
"aggregation_filters": list(
converting_generator(source["aggregations"], convert_map, remove_map, add_map)
)
}
for inner_dict in converted["aggregation_filters"]:
for even_inner_dict in inner_dict["values"]:
even_inner_dict["paragraph"] = None
print(json.dumps(converted, indent=2))
Output:
{
"aggregation_filters": [
{
"values": [
{
"title": "ADL",
"count": 1,
"paragraph": null
},
{
"title": "SDD",
"count": 1,
"paragraph": null
},
{
"title": "JJD",
"count": 1,
"paragraph": null
}
],
"name": "Changed A",
"fieldName": "A"
},
{
"values": [
{
"title": "ABC",
"count": 1,
"paragraph": null
},
{
"title": "CDE",
"count": 1,
"paragraph": null
},
{
"title": "FGH",
"count": 1,
"paragraph": null
}
],
"name": "Changed B",
"fieldName": "B"
},
{
"values": [
{
"title": "XYX",
"count": 1,
"paragraph": null
},
{
"title": "NXS",
"count": 1,
"paragraph": null
}
],
"name": "Changed C",
"fieldName": "C"
}
]
}
Always show your code, would be nice if that's a working one - to show that you've put at least that worth of the effort on your problem.
I don't bother it as this feels like puzzle solving, but others may not.
I have 2 lists, looking like:
temp_data:
{
"id": 1,
"name": "test (replaced)",
"code": "test",
"last_update": "2020-01-01",
"online": false,
"data": {
"temperature": [
{
"date": "2019-12-17",
"value": 23.652905748126333
},
...
]}
hum_data:
{
"id": 1,
"name": "test (replaced)",
"code": "test",
"last_update": "2020-01-01",
"online": false,
"data": {
"humidity": [
{
"date": "2019-12-17",
"value": 23.652905748126333
},
...
]}
I need to merge the 2 lists to 1 without duplicating data. What is the easiest/efficient way? After merging, I want something like this:
{
"id": 1,
"name": "test",
"code": "test",
"last_update": "2020-01-01",
"online": false,
"data": {
"temperature": [
{
"date": "2019-12-17",
"value": 23.652905748126333
},
...
],
"humidity": [
{
"date": "2019-12-17",
"value": 23.652905748126333
},
...
Thanks for helping.
If your lists hum_data and temp_data are not sorted then first sort them and then concatenate the dictionaries pair-wise.
# To make comparisons for sorting
compare_function = lambda value : value['id']
# sort arrays before to make later concatenation easier
temp_data.sort(key=compare_function)
hum_data.sort(key=compare_function)
combined_data = temp_data.copy()
# concatenate the dictionries using the update function
for hum_row, combined_row in zip(hum_data, combined_data):
combined_row['data'].update(hum_row['data'])
# combined hum_data and temp_data
combined_data
If the lists are already sorted then you just need to concatenate dictionary by dictionary.
combined_data = temp_data.copy()
# concatenate the dictionries using the update function
for hum_row, combined_row in zip(hum_data, combined_data):
combined_row['data'].update(hum_row['data'])
# combined hum_data and temp_data
combined_data
With that code I got the following result:
[
{
'id': 1,
'name': 'test (replaced)',
'code': 'test',
'last_update': '2020-01-01',
'online': False,
'data': {
'temperature': [{'date': '2019-12-17', 'value': 1}],
'humidity': [{'date': '2019-12-17', 'value': 1}]}
},
{
'id': 2,
'name': 'test (replaced)',
'code': 'test',
'last_update': '2020-01-01',
'online': False,
'data': {
'temperature': [{'date': '2019-12-17', 'value': 2}],
'humidity': [{'date': '2019-12-17', 'value': 2}]}
}
]
I have a json array as
{
"query": {
"bool": {
"must": [],
"should": [
{
"match": {
"Name": {
"query": "Nametest",
"fuzziness": 3,
"boost": 5
}
}
},
{
"match": {
"Address": {
"query": "NONE",
"fuzziness": 3,
"boost": 4
}
}
},
{
"match": {
"Site": {
"query": "Adeswfvfv",
"fuzziness": 3,
"boost": 4
}
}
},
{
"match": {
"Phone": {
"query": "5680728.00",
"fuzziness": 2,
"boost": 4
}
}
}
],
"minimum_should_match": 2
}
}
}
So What i wanna do is if In json['query']['bool']['should'] if "query" is "NONE" then I wanna remove that json array and the new json will be
{
"query": {
"bool": {
"must": [],
"should": [
{
"match": {
"Name": {
"query": "Nametest",
"fuzziness": 3,
"boost": 5
}
}
},
{
"match": {
"Site": {
"query": "Adeswfvfv",
"fuzziness": 3,
"boost": 4
}
}
},
{
"match": {
"Phone": {
"query": "5680728.00",
"fuzziness": 2,
"boost": 4
}
}
}
],
"minimum_should_match": 2
}
}
}
I have tried iterating over the json and used del(jsonarray) and pop(jsonarray) but nothing seeems to help out?
tried with python json library but failed
for e in q['query']['bool']['should']:
... if "NONE" in str(e['match']):
... del(e)
This should help.
import pprint
d = {'query': {'bool': {'minimum_should_match': 2, 'should': [{'match': {'Name': {'query': 'Nametest', 'boost': 5, 'fuzziness': 3}}}, {'match': {'Address': {'query': 'NONE', 'boost': 4, 'fuzziness': 3}}}, {'match': {'Site': {'query': 'Adeswfvfv', 'boost': 4, 'fuzziness': 3}}}, {'match': {'Phone': {'query': '5680728.00', 'boost': 4, 'fuzziness': 2}}}], 'must': []}}}
d["query"]['bool']['should'] = [i for i in d["query"]['bool']['should'] if list(i['match'].items())[0][1]["query"] != 'NONE']
pprint.pprint(d)
Output:
{'query': {'bool': {'minimum_should_match': 2,
'must': [],
'should': [{'match': {'Name': {'boost': 5,
'fuzziness': 3,
'query': 'Nametest'}}},
{'match': {'Site': {'boost': 4,
'fuzziness': 3,
'query': 'Adeswfvfv'}}},
{'match': {'Phone': {'boost': 4,
'fuzziness': 2,
'query': '5680728.00'}}}]}}}
I write this,but this seems complex
for p,c in enumerate(json['query']['bool']['should']):
if list(c["match"].values())[0]["query"] == "NONE":
json['query']['bool']['should'].pop(p)
print(json)
I have a list of dictionary and I want to get only a specific item from each dictionary. My data pattern is:
data = [
{
"_id": "uuid",
"_index": "my_index",
"_score": 1,
"_source": {
"id" : 1,
"price": 100
}
},
{
"_id": "uuid",
"_index": "my_index",
"_score": 1,
"_source": {
"id" : 2,
"price": 150
}
},
{
"_id": "uuid",
"_index": "my_index",
"_score": 1,
"_source": {
"id" : 3,
"price": 90
}
}
]
My desired output:
formatted_data = [
{
"id": 1,
"price": 100
},
{
"id": 2,
"price": 150
},
{
"id": 3,
"price": 90
}
]
To formate data I have used iteration (for) like
formatted_data = []
for item in data:
formatted_data.append(item['_source'])
In PHP I can use array_column() instead of for loop. So what will be the alternative of for in python3 in my case?
Thanks in advance.
You can use list comprehension:
In [11]: [e['_source'] for e in data]
Out[11]: [{'id': 1, 'price': 100}, {'id': 2, 'price': 150}, {'id': 3, 'price': 90}]