The Problem
I have an API that returns mongoengine results in JSON format, but the frontend (outside of my scope) is unhappy with the format, namely that values like {"$oid": "60e4a07097ddf1e20fbaaaa0"} exist.
Here's an example response from hitting a GET endpoint:
[
{
"name": "Double R Diner",
"permissions": [
{
"user": {
"$oid": "60de35e078ba4f6f3f41e519"
},
"role": {
"name": "admin",
"description": "account creator"
}
},
{
"user": {
"$oid": "60de35e078ba4f6f3f41e516"
},
"role": {
"name": "admin",
"description": "Double R Diner admin"
}
}
],
"buildings": [
{
"$oid": "60de35e078ba4f6f3f41e51e"
}
],
"address": {
"country": "United States",
"province_state": "WA",
"city": "Twin Peaks"
},
"main_contact": {
"$oid": "60de35e078ba4f6f3f41e519"
},
"_id": {
"$oid": "60de35e078ba4f6f3f41e520"
}
}
]
The nesting of $oid fields is where the problem lies, as the frontend considers this an over-complication that locks us into this weird mongoengine format standard. This nesting should be removed, with the $oid string values replacing their nesting dict where possible. As you can see, nesting can occur within dicts (main_contact) and/or lists (buildings).
The desired outcome looks like this:
[
{
"name": "Double R Diner",
"permissions": [
{
"user": "60de35e078ba4f6f3f41e519",
"role": {
"name": "admin",
"description": "account creator"
}
},
{
"user": "60de35e078ba4f6f3f41e516",
"role": {
"name": "admin",
"description": "Double R Diner admin"
}
}
],
"buildings": [
"60de35e078ba4f6f3f41e51e"
],
"address": {
"country": "United States",
"province_state": "WA",
"city": "Twin Peaks"
},
"main_contact": "60de35e078ba4f6f3f41e519",
"_id": "60de35e078ba4f6f3f41e520"
}
]
I think this is a recursion problem, and recursion breaks my brain.
The Question
How can I best achieve the above transformation; simplifying the response data structure and allowing the front end to be unaware of mongoengine $oid dicts regardless of how deeply nested these ObjectIds are?
I'm sure there is a much better way to do this, but I seem to have solved this problem with the following implementation:
def simplify_ids(obj):
if type(obj) not in [list, dict]:
return obj
if type(obj) == dict:
if len(obj) == 1 and '$oid' in obj.keys():
obj = obj['$oid']
return obj
else:
for key, value in obj.items():
obj[key] = simplify_ids(value)
elif type(obj) == list:
for index, nested_obj in enumerate(obj):
obj[index] = simplify_ids(nested_obj)
return obj
def clean_query_results(query_results):
if type(query_results) != QuerySet:
raise TypeError(f'Input must be a mongoengine QuerySet, not {type(query_results)}')
result_string = query_results.to_json()
loaded = json.loads(result_string) # List
cleaned = list()
for obj in loaded:
cleaned.append(simplify_ids(obj))
return json.dumps(cleaned)
The calling endpoint needs only to change the Response line from:
return Response(query_results.to_json(), mimetype='application/json', status=200)
to:
return Response(clean_query_results(query_results), mimetype='application/json', status=200)
Related
Trying to update JSON attribute value in line using JPATH.
Trying a solution in Python but also assessing Snowpark alternative(assuming data loaded in a table in variant column).
Python code is working for Objects but failing in case of arrays involved.
Python code:
json={
"ID": "1",
"NAME": { "FIRST_NAME": "ABC", "LAST_NAME": "XYZ" },
"ADDR": [
{ "TYPE": "HOME", "ADDR_L1": "SDGSG", "CITY": "AFAFA" },
{ "TYPE": "OFFC", "ADDR_L1": "AFASF", "CITY": "SDGSDG" }
],
"CONTACT": { "CONTACTS": [{ "TYPE": "A" }, { "TYPE": "B" }, { "TYPE": "C" }] },
"LEVEL1OBJ": {
"LEVEL2ARR": [{ "LEVEL3OBJ": "A" }, { "LEVEL3OBJ": "B" }],
"LEVEL2ARR_1":[{"LEVEL3ARR":[{"LEVEL4OBJ":"A"},{"LEVEL4OBJ":"B"}]},{"LEVEL3ARR":[{"LEVEL4OBJ":"C"},{"LEVEL4OBJ":"D"}]}],
"LEVEL2OBJ": "GFDB"
}
}
#Below input works
#keys=['NAME','FIRST_NAME']
#Below doesnt work
#keys=['ADDR','0','ADDR_L1']
#keys=['CONTACT','CONTACTS','0','TYPE']
keys=['LEVEL1OBJ','LEVEL2ARR_1','0','LEVEL3ARR','0','LEVEL4OBJ']
from functools import reduce
import operator
def get_by_path(root, items):
return reduce(operator.getitem, items, root)
def set_by_path(root, items, value):
get_by_path(root, items[:-1])[items[-1]] = value
set_by_path(json,keys,'')
print(json)
Does anyone have had experience with this?
What could be the code in Snowpark?
I have a python dictionary that looks something like this:
[
{
"timestamp": 1621559698154,
"user": {
"uri": "spotify:user:xxxxxxxxxxxxxxxxxxxx",
"name": "Panda",
"imageUrl": "https://i.scdn.co/image/ab67757000003b82b54c68ed19f1047912529ef4"
},
"track": {
"uri": "spotify:track:6SJSOont5dooK2IXQoolNQ",
"name": "Dirty",
"imageUrl": "http://i.scdn.co/image/ab67616d0000b273a36e3d46e406deebdd5eafb0",
"album": {
"uri": "spotify:album:0NMpswZbEcswI3OIe6ml3Y",
"name": "Dirty (Live)"
},
"artist": {
"uri": "spotify:artist:4ZgQDCtRqZlhLswVS6MHN4",
"name": "grandson"
},
"context": {
"uri": "spotify:artist:4ZgQDCtRqZlhLswVS6MHN4",
"name": "grandson",
"index": 0
}
}
},
{
"timestamp": 1621816159299,
"user": {
"uri": "spotify:user:xxxxxxxxxxxxxxxxxxxxxxxx",
"name": "maja",
"imageUrl": "https://i.scdn.co/image/ab67757000003b8286459151d5426f5a9e77cfee"
},
"track": {
"uri": "spotify:track:172rW45GEnGoJUuWfm1drt",
"name": "Your Best American Girl",
"imageUrl": "http://i.scdn.co/image/ab67616d0000b27351630f0f26aff5bbf9e10835",
"album": {
"uri": "spotify:album:16i5KnBjWgUtwOO7sVMnJB",
"name": "Puberty 2"
},
"artist": {
"uri": "spotify:artist:2uYWxilOVlUdk4oV9DvwqK",
"name": "Mitski"
},
"context": {
"uri": "spotify:playlist:0tol7yRYYfiPJ17BuJQKu2",
"name": "I Bet on Losing Dogs",
"index": 0
}
}
}
]
How can I get, for example, the group of values for user.name "Panda" and then get that specific "track" list? I can't parse through the list by index because the list order changes randomly.
If you are only looking for "Panda", then you can just loop over the list, check whether the name is "Panda", and then retrieve the track list accordingly.
Otherwise, that would be inefficient if you want to do that for many different users. I would first make a dict that maps user to its index in the list, and then use that for each user (I am assuming that the list does not get modified while you execute the code, although it can be modified between executions.)
user_to_id = {data[i]['user']['name']: i for i in range(len(data))} # {'Panda': 0, 'maja': 1}
def get_track(user):
return data[user_to_id[user]]['track']
print(get_track('maja'))
print(get_track('Panda'))
where data is the list you provided.
Or, perhaps just make a dictionary of tracks directly:
tracks = {item['user']['name']: item['track'] for item in data}
print(tracks['Panda'])
If you want to get list of tracks for user Panda:
tracks = [entry['track'] for entry in data if entry['user']['name'] == 'Panda']
I want to exchange 2 json data's value. But keys of these datas are different from each other. I don't know how can I exchange data value between them.
sample json1: A
{
"contact_person":"Mahmut Kapur",
"contact_people": [
{
"email": "m#gmail.com",
"last_name": "Kapur"
}
],
"addresses": [
{
"city": "istanbul",
"country": "CA",
"first_name": "Mahmut",
"street1": "adres 1",
"zipcode": "34678",
"id": "5f61f72b8348230004f149fd"
}
]
"created_at": "2020-09-16T07:29:47.244-04:00",
"updated_at": "2020-09-16T07:32:50.567-04:00",
}
sample json2: B
The values in this example are: Represents the keys in the A json.
{
"Customer":{
"DisplayName":"contact_person",
"PrimaryEmailAddr":{
"Address":"contact_people/email"
},
"FamilyName":"contact_people/last_name",
"BillAddr":{
"City":"addresses/city",
"CountrySubDivisionCode":"addresses/country",
"Line1":"addresses/street1",
"PostalCode":"addresses/zipcode",
"Id":"addresses/id"
},
"GivenName":"addresses/first_name",
"MetaData":{
"CreateTime":"created_at",
"LastUpdatedTime":"updated_at"
}
}
}
The outcome needs to be:
{
"Customer":{
"DisplayName":"Mahmut Kapur",
"PrimaryEmailAddr":{
"Address":"m#gmail.com"
},
"FamilyName":"Kapur",
"BillAddr":{
"City":"istanbul",
"CountrySubDivisionCode":"CA",
"Line1":"adres 1",
"PostalCode":"34678",
"Id":"5f61f72b8348230004f149fd"
},
"GivenName":"Mahmut",
"MetaData":{
"CreateTime":"2020-09-16T07:29:47.244-04:00",
"LastUpdatedTime":"2020-09-16T07:32:50.567-04:00"
}
}
}
So the important thing here is to match the keys. I hope I was able to explain my problem.
This code can do the work for you. I dont know if someone can make this code shorter for you. It basically searches for dict and list till the leaf level and acts accordingly.
a={
"contact_person":"Mahmut Kapur",
"contact_people": [
{
"email": "m#gmail.com",
"last_name": "Kapur"
}
],
"addresses": [
{
"city": "istanbul",
"country": "CA",
"first_name": "Mahmut",
"street1": "adres 1",
"zipcode": "34678",
"id": "5f61f72b8348230004f149fd"
}
],
"created_at": "2020-09-16T07:29:47.244-04:00",
"updated_at": "2020-09-16T07:32:50.567-04:00",
}
b={
"Customer":{
"DisplayName":"contact_person",
"PrimaryEmailAddr":{
"Address":"contact_people/email"
},
"FamilyName":"contact_people/last_name",
"BillAddr":{
"City":"addresses/city",
"CountrySubDivisionCode":"addresses/country",
"Line1":"addresses/street1",
"PostalCode":"addresses/zipcode",
"Id":"addresses/id"
},
"GivenName":"addresses/first_name",
"MetaData":{
"CreateTime":"created_at",
"LastUpdatedTime":"updated_at"
}
}
}
c={}
for keys in b:
if isinstance(b[keys], dict):
for items in b[keys]:
if isinstance(b[keys][items], dict):
for leaf in b[keys][items]:
if "/" in b[keys][items][leaf]:
getter=b[keys][items][leaf].split("/")
b[keys][items][leaf]=a[getter[0]][0][getter[1]]
else:
b[keys][items][leaf]=a[b[keys][items][leaf]]
else:
if "/" in b[keys][items]:
getter=b[keys][items].split("/")
b[keys][items]=a[getter[0]][0][getter[1]]
else:
b[keys][items]=a[b[keys][items]]
else:
if "/" in b[keys]:
getter=b[keys].split("/")
b[keys]=a[getter[0]][0][getter[1]]
else:
b[keys]=a[b[keys]]
print(json.dumps(b,indent=4))
I have some data from a project where the variables can change from motorcycle and car. I need to get the name out of them and that value is inside the variable.
This is not the data i will be using but it has the same structure, the "official" data is some persional information so i changed it to some random values. I can not change the structure of the JSON data since this is the way the serveradmins decided to structure it for some reason.
This is my python code:
import json
with open('exampleData.json') as j:
data = json.load(j)
name = 0
Vehicle = 0
for x in data:
print(data['persons'][x]['name'])
for i in data['persons'][x]['things']["Vehicles"]:
print(data['persons'][x]['things']['Vehicles'][i]['type']['name'])
print("\n")
This is my Json data i extracted from the file "ExampleData.json"(sorry for long but it is kinda complex and necessary to understand the problem):
{
"total": 2,
"persons": [
{
"name": "Sven Svensson",
"things": {
"House": "apartment",
"Vehicles": [
{
"id": "46",
"type": {
"name": "Kawasaki ER6N",
"type": "motorcyle"
},
"Motorcycle": {
"plate": "aaa111",
"fields": {
"brand": "Kawasaki",
"status": "in shop"
}
}
},
{
"id": "44",
"type": {
"name": "BMW m3",
"type": "Car"
},
"Car": {
"plate": "bbb222",
"fields": {
"brand": "BMW",
"status": "in garage"
}
}
}
]
}
},
{
"name": "Eric Vivian Matthews",
"things": {
"House": "House",
"Vehicles": [
{
"id": "44",
"type": {
"name": "Volvo XC90",
"type": "Car"
},
"Car": {
"plate": "bbb222",
"fields": {
"brand": "Volvo",
"status": "in garage"
}
}
}
]
}
}
]
}
I want it to print out something like this :
Sven Svensson
Bmw M3
Kawasaki ER6n
Eric Vivian Matthews
Volvo XC90
but i get this error:
print(data['persons'][x]['name'])
TypeError: list indices must be integers or slices, not str
Process finished with exit code 1
What you need is
for person in data["persons"]:
for vehicle in person["things"]["vehicles"]:
print(vehicle["type"]["name"])
type = vehicle["type"]["type"]
print(vehicle[type]["plate"])
Python for loop does not return the key but rather an object here:
for x in data:
Referencing an object as key
print(data['persons'][x]['name'])
Is causing the error
What you need is to use the returning json object and iterate over them like so:
for x in data['persons']:
print(x['name'])
for vehicle in x['things']['Vehicles']:
print(vehicle['type']['name'])
print('\n')
I am trying to retain the whole contents of a nested dictionary but only with its contents reordered..
This is an example of my nested dictionaries (pardon the long example..) -
{
"pages": {
"rotatingTest": {
"elements": {
"apvfafwkbnjn2bjt": {
"name": "animRot_tilt40_v001",
"data": {
"description": "tilt testing",
"project": "TEST",
"created": "26/11/18 16:32",
},
"type": "AnimWidget",
"uid": "apvfafwkbnjn2bjt"
},
"p0pkje1hjcc9jukq": {
"name": "poseRot_positionD_v003",
"data": {
"description": "posing test for positionD",
"created": "10/01/18 14:16",
"project": "TEST",
},
"type": "PosedWidget",
"uid": "p0pkje1hjcc9jukq"
},
"k1gzzc5uy1ynqtnj": {
"name": "animRot_positionH_v001",
"data": {
"description": "rotational posing test for positionH",
"created": "13/06/18 14:19",
"project": "TEST",
},
"type": "AnimWidget",
"uid": "k1gzzc5uy1ynqtnj"
}
}
},
"panningTest": {
"elements": {
"7lyuri8g8u5ctwsa": {
"name": "posePan_positionZ_v001",
"data": {
"description": "panning test for posZ",
"created": "04/10/18 12:43",
"project": "TEST",
},
"type": "PosedWidget",
"uid": "7lyuri8g8u5ctwsa"
}
}
},
"zoomingTest": {
"elements": {
"prtn0i6ehudhz475": {
"name": "posZoom_positionH_v010",
"data": {
"description": "zoom test",
"created": "11/10/18 12:42",
"project": "TEST",
},
"type": "PosedWidget",
"uid": "prtn0i6ehudhz475"
}
}
}
},
"page_order": [
"rotatingTest",
"zoomingTest",
"panningTest"
]
}
and this is my code:
for k1, v1 in test_dict.get('pages', {}).items():
return (sorted(v1.get('elements').items(), key=lambda (k2,v2): v2['data']['created']))
In the code, keys such as the page_order, pages etc are missing...
Or if there is/ are any commands where it will enables me to retain the 'whole' of the dictionary?
Appreciate in advance for any advice.
If you're using Python 3.7, a dict will preserve insert order. Otherwise, you need to use an OrderedDict.Additionally, you need to convert the date string to a date to get the correct sort order:
from datetime import datetime
def sortedPage(d):
return {k: {'elements': dict(sorted(list(v['elements'].items()), key=lambda tuple: datetime.strptime(tuple[1]['data']['created'], '%d/%m/%y %H:%M')))} for k,v in d.items()}
output = {k: sortedPage(v) if k == 'pages' else v for k,v in input.items()}