Related
I just learnt django and I am getting data from api and looping through the json and appending the data into the list. but When I use .map() function in react then the data is appending in list (from for loop) like
[
{
"results": {
"id": 544,
"name": "User_1",
}
},
{
"results": {
"id": 218,
"name": "User_2",
}
},
{
"results": {
"id": 8948,
"name": "User_3",
}
},
{
"results": {
"id": 9,
"name": "User_4",
}
},
]
It is not appending like (Like I want)
[
results : [
{
"id": 544,
"name": "User_1"
},
{
"id": 218,
"name": "User_2"
},
{
"id": 8948,
"name": "User_3"
},
{
"id": 9,
"name": "User_4"
}
],
"length_of_results": 25,
]
views.py
def extract_view(request):
results_list = []
// api url for explanation only
get_response = "https://api.punkapi.com/v2/beers"
if get_response.status_code == 200:
for result in get_response.json():
results_list.append({"results": result})
results_list.append({"length_of_results": len(results_list)})
return Response({"data": results_list})
I know, In for loop it is appending every result within it with every iteration but I also want to assign all the responses within results list. Because I will add a append another field after for loop.
I have tried many times but it is still not working.
You can solve it by map function iterating over list:
dict(results=list(map(lambda x: x["results"], response)))
Full working example:
response = [
{
"results": {
"id": 544,
"name": "User_1",
}
},
{
"results": {
"id": 218,
"name": "User_2",
}
},
{
"results": {
"id": 8948,
"name": "User_3",
}
},
{
"results": {
"id": 9,
"name": "User_4",
}
},
]
results = dict(results=list(map(lambda x: x["results"], response)))
results["length_of_results"] = len(results["results"])
>> {'results': [{'id': 544, 'name': 'User_1'},
>> {'id': 218, 'name': 'User_2'},
>> {'id': 8948, 'name': 'User_3'},
>> {'id': 9, 'name': 'User_4'}],
>> 'length_of_results': 4}
By doing
results_list.append({"results": result})
you are creating a new dictionary with the value being result which I believe is a dictionary itself. So you should be able to just do this:
if get_response.status_code == 200:
for result in get_response.json():
results_list.append(result)
I am calling an API and getting a response like the below.
{
"status": 200,
"errmsg": "OK",
"data": {
"total": 12,
"items": [{
"id": 11,
"name": "BBC",
"priority": 4,
"levelStr": "All",
"escalatingChainId": 3,
"escalatingChain": {
"inAlerting": false,
"throttlingAlerts": 20,
"enableThrottling": true,
"name": "Example123",
"destination": [],
"description": "",
"ccdestination": [],
"id": 3,
"throttlingPeriod": 10
}
},
{
"id": 21,
"name": "CNBC",
"priority": 4,
"levelStr": "All",
"escalatingChainId": 3,
"escalatingChain": {
"inAlerting": false,
"throttlingAlerts": 20,
"enableThrottling": true,
"name": "Example456",
"destination": [],
"description": "",
"ccdestination": [],
"id": 3,
"throttlingPeriod": 10
}
}
]
}
}
I need to clean-up this JSON a bit and produce a simple JSON like below where escalatingChainName is the name in the escalatingChain list so that I can write this into a CSV file.
{
"items": [{
"id": 11,
"name": "BBC",
"priority": 4,
"levelStr": "All",
"escalatingChainId": 3,
"escalatingChainName": "Example123"
},
{
"id": 21,
"name": "CNBC",
"priority": 4,
"levelStr": "All",
"escalatingChainId": 3,
"escalatingChainName": "Example456"
}
]
}
Is there a JSON function that I can use to copy only the necessary key-value or nested key-values to a new JSON object?
With the below code, I am able to get the details list.
json_response = response.json()
items = json_response['data']
details = items['items']
I can print individual list items using
for x in details:
print(x)
How do I take it from here to pull only the necessary fields like id, name, priority and the name from escalatingchain to create a new list or JSON?
There is no existing function that will do what you want, so you'll need to write one. Fortunately that's not too hard in this case — basically you just create a list of new items by extracting the pieces of data you want from the existing ones.
import json
json_response = """\
{
"status": 200,
"errmsg": "OK",
"data": {
"total": 12,
"items": [{
"id": 11,
"name": "BBC",
"priority": 4,
"levelStr": "All",
"escalatingChainId": 3,
"escalatingChain": {
"inAlerting": false,
"throttlingAlerts": 20,
"enableThrottling": true,
"name": "Example123",
"destination": [],
"description": "",
"ccdestination": [],
"id": 3,
"throttlingPeriod": 10
}
},
{
"id": 21,
"name": "CNBC",
"priority": 4,
"levelStr": "All",
"escalatingChainId": 3,
"escalatingChain": {
"inAlerting": false,
"throttlingAlerts": 20,
"enableThrottling": true,
"name": "Example456",
"destination": [],
"description": "",
"ccdestination": [],
"id": 3,
"throttlingPeriod": 10
}
}
]
}
}
"""
response = json.loads(json_response)
cleaned = []
for item in response['data']['items']:
cleaned.append({'id': item['id'],
'name': item['name'],
'priority': item['priority'],
'levelStr': item['levelStr'],
'escalatingChainId': item['escalatingChainId'],
'escalatingChainName': item['escalatingChain']['name']})
print('cleaned:')
print(json.dumps(cleaned, indent=4))
You can try:
data = {
"status": 200,
"errmsg": "OK",
"data": {
"total": 12,
"items": [{
"id": 11,
"name": "BBC",
"priority": 4,
"levelStr": "All",
"escalatingChainId": 3,
"escalatingChain": {
"inAlerting": False,
"throttlingAlerts": 20,
"enableThrottling": True,
"name": "Example123",
"destination": [],
"description": "",
"ccdestination": [],
"id": 3,
"throttlingPeriod": 10
}
},
{
"id": 21,
"name": "CNBC",
"priority": 4,
"levelStr": "All",
"escalatingChainId": 3,
"escalatingChain": {
"inAlerting": False,
"throttlingAlerts": 20,
"enableThrottling": True,
"name": "Example456",
"destination": [],
"description": "",
"ccdestination": [],
"id": 3,
"throttlingPeriod": 10
}
}
]
}
}
for single_item in data["data"]["items"]:
print(single_item["id"])
print(single_item["name"])
print(single_item["priority"])
print(single_item["levelStr"])
print(single_item["escalatingChain"]["inAlerting"])
# and so on
Two ways of approaching this depending on whether your dealing with a variable or .json file using python list and dictionary comprehension:
Where data variable of type dictionary (nested) already defined:
# keys you want
to_keep = ['id', 'name', 'priority', 'levelStr', 'escalatingChainId',
'escalatingChainName']
new_data = [{k:v for k,v in low_dict.items() if k in to_keep}
for low_dict in data['data']['items']]
# where item is dictionary at lowest level
escalations = [{v+'Name':k[v]['name']} for k in data['data']['items']
for v in k if type(k[v])==dict]
# merge both lists of python dictionaries to produce flattened list of dictionaries
new_data = [{**new,**escl} for new,escl in zip(new_data,escalations)]
Or (and since your refer json package) if you have save the response to as a .json file:
import json
with open('response.json', 'r') as handl:
data = json.load(handl)
to_keep = ['id', 'name', 'priority', 'levelStr', 'escalatingChainId',
'escalatingChainName']
new_data = [{k:v for k,v in low_dict.items() if k in to_keep}
for low_dict in data['data']['items']]
escalations = [{v+'Name':k[v]['name']} for k in data['data']['items']
for v in k if type(k[v])==dict]
new_data = [{**new,**escl} for new,escl in zip(new_data,escalations)]
Both produce output:
[{'id': 11,
'name': 'BBC',
'priority': 4,
'levelStr': 'All',
'escalatingChainId': 3,
'escalatingChainName': 'Example123'},
{'id': 21,
'name': 'CNBC',
'priority': 4,
'levelStr': 'All',
'escalatingChainId': 3,
'escalatingChainName': 'Example456'}]
I have a dataframe like below, where each 'level' drills down into more detail, with the last level having an id value.
data = [
{'id': 1, 'level_1': 'Animals', 'level_2': 'Carnivores', 'level_3': 'Felidae', 'level_4', 'Siamese Cat'},
{'id': 2, 'level_1': 'Animals', 'level_2': 'Carnivores', 'level_3': 'Felidae', 'level_4', 'Javanese Cat'},
{'id': 3, 'level_1': 'Animals', 'level_2': 'Carnivores', 'level_3': 'Ursidae', 'level_4', 'Polar Bear'},
{'id': 4, 'level_1': 'Animals', 'level_2': 'Carnivores', 'level_3': 'Canidae', 'level_4', 'Labradore Retriever'},
{'id': 5, 'level_1': 'Animals', 'level_2': 'Carnivores', 'level_3': 'Canidae', 'level_4', 'Golden Retriever'}
]
I want to turn this into a nested dictionary of parent / child relationships like below.
var data = {
"name": "Animals",
"children": [
{
"name": "Carnivores",
"children": [
{
"name": "Felidae",
"children": [
{
"id": 1,
"name": "Siamese Cat",
"children": []
},
{
"id": 2,
"name": "Javanese Cat",
"children": []
}
]
},
{
"name": "Ursidae",
"children": [
{
"id": 3,
"name": "Polar Bear",
"children": []
}
]
},
{
"name": "Canidae",
"children": [
{
"id": 4,
"name": "Labradore Retriever",
"children": []
},
{
"id": 5,
"name": "Golden Retriever",
"children": []
}
]
}
]
}
]
}
I've tried several approaches of grouping the dataframe and also looping over individual rows, but haven't been able to find a working solution yet. Any help would be greatly appreciated!
The answer of #Timus mimics your intention, however you might encounter some difficulties searching this dictionary as each level has a key name and a key children. If this is what you intended ignore my answer. However, if you would like to create a dictionary in which you can more easily search through unique keys you can try:
df = df.set_index(['level_1', 'level_2', 'level_3', 'level_4'])
def make_dictionary(df):
if df.index.nlevels == 1:
return df.to_dict()
dictionary = {}
for key in df.index.get_level_values(0).unique():
sub_df = df.xs(key)
dictionary[key] = df_to_dict(sub_df)
return dictionary
make_dictionary(df)
It requires setting the different levels as index, and you will end up with a slightly different dictionary:
{'Animals':
{'Carnivores':
{'Felidae':
{'id': {'Siamese Cat': 1,
'Javanese Cat': 2}},
'Ursidae':
{'id': {'Polar Bear': 3}},
'Canidae':
{'id': {'Labradore Retriever': 4,
'Golden Retriever': 5}}}
}
}
EDIT: Had to make an adjustment, because the result wasn't exactly as expected.
Here's an attempt that produces the expected output (if I haven't made a mistake, which wouldn't be a surprise, because I've made several on the way):
def pack_level(df):
if df.columns[0] == 'id':
return [{'id': i, 'name': name, 'children': []}
for i, name in zip(df[df.columns[0]], df[df.columns[1]])]
return [{'name': df.iloc[0, 0],
'children': [entry for lst in df[df.columns[1]]
for entry in lst]}]
df = pd.DataFrame(data)
columns = list(df.columns[1:])
df = df.groupby(columns[:-1]).apply(pack_level)
for i in range(1, len(columns) - 1):
df = (df.reset_index(level=-1, drop=False).groupby(columns[:-i])
.apply(pack_level)
.reset_index(level=-1, drop=True))
var_data = {'name': df.index[0], 'children': df.iloc[0]}
The result looks a bit different at first glance, but that should be only due to the sorting (from printing):
{
"children": [
{
"children": [
{
"children": [
{
"children": [],
"id": 4,
"name": "Labradore Retriever"
},
{
"children": [],
"id": 5,
"name": "Golden Retriever"
}
],
"name": "Canidae"
},
{
"children": [
{
"children": [],
"id": 1,
"name": "Siamese Cat"
},
{
"children": [],
"id": 2,
"name": "Javanese Cat"
}
],
"name": "Felidae"
},
{
"children": [
{
"children": [],
"id": 3,
"name": "Polar Bear"
}
],
"name": "Ursidae"
}
],
"name": "Carnivores"
}
],
"name": "Animals"
}
I've tried to be as generic as possible, but the first column has to be named id (as in your sample).
Example below of what I'm trying but Im having trouble figuring out the addFields part.
I'd like to add the created_at and amount fields from the matching expr in the pipeline from $$items. Currently it returns every created_at in the $$items
User model
class User(BaseModel):
__schema__ = DictField(dict(
user_id=StringField(required=True),
name=StringField(required=True),
item_list=ListField(default=[])
))
class InventoryAggregateManipulator(Manipulator):
def transform_outgoing(self, doc, model):
cur = User.aggregate([
{"$lookup": {
"from": "item",
"let": {"items": "$item_list"},
"pipeline": [
{"$match": {"$expr": {"$in": ["$_id", "$$items.id"]}}},
{"$project": {"name": "$name", 'rate': "$rate", "payout": "$payout"}},
{"$addFields": {"last_run": "$$items.created_at"}}
],
"as": "inventory"
}},
{"$match": {"_id": doc['_id']}}
])
for doc in cur:
return doc
example item collection:
[
{
"_id": 1,
"name": "some_item",
"rate": 60,
"payout": 15
},
{
"_id": 2,
"name": "another_item",
"rate": 30,
"payout": 20
}
]
example user:
{
'_id': 1,
'user_id': 1234,
'name': 'user123',
'item_list':[
{
"id": 1,
"created_at": datetime,
"amount": 3
},
{
"id": 2,
"created_at": datetime,
"amount": 5
}
]
}
expected result
{
'user_id': 1234,
'name': 'user123',
'item_list':[
{
"id": 1,
"created_at": datetime,
"amount": 3
},
{
"id": 2,
"created_at": datetime,
"amount": 5
}
],
'inventory':[
{
"name": "some_item",
"rate": 60,
"payout": 15,
"last_run": [datetime from item_list],
"amount": [amount from item_list]
},
{
"name": "another_item",
"rate": 30,
"payout": 20,
"last_run": [datetime from item_list],
"amount": [amount from item_list]
}
]
}
As part of a Python program, I want to merge JSON objects that contain identically structured data. For instance:
{
"responseStatus": "SUCCESS",
"responseDetails": {
"total": 5754,
},
"data": [
{
"id": 1324651
},
{
"id": 5686131
}
]
}
What I want to do is to add the content of the data array of my section object into the data array of my first object.
So, assuming:
thejson1 = json.loads({"responseStatus": "SUCCESS","responseDetails": {"total": 5754,},"data": [{"id": 1324651},{"id": 5686131}]})
thejson2 = json.loads({"responseStatus": "SUCCESS","responseDetails": {"total": 1234,},"data": [{"id": 2165735},{"id": 2133256}]})
I thought that executing:
thejson1["data"].append(thejson2["data"])
Would expand thejson1 into:
{
"responseStatus": "SUCCESS",
"responseDetails": {
"total": 5754,
},
"data": [
{
"id": 1324651
},
{
"id": 5686131
},
{
"id": 2165735
},
{
"id": 2133256
}
]
}
But what it does instead is add thejson2 data as an array within the data array of thejson1:
{
"responseStatus": "SUCCESS",
"responseDetails": {
"total": 5754,
},
"data": [
{
"id": 1324651
},
{
"id": 5686131
},
[
{
"id": 2165735
},
{
"id": 2133256
}
]
]
}
So, what am I doing wrong? It looks like append adds the data array of the second JSON object instead of its content, but note that I can't know in advance the contents of the "data" array in my JSON input, so I can't write code that specifically loops in the "id" objects to add them one by one.
Thanks in advance!
R.
You're looking for extend, not append.
thejson1["data"].extend(thejson2["data"])
append takes the single argument and insert it to the end. While extend extends the list by adding all the individual values in the argument list to the end.
# example:
a=[1, 2, 3]
b = a[:].append([4, 5])
# b = [1, 2, 3, [4, 5]]
c = a[:].extend([4, 5])
# c = [1, 2, 3, 4, 5]
thejson1 = {"responseStatus": "SUCCESS","responseDetails": {"total": 5754,},"data": [{"id": 1324651},{"id": 5686131}]}
thejson2 = {"responseStatus": "SUCCESS","responseDetails": {"total": 1234,},"data": [{"id": 2165735},{"id": 2133256}]}
thejson1["data"] += thejson2["data"]
Output:
{'responseDetails': {'total': 5754}, 'data': [{'id': 1324651}, {'id': 5686131}, {'id': 2165735}, {'id': 2133256}], 'responseStatus': 'SUCCESS'}
You can also use += to extend.