Develop Reference

Parse complex JSON in Python

EDITED WITH LARGER JSON:
I have the following JSON and I need to get id element: 624ff9f71d847202039ec220
results": [
{
"id": "62503d2800c0d0004ee4636e",
"name": "2214524",
"settings": {
"dataFetch": "static",
"dataEntities": {
"variables": [
{
"id": "624ffa191d84720202e2ed4a",
"name": "temp1",
"device": {
"id": "624ff9f71d847202039ec220",
"name": "282c0240ea4c",
"label": "282c0240ea4c",
"createdAt": "2022-04-08T09:01:43.547702Z"
},
"chartType": "line",
"aggregationMethod": "last_value"
},
{
"id": "62540816330443111016e38b",
"device": {
"id": "624ff9f71d847202039ec220",
"name": "282c0240ea4c",
},
"chartType": "line",
}
]
}
...
Here is my code (EDITED)
url = "API_URL"
response = urllib.urlopen(url)
data = json.loads(response.read().decode("utf-8"))
print url
all_ids = []
for i in data['results']: # i is a dictionary
for variable in i['settings']['dataEntities']['variables']:
print(variable['id'])
all_ids.append(variable['id'])
But I have the following error:
for variable in i['settings']['dataEntities']['variables']:
KeyError: 'dataEntities'
Could you please help?
Thanks!!

What is it printing when you print(fetc)? If you format the json, it will be easier to read, the current nesting is very hard to comprehend.
fetc is a string, not a dict. If you want the dict, you have to use the key.
Try:
url = "API_URL"
response = urllib.urlopen(url)
data = json.loads(response.read().decode("utf-8"))
print url
for i in data['results']:
print(json.dumps(i['settings']))
print(i['settings']['dataEntities']
EDIT: To get to the id field, you'll need to dive further.
i['settings']['dataEntities']['variables'][0]['id']
So if you want all the ids you'll have to loop over the variables (assuming the list is more than one)`, and if you want them for all the settings, you'll need to loop over that too.
Full solution for you to try (EDITED after you uploaded the full JSON):
url = "API_URL"
response = urllib.urlopen(url)
data = json.loads(response.read().decode("utf-8"))
print url
all_ids = []
for i in data['results']: # i is a dictionary
for variable in i['settings']['dataEntities']['variables']:
print(variable['id'])
all_ids.append(variable['id'])
all_ids.append(variable['device']['id']
Let me know if that works.

The shared JSON is not valid. A valid JSON similar to yours is:
{
"results": [
{
"settings": {
"dataFetch": "static",
"dataEntities": {
"variables": [
{
"id": "624ffa191d84720202e2ed4a",
"name": "temp1",
"span": "inherit",
"color": "#2ccce4",
"device": {
"id": "624ff9f71d847202039ec220"
}
}
]
}
}
}
]
}
In order to get a list of ids from your JSON you need a double for cycle. A Pythonic code to do that is:
all_ids = [y["device"]["id"] for x in my_json["results"] for y in x["settings"]["dataEntities"]["variables"]]
Where my_json is your initial JSON.

Creating a Dataframe from nested JSON API output

I am using the Qualtrics API to pull some data for work. The results I have been receiving comes back in a JSON format and I would like to transform the data into a dataframe. I'm working inside a jupyter notebook within Alteryx. I plan to export the dataframe in Alteryx to do work elsewhere..all I need to do is get it into shape. I receive the same response as the example I have posted below from Qualtrics website. Does anyone know how I can take the fields under the nested "elements" section and create a dataframe? I would like to make a dataframe of the contact fields I receive back.
I have tried the following:
jdata = json.loads(response.text)
df = pd.DataFrame(jdata)
print(df)
But I am getting a dataframe of the entire json response.
Example Response:
{
"meta": {
"httpStatus": "200 - OK",
"requestId": "7de14d38-f5ed-49d0-9ff0-773e12b896b8"
},
"result": {
"elements": [
{
"contactId": "CID_123456",
"email": "js#example.com",
"extRef": "1234567",
"firstName": "James",
"language": "en",
"lastName": "Smith",
"phone": "8005552000",
"unsubscribed": false
},
{
"contactId": "CID_3456789",
"email": "person#example.com",
"extRef": "12345678",
"firstName": "John",
"language": "en",
"lastName": "Smith",
"phone": "8005551212",
"unsubscribed": true
}
],
"nextPage": null
}
}

jdata = json.loads(response.text)
df = pd.json_normalize(jdata, record_path=['result', 'elements'])
In fact, if jdata is a list of dict, this method is also available.

try
jdata = json.loads(response.text)
elements= jdata['result']['elements']
df = pd.DataFrame(elements)
print(df)

Parsing list of dictionaries in a dictionary to retrieve a specific key's value from each dictionary

I got a JSON response and converted it to a python dictionary using json.loads(). So the dictionary looks like this:
{u'body': u'[{"id":"1","entity":"zone","status":"PROCESSING","url":null,"createdOn":"2019-10-11T05:49:11Z"},{"id":"2","entity":"floor","status":"FAILED","url":null,"createdOn":"2019-10-11T05:49:15Z"},{"id":"3","entityType":"apartment","status":"SUCCESS","url":null,"createdOn":"2019-10-11T05:49:18Z"}]',u'isBase64Encoded': False, u'statusCode': 200}
I named this as testStatusList. I want to retrieve the value of "status" key of every dictionary inside "body". I was able to retrieve the "body" by giving body = testStatusList['body']. Now, the dictionary looks like:
[
{
"id": "1",
"entityType": "zone",
"status": "PROCESSING",
"url": null,
"createdOn": "2019-03-07T12:47:10Z"
},
{
"id": "2",
"entityType": "floor",
"status": "FAILED",
"url": null,
"createdOn": "2019-08-19T16:46:13Z"
},
{
"id": "3",
"entityType": "apartment",
"status": "SUCCESS",
"url": null,
"createdOn": "2019-08-19T16:46:13Z"
}
]
I tried out this solution [Parsing a dictionary to retrieve a key in Python 3.6
testStatusList= json.loads(status_response['Payload'].read())
body = testStatusList['body']
status =[]
for b in body:
for k,v in b.items():
if k == 'status':
status.append(v)
but I keep getting AttributeError: 'unicode' object has no attribute 'items'. Is there a different method to get items for unicode objects?
So I basically want to retrieve all the statuses i.e., PROCESSING, FAILED AND SUCCESS so that I can put an 'if' condition to display appropriate messages when something failed for that particular "id". I am very unsure about my approach as I am totally new to Python. Any help would be much appreciated thanks!

body is still a (unicode) string in your top blob. Use json.loads again on that string:
body = """[
{
"id": "1",
"entityType": "zone",
"status": "PROCESSING",
"url": null,
"createdOn": "2019-03-07T12:47:10Z"
},
{
"id": "2",
"entityType": "floor",
"status": "FAILED",
"url": null,
"createdOn": "2019-08-19T16:46:13Z"
},
{
"id": "3",
"entityType": "apartment",
"status": "SUCCESS",
"url": null,
"createdOn": "2019-08-19T16:46:13Z"
}
]"""
import json
body = json.loads(body)
status =[]
for b in body:
for k,v in b.items():
if k == 'status':
status.append(v)
print(status)
Result:
['PROCESSING', 'FAILED', 'SUCCESS']

Mitmproxy tampering with GET and POST request/ response in one script

A POST request sent to a certain URL (http://test.com) looks like this:
{
"messageType": "OK",
"city": {
"Name": "Paris",
"Views": {
"1231": {
"id": 4234,
"enableView": false
},
},
"Views": [5447, 8457],
"messages": [{
"id": "message_6443",
"eTag": 756754338
}]
},
"client": {
"Id": 53,
"email": "test#test.us",
"firstName": "test",
"lastName": "test",
"id": 52352352,
"uuid": "5631f-grdeh4",
"isAdmin": false
}
}
I need to intercept the request and change isAdmin to true.
And a GET request to a certain URL (https://test.com/profiles/{Random_Numbers}/{id}) has a (encoded) response like this:
{
"id": 0,
"Code": "Admin",
"display": "RRRR"
}
I need to change id to 5.
So basically I need to write one script that will do both of these actions.
So far I have tried to take advantage of some examples on GitHub, but I haven't gotten it so far:
from libmproxy.protocol.http import decoded
def start(context, argv):
if len(argv) != 3:
raise ValueError('Usage: -s "modify-response-body.py old new"')
context.old, context.new = argv[1], argv[2]
def response(context, flow):
with decoded(flow.response): # automatically decode gzipped responses.
flow.response.content = flow.response.content.replace(context.old, context.new)`
How do I implement this for my scenario?
Probably using the libmproxy to get http-request and response would be a better idea, maybe.

The script you posted and Python's JSON module should get you pretty far:
def response(context, flow):
if flow.request.url == "...": # optionally filter based on some criteria...
with decoded(flow.response): # automatically decode gzipped responses.
data = json.loads(flow.response.content)
data["foo"] = "bar"
flow.response.content = json.dumps(data)

Parse Through Json Python

This the json file, I want to access the token ID and write it to a file , I try this method but it fails:
python:
data=json.loads(content.getvalue())
result=json.dumps(data, indent=4, separators = (', ', ': '))
f=open("/home/moby/writing.txt","w")
f.write(result.access.token.id)
f.close()
json:
{
"access": {
"token": {
"issued_at": "2013-12-19T05:16:05.901222",
"expires": "2013-12-20T05:16:05Z",
"id": "d08249e885b24d248f7935a1aa528e28",
"tenant": {
"enabled": true,
"description": null,
"name": "admin",
"id": "de30c00c1a8a488999bfb557f8748222"
}
}
}
}
Any Help? Thanks in advance

Um...
with open("/home/moby/writing.txt","w") as f:
f.write(data[u'access'][u'token'][u'id'])

json.loads produces a dictionary, and that's not how you use a dictionary.
Get the data you need by using the keys:
.write(data[u'result'][u'access'][u'token'][u'id'])