Develop Reference

Parsing JSON with Python to retrieve values within a dictionary

I am trying to parse JSON and retrieve a certain value (ID) but I am getting the following error:
TypeError: string indices must be integers
The following is my code and JSON:
import json
# JSON string
info_symbol = '{"status":{"timestamp":"2023-01-21T15:18:43.937Z","error_code":0,"error_message":null,"elapsed":21,"credit_count":1,"notice":null},"data":{"BITCOIN":{"id":15907,"name":"HarryPotterObamaSonic10Inu","symbol":"BITCOIN"}}}'
# Convert string to Python dict
test_dict = json.loads(info_symbol)
print(test_dict['data'])
for name in test_dict['data']['BITCOIN']:
print(name['id'])
Id like to grab solely the ID - so: 15907
"id":15907,

You don't need the looping in that case to get only the id. If you need all the key and values from the dictionary then you can iterate like below-
import json
# JSON string
info_symbol = '{
"status": {
"timestamp": "2023-01-21T15:18:43.937Z",
"error_code": 0,
"error_message": null,
"elapsed": 21,
"credit_count": 1,
"notice": null
},
"data": {
"BITCOIN": {
"id": 15907,
"name": "HarryPotterObamaSonic10Inu",
"symbol": "BITCOIN"
}
}
}'
# Convert string to Python dict
test_dict = json.loads(info_symbol)
data = test_dict['data']['BITCOIN']
print(f"only id = {data['id']}")
print("--------")
for key, value in test_dict['data']['BITCOIN'].items():
print(key, value)
Output
only id = 15907
--------
id 15907
name HarryPotterObamaSonic10Inu
symbol BITCOIN

for name in test_dict['data']['BITCOIN']:
print(name)
output
{'BITCOIN': {'id': 15907, 'name': 'HarryPotterObamaSonic10Inu', 'symbol': 'BITCOIN'}}
id
name
symbol
So, if you do name['id'], you will get id in first iteration and in second iteration it will throw error
you can get value of id by:
for key, value in test_dict['data']['BITCOIN'].items():
if key=='id':
print(value)
#15907
with comprehension:
[value for key, value in test_dict['data']['BITCOIN'].items() if key=='id']
[15907]

Keys Error
Fellow, test_dict['data]['BITCOIN'] is represented as:
{
'id': 15907,
'name': 'HarryPotterObamaSonic10Inu',
'symbol': 'BITCOIN'
}
Your are iterating the above dictionary, so when you print(name[id]), name iterator take the key:Value of id, name, and symbol.
Literally tour code is accesing as id.['id'], name.['id'], symbol.['id]'.
SOLUTION
just access to the id adding one more bracket:
print( test_dict['data']['BITCOIN']['id'] )
get rid of for loop.

Getting value from a JSON file based on condition

In python I'm trying to get the value(s) of the key "relativePaths" from a JSON element if that element contains the value "concept" for the key "tags". The JSON file has the following format.
]
},
{
"fileName": "#Weizman.2011",
"relativePath": "Text/#Weizman.2011.md",
"tags": [
"text",
"concept"
],
"frontmatter": {
"authors": "Weizman",
"year": 2011,
"position": {
"start": {
"line": 0,
"col": 0,
"offset": 0
},
"end": {
"line": 4,
"col": 3,
"offset": 120
}
}
},
"aliases": [
"The least of all possible evils - humanitarian violence from Arendt to Gaza"
],
I have tried the following codes:
import json
with open("/Users/metadata.json") as jsonFile:
data = json.load(jsonFile)
for s in range(len(data)):
if 'tags' in s in range(len(data)):
if data[s]["tags"] == "concept":
files = data[s]["relativePaths"]
print(files)
Which results in the error message:
TypeError: argument of type 'int' is not iterable
I then tried:
with open("/Users/metadata.json") as jsonFile:
data = json.load(jsonFile)
for s in str(data):
if 'tags' in s in str(data):
print(s["relativePaths"])
That code seems to work. But I don't get any output from the print command. What am I doing wrong?

Assuming your json is a list of the type you put on your question, you can get those values like this:
with open("/Users/metadata.json") as jsonFile:
data = json.load(jsonFile)
for item in data: # Assumes the first level of the json is a list
if ('tags' in item) and ('concept' in item['tags']): # Assumes that not all items have a 'tags' entry
print(item['relativePaths']) # Will trigger an error if relativePaths is not in the dictionary

Figured it
import json
f = open("/Users/metadata.json")
# returns JSON object as
# a dictionary
data = json.load(f)
# Iterating through the json
# list
for i in data:
if "tags" in i:
if "concept" in i["tags"]:
print(i["relativePaths"])
# Closing file
f.close()

I think this will do what you want. It is more "pythonic" because it doesn't use numerical indices to access elements of the list — making it easier to write and read).
import json
with open("metadata.json") as jsonFile:
data = json.load(jsonFile)
for elem in data:
if 'tags' in elem and 'concept' in elem['tags']:
files = elem["relativePath"]
print(files)

Generating specific fields from JSON objects

I have a rather massive JSON objects and I'm trying to generate a specific JSON using only certain elements from it.
My current code looks like this:
data = get.req("/v2/users")
data = json.loads(data)
print (type(data)) # This returns <class 'list'>
print (data) # Below for what this returns
all_data = []
for d in data:
login_value = d['login']
if login_value.startswith('fe'):
continue
s = get.req("/v2/users/" + str(login_value)) # Sending another request with each
# login from the first request
all_data.append(s)
print (all_data) # Below for what this looks like this
print (data) before json.loads is str for the information, it returns data like this:
[
{
"id": 68663,
"login": "test1",
"url": "https://x.com/test"
},
{
"id": 67344,
"login": "test2",
"url": "https://x.com/test"
},
{
"id": 66095,
"login": "hi",
"url": "https://x.com/test"
}
]
print (all_data) returns a similar result to this for every user a request was sent for the first time
[b'{"id":68663,"email":"x#gmail.com","login":"xg","phone":"hidden","fullname":"xg gx","image_url":"https://imgur.com/random.png","mod":false,"points":5,"activity":0,"groups":['skill': 'archery']}
And this repeats for every user.
What I'm attempting to do is filtering by a few fields from all those results I received, so the final JSON I have will look something like this
[
{
"email": "x#gmail.com",
"fullname": "xg gf",
"points": 5,
"image_url", "https://imgur.com/random.pmg"
},
{
... similar json for the next user and so on
}
]
I feel as if the way I'm iterating over the data might be inefficient, so if you could guide me to a better way it would be wonderful.

In order for you to fetch login value, you have to iterate over data atleast once and for fetching details for every user, you have to make one call and that is exactly what you have done.
After you receive the user details instead of appending the whole object to the all_data list just take the field you need and construct a dict of it and then append it to all_data.
So your code has time complexity of O(n) which is best I understand.
Edit :
For each user you are receiving a byte response like below.
byte_response = [ b'{"id":68663,"email":"x#gmail.com","login":"xg","phone":"hidden","fullname":"xg gx","image_url":"https://imgur.com/random.png","mod":false,"points":5,"activity":0,"groups":[]}']
I'm not sure why would you get a response in a list [], but if it like that then take byte_response[0] so that we have the actual byte data like below.
byte_response = b'{"id":68663,"email":"x#gmail.com","login":"xg","phone":"hidden","fullname":"xg gx","image_url":"https://imgur.com/random.png","mod":false,"points":5,"activity":0,"groups":[]}'
response_decoded = byte_response.decode("utf-8") #decode it
import json
json_object_in_dict_form = json.loads(response_decoded) #convert it into dictionary
and then...
json_object_in_dict_form['take the field u want']

you can write:
data = get.req("/v2/users")
data = json.loads(data)
all_data = []
for d in data:
...
s = get.req("/v2/users/" + str(login_value))
new_data = {
'email': s['email'],
'fullname': s['fullname'],
'points': s['points'],
'image_url': s['image_url']
}
all_data.append(new_data)
print (all_data)
or you can make it fancy using an array with the fields you need:
data = get.req("/v2/users")
data = json.loads(data)
all_data = []
fields = ['email', 'fullname', 'point', 'image_url']
for d in data:
...
s = get.req("/v2/users/" + str(login_value))
new_data = dict()
for field in fields:
new_data[field] = s[field]
all_data.append(new_data)
print (all_data)

error in accessing a nested json within a list in a json in python

I have this sample json data that is coming as a response from a POST request- response is the name of below sample json data variable:
{
"statusCode": "OK",
"data": [
{
"id": -199,
"result": {
"title": "test1",
"group": "test_grp2"
}
},
{
"id": -201,
"result": {
"title": "test2",
"group": "test_grp2"
}
}
]
}
Now what I want to do is form a list of list where each list will have id,title,and group values from above json data. Here is what my code looks like:
def get_list():
# code to get the response from POST request
json_data = json.loads(response.text)
group_list = []
if json_data['statusCode'] == 'OK':
for data in json_data['data']:
print(data)
for result_data in data['result']:
title = result_data['title']
group = result_data['group']
group_list.append([data['id'],title,group])
return(group_list)
When I execute this I get error as list indices must be int not str at line title = result_data['title]. How can I form the list of list above?

data['result'] in your case is a dictionary. When you iterate over it using for result_data in data['result'], you get the keys - in other words, result_data is a string and it does not support string indexing.
Instead, you meant:
for data in json_data['data']:
result_data = data['result']
title = result_data['title']
group = result_data['group']
Or, alternatively, you can make use of a list comprehension:
group_list = [(data['id'], data['result']['title'], data['result']['group'])
for data in json_data['data']]
print(group_list)

You can extract the list of lists directly using a list comprehension:
>>> [[d['id'], d['result']['title'], d['result']['group']] for d in json_data['data']]
[[-199, 'test1', 'test_grp2'], [-201, 'test2', 'test_grp2']]

how to parse json where key is variable in python?

i am parsing a log file which is in json format,
and contains data in the form of key : value pair.
i was stuck at place where key itself is variable. please look at the attached code
in this code i am able to access keys like username,event_type,ip etc.
problem for me is to access the values inside the "submission" key where
i4x-IITB-CS101-problem-33e4aac93dc84f368c93b1d08fa984fc_2_1 is a variable key which will change for different users,
how can i access it as a variable ?
{
"username": "batista",
"event_type": "problem_check",
"ip": "127.0.0.1",
"event": {
"submission": {
"i4x-IITB-CS101-problem-33e4aac93dc84f368c93b1d08fa984fc_2_1": {
"input_type": "choicegroup",
"question": "",
"response_type": "multiplechoiceresponse",
"answer": "MenuInflater.inflate()",
"variant": "",
"correct": true
}
},
"success": "correct",
"grade": 1,
"correct_map": {
"i4x-IITB-CS101-problem-33e4aac93dc84f368c93b1d08fa984fc_2_1": {
"hint": "",
"hintmode": null,
"correctness": "correct",
"npoints": null,
"msg": "",
"queuestate": null
}
}
this is my code how i am solving it :
import json
import pprint
with open("log.log") as infile:
# Loop until we have parsed all the lines.
for line in infile:
# Read lines until we find a complete object
while (True):
try:
json_data = json.loads(line)
username = json_data['username']
print "username :- " + username
except ValueError:
line += next(infile)
how can i access i4x-IITB-CS101-problem-33e4aac93dc84f368c93b1d08fa984fc_2_1 key and
data inside this key ??

You don't need to know the key in advance, you can simply iterate over the dictionary:
for k,v in obj['event']['submission'].iteritems():
print(k,v)

Suppose you have a dictionary of type d = {"a":"b"} then d.popitem() would give you a tuple ("a","b") which is (key,value). So using this you can access key-value pairs without knowing the key.
In you case if j is the main dictionary then j["event"]["submission"].popitem() would give you tuple
("i4x-IITB-CS101-problem-33e4aac93dc84f368c93b1d08fa984fc_2_1": {
"input_type": "choicegroup",
"question": "",
"response_type": "multiplechoiceresponse",
"answer": "MenuInflater.inflate()",
"variant": "",
"correct": true
})
Hope this is what you were asking.

using python json module you'll end up with a dictionary of parsed values from the above JSON data
import json
parsed = json.loads(this_sample_data_in_question)
# parsed is a dictionary, so are "correct_map" and "submission" dictionary keys within "event" key
So you could iterate over the key, values of the data as a normal dictionary, say like this:
for k, v in parsed.items():
print k, v
Now you could find the (possible different values) of "i4x-IITB-CS101-problem-33e4aac93dc84f368c93b1d08fa984fc_2_1" key in a quick way like this:
import json
parsed = json.loads(the_data_in_question_as_string)
event = parsed['event']
for key, val in event.items():
if key in ('correct_map', 'submission'):
section = event[key]
for possible_variable_key, its_value in section.items():
print possible_variable_key, its_value
Of course there might be better way of iterating over the dictionary, but that one you could choose based on your coding taste, or performance if you have a fairly larger kind of data than the one posted in here.