Develop Reference

Deeply nested json - a list within a dictionary to Pandas DataFrame

I'm trying to parse nested json results.
data = {
"results": [
{
"components": [
{
"times": {
"periods": [
{
"fromDayOfWeek": 0,
"fromHour": 12,
"fromMinute": 0,
"toDayOfWeek": 4,
"toHour": 21,
"toMinute": 0,
"id": 156589,
"periodId": 20855
}
],
}
}
],
}
],
}
I can get to and create dataframes for "results" and "components" lists, but cannot get to "periods" due to the "times" dict. So far I have this:
df = pd.json_normalize(data, record_path = ['results','components'])
Need a separate "periods" dataframe with the included column names and values. Would appreciate your help on this. Thank you!

I results
II components
III times
IIII periods
The normalize should be correct way:
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.json_normalize.html
There is 4 level of nesting. There can be x components in results and y times in components - however that type of nesting is overengineering?
The simplest way of getting data is:
print data['a']['b']['c']['d'] (...)
in your case:
print data['results']['components']['times']['periods']
You can access the specific nested level by this piece of code:
def GetPropertyFromPeriods (property):
propertyList = []
for x in data['results']['components']['times']:
singleProperty = photoURL['periods'][property]
propertyList.append(singleProperty)
return propertyList
This give you access to one property inside periods (fromDayOfWeek, fromHour, fromMinute)
After coverting json value, transform it into pandas dataframe:
print pd.DataFrame(data, columns=["columnA", "columnB”])
If stuck:
How to Create a table with data from JSON output in Python
Python - How to convert JSON File to Dataframe
pandas documentation:
pandas.DataFrame.from_dict
pandas.json_normalize

Prevent Pandas to_json() from wrapping lists in single quotes

I am trying to generate a JSON blob that contains lists generated from series of data within my dataframe. When I run to_json(orient='values') it appropriately transforms the series to a list with just the series values, but then wraps the list in single quotes making it difficult to parse this data if anyone handle it. Is there a good way to convert the series to a JSON object, but not wrap the lists in single quotes?
Code:
# Create empty JSON blob
output_json = {}
# Add to JSON output
output_json['count_time_fitness_timeseries'] = {
'x': df.fitness_date.to_json(orient='values'),
'y': df.minutes_fitness.to_json(orient='values'),
'labels': df.minutes_fitness_hhmm.to_json(orient='values')
}
print(output_json)
Example:
{
'count_time_fitness_timeseries': {
'x': '["2020-04-01","2020-04-02","2020-04-03","2020-04-04","2020-04-05","2020-04-06","2020-04-07","2020-04-08","2020-04-09","2020-04-10","2020-04-11","2020-04-12","2020-04-13","2020-04-14","2020-04-15","2020-04-16","2020-04-17","2020-04-18","2020-04-19","2020-04-20","2020-04-21","2020-04-22","2020-04-23","2020-04-24","2020-04-25","2020-04-26","2020-04-27","2020-04-28","2020-04-29","2020-04-30","2020-05-01","2020-05-02","2020-05-03","2020-05-04","2020-05-05","2020-05-06","2020-05-07","2020-05-08","2020-05-09","2020-05-10","2020-05-11","2020-05-12","2020-05-13","2020-05-14","2020-05-15","2020-05-16","2020-05-17","2020-05-18","2020-05-19","2020-05-20","2020-05-21","2020-05-22","2020-05-23","2020-05-24","2020-05-25","2020-05-26","2020-05-27","2020-05-28","2020-05-29","2020-05-30","2020-05-31","2020-06-01","2020-06-02","2020-06-03","2020-06-04","2020-06-05","2020-06-06","2020-06-07","2020-06-08","2020-06-09","2020-06-10","2020-06-11","2020-06-12","2020-06-13","2020-06-14","2020-06-15","2020-06-16","2020-06-17","2020-06-18","2020-06-19","2020-06-20","2020-06-21","2020-06-22","2020-06-23","2020-06-24","2020-06-25","2020-06-26","2020-06-27","2020-06-28","2020-06-29","2020-06-30","2020-07-01"]',
'y': '[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,461.0,421.0,519.0,502.0,511.0,513.0,496.0,480.0,364.0,498.0,467.0,477.0,431.0,419.0,471.0,391.0,481.0,494.0,506.0,464.0,474.0,383.0,385.0,470.0,465.0,574.0,473.0,431.0,497.0,null,482.0,492.0,494.0,469.0,395.0,427.0,346.0,416.0,461.0,486.0,451.0,533.0,null,462.0,461.0,477.0,458.0,484.0,389.0,null,472.0,462.0,486.0,489.0,483.0,426.0,453.0,489.0,467.0,474.0,451.0,450.0,470.0,null,247.0,502.0,464.0]',
'labels': '[null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"7:41","7:01","8:39","8:22","8:31","8:33","8:16","8:00","6:04","8:18","7:47","7:57","7:11","6:59","7:51","6:31","8:01","8:14","8:26","7:44","7:54","6:23","6:25","7:50","7:45","9:34","7:53","7:11","8:17",null,"8:02","8:12","8:14","7:49","6:35","7:07","5:46","6:56","7:41","8:06","7:31","8:53",null,"7:42","7:41","7:57","7:38","8:04","6:29",null,"7:52","7:42","8:06","8:09","8:03","7:06","7:33","8:09","7:47","7:54","7:31","7:30","7:50",null,"4:07","8:22","7:44"]'
}

Try this:
import json
#.... your code
print(json.dumps(output_json))

Map JSON values into another JSON object in Python

Background to this question is I have JSON Responses A and wish to send Response A's values only to JSON format B. JSON format B has different fields.
JSON Response A
{
"res_comment": "work description",
"res_id": "62",
"res_priority": "P2",
"res_qid": "INC0140315"
}
....... etc x n
Looking to map the values in JSON Response A to the below JSON format B.
JSON format B
{
"ServiceIssueCategoryID": "62", **#res_id**
"ServicePriorityCode": "P2", **#res_priority**
"ServiceNowID_KUT": "INC0140315", **#res_qid**
"ServiceRequestTextCollection":[
{
"TypeCode": "10004", **value does not change**
"Text": "work description" **#res_comment**
}
]
}
## update
To be clear:
I have made a single map of Values (between one set of curly brackets) to Format B. Then I used the below to update Format B:
with open("sapFormat.json", "r+") as jsonFile:
data = json.load(jsonFile)
tmp1 = data['ServiceIssueCategoryID']
data["ServiceIssueCategoryID"] = res_id
.... other fields
Works fine, however I have 100's of these individual 'res_id'. Not sure how to loop through all individually and add individually. The function that pulls the Values, takes all Values, e.g. 100 x res_id, need to take 1 res_id, update B and move to the next res_id, update etc

pandas change the order of columns

In my project I'm using flask I get a JSON (by REST API) that has data that I should convert to a pandas Dataframe.
The JSON looks like:
{
"entity_data":[
{"id": 1, "store": "a", "marker": "a"}
]
}
I get the JSON and extract the data:
params = request.json
entity_data = params.pop('entity_data')
and then I convert the data into a pandas dataframe:
entity_ids = pd.DataFrame(entity_data)
the result looks like this:
id marker store
0 1 a a
This is not the original order of the columns. I'd like to change the order of the columns as in the dictionary.
help?

Use OrderedDict for an ordered dictionary
You should not assume dictionaries are ordered. While dictionaries are insertion ordered in Python 3.7, whether or not libraries maintain this order when reading json into a dictionary, or converting the dictionary to a Pandas dataframe, should not be assumed.
The most reliable solution is to use collections.OrderedDict from the standard library:
import json
import pandas as pd
from collections import OrderedDict
params = """{
"entity_data":[
{"id": 1, "store": "a", "marker": "a"}
]
}"""
# replace myjson with request.json
data = json.loads(params, object_pairs_hook=OrderedDict)
entity_data = data.pop('entity_data')
df = pd.DataFrame(entity_data)
print(df)
# id store marker
# 0 1 a a

Just add the column names parameter.
entity_ids = pd.DataFrame(entity_data, columns=["id","store","marker"])

Assuming you have access to JSON sender, you can send the order in the JSON itself.
like
`{
"order":['id','store','marker'],
"entity_data":{"id": [1,2], "store": ["a","b"],
"marker": ["a","b"]}
}
then create DataFrame with columns specified. as said by Chiheb.K.
import pandas as pd
params = request.json
entity_data = params.pop('entity_data')
order = params.pop('order')
entity_df=pd.DataFrame(data,columns=order)
if you cannot explicitly specify the order in the JSON. see this answer to specify object_pairs_hook in
JSONDecoder to get an OrderedDict and then create the DataFrame

Create a data frame from a complex nested dictionary?

I have a big nested, then nested then nested json file saved as .txt format. I need to access some specific key pairs and crate a data frame or another transformed json object for further use. Here is a small sample with 2 key pairs.
[
{
"ko_id": [819752],
"concepts": [
{
"id": ["11A71731B880:http://ontology.intranet.com/Taxonomy/116#en"],
"uri": ["http://ontology.intranet.com/Taxonomy/116"],
"language": ["en"],
"prefLabel": ["Client coverage & relationship management"]
}
]
},
{
"ko_id": [819753],
"concepts": [
{
"id": ["11A71731B880:http://ontology.intranet.com/Taxonomy/116#en"],
"uri": ["http://ontology.intranet.com/Taxonomy/116"],
"language": ["en"],
"prefLabel": ["Client coverage & relationship management"]
}
]
}
]
The following code load the data as list but I need to access to the data probably as a dictionary and I need the "ko_id", "uri" and "prefLabel" from each key pair and put it to a pandas data frame or a dictionary for further analysis.
with open('sample_data.txt') as data_file:
json_sample = js.load(data_file)
The following code gives me the exact value of the first element. But donot actually know how to put it together and build the ultimate algorithm to create the dataframe.
print(sample_dict["ko_id"][0])
print(sample_dict["concepts"][0]["prefLabel"][0])
print(sample_dict["concepts"][0]["uri"][0])

for record in sample_dict:
df = pd.DataFrame(record['concepts'])
df['ko_id'] = record['ko_id']
final_df = final_df.append(df)

You can pass the data to pandas.DataFrame using a generator:
import pandas as pd
import json as js
with open('sample_data.txt') as data_file:
json_sample = js.load(data_file)
df = pd.DataFrame(data = ((key["ko_id"][0],
key["concepts"][0]["prefLabel"][0],
key["concepts"][0]["uri"][0]) for key in json_sample),
columns = ("ko_id", "prefLabel", "uri"))
Output:
>>> df
ko_id prefLabel uri
0 819752 Client coverage & relationship management http://ontology.intranet.com/Taxonomy/116
1 819753 Client coverage & relationship management http://ontology.intranet.com/Taxonomy/116