Parsing JSON in Python and Converting to Excel - python

I am attempting to parse this JSON and convert it to Excel and only print out certain columns using Python but I keep getting an TypeError: string indices must be integers. The JSON file continues to repeat what is inside "d" over and over again.
JSON:
{
"d": [
{
"__type": "New Cust",
"Description": "TV Purchase",
"End": "/Date(1624962600000)/",
"ID": 1,
"Resources": [
{
"Available": true,
"Key": "12345",
"Text": "John Doe",
"Type": "SalesProvider",
"CalendarID": "1234"
}
],
"Subject": "Meeting with Sam",
"Visible": true,
"AppStatus": "5",
"StartTime": "06/01/2021 10:30:00",
"AppID": "1",
"ServiceID": "7",
"NotesCount": "0",
"CustomerID": "1",
"AppType": "NNR",
"IsEditedThis": "False",
"BusinessPackageID": "0",
"PopupNotesCount": "0",
"EventType": "1",
"SPName": "John Doe",
"SPID": "12345",
"EventCapacity": "0",
"IsPaid": "False",
"IsWorkShop": "False",
"FormStatus": "0",
"PaidType": "1",
"AppComment": "",
"CustName": "Sam Will",
"ResourceID": "",
"CssClass": "rsCategoryBlue",
"ServiceName": "TV Sale",
"NoOfAttendees": null,
"StreamingNoOfAttendees": null,
"StreamingStatus": "0",
"StreamingEventCapacity": "",
"Photo": "",
"PersonalOffType": null,
"ResourceName": null,
"IsShowCheckIn": false,
"PaymentStatus": 0
},
{
"__type": "New Cust",
"Description": "Receiver Purchase",
"End": "/Date(1624962600000)/",
"ID": 1,
"Resources": [
{
"Available": true,
"Key": "12345",
"Text": "John Doe",
"Type": "SalesProvider",
"CalendarID": "1234"
}
],
"Subject": "Meeting with Bill",
"Visible": true,
"AppStatus": "5",
"StartTime": "07/02/2021 9:30:00",
"AppID": "1",
"ServiceID": "7",
"NotesCount": "0",
"CustomerID": "1",
"AppType": "NNR",
"IsEditedThis": "False",
"BusinessPackageID": "0",
"PopupNotesCount": "0",
"EventType": "1",
"SPName": "John Doe",
"SPID": "12345",
"EventCapacity": "0",
"IsPaid": "False",
"IsWorkShop": "False",
"FormStatus": "0",
"PaidType": "1",
"AppComment": "",
"CustName": "Bill Tom",
"ResourceID": "",
"CssClass": "rsCategoryBlue",
"ServiceName": "Audio Sale",
"NoOfAttendees": null,
"StreamingNoOfAttendees": null,
"StreamingStatus": "0",
"StreamingEventCapacity": "",
"Photo": "",
"PersonalOffType": null,
"ResourceName": null,
"IsShowCheckIn": false,
"PaymentStatus": 0
}
]
}
Python Code:
import json
import pandas as pd
f = open('JSON.txt', 'r')
data = json.loads(f.read())
l = []
for profile in data['d']:
l.append(profile["Subject"]["StartTime"]["IsPaid"]["CustName"]["ServiceName"])
df1 = pd.DataFrame(l)
print(df1)
df1.to_excel('df1.xlsx')
I do not need the "Resources": [] info I just need certain parameters outside it in the JSON object. I am having difficulty parsing the JSON any help would be appreciated.

you can use a combination of the json standard library and pd.json_normalize
import json
import pandas as pd
parsed_json = json.loads('JSON.txt')
df = pd.json_normalize(parsed_json['d'],record_path='Resources')
print(df)
Available Key Text Type CalendarID
0 True 12345 John Doe SalesProvider 1234
1 True 12345 John Doe SalesProvider 1234
then pass it to excel
df.to_excel(...,index=False)
Going back to your issues, it seems like your trying to grab a bunch of fields like a list, when in reality you're attempting to find an attribute from a single data type object.
print(data['d'][0]['Subject'])
'Meeting with Sam'
This has no other nested items so you'll naturally get an error.
the error TypeError: string indices must be integers is telling you you can only slice this object with integers i.e
#reversing the string.
print(data['d'][0]['Subject'][::-1])
maS htiw gniteeM
or
#first two characters of the string.
print(data['d'][0]['Subject'][:2])
Me
if you want to grab only a subset of columns from the top level you could do :
cols = ['Subject', 'StartTime', 'IsPaid', 'CustName', 'ServiceName']
df = pd.json_normalize(parsed_json['d'],)[cols]
print(df)
Subject StartTime IsPaid CustName ServiceName
0 Meeting with Sam 06/01/2021 10:30:00 False Sam Will TV Sale
1 Meeting with Bill 07/02/2021 9:30:00 False Bill Tom Audio Sale

Related

Json data parsing using python to get the specific headers value

I have a Following Json File format in which the data will be having Dynamic update.
I have to parse the Json file using python with the following scenario
If the status: "PASS", then the results of value should be maths,q1 ,q2
Kindly me with this scenario using python
{
"quiz": {
"sport": {
"q1": {
"question": "Which one is correct team name in NBA?",
"options": [
"New York Bulls",
"Los Angeles Kings",
"Golden State Warriros",
"Huston Rocket"
],
"answer": "Huston Rocket"
},
"status": "BLOCK"
},
"maths": {
"q1": {
"question": "5 + 7 = ?",
"options": [
"10",
"11",
"12",
"13"
],
"answer": "12"
},
"q2": {
"question": "12 - 8 = ?",
"options": [
"1",
"2",
"3",
"4"
],
"answer": "4"
},
"status": "PASS"
}
}
}
Hope below code gives you the expected output
import json
file = open('data.json')
data = json.load(file)
keys = []
for i in data['quiz']:
if data['quiz'][i]['status'] != 'PASS':
keys.append(i)
for key in keys:
del data['quiz'][key]
print(data)
for key in data['quiz'].keys():
print(key)
for key1 in data['quiz'][key].keys():
if key1 != 'status':
print(key1)

Filter json response from http request to only specific value then select the largest 'id' value from all results

I have a Python script where I would like to filter Python with a http get and I would like to filter the response data for only specific values. The json response example is below:
{
"id": "38",
"name": "Report1",
"description": "",
"reportDefinitionID": "-1",
"jobID": "105600",
"type": "csv",
"status": "Completed",
"creator": {
"id": "1",
"username": "btest",
"firstname": "bob",
"lastname": "test"
},
{
"id": "39",
"name": "Report2",
"description": "",
"reportDefinitionID": "-1",
"jobID": "113218",
"type": "csv",
"status": "Completed"
"creator": {
"id": "1",
"username": "btest1",
"firstname": "Bob",
"lastname": "test1"
},
"id": "49",
"name": "Report1",
"description": "",
"reportDefinitionID": "-1",
"jobID": "113219",
"type": "csv",
"status": "Completed"
"creator": {
"id": "1",
"username": "btest1",
"firstname": "Bob",
"lastname": "test1"
}
I would like to filter the above json to only show a report by name. For example if there is a Python filter that would only allow me to filter for a report by the name of "Report1". If I filtered on name of "Report1". I would expect to following to be to be returned below:
{
"id": "38",
"name": "Report1",
"description": "",
"reportDefinitionID": "-1",
"jobID": "105600",
"type": "csv",
"status": "Completed",
"creator": {
"id": "1",
"username": "btest",
"firstname": "bob",
"lastname": "test"
},
"id": "49",
"name": "Report1",
"description": "",
"reportDefinitionID": "-1",
"jobID": "113219",
"type": "csv",
"status": "Completed"
"creator": {
"id": "1",
"username": "btest1",
"firstname": "Bob",
"lastname": "test1"
}
For the final part of the script I would like to compare the 'id' field to show the largest value for example id 38 vs id 49 and then output the json for the largest in this case id 49. I would like it output
},
"id": "49",
"name": "Report1",
"description": "",
"reportDefinitionID": "-1",
"jobID": "113219",
"type": "csv",
"status": "Completed"
"creator": {
"id": "1",
"username": "btest1",
"firstname": "Bob",
"lastname": "test1"
}
For the last part i would just like to save the id value '49' to a variable in Python.
So far what I have below is:
response_data = response.json()
input_dict = json.dumps(response_data)
input_transform = json.loads(input_dict)
# Filter python objects with list comprehensions
sort1 = sorted([r.get("id") for r in input_transform if r.get("name") == "Report1"], reverse=True)[0]
# Print sorted JSON
print(sort1)
I updated my code and now I'm getting the error below:
'str' object has no attribute 'get'
I researched it and can not figure out what I'm doing now and how to get past it.
You need to get the ID in the listcomp as bellow:
sorted([r.get("id") for r in sample if r.get("name") == "Report1"], reverse=True)[0]

Python 3 Trying to access json array keep getting key error

Im using python 3 and am trying to access some information from a json query. Currently I can access data within payments by looping through. I try to do the same to access order_status and it doesn't work.
Json:
___order details___
{
"message_type_name": "OrderPlaced",
"order": {
"adcode": "",
"adcode_id": 0,
"billing_address": {
"address_line_1": "123 east street",
"address_line_2": "apt one"
},
"campaign_code": "",
"channel": "Online",
"customer": {
"adcode": "",
"adcode_id": 0,
"affiliate_id": 0,
"alternate_phone_number": ""
},
"device": "None",
"discount_total": 0.0,
"discounted_shipping_total": 0.0,
"items": [
{
"admin_comments": "",
"cost": 90.0,
"created_at": "2018-12-04T17:14:55.1128646-06:00"
}
],
"manufacturer_invoice_amount": null,
"manufacturer_invoice_number": "",
"manufacturer_invoice_paid": false,
"order_status": {
"color": "#A4E065",
"created_at": null,
"email_template_id": null,
"id": 6,
"is_cancelled": false,
"is_declined": false,
"is_fully_refunded": false,
"is_open": true,
"is_partially_refunded": false,
"is_quote_status": false,
"is_shipped": false,
"name": "Awaiting Payment",
"updated_at": "2015-10-12T22:07:47.487-05:00"
},
"order_status_id": 6,
"order_status_last_changed_at": "2018-12-04T17:14:55.0503538-06:00",
"order_type": "Order",
"payments": [
{
"amount": 100.00,
"approved_at": "",
"authorization_code": ""
"
}
],
"ppc_keyword": "",
"previous_order_status_id": 6,
"shipments": [],
"shipping_address": {
"comments": "",
"company": "",
"country": "United States"
},
"shipping_total": 0.0,
"source": "Google [organic]"
},
"store_id": 1
}
I have a variable called data: data = json.loads(self.rfile.read( length ).decode('utf-8'))
I have another variable order_payments = data['payments']
I can loop through that and access is_declined
I want to have a variable called order_status = data['order_status'], and then loop through that to access name. I get a key error on order_status = data['order_status']and I dont know why.

can't print the value of json object in django

I have this json object in ajax_data variable
{
"columns[0][data]": "0",
"columns[1][name]": "",
"columns[5][searchable]": "true",
"columns[5][name]": "",
"columns[4][search][regex]": "false",
"order[0][dir]": "asc",
"length": "10",
}
I have converted it using json.loads() function like.
ajax_data = json.loads(ajax_data)
I want to get the value if "order[0][dir]" and "columns[0][data]" but if i print it using
ajax_data['order'][0]['dir]
its giving error :
KeyError at /admin/help
'order'
But same code works if i access it for length key then it works.
The keys you have used are actually not a good way of implementation.
{
"columns[0][data]": "0",
"columns[1][name]": "",
"columns[5][searchable]": "true",
"columns[5][name]": "",
"columns[4][search][regex]": "false",
"order[0][dir]": "asc",
"length": "10",
}
Instead of this you should hav gone for
{
"columns": [
{"data": "0", "name": "", "searchable": "true", "name": "", "search": {
"regex": "false"}
},
{"data": "0", "name": "", "searchable": "true", "name": ""," search": {
"regex": "false"}},
{"data": "0", "name": "", "searchable": "true", "name": "", "search": {
"regex": "false"}},
{"data": "0", "name": "", "searchable": "true", "name": "", "search": {
"regex": "false"}},
{"data": "0", "name": "", "searchable": "true", "name": "", "search": {
"regex": "false"}},
{"data": "0", "name": "", "searchable": "true", "name": "", "search": {
"regex": "false"}},
],
"order": [
{"dir": "asc"}
],
"length": "10"
}
In this case ajax_data['order'][0]['dir] will result in value "asc"
For your current implementation the key is "order[0][dir]"
That is go for
ajax_data["order[0][dir]"]
Hope you understood the issue.
Structuring of json is very important when dealing with APIs. Try to restructure your json which will help for future too.
That's because length is a key in that json object, and order is not. The key names are the entire strings inside the quotes: columns[0][data], order[0][dir], etc.
Those are unusual key names, but perfectly valid.

Parse JSON using python to fetch value based on condition

I am new to python and trying to parse the json file and fetch the required field based on condition.
eg., if status = true, then
print name
Json file:
[
{
"id": "12345",
"name": "London",
"active": true,
"status": "true",
"version": "1.0",
"tags": [
]
},
{
"id": "12457",
"name": "Newyork",
"active": true,
"status": "false",
"version": "1.1",
"tags": [
]
},
]
expected output:
name : London
Please help me on this. Thank you in advance.
>>> import json
>>> obj = json.loads('[ { "id": "12345", "name": "London", "active": true, "status": "true", "version": "1.0", "tags": [ ] }, { "id": "12457", "name": "Newyork", "active": true, "status": "false", "version": "1.1", "tags": [ ] } ]')
>>> print "Names:", ",".join(x["name"] for x in obj if x["status"] == "true")
Names: London
Your JSON is invalid. Remove the comma as below:
[
{
"id": "12345",
"name": "London",
"active": true,
"status": "true",
"version": "1.0",
"tags": [
]
},
{
"id": "12457",
"name": "Newyork",
"active": true,
"status": "false",
"version": "1.1",
"tags": [
]
},
^__________Remove this comma!
]
You can get all info about json parsing here.
http://docs.python.org/3.3/library/json.html

Categories

Resources