Develop Reference

Saving JSON File with Many Nested Objects to a List - python

I currently have a JSON file that has the following format. Keep in mind that this is not the entire file. The entire JSON file is comprised of hundreds of lists of key, value pairs that follow one after another between square brackets [][][]...etc. I am trying to store each of these individual lists e.g. the list below in a structure so that I may iterate the structure and parse each list for filename, labels etc. I initially tried to import this using json.loads() but I am having issues due the nested nature of the objects. I would appreciate any ideas/insight on how I can go about importing this file into a list or other appropriate python structure.
[
{
"File_Name": "1.jpg",
"Analysis": {
"Labels": [
{
"Confidence": 94.77251434326172,
"Name": "Flora"
},
{
"Confidence": 94.77251434326172,
"Name": "Grass"
},
{
"Confidence": 94.77251434326172,
"Name": "Plant"
},
{
"Confidence": 78.49254608154297,
"Name": "Animal"
},
{
"Confidence": 78.49254608154297,
"Name": "Cheetah"
},
{
"Confidence": 78.49254608154297,
"Name": "Mammal"
},
{
"Confidence": 78.49254608154297,
"Name": "Wildlife"
},
{
"Confidence": 69.79740142822266,
"Name": "Field"
},
{
"Confidence": 69.79740142822266,
"Name": "Grassland"
},
{
"Confidence": 69.79740142822266,
"Name": "Outdoors"
},
{
"Confidence": 67.31356048583984,
"Name": "Leisure Activities"
},
{
"Confidence": 67.31356048583984,
"Name": "Walking"
},
{
"Confidence": 57.44683837890625,
"Name": "Jaguar"
},
{
"Confidence": 57.44683837890625,
"Name": "Leopard"
},
{
"Confidence": 57.44683837890625,
"Name": "Panther"
},
{
"Confidence": 55.88261032104492,
"Name": "Bush"
},
{
"Confidence": 55.88261032104492,
"Name": "Vegetation"
},
{
"Confidence": 53.4413948059082,
"Name": "Lawn"
}
],
"ResponseMetadata": {
"RetryAttempts": 0,
"HTTPStatusCode": 200,
"RequestId": "978e32e4-1da8-11e8-a380-cd680f89684e",
"HTTPHeaders": {
"date": "Thu, 01 Mar 2018 23:30:59 GMT",
"x-amzn-requestid": "978e32e4-1da8-11e8-a380-cd680f89684e",
"content-length": "947",
"content-type": "application/x-amz-json-1.1",
"connection": "keep-alive"
}
},
"OrientationCorrection": "ROTATE_0"
}
}
][
{
"File_Name": "2.jpg",
"Analysis": {
"Labels": [
{
"Confidence": 98.57389068603516,
"Name": "Astronomy"
},
{
"Confidence": 98.57389068603516,
"Name": "Galaxy"
},
{
"Confidence": 98.57389068603516,
"Name": "Nebula"
},
{
"Confidence": 98.57389068603516,
"Name": "Night"
},
{
"Confidence": 98.57389068603516,
"Name": "Outdoors"
},
{
"Confidence": 98.57389068603516,
"Name": "Outer Space"
},
{
"Confidence": 98.57389068603516,
"Name": "Space"
},
{
"Confidence": 98.57389068603516,
"Name": "Universe"
}
],
"ResponseMetadata": {
"RetryAttempts": 0,
"HTTPStatusCode": 200,
"RequestId": "98d2c109-1da8-11e8-a2d9-b91cf22c7f33",
"HTTPHeaders": {
"date": "Thu, 01 Mar 2018 23:30:59 GMT",
"x-amzn-requestid": "98d2c109-1da8-11e8-a2d9-b91cf22c7f33",
"content-length": "449",
"content-type": "application/x-amz-json-1.1",
"connection": "keep-alive"
}
},
"OrientationCorrection": "ROTATE_0"
}
},
{
"File_Name": "2.jpg",
"Analysis": {
"Labels": [
{
"Confidence": 98.57389068603516,
"Name": "Astronomy"
},
{
"Confidence": 98.57389068603516,
"Name": "Galaxy"
},
{
"Confidence": 98.57389068603516,
"Name": "Nebula"
},
{
"Confidence": 98.57389068603516,
"Name": "Night"
},
{
"Confidence": 98.57389068603516,
"Name": "Outdoors"
},
{
"Confidence": 98.57389068603516,
"Name": "Outer Space"
},
{
"Confidence": 98.57389068603516,
"Name": "Space"
},
{
"Confidence": 98.57389068603516,
"Name": "Universe"
}
],
"ResponseMetadata": {
"RetryAttempts": 0,
"HTTPStatusCode": 200,
"RequestId": "98d2c109-1da8-11e8-a2d9-b91cf22c7f33",
"HTTPHeaders": {
"date": "Thu, 01 Mar 2018 23:30:59 GMT",
"x-amzn-requestid": "98d2c109-1da8-11e8-a2d9-b91cf22c7f33",
"content-length": "449",
"content-type": "application/x-amz-json-1.1",
"connection": "keep-alive"
}
},
"OrientationCorrection": "ROTATE_0"
}
}
]

big_json_file = json.loads(file_string)
big_list_of_labels = []
for file in big_json_file:
big_list_of_labels.append(file['Analysis']['Labels'])
Or if you want to store the file name and the list I'd recommend something like:
my_processed_dict = {}
for file in big_json_file:
my_processed_dict[file['File_Name']] = file['Analysis']['Labels']
where you can iterate over my_processed_dict with:
for key, value in my_processed_dict.items():
# value is the list of confidence values!
pass

Related

The 3rd party system I am using (vendor product) still uses Python 2.7 and doesn't support Python 3+ so bear with me, I'm fully aware Python 3 is out and this is a limitation of the system I have to use rather than a choice.
I am trying to do an integration between this third party product and MS teams - basically, the third party system provides data, I read this into my Python script and output a message to Teams using a webhook. It mostly works, but I'm struggling to load in some of the variables from the systems data.
For example, in my code, I use the following:
messageID='"{}"'.format(item["messageId"])
recipient='"{}"'.format(item["recipient"]["email"])
subject='"{}"'.format(item["subject"])
sender='"{}"'.format(item["sender"]["email"])
which has output like this:
messageId="34239482030783472#test.net"
recipient="testuser#domain.com"
subject="Email subject here"
sender="sender#domain2.com"
This is all fine, the trouble comes when I need to format my string to post to the Teams webhook.
It currently looks like:
teams_card='{"#type": "MessageCard","#context": "http://schema.org/extensions","themeColor": "0076D7","summary": “PTR”,”sections": [{"activityTitle": "PTR Incident Created","activitySubtitle": “End “User Exposed to Phishing Threat,”facts": [{"name": “Message” ID,”value": %s}, {"name": "Subject”,”value": %s},{“name": "End User","value": %s},{“name": “sender”,”value": %s}],”markdown": true}],"potentialAction": [{"#type": "OpenUri","name": "View Related Emails","targets": [{"os": "default","uri": "https://maskedurlhere.com”}]}]}’ % (messageId,subject,recipient,sender)
which throws an error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: not enough arguments for format string
I tried to use .format option also, but this fails with a different error:
teams_card='{"#type": "MessageCard","#context": "http://schema.org/extensions","themeColor": "0076D7","summary": “PTR”,”sections": [{"activityTitle": "PTR Incident Created","activitySubtitle": “End “User Exposed to Phishing Threat,”facts": [{"name": “Message” ID,”value": %s}, {"name": "Subject”,”value": %s},{“name": "End User","value": %s},{“name": “sender”,”value": %s}],”markdown": true}],"potentialAction": [{"#type": "OpenUri","name": "View Related Emails","targets": [{"os": "default","uri": "https://maskedurlhere.com”}]}]}’.format(messageId,subject,recipient,sender)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: '"#type"'
The teams card variable is fine and posts to Teams successfully when it's just text, but trying to load in these variables doesn't seem to work at all.
Any ideas?

To pass dynamic values in Json you need to use format like ${value}
Please follow below example json format
Template JSON
{
"type": "AdaptiveCard",
"body": [
{
"type": "Container",
"style": "emphasis",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"size": "Large",
"weight": "Bolder",
"text": "**EXPENSE APPROVAL**",
"wrap": true
}
],
"width": "stretch"
},
{
"type": "Column",
"items": [
{
"type": "Image",
"url": "${status_url}",
"altText": "${status}",
"height": "30px"
}
],
"width": "auto"
}
]
}
],
"bleed": true
},
{
"type": "Container",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"size": "ExtraLarge",
"text": "${purpose}",
"wrap": true
}
],
"width": "stretch"
},
{
"type": "Column",
"items": [
{
"type": "ActionSet",
"actions": [
{
"type": "Action.OpenUrl",
"title": "EXPORT AS PDF",
"url": "https://adaptivecards.io"
}
]
}
],
"width": "auto"
}
]
},
{
"type": "TextBlock",
"spacing": "Small",
"size": "Small",
"weight": "Bolder",
"color": "Accent",
"text": "[${code}](https://adaptivecards.io)",
"wrap": true
},
{
"type": "FactSet",
"spacing": "Large",
"facts": [
{
"title": "Submitted By",
"value": "**${created_by_name}** ${creater_email}"
},
{
"title": "Duration",
"value": "${formatTicks(min(select(expenses, x, int(x.created_by))), 'yyyy-MM-dd')} - ${formatTicks(max(select(expenses, x, int(x.created_by))), 'yyyy-MM-dd')}"
},
{
"title": "Submitted On",
"value": "${formatDateTime(submitted_date, 'yyyy-MM-dd')}"
},
{
"title": "Reimbursable Amount",
"value": "$${formatNumber(sum(select(expenses, x, if(x.is_reimbursable, x.total, 0))), 2)}"
},
{
"title": "Awaiting approval from",
"value": "**${approver}** ${approver_email}"
},
{
"title": "Submitted to",
"value": "**${other_submitter}** ${other_submitter_email}"
}
]
}
]
},
{
"type": "Container",
"spacing": "Large",
"style": "emphasis",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"weight": "Bolder",
"text": "DATE",
"wrap": true
}
],
"width": "auto"
},
{
"type": "Column",
"spacing": "Large",
"items": [
{
"type": "TextBlock",
"weight": "Bolder",
"text": "CATEGORY",
"wrap": true
}
],
"width": "stretch"
},
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"weight": "Bolder",
"text": "AMOUNT",
"wrap": true
}
],
"width": "auto"
}
]
}
],
"bleed": true
},
{
"$data": "${expenses}",
"type": "Container",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"text": "${formatDateTime(created_time, 'MM-dd')}",
"wrap": true
}
],
"width": "auto"
},
{
"type": "Column",
"spacing": "Medium",
"items": [
{
"type": "TextBlock",
"text": "${description}",
"wrap": true
}
],
"width": "stretch"
},
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"text": "$${formatNumber(total, 2)}",
"wrap": true
}
],
"width": "auto"
},
{
"type": "Column",
"spacing": "Small",
"selectAction": {
"type": "Action.ToggleVisibility",
"targetElements": [
"cardContent${$index}",
"chevronDown${$index}",
"chevronUp${$index}"
]
},
"verticalContentAlignment": "Center",
"items": [
{
"type": "Image",
"id": "chevronDown${$index}",
"url": "https://adaptivecards.io/content/down.png",
"width": "20px",
"altText": "${description} $${total} collapsed"
},
{
"type": "Image",
"id": "chevronUp${$index}",
"url": "https://adaptivecards.io/content/up.png",
"width": "20px",
"altText": "${description} $${total} expanded",
"isVisible": false
}
],
"width": "auto"
}
]
},
{
"type": "Container",
"id": "cardContent${$index}",
"isVisible": false,
"items": [
{
"type": "Container",
"items": [
{
"$data": "${custom_fields}",
"type": "TextBlock",
"text": "* ${value}",
"isSubtle": true,
"wrap": true
},
{
"type": "Container",
"items": [
{
"type": "Input.Text",
"id": "comment${$index}",
"placeholder": "Add your comment here."
}
]
}
]
},
{
"type": "Container",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"items": [
{
"type": "ActionSet",
"actions": [
{
"type": "Action.Submit",
"title": "Send",
"data": {
"id": "_qkQW8dJlUeLVi7ZMEzYVw",
"action": "comment",
"lineItem": 1
}
}
]
}
],
"width": "auto"
}
]
}
]
}
]
}
]
},
{
"type": "ColumnSet",
"spacing": "Large",
"separator": true,
"columns": [
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"horizontalAlignment": "Right",
"text": "Total Expense Amount \t",
"wrap": true
},
{
"type": "TextBlock",
"horizontalAlignment": "Right",
"text": "Non-reimbursable Amount",
"wrap": true
},
{
"type": "TextBlock",
"horizontalAlignment": "Right",
"text": "Advance Amount",
"wrap": true
}
],
"width": "stretch"
},
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"text": "$${formatNumber(sum(select(expenses, x, x.total)), 2)}",
"wrap": true
},
{
"type": "TextBlock",
"text": "(-) $${formatNumber(sum(select(expenses, x, if(x.is_reimbursable, 0, x.total))), 2)} \t",
"wrap": true
},
{
"type": "TextBlock",
"text": "(-) 0.00 \t",
"wrap": true
}
],
"width": "auto"
},
{
"type": "Column",
"width": "auto"
}
]
},
{
"type": "Container",
"style": "emphasis",
"items": [
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"horizontalAlignment": "Right",
"text": "Amount to be Reimbursed",
"wrap": true
}
],
"width": "stretch"
},
{
"type": "Column",
"items": [
{
"type": "TextBlock",
"weight": "Bolder",
"text": "$${formatNumber(sum(select(expenses, x, if(x.is_reimbursable, x.total, 0))), 2)}",
"wrap": true
}
],
"width": "auto"
},
{
"type": "Column",
"width": "auto"
}
]
}
],
"bleed": true
},
{
"type": "ColumnSet",
"columns": [
{
"type": "Column",
"selectAction": {
"type": "Action.ToggleVisibility",
"targetElements": [
"cardContent4",
"showHistory",
"hideHistory"
]
},
"verticalContentAlignment": "Center",
"items": [
{
"type": "TextBlock",
"id": "showHistory",
"horizontalAlignment": "Right",
"color": "Accent",
"text": "Show history",
"wrap": true
},
{
"type": "TextBlock",
"id": "hideHistory",
"horizontalAlignment": "Right",
"color": "Accent",
"text": "Hide history",
"wrap": true,
"isVisible": false
}
],
"width": 1
}
]
},
{
"type": "Container",
"id": "cardContent4",
"isVisible": false,
"items": [
{
"type": "Container",
"items": [
{
"type": "TextBlock",
"text": "* Expense submitted by **${created_by_name}** on {{DATE(${formatDateTime(created_date, 'yyyy-MM-ddTHH:mm:ssZ')}, SHORT)}}",
"isSubtle": true,
"wrap": true
},
{
"type": "TextBlock",
"text": "* Expense ${expenses[0].status} by **${expenses[0].approver}** on {{DATE(${formatDateTime(approval_date, 'yyyy-MM-ddTHH:mm:ssZ')}, SHORT)}}",
"isSubtle": true,
"wrap": true
}
]
}
]
},
{
"type": "Container",
"items": [
{
"type": "ActionSet",
"actions": [
{
"type": "Action.Submit",
"title": "Approve",
"style": "positive",
"data": {
"id": "_qkQW8dJlUeLVi7ZMEzYVw",
"action": "approve"
}
},
{
"type": "Action.ShowCard",
"title": "Reject",
"style": "destructive",
"card": {
"type": "AdaptiveCard",
"body": [
{
"type": "Input.Text",
"id": "RejectCommentID",
"placeholder": "Please specify an appropriate reason for rejection.",
"isMultiline": true
}
],
"actions": [
{
"type": "Action.Submit",
"title": "Send",
"data": {
"id": "_qkQW8dJlUeLVi7ZMEzYVw",
"action": "reject"
}
}
],
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json"
}
}
]
}
]
}
],
"$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
"version": "1.2",
"fallbackText": "This card requires Adaptive Cards v1.2 support to be rendered properly."
}
Data Json
{
"code": "ER-13052",
"message": "success",
"created_by_name" : "Matt Hidinger",
"created_date" : "2019-07-15T18:33:12+0800",
"submitted_date": "2019-04-14T18:33:12+0800",
"creater_email" : "matt#contoso.com",
"status" : "Pending",
"status_url" : "https://adaptivecards.io/content/pending.png",
"approver": "Thomas",
"purpose" : "Trip to UAE",
"approval_date" : "2019-07-15T22:33:12+0800",
"approver" : "Thomas",
"approver_email" : "thomas#contoso.com",
"other_submitter" : "David",
"other_submitter_email" : "david#contoso.com",
"expenses": [
{
"expense_id": "16367000000083065",
"approver" : "Thomas",
"date": "2017-02-21",
"description": "Air Travel Expense",
"created_by": "636965431200000000",
"created_by_name": "PATRICIA",
"employee_number": "E001",
"currency_id": "16367000000000097",
"currency_code": "USD",
"paid_through_account_id": "16367000000036003",
"paid_through_account_name": "Employee Reimbursements",
"bcy_total": 13900.79,
"bcy_subtotal": 13900.79,
"total": 300,
"total_without_tax": 300,
"is_billable": true,
"is_reimbursable": true,
"reference_number": "DD145",
"due_days": "Due in 15 days",
"merchant_id": "16367000000074027",
"merchant_name": "ABS Solutions",
"status": "approved",
"created_time": "2019-06-19T18:33:12+0800",
"last_modified_time": "2017-02-21T18:42:46+0530",
"receipt_name": "receipt1.jpg",
"report_id": "16367000000083075",
"mileage_type": "non_mileage",
"report_name": "Purchase",
"is_receipt_only": false,
"distance": 0,
"per_diem_rate": 0,
"per_diem_days": 0,
"per_diem_id": "",
"per_diem_name": "",
"expense_type": "non_mileage",
"location": "Washington",
"receipt_type": "jpg",
"policy_violated": false,
"comments_count": 0,
"report_status": "submitted",
"price_precision": 2,
"mileage_rate": 0,
"mileage_unit": "km",
"receipt_status": "processed",
"is_uncategorized": false,
"is_expired": false,
"gl_code": "LG001",
"exchange_rate": 66.943366,
"start_reading": "",
"end_reading": "",
"payment_mode": "Check",
"customer_id": "27927000000075081",
"customer_name": "ACME Corp.",
"custom_fields": [
{
"customfield_id": "16367000000277001",
"label": "Other Name",
"value": "Leg 1 on Tue, Jun 19th, 2019 at 6:00 AM."
},
{
"customfield_id": "16367000000277001",
"label": "Other Name",
"value": "Leg 2 on Tue, Jun 19th, 2019 at 7:15 PM."
}
],
"project_id": "27927000001243001",
"project_name": "Coffee Research",
"transaction_description": "",
"tax_id": "16367000000086001",
"tax_name": "Sales Tax",
"tax_percentage": 2,
"amount": 207.65,
"is_inclusive_tax": false,
"vehicle_type": "Bike",
"vehicle_id": "17456000000078029",
"fuel_type": "lpg",
"engine_capacity_range": "between_1401cc_and_1600cc",
"is_personal": false,
"policy_id": "16367000000092011",
"policy_name": "LIC",
"documents": [
{
"file_name": "receipt1.jpg",
"file_size_formatted": "71.8 KB",
"attachment_order": 1,
"document_id": "16367000000083071"
}
],
"reimbursement_reference": "",
"reimbursement_date": "",
"reimbursement_paid_through_account_id": "",
"reimbursement_paid_through_account_name": "",
"reimbursement_currency_id": "",
"reimbursement_currency_code": ""
},
{
"expense_id": "16367000000083065",
"date": "2019-06-19",
"description": "Auto Mobile Expense",
"created_by": "636965431200000000",
"created_by_name": "PATRICIA",
"employee_number": "E001",
"currency_id": "16367000000000097",
"currency_code": "USD",
"paid_through_account_id": "16367000000036003",
"paid_through_account_name": "Employee Reimbursements",
"bcy_total": 13900.79,
"bcy_subtotal": 13900.79,
"total": 100,
"total_without_tax": 100,
"is_billable": true,
"is_reimbursable": true,
"reference_number": "DD145",
"due_days": "Due in 15 days",
"merchant_id": "16367000000074027",
"merchant_name": "ABS Solutions",
"status": "submitted",
"created_time": "2019-06-19T18:33:12+0800",
"last_modified_time": "2017-02-21T18:42:46+0530",
"receipt_name": "receipt1.jpg",
"report_id": "16367000000083075",
"mileage_type": "non_mileage",
"report_name": "Purchase",
"is_receipt_only": false,
"distance": 0,
"per_diem_rate": 0,
"per_diem_days": 0,
"per_diem_id": "",
"per_diem_name": "",
"expense_type": "non_mileage",
"location": "Washington",
"receipt_type": "jpg",
"policy_violated": false,
"comments_count": 0,
"report_status": "submitted",
"price_precision": 2,
"mileage_rate": 0,
"mileage_unit": "km",
"receipt_status": "processed",
"is_uncategorized": false,
"is_expired": false,
"gl_code": "LG001",
"exchange_rate": 66.943366,
"start_reading": "",
"end_reading": "",
"payment_mode": "Check",
"customer_id": "27927000000075081",
"customer_name": "ACME Corp.",
"custom_fields": [
{
"customfield_id": "16367000000277001",
"label": "Other Name",
"value": " Contoso Car Rentrals, Tues 6/19 at 7:00 AM"
}
],
"project_id": "27927000001243001",
"project_name": "Coffee Research",
"transaction_description": "",
"tax_id": "16367000000086001",
"tax_name": "Sales Tax",
"tax_percentage": 2,
"amount": 207.65,
"is_inclusive_tax": false,
"vehicle_type": "Bike",
"vehicle_id": "17456000000078029",
"fuel_type": "lpg",
"engine_capacity_range": "between_1401cc_and_1600cc",
"is_personal": false,
"policy_id": "16367000000092011",
"policy_name": "LIC",
"documents": [
{
"file_name": "receipt1.jpg",
"file_size_formatted": "71.8 KB",
"attachment_order": 1,
"document_id": "16367000000083071"
}
],
"reimbursement_reference": "",
"reimbursement_date": "",
"reimbursement_paid_through_account_id": "",
"reimbursement_paid_through_account_name": "",
"reimbursement_currency_id": "",
"reimbursement_currency_code": ""
},
{
"expense_id": "16367000000083065",
"date": "2019-06-21",
"description": "Excess Baggage Cost",
"created_by": "636967159200000000",
"created_by_name": "PATRICIA",
"employee_number": "E001",
"currency_id": "16367000000000097",
"currency_code": "USD",
"paid_through_account_id": "16367000000036003",
"paid_through_account_name": "Employee Reimbursements",
"bcy_total": 13900.79,
"bcy_subtotal": 13900.79,
"total": 50.38,
"total_without_tax": 4.3,
"is_billable": true,
"is_reimbursable": false,
"reference_number": "DD145",
"due_days": "Due in 15 days",
"merchant_id": "16367000000074027",
"merchant_name": "ABS Solutions",
"status": "submitted",
"created_time": "2019-06-21T18:33:12+0800",
"last_modified_time": "2017-02-21T18:42:46+0530",
"receipt_name": "receipt1.jpg",
"report_id": "16367000000083075",
"mileage_type": "non_mileage",
"report_name": "Purchase",
"is_receipt_only": false,
"distance": 0,
"per_diem_rate": 0,
"per_diem_days": 0,
"per_diem_id": "",
"per_diem_name": "",
"expense_type": "non_mileage",
"location": "Washington",
"receipt_type": "jpg",
"policy_violated": false,
"comments_count": 0,
"report_status": "submitted",
"price_precision": 2,
"mileage_rate": 0,
"mileage_unit": "km",
"receipt_status": "processed",
"is_uncategorized": false,
"is_expired": false,
"gl_code": "LG001",
"exchange_rate": 66.943366,
"start_reading": "",
"end_reading": "",
"payment_mode": "Check",
"customer_id": "27927000000075081",
"customer_name": "ACME Corp.",
"custom_fields": [
],
"project_id": "27927000001243001",
"project_name": "Coffee Research",
"transaction_description": "",
"tax_id": "16367000000086001",
"tax_name": "Sales Tax",
"tax_percentage": 2,
"amount": 207.65,
"is_inclusive_tax": false,
"vehicle_type": "Bike",
"vehicle_id": "17456000000078029",
"fuel_type": "lpg",
"engine_capacity_range": "between_1401cc_and_1600cc",
"is_personal": false,
"policy_id": "16367000000092011",
"policy_name": "LIC",
"documents": [
{
"file_name": "receipt1.jpg",
"file_size_formatted": "71.8 KB",
"attachment_order": 1,
"document_id": "16367000000083071"
}
],
"reimbursement_reference": "",
"reimbursement_date": "",
"reimbursement_paid_through_account_id": "",
"reimbursement_paid_through_account_name": "",
"reimbursement_currency_id": "",
"reimbursement_currency_code": ""
}
]
}
Please go through this for more info.

I just downloaded some json from spotify and took a look into the pd.normalize_json().
But if I normalise the data i still have dictionaries within my dataframe. Also setting the level doesnt help.
DATA I want to have in my dataframe:
{
"collaborative": false,
"description": "",
"external_urls": {
"spotify": "https://open.spotify.com/playlist/5"
},
"followers": {
"href": null,
"total": 0
},
"href": "https://api.spotify.com/v1/playlists/5?additional_types=track",
"id": "5",
"images": [
{
"height": 640,
"url": "https://i.scdn.co/image/a",
"width": 640
}
],
"name": "Another",
"owner": {
"display_name": "user",
"external_urls": {
"spotify": "https://open.spotify.com/user/user"
},
"href": "https://api.spotify.com/v1/users/user",
"id": "user",
"type": "user",
"uri": "spotify:user:user"
},
"primary_color": null,
"public": true,
"snapshot_id": "M2QxNTcyYTkMDc2",
"tracks": {
"href": "https://api.spotify.com/v1/playlists/100&additional_types=track",
"items": [
{
"added_at": "2020-12-13T18:34:09Z",
"added_by": {
"external_urls": {
"spotify": "https://open.spotify.com/user/user"
},
"href": "https://api.spotify.com/v1/users/user",
"id": "user",
"type": "user",
"uri": "spotify:user:user"
},
"is_local": false,
"primary_color": null,
"track": {
"album": {
"album_type": "album",
"artists": [
{
"external_urls": {
"spotify": "https://open.spotify.com/artist/1dfeR4Had"
},
"href": "https://api.spotify.com/v1/artists/1dfDbWqFHLkxsg1d",
"id": "1dfeR4HaWDbWqFHLkxsg1d",
"name": "Q",
"type": "artist",
"uri": "spotify:artist:1dfeRqFHLkxsg1d"
}
],
"available_markets": [
"CA",
"US"
],
"external_urls": {
"spotify": "https://open.spotify.com/album/6wPXmlLzZ5cCa"
},
"href": "https://api.spotify.com/v1/albums/6wPXUJ9LzZ5cCa",
"id": "6wPXUmYJ9zZ5cCa",
"images": [
{
"height": 640,
"url": "https://i.scdn.co/image/ab676620a47",
"width": 640
},
{
"height": 300,
"url": "https://i.scdn.co/image/ab67616d0620a47",
"width": 300
},
{
"height": 64,
"url": "https://i.scdn.co/image/ab603e6620a47",
"width": 64
}
],
"name": "The (Deluxe ",
"release_date": "1920-07-17",
"release_date_precision": "day",
"total_tracks": 15,
"type": "album",
"uri": "spotify:album:6m5cCa"
},
"artists": [
{
"external_urls": {
"spotify": "https://open.spotify.com/artist/1dg1d"
},
"href": "https://api.spotify.com/v1/artists/1dsg1d",
"id": "1dfeR4HaWDbWqFHLkxsg1d",
"name": "Q",
"type": "artist",
"uri": "spotify:artist:1dxsg1d"
}
],
"available_markets": [
"CA",
"US"
],
"disc_number": 1,
"duration_ms": 21453,
"episode": false,
"explicit": false,
"external_ids": {
"isrc": "GBU6015"
},
"external_urls": {
"spotify": "https://open.spotify.com/track/5716J"
},
"href": "https://api.spotify.com/v1/tracks/5716J",
"id": "5716J",
"is_local": false,
"name": "Another",
"popularity": 73,
"preview_url": null,
"track": true,
"track_number": 3,
"type": "track",
"uri": "spotify:track:516J"
},
"video_thumbnail": {
"url": null
}
}
],
"limit": 100,
"next": null,
"offset": 0,
"previous": null,
"total": 1
},
"type": "playlist",
"uri": "spotify:playlist:fek"
}
So what are best practices to read nested data like this into one dataframe in pandas?
I'm glad for any advice.
EDIT:
so basically I want all keys as columns in my dataframe. But with normalise it stops at "tracks.items" and if I normalise this again i have the recursive problem again.

It depends on the information you are looking for. Take a look at pandas.read_json() to see if that can work. Also you can select data as such
json_output = {"collaborative": 'false',"description": "", "external_urls": {"spotify": "https://open.spotify.com/playlist/5"}}
df['collaborative'] = json_output['collaborative'] #set value of your df to value of returned json values

{
"type": "Data",
"version": "1.0",
"box": {
"identifier": "abcdef",
"serial": "12345678"
},
"payload": {
"Type": "EL",
"Version": "1",
"Result": "Successful",
"Reference": null,
"Box": {
"Identifier": "abcdef",
"Serial": "12345678"
},
"Configuration": {
"EL": "1"
},
"vent": [
{
"ventType": "Arm",
"Timestamp": "2020-03-18T12:17:04+10:00",
"Parameters": [
{
"Name": "Arm",
"Value": "LT"
},
{
"Name": "Status",
"Value": "LD"
}
]
},
{
"ventType": "Arm",
"Timestamp": "2020-03-18T12:17:24+10:00",
"Parameters": [
{
"Name": "Arm",
"Value": "LT"
},
{
"Name": "Status",
"Value": "LD"
}
]
},
{
"EventType": "TimeUpdateCompleted",
"Timestamp": "2020-03-18T02:23:21.2979668Z",
"Parameters": [
{
"Name": "ActualAdjustment",
"Value": "PT0S"
},
{
"Name": "CorrectionOffset",
"Value": "PT0S"
},
{
"Name": "Latency",
"Value": "PT0.2423996S"
}
]
}
]
}
}

If you're looking to transfer information from a JSON file to a CSV, then you can use the following code to read in a JSON file into a dictionary in Python:
import json
with open('data.txt') as json_file:
data_dict = json.load(json_file)
You could then convert this dictionary into a list with either data_dict.items() or data_dict.values().
Then you just need to write this list to a CSV file which you can easily do by just looping through the list.

The following deploys a azure function that run the specified C#. How do I do the same for a function that should run python?
I tried just changing the name to __init__.py as is generated when you use the azure-function-core-tools func command with the --python switch, but couldn't even find error messages as to why things weren't working.
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"appName": {
"type": "string",
"metadata": {
"description": "The name of the function app that you wish to create."
}
},
"storageAccountType": {
"type": "string",
"defaultValue": "Standard_LRS",
"allowedValues": [
"Standard_LRS",
"Standard_GRS",
"Standard_ZRS",
"Premium_LRS"
],
"metadata": {
"description": "Storage Account type"
}
}
},
"variables": {
"functionAppName": "[parameters('appName')]",
"hostingPlanName": "[parameters('appName')]",
"storageAccountName": "[concat(uniquestring(resourceGroup().id), 'azfunctions')]"
},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts",
"name": "[variables('storageAccountName')]",
"apiVersion": "2015-06-15",
"location": "[resourceGroup().location]",
"properties": {
"accountType": "[parameters('storageAccountType')]"
}
},
{
"type": "Microsoft.Web/serverfarms",
"apiVersion": "2015-04-01",
"name": "[variables('hostingPlanName')]",
"location": "[resourceGroup().location]",
"properties": {
"name": "[variables('hostingPlanName')]",
"computeMode": "Dynamic",
"sku": "Dynamic"
}
},
{
"apiVersion": "2015-08-01",
"type": "Microsoft.Web/sites",
"name": "[variables('functionAppName')]",
"location": "[resourceGroup().location]",
"kind": "functionapp",
"properties": {
"name": "[variables('functionAppName')]",
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]"
},
"dependsOn": [
"[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]",
"[resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName'))]"
],
"resources": [
{
"apiVersion": "2016-03-01",
"name": "appsettings",
"type": "config",
"dependsOn": [
"[resourceId('Microsoft.Web/sites', variables('functionAppName'))]",
"[resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName'))]"
],
"properties": {
"AzureWebJobsStorage": "[concat('DefaultEndpointsProtocol=https;AccountName=',variables('storageAccountName'),';AccountKey=',listkeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2015-05-01-preview').key1,';')]",
"AzureWebJobsDashboard": "[concat('DefaultEndpointsProtocol=https;AccountName=',variables('storageAccountName'),';AccountKey=',listkeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2015-05-01-preview').key1,';')]",
"FUNCTIONS_EXTENSION_VERSION": "latest"
}
},
{
"apiVersion": "2015-08-01",
"name": "TestFunctionCM",
"type": "functions",
"dependsOn": [
"[resourceId('Microsoft.Web/sites', variables('functionAppName'))]"
],
"properties": {
"config": {
"bindings": [
{
"authLevel": "anonymous",
"name": "req",
"type": "httpTrigger",
"direction": "in"
},
{
"name": "res",
"type": "http",
"direction": "out"
}
]
},
"files": {
"run.csx": "using System.Net;\r\n\r\n public static HttpResponseMessage Run(HttpRequestMessage req, TraceWriter log)\r\n\r\n {\r\n\r\nreturn req.CreateResponse(\"Hello from MyFunction\", HttpStatusCode.OK);\r\n\r\n }"
}
}
}
]
}
]
}
Thank you.

You will probably need the following:
Runtime under appsettings
"FUNCTIONS_WORKER_RUNTIME": "python"
My template looks bit different but does deploy a python function, here is the resource from the same:
{
"type": "Microsoft.Web/sites",
"apiVersion": "2018-11-01",
"name": "[parameters('name')]",
"location": "[parameters('location')]",
"dependsOn": [
"microsoft.insights/components/mycoolfunction",
"[concat('Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]",
"[concat('Microsoft.Storage/storageAccounts/', parameters('storageAccountName'))]"
],
"tags": {},
"kind": "functionapp,linux",
"properties": {
"name": "[parameters('name')]",
"siteConfig": {
"appSettings": [
{
"name": "FUNCTIONS_WORKER_RUNTIME",
"value": "python"
},
{
"name": "FUNCTIONS_EXTENSION_VERSION",
"value": "~2"
},
{
"name": "APPINSIGHTS_INSTRUMENTATIONKEY",
"value": "[reference('microsoft.insights/components/mycoolfunction', '2015-05-01').InstrumentationKey]"
},
{
"name": "AzureWebJobsStorage",
"value": "[concat('DefaultEndpointsProtocol=https;AccountName=',parameters('storageAccountName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2019-06-01').keys[0].value,';EndpointSuffix=','core.windows.net')]"
}
]
},
"serverFarmId": "[concat('/subscriptions/', parameters('subscriptionId'),'/resourcegroups/', parameters('serverFarmResourceGroup'), '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]",
"hostingEnvironment": "[parameters('hostingEnvironment')]",
"clientAffinityEnabled": false
}
}

I am trying to remove the outer (parent) layer of a JSON file so that I can process it, however I have no idea how.
As you will see by the code below, the outer 2 most layers are 2 dictionaries, however, python says the 2nd dictionary ("item") is just a string when I call its type. Am I incorrect in how I interpret the structure?
sample_object6 = {
"items":
{
"item":
[
{
"id": "0001",
"type": "donut",
"name": "Cake",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
{ "id": "1002", "type": "Chocolate" },
{ "id": "1003", "type": "Blueberry" },
{ "id": "1004", "type": "Devil's Food" }
]
},
"topping":
[
{ "id": "5001", "type": "None" },
{ "id": "5002", "type": "Glazed" },
{ "id": "5005", "type": "Sugar" },
{ "id": "5007", "type": "Powdered Sugar" },
{ "id": "5006", "type": "Chocolate with Sprinkles" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
]
},
{
"id": "0002",
"type": "donut",
"name": "Raised",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" }
]
},
"topping":
[
{ "id": "5001", "type": "None" },
{ "id": "5002", "type": "Glazed" },
{ "id": "5005", "type": "Sugar" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
]
},
{
"id": "0003",
"type": "donut",
"name": "Old Fashioned",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
{ "id": "1002", "type": "Chocolate" }
]
},
"topping":
[
{ "id": "5001", "type": "None" },
{ "id": "5002", "type": "Glazed" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
]
},
{
"id": "0004",
"type": "bar",
"name": "Bar",
"ppu": 0.75,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
]
},
"topping":
[
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
],
"fillings":
{
"filling":
[
{ "id": "7001", "name": "None", "addcost": 0 },
{ "id": "7002", "name": "Custard", "addcost": 0.25 },
{ "id": "7003", "name": "Whipped Cream", "addcost": 0.25 }
]
}
},
{
"id": "0005",
"type": "twist",
"name": "Twist",
"ppu": 0.65,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
]
},
"topping":
[
{ "id": "5002", "type": "Glazed" },
{ "id": "5005", "type": "Sugar" },
]
},
{
"id": "0006",
"type": "filled",
"name": "Filled",
"ppu": 0.75,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
]
},
"topping":
[
{ "id": "5002", "type": "Glazed" },
{ "id": "5007", "type": "Powdered Sugar" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
],
"fillings":
{
"filling":
[
{ "id": "7002", "name": "Custard", "addcost": 0 },
{ "id": "7003", "name": "Whipped Cream", "addcost": 0 },
{ "id": "7004", "name": "Strawberry Jelly", "addcost": 0 },
{ "id": "7005", "name": "Rasberry Jelly", "addcost": 0 }
]
}
}
]
}
}
I thought that it might be possible to store the nested portion starting at the first list (right after 'item') in a variable and then work with this but if I can't get python to see that item is a dictionary inside the items dictionary, then I fear I am at a loss with how to proceed.
Does anyone know what I am doing wrong?
Thank you in advance!
As far as the processing goes, there has been none because I could not even get the string to read as a dictionary appropriately.
This is what I tried to test if it was a dictionary:
for i in sample_object6:
print(i + str(type(i)))
for n in i["item"]:
print(n + str(type(n)))

After submitting the same code that I thought I had already submitted, I noticed that python is interpreting the object correctly. I have some obvious fundamental gaps in how to work in python and I'm sorry I took it to the forum.
For the record (and for future python newbies out there like me), I used the following code which returned the proper class types:
#this returned a class type of dictionary
print(type(sample_object6["items"]))
#this returned a class type of list
print(type(sample_object6["items"]["item"]))
Thank you SungJin Steve Yoo & Pm2Ring for your help.