I'm trying to write code to use data to generate a report. Instead of iterating through the dictionary, I wanted to use Pandas this time.
So, the first issue I faced was null in the data. I corrected it using json.loads().
I am trying to understand how can I get the nested JSON to Pandas.
How should I go about getting the below data to pandas? read_json() throws an error.
{
"Scans": [
{
"Targets": [
{
"Id": "5a8f415a-5146-4e39-8827-33cdb14ad478",
"Host": "xyz.com"
}
],
"Id": "51b13233-b3de-4f26-81d6-0001314f635f",
"Status": 32,
"StartTime": "2021-05-25T16:00:13.16",
"WindowScanStart": "2021-05-25T16:00:00.76",
"WindowScanStop": null,
"StartedBy": "8cafa5bc-b0df-496f-bab7-74f71eeadf9d",
"StoppedTime": null,
"CompletionTime": "2021-05-25T17:18:43.007",
"IsApproveRequired": false,
"IsMonitoring": false,
"IsUploaded": false,
"IsImported": true,
"SubStatus": null
},
{
"Targets": [
{
"Id": "6108c410-7d5c-41c9-979a-c8bf70e6f6bf",
"Host": "abc.com"
}
],
"Id": "de3a98a3-cb95-42ce-874b-00037347ebba",
"Status": 72,
"StartTime": "2021-06-07T19:50:01.85",
"WindowScanStart": "2021-06-07T19:49:44.517",
"WindowScanStop": null,
"StartedBy": "b6a3f887-0b4a-43e1-ba5b-fd93e60e58b6",
"StoppedTime": null,
"CompletionTime": "2021-06-07T19:50:53.667",
"IsApproveRequired": false,
"IsMonitoring": false,
"IsUploaded": false,
"IsImported": false,
"SubStatus": null
},
],
"IsSuccess": true,
"Reason": null,
"ErrorMessage": null
}
I am able to handle nulls using json.loads(). How can I create a Pandas Dataframe from this?
I suspect your problem is the comma after the } in the line before the ],
There is no following element, so the comma is not correct.
Remove that and try again.
{
"Scans": [
{
"Targets": [
{
"Id": "5a8f415a-5146-4e39-8827-33cdb14ad478",
"Host": "xyz.com"
}
],
"Id": "51b13233-b3de-4f26-81d6-0001314f635f",
"Status": 32,
"StartTime": "2021-05-25T16:00:13.16",
"WindowScanStart": "2021-05-25T16:00:00.76",
"WindowScanStop": null,
"StartedBy": "8cafa5bc-b0df-496f-bab7-74f71eeadf9d",
"StoppedTime": null,
"CompletionTime": "2021-05-25T17:18:43.007",
"IsApproveRequired": false,
"IsMonitoring": false,
"IsUploaded": false,
"IsImported": true,
"SubStatus": null
},
{
"Targets": [
{
"Id": "6108c410-7d5c-41c9-979a-c8bf70e6f6bf",
"Host": "abc.com"
}
],
"Id": "de3a98a3-cb95-42ce-874b-00037347ebba",
"Status": 72,
"StartTime": "2021-06-07T19:50:01.85",
"WindowScanStart": "2021-06-07T19:49:44.517",
"WindowScanStop": null,
"StartedBy": "b6a3f887-0b4a-43e1-ba5b-fd93e60e58b6",
"StoppedTime": null,
"CompletionTime": "2021-06-07T19:50:53.667",
"IsApproveRequired": false,
"IsMonitoring": false,
"IsUploaded": false,
"IsImported": false,
"SubStatus": null
}
],
"IsSuccess": true,
"Reason": null,
"ErrorMessage": null
}
The program for creating the database, I am trying to do that in python. I have a json file whose contents will be used to populate the db.
In this we need to parse a JSON file, create a relational database dbjson corresponding to the JSON file, and upload the data into the dbjson.
The following is the formatted json file:
{
"cursor": {
"prev": null,
"hasNext": false,
"next": "1542275432008325:1:0",
"hasPrev": false,
"total": null,
"id": "1542275432008325:1:0",
"more": false
},
"code": 0,
"response": [
{
"editableUntil": "2018-11-22T11:20:37",
"dislikes": 0,
"numReports": 0,
"likes": 1,
"message": "<p>A sinking pound, resignations by the bucketful railroading through a so-called agreement nobody wants. For the good of the country Mrs. May 'go and go now'.</p>",
"id": "4196155749",
"createdAt": "2018-11-15T11:20:37",
"author": {
"username": "mnaid-233509073ed3432027d48b1a83f5fbd2",
"about": "",
"name": "baggiebuoy",
"disable3rdPartyTrackers": true,
"isPowerContributor": false,
"joinedAt": "2017-05-03T10:40:52",
"profileUrl": "https://disqus.com/by/mnaid-233509073ed3432027d48b1a83f5fbd2/",
"url": "",
"location": "",
"isPrivate": false,
"signedUrl": "",
"isPrimary": true,
"isAnonymous": false,
"id": "250728493",
"avatar": {
"small": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-233509073ed3432027d48b1a83f5fbd2.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar32.png"
},
"isCustom": false,
"permalink": "https://disqus.com/api/users/avatars/mnaid-233509073ed3432027d48b1a83f5fbd2.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png",
"large": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-233509073ed3432027d48b1a83f5fbd2.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png"
}
}
},
"media": [],
"isSpam": false,
"isDeletedByAuthor": false,
"isDeleted": false,
"parent": null,
"isApproved": true,
"isFlagged": false,
"raw_message": "A sinking pound, resignations by the bucketful railroading through a so-called agreement nobody wants. For the good of the country Mrs. May 'go and go now'.",
"isHighlighted": false,
"canVote": false,
"thread": "7046384220",
"forum": "expressandstar",
"points": 1,
"moderationLabels": [],
"isEdited": true,
"sb": false
},
{
"editableUntil": "2018-11-22T10:37:59",
"dislikes": 0,
"numReports": 0,
"likes": 0,
"message": "<p>This could be heaven, or this could be hell.</p>",
"id": "4196048572",
"createdAt": "2018-11-15T10:37:59",
"author": {
"username": "mnaid-29e1c59be16c852670e3be302e8c303b",
"about": "",
"name": "Mordecai",
"disable3rdPartyTrackers": false,
"isPowerContributor": false,
"joinedAt": "2017-05-03T13:01:02",
"profileUrl": "https://disqus.com/by/mnaid-29e1c59be16c852670e3be302e8c303b/",
"url": "",
"location": "",
"isPrivate": false,
"signedUrl": "",
"isPrimary": true,
"isAnonymous": false,
"id": "250739494",
"avatar": {
"small": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-29e1c59be16c852670e3be302e8c303b.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar32.png"
},
"isCustom": false,
"permalink": "https://disqus.com/api/users/avatars/mnaid-29e1c59be16c852670e3be302e8c303b.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png",
"large": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-29e1c59be16c852670e3be302e8c303b.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png"
}
}
},
"media": [],
"isSpam": false,
"isDeletedByAuthor": false,
"isDeleted": false,
"parent": 4196013534,
"isApproved": true,
"isFlagged": false,
"raw_message": "This could be heaven, or this could be hell.",
"isHighlighted": false,
"canVote": false,
"thread": "7046384220",
"forum": "expressandstar",
"points": 0,
"moderationLabels": [],
"isEdited": false,
"sb": false
},
{
"editableUntil": "2018-11-22T10:36:50",
"dislikes": 0,
"numReports": 0,
"likes": 0,
"message": "<p>The crappest of crap deals.<br>Will never get through Parliament.</p><p>You are the weakest link May, goodbye.</p>",
"id": "4196044068",
"createdAt": "2018-11-15T10:36:50",
"author": {
"username": "mnaid-29e1c59be16c852670e3be302e8c303b",
"about": "",
"name": "Mordecai",
"disable3rdPartyTrackers": false,
"isPowerContributor": false,
"joinedAt": "2017-05-03T13:01:02",
"profileUrl": "https://disqus.com/by/mnaid-29e1c59be16c852670e3be302e8c303b/",
"url": "",
"location": "",
"isPrivate": false,
"signedUrl": "",
"isPrimary": true,
"isAnonymous": false,
"id": "250739494",
"avatar": {
"small": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-29e1c59be16c852670e3be302e8c303b.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar32.png"
},
"isCustom": false,
"permalink": "https://disqus.com/api/users/avatars/mnaid-29e1c59be16c852670e3be302e8c303b.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png",
"large": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-29e1c59be16c852670e3be302e8c303b.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png"
}
}
},
"media": [],
"isSpam": false,
"isDeletedByAuthor": false,
"isDeleted": false,
"parent": null,
"isApproved": true,
"isFlagged": false,
"raw_message": "The crappest of crap deals.\nWill never get through Parliament.\n\nYou are the weakest link May, goodbye.",
"isHighlighted": false,
"canVote": false,
"thread": "7046384220",
"forum": "expressandstar",
"points": 0,
"moderationLabels": [],
"isEdited": false,
"sb": false
},
{
"editableUntil": "2018-11-22T10:28:13",
"dislikes": 0,
"numReports": 0,
"likes": 0,
"message": "<p>We are all just prisoners here of our own device. 😁</p>",
"id": "4196013534",
"createdAt": "2018-11-15T10:28:13",
"author": {
"username": "mnaid-42ba513c42a0fd6558aa44b1de658140",
"about": "",
"name": "chaffwolf",
"disable3rdPartyTrackers": true,
"isPowerContributor": false,
"joinedAt": "2017-12-20T16:45:10",
"profileUrl": "https://disqus.com/by/mnaid-42ba513c42a0fd6558aa44b1de658140/",
"url": "",
"location": "",
"isPrivate": false,
"signedUrl": "",
"isPrimary": true,
"isAnonymous": false,
"id": "274657487",
"avatar": {
"small": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-42ba513c42a0fd6558aa44b1de658140.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar32.png"
},
"isCustom": false,
"permalink": "https://disqus.com/api/users/avatars/mnaid-42ba513c42a0fd6558aa44b1de658140.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png",
"large": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-42ba513c42a0fd6558aa44b1de658140.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png"
}
}
},
"media": [],
"isSpam": false,
"isDeletedByAuthor": false,
"isDeleted": false,
"parent": 4195981074,
"isApproved": true,
"isFlagged": false,
"raw_message": "We are all just prisoners here of our own device. 😁",
"isHighlighted": false,
"canVote": false,
"thread": "7046384220",
"forum": "expressandstar",
"points": 0,
"moderationLabels": [],
"isEdited": false,
"sb": false
},
{
"editableUntil": "2018-11-22T10:27:03",
"dislikes": 0,
"numReports": 0,
"likes": 0,
"message": "<p>Should be no deal end off.<br>Nobody could possibly be the winner so it's simple leave and that's it .<br>Their will always be wannabes who think they can do better in it for only their own powers not for the general country or the people of it .,</p>",
"id": "4196012501",
"createdAt": "2018-11-15T10:27:03",
"author": {
"username": "mnaid-90ecce8d5dad4396f681182cb470872c",
"about": "",
"name": "wanderer in eire",
"disable3rdPartyTrackers": true,
"isPowerContributor": false,
"joinedAt": "2017-05-12T20:20:43",
"profileUrl": "https://disqus.com/by/mnaid-90ecce8d5dad4396f681182cb470872c/",
"url": "",
"location": "",
"isPrivate": false,
"signedUrl": "",
"isPrimary": true,
"isAnonymous": false,
"id": "251694793",
"avatar": {
"small": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-90ecce8d5dad4396f681182cb470872c.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar32.png"
},
"isCustom": false,
"permalink": "https://disqus.com/api/users/avatars/mnaid-90ecce8d5dad4396f681182cb470872c.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png",
"large": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-90ecce8d5dad4396f681182cb470872c.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png"
}
}
},
"media": [],
"isSpam": false,
"isDeletedByAuthor": false,
"isDeleted": false,
"parent": null,
"isApproved": true,
"isFlagged": false,
"raw_message": "Should be no deal end off.\nNobody could possibly be the winner so it's simple leave and that's it .\nTheir will always be wannabes who think they can do better in it for only their own powers not for the general country or the people of it .,",
"isHighlighted": false,
"canVote": false,
"thread": "7046384220",
"forum": "expressandstar",
"points": 0,
"moderationLabels": [],
"isEdited": false,
"sb": false
},
{
"editableUntil": "2018-11-22T10:26:42",
"dislikes": 0,
"numReports": 0,
"likes": 0,
"message": "<p>Damned if she does, damned if she doesn't. All for a cause she does not believe in.<br>She will go down in history whatever happens. <br>The question is: who better to lead the country at such an important juncture??<br>Answers on a postcard!</p>",
"id": "4196012237",
"createdAt": "2018-11-15T10:26:42",
"author": {
"username": "mnaid-42ba513c42a0fd6558aa44b1de658140",
"about": "",
"name": "chaffwolf",
"disable3rdPartyTrackers": true,
"isPowerContributor": false,
"joinedAt": "2017-12-20T16:45:10",
"profileUrl": "https://disqus.com/by/mnaid-42ba513c42a0fd6558aa44b1de658140/",
"url": "",
"location": "",
"isPrivate": false,
"signedUrl": "",
"isPrimary": true,
"isAnonymous": false,
"id": "274657487",
"avatar": {
"small": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-42ba513c42a0fd6558aa44b1de658140.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar32.png"
},
"isCustom": false,
"permalink": "https://disqus.com/api/users/avatars/mnaid-42ba513c42a0fd6558aa44b1de658140.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png",
"large": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-42ba513c42a0fd6558aa44b1de658140.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png"
}
}
},
"media": [],
"isSpam": false,
"isDeletedByAuthor": false,
"isDeleted": false,
"parent": null,
"isApproved": true,
"isFlagged": false,
"raw_message": "Damned if she does, damned if she doesn't. All for a cause she does not believe in.\nShe will go down in history whatever happens. \nThe question is: who better to lead the country at such an important juncture??\nAnswers on a postcard!",
"isHighlighted": false,
"canVote": false,
"thread": "7046384220",
"forum": "expressandstar",
"points": 0,
"moderationLabels": [],
"isEdited": false,
"sb": false
},
{
"editableUntil": "2018-11-22T09:50:32",
"dislikes": 0,
"numReports": 0,
"likes": 0,
"message": "<p>Theresa May is using The Eagles Hotel California lyrics as the inspiration for her Brexit plan.</p><p>You can check out any time you like, but you can never leave.</p>",
"id": "4195981074",
"createdAt": "2018-11-15T09:50:32",
"author": {
"username": "mnaid-e82a88d937e60267fd2c866b01131ada",
"about": "",
"name": "Olly the cat",
"disable3rdPartyTrackers": false,
"isPowerContributor": false,
"joinedAt": "2017-05-03T10:27:23",
"profileUrl": "https://disqus.com/by/mnaid-e82a88d937e60267fd2c866b01131ada/",
"url": "",
"location": "",
"isPrivate": false,
"signedUrl": "",
"isPrimary": true,
"isAnonymous": false,
"id": "250727584",
"avatar": {
"small": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-e82a88d937e60267fd2c866b01131ada.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar32.png"
},
"isCustom": false,
"permalink": "https://disqus.com/api/users/avatars/mnaid-e82a88d937e60267fd2c866b01131ada.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png",
"large": {
"permalink": "https://disqus.com/api/users/avatars/mnaid-e82a88d937e60267fd2c866b01131ada.jpg",
"cache": "//a.disquscdn.com/1541535443/images/noavatar92.png"
}
}
},
"media": [],
"isSpam": false,
"isDeletedByAuthor": false,
"isDeleted": false,
"parent": null,
"isApproved": true,
"isFlagged": false,
"raw_message": "Theresa May is using The Eagles Hotel California lyrics as the inspiration for her Brexit plan.\n\nYou can check out any time you like, but you can never leave.",
"isHighlighted": false,
"canVote": false,
"thread": "7046384220",
"forum": "expressandstar",
"points": 0,
"moderationLabels": [],
"isEdited": false,
"sb": false
}
]
}
This data must be used to populate the database.
The following is the implementation I have worked out:
import csv, json, sys
if sys.argv[1] is not None and sys.argv[2] is not None:
fileInput = sys.argv[1]
fileOutput = sys.argv[2]
inputFile = open(fileInput)
outputFile = open(fileOutput, 'w')
data = json.load(inputFile)
inputFile.close()
output = csv.writer(outputFile)
output.writerow(data.keys())
#output.writerow(data[0].keys()) # header row
for row in data:
output.writerow(row.values())
I tried to parse data using json package in python but have not been able to successfully implement it.
This seems a similar problem.
Have you tried it? Please edit with your code and errors
Im using python 3 and am trying to access some information from a json query. Currently I can access data within payments by looping through. I try to do the same to access order_status and it doesn't work.
Json:
___order details___
{
"message_type_name": "OrderPlaced",
"order": {
"adcode": "",
"adcode_id": 0,
"billing_address": {
"address_line_1": "123 east street",
"address_line_2": "apt one"
},
"campaign_code": "",
"channel": "Online",
"customer": {
"adcode": "",
"adcode_id": 0,
"affiliate_id": 0,
"alternate_phone_number": ""
},
"device": "None",
"discount_total": 0.0,
"discounted_shipping_total": 0.0,
"items": [
{
"admin_comments": "",
"cost": 90.0,
"created_at": "2018-12-04T17:14:55.1128646-06:00"
}
],
"manufacturer_invoice_amount": null,
"manufacturer_invoice_number": "",
"manufacturer_invoice_paid": false,
"order_status": {
"color": "#A4E065",
"created_at": null,
"email_template_id": null,
"id": 6,
"is_cancelled": false,
"is_declined": false,
"is_fully_refunded": false,
"is_open": true,
"is_partially_refunded": false,
"is_quote_status": false,
"is_shipped": false,
"name": "Awaiting Payment",
"updated_at": "2015-10-12T22:07:47.487-05:00"
},
"order_status_id": 6,
"order_status_last_changed_at": "2018-12-04T17:14:55.0503538-06:00",
"order_type": "Order",
"payments": [
{
"amount": 100.00,
"approved_at": "",
"authorization_code": ""
"
}
],
"ppc_keyword": "",
"previous_order_status_id": 6,
"shipments": [],
"shipping_address": {
"comments": "",
"company": "",
"country": "United States"
},
"shipping_total": 0.0,
"source": "Google [organic]"
},
"store_id": 1
}
I have a variable called data: data = json.loads(self.rfile.read( length ).decode('utf-8'))
I have another variable order_payments = data['payments']
I can loop through that and access is_declined
I want to have a variable called order_status = data['order_status'], and then loop through that to access name. I get a key error on order_status = data['order_status']and I dont know why.
How can I convert the stripe model object into JSON to receive complete hierarchical data at client end?
stripeCustomer = stripe.Customer.retrieve(<stripe customer id>)
sendResponseToClient(stripeCustomer)
I am receiving only 1st level of data as json at client end, second level data from Stripe JSON object is not formatted.
JSON Data Example: (2nd level of data is not received at client end,)
Customer customer id=cus_DTWEPsfrHx3ikZ at 0x00000a> JSON: {
"id": "cus_DTWEPsfrHx3ikZ",
"object": "customer",
"account_balance": 0,
"created": 1535093686,
"currency": "usd",
"default_source": null,
"delinquent": false,
"description": null,
"discount": null,
"email": "rakesh16+test9#gmail.com",
"invoice_prefix": "E91FF30",
"livemode": false,
"metadata": {
},
"shipping": null,
"sources": {
"object": "list",
"data": [
],
"has_more": false,
"total_count": 0,
"url": "/v1/customers/cus_DTWEPsfrHx3ikZ/sources"
},
"subscriptions": {
"object": "list",
"data": [
{
"id": "sub_DTWEALN3urFael",
"object": "subscription",
"application_fee_percent": null,
"billing": "charge_automatically",
"billing_cycle_anchor": 1535093688,
"cancel_at_period_end": false,
"canceled_at": null,
"created": 1535093688,
"current_period_end": 1537772088,
"current_period_start": 1535093688,
"customer": "cus_DTWEPsfrHx3ikZ",
"days_until_due": null,
"discount": null,
"ended_at": null,
"items": {
"object": "list",
"data": [
{
"id": "si_DTWEuZaU4pw9Cv",
"object": "subscription_item",
"created": 1535093688,
"metadata": {
},
"plan": {
"id": "plan_free",
"object": "plan",
"active": true,
"aggregate_usage": null,
"amount": 0,
"billing_scheme": "per_unit",
"created": 1535008667,
"currency": "usd",
"interval": "month",
"interval_count": 1,
"livemode": false,
"metadata": {
},
"nickname": "free",
"product": "prod_DT8B8auk3CRNdw",
"tiers": null,
"tiers_mode": null,
"transform_usage": null,
"trial_period_days": null,
"usage_type": "licensed"
},
"quantity": 1,
"subscription": "sub_DTWEALN3urFael"
}
],
"has_more": false,
"total_count": 1,
"url": "/v1/subscription_items?subscription=sub_DTWEALN3urFael"
},
"livemode": false,
"metadata": {
},
"plan": {
"id": "plan_free",
"object": "plan",
"active": true,
"aggregate_usage": null,
"amount": 0,
"billing_scheme": "per_unit",
"created": 1535008667,
"currency": "usd",
"interval": "month",
"interval_count": 1,
"livemode": false,
"metadata": {
},
"nickname": "free",
"product": "prod_DT8B8auk3CRNdw",
"tiers": null,
"tiers_mode": null,
"transform_usage": null,
"trial_period_days": null,
"usage_type": "licensed"
},
"quantity": 1,
"start": 1535093688,
"status": "active",
"tax_percent": null,
"trial_end": null,
"trial_start": null
}
],
"has_more": false,
"total_count": 1,
"url": "/v1/customers/cus_DTWEPsfrHx3ikZ/subscriptions"
},
"tax_info": null,
"tax_info_verification": null
}
You can convert the returned Stripe object model to JSON using the following technique to ignore non-serializable fields:
stripeObject = stripe.SomeAPICall(...)
jsonEncoded = json.dumps(stripeObject, default=lambda o: '<not serializable>')
pythonDict = json.loads(jsonEncoded)
you can to_dict() api which will convert the stripe model object into dictionary format and then can be eventually converted into JSON string.