Develop Reference

How to iterate over a 2D array in Python

I am making a get request to receive data. The requests and response is below. I am trying to iterate over the results and do 2 things:
Iterate over an undefined number of pages (max 100 results per page so I need to paginate over my response to get all results).
Print/Return the list of orders where CombinedPaymentStatus = PAID.
I'm sure my request isn't complete however I'm stuck at this point. I have gotten this error most recently:
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 129 (char 128).
I thought using the variable paymentStatus would help, the index [1] is static so will not iterate over each item in the array.
Request:
import requests
import json
import config
url = 'https://inventory.dearsystems.com/ExternalApi/v2/saleList?Page=25&Limit=100&CreatedSince=2021-05-01'
payload = {}
headers = {
"Content-Type": "application/json",
"api-auth-accountid": config.liveAccountid,
"api-auth-applicationkey": config.liveApplicationKey
}
response = requests.request("GET", url, headers=headers, data=payload)
paymentStatus = response.json()["SaleList"][1]["CombinedPaymentStatus"]
total = response.json()["Total"]
page = response.json()["Page"]
for paymentStatus in response:
json.loads(paymentStatus)
print(paymentStatus)
Response:
{
"Total": 6471,
"Page": 1,
"SaleList": [
{
"SaleID": "6222cb47-0e1e-450d-af6d-0cefcee9eef7",
"OrderNumber": "SO-00092",
"Status": "ORDERED",
"OrderDate": "2017-09-29T00:00:00Z",
"InvoiceDate": null,
"Customer": "Rock Star Transport",
"CustomerID": "74de0528-fae7-4233-9222-e92110d57f5a",
"InvoiceNumber": null,
"CustomerReference": "",
"InvoiceAmount": 0,
"PaidAmount": 0,
"InvoiceDueDate": null,
"ShipBy": null,
"BaseCurrency": "RUB",
"CustomerCurrency": "RUB",
"CreditNoteNumber": null,
"Updated": "2017-09-29T03:03:13.913Z",
"QuoteStatus": "AUTHORISED",
"OrderStatus": "AUTHORISED",
"CombinedPickingStatus": "PICKED",
"CombinedPaymentStatus": "PAID",
"CombinedTrackingNumbers": "",
"CombinedPackingStatus": "PACKED",
"CombinedShippingStatus": "SHIPPING",
"CombinedInvoiceStatus": "INVOICED",
"CreditNoteStatus": "NOT AVAILABLE",
"FulFilmentStatus": "NOT FULFILLED",
"Type": "Advanced Sale",
"SourceChannel": "Amazon_US",
"ExternalID": null,
"OrderLocationID": "8b5d4343-c007-43d7-8e8f-6fa6c0d29f22"
},
{
"SaleID": "a6558396-8893-479b-bca9-f89ea3e54633",
"OrderNumber": "SO-00091",
"Status": "BACKORDERED",
"OrderDate": "2017-09-28T00:00:00Z",
"InvoiceDate": "2017-09-28T00:00:00Z",
"Customer": "Rock Star Transport",
"CustomerID": "9a4513e9-a7a4-4ee5-b240-84cfd8944cde",
"InvoiceNumber": "INV-06073",
"CustomerReference": "19614",
"InvoiceAmount": 0,
"PaidAmount": 0,
"InvoiceDueDate": "2017-10-02T00:00:00Z",
"ShipBy": "2017-09-29T00:00:00Z",
"BaseCurrency": "RUB",
"CustomerCurrency": "AUD",
"CreditNoteNumber": null,
"Updated": "2017-09-29T03:00:48.043Z",
"QuoteStatus": "NOT AVAILABLE",
"OrderStatus": "AUTHORISED",
"CombinedPickingStatus": "NOT PICKED",
"CombinedPaymentStatus": "UNPAID",
"CombinedTrackingNumbers": "",
"CombinedPackingStatus": "NOT PACKED",
"CombinedShippingStatus": "NOT SHIPPED",
"CombinedInvoiceStatus": "NOT INVOICED",
"CreditNoteStatus": "NOT AVAILABLE",
"FulFilmentStatus": "NOT FULFILLED",
"Type": "Simple Sale",
"SourceChannel": null,
"ExternalID": null,
"OrderLocationID": "8b5d4343-c007-43d7-8e8f-6fa6c0d29f22"
}
]
}

Parse JSON from URL and skip first line with Python

I have a URL which contains some JSON data. I would like to parse this data and convert to a dictionary using Python. The first line of the data on the webpage is not in JSON format, so I would like to skip the first line before parsing. The data on the webpage looks like the following:
expected 1 issue, got 1
{
"Issues": [
{
"issue": {
"assignedTo": {
"iD": "2",
"name": "industry"
},
"count": "1117",
"logger": "errors",
"metadata": {
"function": "_execute",
"type": "IntegrityError",
"value": "duplicate key value violates unique constraint \nDETAIL: Key (id, date, reference)=(17, 2020-08-03, ZER) already exists.\n"
},
"stats": {},
"status": "unresolved",
"type": "error"
},
"Events": [
{
"message": "Unable to record contract details",
"tags": {
"environment": "worker",
"handled": "yes",
"level": "error",
"logger": "errors",
"mechanism": "logging",
},
"Messages": null,
"Stacktraces": null,
"Exceptions": null,
"Requests": null,
"Templates": null,
"Users": null,
"Breadcrumbs": null,
"Context": null
},
],
"fetch_time": "2020-07-20"
}
]
}
And I have tried running this script:
with urllib.request.urlopen("[my_url_here]") as url:
if(url.getcode()==200):
for _ in range(1):
next(url)
data = url.read()
json=json.loads(data)
else:
print("Error receiving data", url.getcode())
But am met with the error:
Traceback (most recent call last):
File "<stdin>", line 6, in <module>
File
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I get the same error when I run it without using
for _ in range(2):
next(url)
... but with the last line as 'Expecting value: line 2 column 1 (char 1)'.
Any advice? Thanks

You can Remove the First line by the following code.
Code:
data = ''.join(data.split('\n')[1:])
print(data)
Output:
{ "Issues": [ { "issue": { "assignedTo": { "iD": "2", "name": "industry" }, "count": "1117", "logger": "errors", "metadata": { "function": "_execute", "type": "IntegrityError", "value": "duplicate key value violates unique constraint DETAIL: Key (id, date, reference)=(17, 2020-08-03, ZER) already exists." }, "stats": {}, "status": "unresolved", "type": "error" }, "Events": [ { "message": "Unable to record contract details", "tags": { "environment": "worker", "handled": "yes", "level": "error", "logger": "errors", "mechanism": "logging", }, "Messages": null, "Stacktraces": null, "Exceptions": null, "Requests": null, "Templates": null, "Users": null, "Breadcrumbs": null, "Context": null }, ], "fetch_time": "2020-07-20" } ]}
As you can see that the we achieved removing first line. But your Parsed Json response have issues. It is not properly formatted. Take a look on the below image.
On the crossed lines we got extra comma letting know the parser there are more instances left still but your response doesn't have any more instance on that scope. So please check your code which used to convert your data to json. If you have doubt please write here. For validating your json you can check on https://jsonlint.com/
I hope it would be helpful... :)

you can try to load the json like this:
json.loads(data.split("\n",1)[1])
this will split the string at the first endline and use the second part of it.
However I discourage this as you can't be sure your server will always reply like this - try to fix the endpoint or find one that returns a valid json reply if you can.
you will still get a json.decoder.JSONDecodeError: Invalid control character at: line 14 column 68 (char 336) because of that \n in the data.

Parsing list of dictionaries in a dictionary to retrieve a specific key's value from each dictionary

I got a JSON response and converted it to a python dictionary using json.loads(). So the dictionary looks like this:
{u'body': u'[{"id":"1","entity":"zone","status":"PROCESSING","url":null,"createdOn":"2019-10-11T05:49:11Z"},{"id":"2","entity":"floor","status":"FAILED","url":null,"createdOn":"2019-10-11T05:49:15Z"},{"id":"3","entityType":"apartment","status":"SUCCESS","url":null,"createdOn":"2019-10-11T05:49:18Z"}]',u'isBase64Encoded': False, u'statusCode': 200}
I named this as testStatusList. I want to retrieve the value of "status" key of every dictionary inside "body". I was able to retrieve the "body" by giving body = testStatusList['body']. Now, the dictionary looks like:
[
{
"id": "1",
"entityType": "zone",
"status": "PROCESSING",
"url": null,
"createdOn": "2019-03-07T12:47:10Z"
},
{
"id": "2",
"entityType": "floor",
"status": "FAILED",
"url": null,
"createdOn": "2019-08-19T16:46:13Z"
},
{
"id": "3",
"entityType": "apartment",
"status": "SUCCESS",
"url": null,
"createdOn": "2019-08-19T16:46:13Z"
}
]
I tried out this solution [Parsing a dictionary to retrieve a key in Python 3.6
testStatusList= json.loads(status_response['Payload'].read())
body = testStatusList['body']
status =[]
for b in body:
for k,v in b.items():
if k == 'status':
status.append(v)
but I keep getting AttributeError: 'unicode' object has no attribute 'items'. Is there a different method to get items for unicode objects?
So I basically want to retrieve all the statuses i.e., PROCESSING, FAILED AND SUCCESS so that I can put an 'if' condition to display appropriate messages when something failed for that particular "id". I am very unsure about my approach as I am totally new to Python. Any help would be much appreciated thanks!

body is still a (unicode) string in your top blob. Use json.loads again on that string:
body = """[
{
"id": "1",
"entityType": "zone",
"status": "PROCESSING",
"url": null,
"createdOn": "2019-03-07T12:47:10Z"
},
{
"id": "2",
"entityType": "floor",
"status": "FAILED",
"url": null,
"createdOn": "2019-08-19T16:46:13Z"
},
{
"id": "3",
"entityType": "apartment",
"status": "SUCCESS",
"url": null,
"createdOn": "2019-08-19T16:46:13Z"
}
]"""
import json
body = json.loads(body)
status =[]
for b in body:
for k,v in b.items():
if k == 'status':
status.append(v)
print(status)
Result:
['PROCESSING', 'FAILED', 'SUCCESS']

Rasa utter_message converting json to string

I want to convert my response in json format but it is converting it into string.
Response expected
{
"recipient_id": "default",
"data": [{
"Name": "John",
"status": "To Be Processed",
"LastUpdatedDate": "2013-05-31 08:40:55.0"
}, {
"Name": "Paul",
"status": "To Be Processed",
"LastUpdatedDate": "2013-06-02 16:03:00.0"
}] };
Response getting
{
"recipient_id": "default",
"data": ""[{
"Name": "John",
"status": "To Be Processed",
"LastUpdatedDate": "2013-05-31 08:40:55.0"
}, {
"Name": "Paul",
"status": "To Be Processed",
"LastUpdatedDate": "2013-06-02 16:03:00.0"
}]""
};
i am using dispatcher.utter_message to send response.

JSON or Python dict / list decoding problem

I have been using the Python script below to try and retrieve and extract some data from Flightradar24, it would appear that it extracts the data in JSON format and will print the data out ok fully using json.dumps, but when I attempt to select the data I want (the status text in this case) using get it gives the following error:
'list' object has no attribute 'get'
Is the Data in JSON or a List ? I'm totally confused now.
I'm fairly new to working with data in JSON format, any help would be appreciated!
Script:
import flightradar24
import json
flight_id = 'BA458'
fr = flightradar24.Api()
flight = fr.get_flight(flight_id)
y = flight.get("data")
print (json.dumps(flight, indent=4))
X= (flight.get('result').get('response').get('data').get('status').get('text'))
print (X)
Sample of output data:
{
"result": {
"request": {
"callback": null,
"device": null,
"fetchBy": "flight",
"filterBy": null,
"format": "json",
"limit": 25,
"page": 1,
"pk": null,
"query": "BA458",
"timestamp": null,
"token": null
},
"response": {
"item": {
"current": 16,
"total": null,
"limit": 25
},
"page": {
"current": 1,
"total": null
},
"timestamp": 1546241512,
"data": [
{
"identification": {
"id": null,
"row": 4852575431,
"number": {
"default": "BA458",
"alternative": null
},
"callsign": null,
"codeshare": null
},
"status": {
"live": false,
"text": "Scheduled",
"icon": null,
"estimated": null,
"ambiguous": false,
"generic": {
"status": {
"text": "scheduled",
"type": "departure",
"color": "gray",
"diverted": null
},

You can use print(type(variable_name)) to see what type it is. The .get(key[,default]) is not supported on lists - it is supported for dict's.
X = (flight.get('result').get('response').get('data').get('status').get('text'))
# ^^^^^^^^ does not work, data is a list of dicts
as data is a list of dicts:
"data": [ # <<<<<< this is a list
{
"identification": {
"id": null,
"row": 4852575431,
"number": {
"default": "BA458",
"alternative": null
},
"callsign": null,
"codeshare": null
},
"status": {
This should work:
X = (flight.get('result').get('response').get('data')[0].get('status').get('text')

The issue, as pointed out by #PatrickArtner, is your data is actually a list rather than a dictionary. As an aside, you may find your code more readable if you were to use a helper function to apply dict.get repeatedly on a nested dictionary:
from functools import reduce
def ng(dataDict, mapList):
"""Nested Getter: Iterate nested dictionary"""
return reduce(dict.get, mapList, dataDict)
X = ng(ng(flight, ['result', 'response', 'data'])[0], ['status'[, 'text']])