Develop Reference

SQLalchemy +Postgresql accessing Array of jsonb elements

I have a query that looks like this.
query = db.query(status_cte.c.finding_status_history)
the finding_status_history column is of type array when I check its .type. It's an array of jsonb objects I can easily change it to be json instead if it's easier. I've also tested this out with it as json.
[
{
"data": [
{
"status": "closed",
"created_at": "2023-01-27T18:05:27.579817",
"previous_status": "open"
},
{
"status": "open",
"created_at": "2023-01-27T18:05:28.694352",
"previous_status": "closed"
}
]
},
...
]
I'm trying to access the first dictionary nested inside data and access the status column.
I've tried to grab it using query = db.query(status_cte.c.finding_status_history[0]) but this returns a list of empty dictionaries like so.
[
{},
{},
{},
{},
{},
{},
{}
]
I'm not sure why that doesn't work as its my impression that i should grab the first entry. I'm assuming i need to access "data" some how first but i've also tried...
query = db.query(status_cte.c.finding_status_history.op('->>')('data')
Which gives me jsonb[] ->> unknown operator doesn't exist. I've tried to type cast data to be that of String and i get the same error but jsonb[] ->> String etc etc
Also when looping through the items for item in query.all() i'm seeing that [0] results in (None,) and [1] results in
({
"status": "closed",
"created_at": "2023-01-27T18:05:27.579817",
"previous_status": "open"
},)
as a tuple...

The secret was that [0] is not the first element. [1] is also noted that [-1] doesn't appear to give me the last element so i also had to order my aggregated json objects.

How to extract data from json and add additional values to the extracted values using python?

I want to parse the value from json response using python and assign additional value to the list
{ "form": [{ "box": [60,120,260,115], "text": "hello", "label": "question", "words": [{ "box": [90,190,160,215 ],"text": "hello"} ], "linking": [[0,13]],"id": 0 }]}
I am trying to parse the value and assign to a variable using python. What I am trying to achieve is:
If the actual output is ([60,120,260,115],hello) I wanted to add few more values to the list: Thus expected output should be:
([60,120,260,120,260,115,60,115],hello)

try this:
tmp_json = { "form": [{ "box": [60,120,260,115], "text": "hello", "label": "question", "words": [{ "box": [90,190,160,215 ],"text": "hello"} ], "linking": [[0,13]],"id": 0 }]}
# Then do whatever you need to do with the list by accessing it as follows
# tmp_json["form"][0]["box"]

you can iterate through all elements of list here and if each item matches required condition extend the existing list with required values.
# Pseudocode
for item in data["form"]:
# check each item's box attribute has all such elements i.e 60,120,260,115
# AND item's text attribute has value "hello"
# If matches then to add extra values to box list you can use <list>.extend([115, 120 etc])
# e.g item["box"].extend([120, 115, 260])

Filter Nested array Python

I have an output array like this after using "json.dumps(response.json(), indent=4"
{
"totalCount": 8,
"hasMore": false,
"firstIndex": 0,
"list": [
{
"id": "7d5bb8asdfasdfasfdasdfasdfasdf",
"name": "Corporate",
"domainType": "AAAAAAAA",
"description": "",
"createdBy": "admin",
"createDatetime": "2020/06/04 17:40:22",
"parentDomainId": "8b208asdfasdfasdfasdfasdfas",
"zoneCount": 2,
"subDomainCount": 1,
"administratorCount": 0,
"apCount": 0,
"zeroTouchStatus": true
},
Now when I try to filter it as follows
print(results['name']) or print(results['list'][0]['name'])
I keep getting this error message:
TypeError: string indices must be integers
This starts with a dict {}, then ther are lists [] of dict {} in here. Based on that it should work. Appreciate any guidance. Thank you.

It is clear by your data you posted that it is still in text format JSON. Convert to a python dict using json.loads()
result = json.loads(response.json())
print(results['name'])
print(results['list'][0]['name'])

The problem is json.dumps(response.json()). The return of this statement is a str so you cannot get items using [] notation. You can convert back it to a dict.

How to push an item to an array and get the index of that item in MongoDB, Projection in find_one_and_update pymongo

I'd like to $push an item into an array and determine the index at which it was inserted. How can I do this with Mongo?
I need this to be atomic as multiple pushes can be happening in parallel on the document.
I'm using Python/PyMongo as the driver.

You can store the size of the array along with the array within the document and get that value after the update:
Sample input document: { '_id: 1', 'arr': [ "apple", "orange" ] }
The update operation - uses a pipeline for the update (available with MongoDB version 4.2):
NEW_ITEM = 'pear'
r = collection.find_one_and_update(
{ '_id': 1 },
[
{
'$set': {
'ix': { '$size': '$arr' },
'arr': { '$concatArrays': [ '$arr', [ NEW_ITEM ] ] }
}
}
],
projection = { '_id': False, 'ix': True },
return_document = ReturnDocument.AFTER
)
Another way is set the index of the newly inserted element within the same update operation (this can be used if array elements are unique):
[
{
'$set': {
'arr': { '$concatArrays': [ '$arr', [ NEW_ITEM ] ] }
},
'$set': {
'ix': { '$indexOfArray': [ '$arr', NEW_ITEM ] }
}
}
]

Updates to a single document in MongoDB are atomic, So if one update operation is writing to a document the following update operation has to wait until the first one finishes. So you can return the updated document & in code get the index of the newly pushed value(As $push will usually push to end of the array).
So when you use MongoDB's aggregation framework for reads - You can use $indexOfArray operator with $project stage to get the index of an element in an array. But projection is aggregation framework can accept lot more operators than projection in .find()'s or .findOneAndUpdate()'s. Getting index of an element in an array might not be possible with update operations, So using below query you can return the complete new array from updated document & using python try to get the index of element in new array.
Sample Doc :
{
_id: ObjectId("5eb773b8c4ec53c0626b167e"),
name: "YesMe",
ids: [1, 2, 3],
};
Query :
db.collection.find_one_and_update(
{ name: "YesMe" },
{ $push: { ids: 4 } },
(projection = { ids: True, _id: False }), // Projecting only `ids` field
(return_document = ReturnDocument.AFTER) // Returns updated doc (In your case updated array)
);
Output :
{ ids : [1, 2, 3, 4] }
Ref : Collection.find_one_and_update

Access JSON nested arrays in Python

I'm new to Python and I'm trying to process something and having no luck finding the answer or if it's already been asked. I'm making a call to an API and receiving some data back as JSON. I'm stripping out certain bits that I don't need with the keys being stripped out and only the values remaining which wouldn't be a problem but I can't get into them as the keys I want to access are nested in an array.
I've been accessing the data and can get up to json.dumps(payload['output']['generic']) but I can't seem to find any information online as to how I can access these last values only.
Apologies in advance if this question already exists.
{
"output": {
"generic": [
{
"response_type": "text",
"text": "hi"
}
],
"intents": [
{
"intent": "CollectionDate",
"confidence": 0.8478035449981689
}
],
"entities": [
{
"entity": "Payslip",
"location": [
19,
26
],
"value": "When is my collection date",
"confidence": 1
}
]
},
"context": {
"global": {
"system": {
"turn_count": 10
}
},
"skills": {
"main skill": {
"user_defined": {
"DemoContext": "Hi!"
},
"system": {}
}
}
}
}
To clarify:
I want to access the "text", "intent" and "confidence"
at the moment I'm printing the value posted and then the responses for the sections I want like the below.
print(x)
print(json.dumps(payload['output']['generic']))
print(json.dumps(payload['output']['intents']))

Use following code to convert the json to a dict first:
json_data = json.loads(str(yourData))
After that, in your case, the outermost key is "output", and it is another dict, so just use json_data['output'] to access the content inside.
For other keys inside of the "output", like "generic", you can see it is an array with the [] brackets. use json_data['output'][index] first to get the content inside, then use the same method you access a dict to access the content inside of keys like this.

They key here is that the Traceback error indicates an issue with indexing a "List"
This is because a "List" type is a valid JSON type, and generic contains a list of length 1, with a dict inside!
>>> payload['output']['generic']['text']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: list indices must be integers or slices, not str
>>> type(payload['output']['generic'])
<class 'list'>
>>> len(payload['output']['generic'])
1
>>> payload['output']['generic'][0]
{'response_type': 'text', 'text': 'hi'}
>>> type(payload['output']['generic'][0])
<class 'dict'>
>>> payload['output']['generic'][0]['text']
'hi'
>>>
So, given your expected input JSON format, you will need to know how to index in to pull each required data point.
There are a few packages, glom is one, that will help you deal with missing values from API generated JSON.