Develop Reference

How does one pass a json file or object in a POST request using Python module 'requests'

I am using a site's REST API's and have been primarily using Python's 'requests' module to GET json responses. The goal of the GET requests are to ultimately pull a user's form response which ends up being a complex json document. To deal with this:
user_form_submission = requests.get('https://www.url/doc.json',
auth = (api_key, secret),
params = params)
python_obj = json.loads(user_form_submission.text)
trimmed_dict = python_obj['key'][0]['keys']
For context, this is what trimmed_dict would look like formatted as .json:
{
"Date": { "value": "2020-04-26", "type": "date" },
"Location": {
"value": "Test ",
"type": "text",
"geostamp": "lat=34.00000, long=-77.00000, alt=17.986118, hAccuracy=65.000000, vAccuracy=10.000000, timestamp=2020-04-26T23:39:56Z"
},
"form": {
"value": [
{
"form_Details": {
"value": [
{
"code": {
"value": "0000000000",
"type": "barcode"
},
"Name": { "value": "bob", "type": "text" }
}
],
"type": "group"
},
"Subtotal": { "value": "4", "type": "decimal" },
"form_detail2": {
"value": [
{
"name": {
"value": "billy",
"type": "text"
},
"code": {
"value": "00101001",
"type": "barcode"
},
"Classification": {
"value": "person",
"type": "select1"
},
"Start_Time": { "value": "19:43:00", "type": "time" },
"time": { "value": "4", "type": "decimal" }
}
],
"type": "subform"}
}
]
}
}
Now I have a portion of the json that contains both the useful and useless. From this point, can I pass this obj in a POST? I've tried every way that I can think of approaching it, and have been shut down.
Understanding how I want to go about this, this is how I thought it would go:
json_post = requests.post(' https://url/api/doc.json',
auth = (api_key, secret),
json = {
"form_id" : 'https://url.form.com/formid',
'payload':{
json.dumps(trimmed_dict)
}})
But, when I do this, I get the following error --
TypeError: Object of type set is not JSON serializable
How can I push this dict through this POST? If there's a more effective way of going about it, I am very open to suggestion.

Try removing the curly braces around json.dumps(trimmed_dict). json.dumps turns your trimmed_dict into a string, which becomes a python set when surrounded with braces.
Additionally you could remove json.dumps and plug the trimmed_dict into the structure directly as the value associated with payload.

Remove the extra {} from the payload. payload itself is a key and json.dumps(trimmed_dict) as a value is enough
json_post = requests.post(' https://url/api/doc.json',
auth = (api_key, secret),
json = {
"form_id" : 'https://url.form.com/formid',
"payload": json.dumps(trimmed_dict)
})

Need read some data from JSON

I need to make a get (id, name, fraction id) for each deputy in this json
{
"id": "75785",
"title": "(за основу)",
"asozdUrl": null,
"datetime": "2011-12-21T12:20:26+0400",
"votes": [
{
"deputy": {
"id": "99111772",
"name": "Абалаков Александр Николаевич",
"faction": {
"id": "72100004",
"title": "КПРФ"
}
},
"result": "accept"
},
{
"deputy": {
"id": "99100491",
"name": "Абдулатипов Рамазан Гаджимурадович",
"faction": {
"id": "72100024",
"title": "ЕР"
}
},
"result": "none"
}
.......,` etc
My code is looks like that:
urlData = "https://raw.githubusercontent.com/data-dumaGovRu/vote/master/poll/2011-12-21/75785.json"
response = urllib.request.urlopen(urlData)
content = response.read()
data = json.loads(content.decode("utf8"))
for i in data:
#print(data["name"])
`
And i dont know what to do with that #print line, how I should write it?

You can access the list containing the deputies with data['votes']. Iterating through the list, you can access the keys you're interested in as you would with dict key lookups. Nested dicts imply you have to walk through the keys starting from the root to your point of interest:
for d in data['votes']:
print(d['deputy']['id'], d['deputy']['name'], d['deputy']['faction']['id'])

Connecting many json files to one

i get many json strings from a mysql DB an should combine them.
For example:
{
"type": "device",
"name": "Lampe",
"controls": [
{
"type": "switch",
"name": "Betrieb",
"topic": "/lampe/schalter"
}
]
}
in combination this devices should get into a array of a json file
{
"name": "Test-System",
"devices": [
{
"type": "device",
"name": "Lampe",
"controls": [
{
"type": "switch",
"name": "Betrieb",
"topic": "/lampe/schalter"
}
]
},
{
other Device
}
]
}
i do not understand how to do this in python
does someone have a idea how to do it ?

The json module can be used.
#!/usr/bin/env python3.5
import json
# Parse each device JSON file.
device1 = json.load(open("device-switch-Lampe.json"))
device2 = json.load(open("device-sensor-Wert.json"))
# more devices ...
obj = {"name": "Test-System", "devices": [device1, device2]}
print(json.dumps(obj))
Output (prettified):
{
"devices": [{
"type": "device",
"controls": [{
"type": "switch",
"topic": "/lampe/schalter",
"name": "Betrieb"
}],
"name": "Lampe"
}, {
"type": "device",
"controls": [{
"type": "sensor",
"topic": "/sensor/wert",
"name": "Wert"
}],
"name": "Sensor"
}],
"name": "Test-System"
}

There are two ways you could do this - by working on strings, or by working with Python-JSON data structures. The former would be something like
# untested code
s = '''{
"name": "Test-System",
"devices": [ '''
while True:
j = get_json_from_DB()
if not j: break # null string or None
s = s + j + ',\n'
s = s[:-2] + ']\n}\n' #[:-2 loses the last ',\n' from the loop
Or if you want to work with Python loaded-JSON then
import json
# untested code
s = {
"name": "Test-System",
"devices": []
}
while True:
j = get_json_from_DB()
if not j: break # null string or None
s['devices'].append( json.loads(j) )
# str = json.dumps(s) # ought to be valid
This latter will validate all your incoming json-strings (json.loads() will throw an exception for any bad JSON) and will be more efficient for large numbers of devices. It's therefore to be preferred unless you are working in a RAM-constrained embedded system with small numbers of devices, where the greater memory footprint of the latter is a problem.

Getting Deeper Level JSON Values in Python

I have a Python script that make an API call to retrieve data from Zendesk. (Using Python 3.x) The JSON object has a structure like this:
{
"id": 35436,
"url": "https://company.zendesk.com/api/v2/tickets/35436.json",
"external_id": "ahg35h3jh",
"created_at": "2009-07-20T22:55:29Z",
"updated_at": "2011-05-05T10:38:52Z",
"type": "incident",
"subject": "Help, my printer is on fire!",
"raw_subject": "{{dc.printer_on_fire}}",
"description": "The fire is very colorful.",
"priority": "high",
"status": "open",
"recipient": "support#company.com",
"requester_id": 20978392,
"submitter_id": 76872,
"assignee_id": 235323,
"organization_id": 509974,
"group_id": 98738,
"collaborator_ids": [35334, 234],
"forum_topic_id": 72648221,
"problem_id": 9873764,
"has_incidents": false,
"due_at": null,
"tags": ["enterprise", "other_tag"],
"via": {
"channel": "web"
},
"custom_fields": [
{
"id": 27642,
"value": "745"
},
{
"id": 27648,
"value": "yes"
}
],
"satisfaction_rating": {
"id": 1234,
"score": "good",
"comment": "Great support!"
},
"sharing_agreement_ids": [84432]
}
Where I am running into issues is in the "custom_fields" section specifically. I have a particular custom field inside of each ticket I need the value for, and I only want that particular value.
To spare you too many specifics of the Python code, I am reading through each value below for each ticket and adding it to an output variable before writing that output variable to a .csv. Here is the particular place the breakage is occuring:
output += str(ticket['custom_fields'][id:23825198]).replace(',', '')+','
All the replace nonsense is to make sure that since it is going into a comma delimited file, any commas inside of the values are removed. Anyway, here is the error I am getting:
output += str(ticket['custom_fields'][id:int(23825198)]).replace(',', '')+','
TypeError: slice indices must be integers or None or have an __index__ method
As you can see I have tried a couple different variations of this to try and resolve the issue, and have yet to find a fix. I could use some help!
Thanks...

Are you using json.loads()? If so you can then get the keys, and do an if statement against the keys. An example on how to get the keys and their respective values is shown below.
import json
some_json = """{
"id": 35436,
"url": "https://company.zendesk.com/api/v2/tickets/35436.json",
"external_id": "ahg35h3jh",
"created_at": "2009-07-20T22:55:29Z",
"updated_at": "2011-05-05T10:38:52Z",
"type": "incident",
"subject": "Help, my printer is on fire!",
"raw_subject": "{{dc.printer_on_fire}}",
"description": "The fire is very colorful.",
"priority": "high",
"status": "open",
"recipient": "support#company.com",
"requester_id": 20978392,
"submitter_id": 76872,
"assignee_id": 235323,
"organization_id": 509974,
"group_id": 98738,
"collaborator_ids": [35334, 234],
"forum_topic_id": 72648221,
"problem_id": 9873764,
"has_incidents": false,
"due_at": null,
"tags": ["enterprise", "other_tag"],
"via": {
"channel": "web"
},
"custom_fields": [
{
"sid": 27642,
"value": "745"
},
{
"id": 27648,
"value": "yes"
}
],
"satisfaction_rating": {
"id": 1234,
"score": "good",
"comment": "Great support!"
},
"sharing_agreement_ids": [84432]
}"""
# load the json object
zenJSONObj = json.loads(some_json)
# Shows a list of all custom fields
print("All the custom field data")
print(zenJSONObj['custom_fields'])
print("----")
# Tells you all the keys in the custom_fields
print("How keys and the values")
for custom_field in zenJSONObj['custom_fields']:
print("----")
for key in custom_field.keys():
print("key:",key," value: ",custom_field[key])
You can then modify the JSON object by doing something like
print(zenJSONObj['custom_fields'][0])
zenJSONObj['custom_fields'][0]['value'] = 'something new'
print(zenJSONObj['custom_fields'][0])
Then re-encode it using the following:
newJSONObject = json.dumps(zenJSONObj, sort_keys=True, indent=4)
I hope this is of some help.

Extracting values from deeply nested JSON structures

This is a structure I'm getting from elsewhere, that is, a list of deeply nested dictionaries:
{
"foo_code": 404,
"foo_rbody": {
"query": {
"info": {
"acme_no": "444444",
"road_runner": "123"
},
"error": "no_lunch",
"message": "runner problem."
}
},
"acme_no": "444444",
"road_runner": "123",
"xyzzy_code": 200,
"xyzzy_rbody": {
"api": {
"items": [
{
"desc": "OK",
"id": 198,
"acme_no": "789",
"road_runner": "123",
"params": {
"bicycle": "2wheel",
"willie": "hungry",
"height": "1",
"coyote_id": "1511111"
},
"activity": "TRAP",
"state": "active",
"status": 200,
"type": "chase"
}
]
}
}
}
{
"foo_code": 200,
"foo_rbody": {
"query": {
"result": {
"acme_no": "260060730303258",
"road_runner": "123",
"abyss": "26843545600"
}
}
},
"acme_no": "260060730303258",
"road_runner": "123",
"xyzzy_code": 200,
"xyzzy_rbody": {
"api": {
"items": [
{
"desc": "OK",
"id": 198,
"acme_no": "789",
"road_runner": "123",
"params": {
"bicycle": "2wheel",
"willie": "hungry",
"height": "1",
"coyote_id": "1511111"
},
"activity": "TRAP",
"state": "active",
"status": 200,
"type": "chase"
}
]
}
}
}
Asking for different structures is out of question (legacy apis etc).
So I'm wondering if there's some clever way of extracting selected values from such a structure.
The candidates I was thinking of:
flatten particular dictionaries, building composite keys, smth like:
{
"foo_rbody.query.info.acme_no": "444444",
"foo_rbody.query.info.road_runner": "123",
...
}
Pro: getting every value with one access and if predictable key is not there, it means that the structure was not there (as you might have noticed, dictionaries may have different structures depending on whether it was successful operation, error happened, etc).
Con: what to do with lists?
Use some recursive function that would do successive key lookups, say by "foo_rbody", then by "query", "info", etc.
Any better candidates?

You can try this rather trivial function to access nested properties:
import re
def get_path(dct, path):
for i, p in re.findall(r'(\d+)|(\w+)', path):
dct = dct[p or int(i)]
return dct
Usage:
value = get_path(data, "xyzzy_rbody.api.items[0].params.bicycle")

Maybe the function byPath in my answer to this post might help you.

You could create your own path mechanism and then query the complicated dict with paths. Example:
/ : get the root object
/key: get the value of root_object['key'], e.g. /foo_code --> 404
/key/key: nesting: /foo_rbody/query/info/acme_no -> 444444
/key[i]: get ith element of that list, e.g. /xyzzy_rbody/api/items[0]/desc --> "OK"
The path can also return a dict which you then run more queries on, etc.
It would be fairly easy to implement recursively.

I think about two more solutions:
You can try package Pynq, described here - structured query language for JSON (in Python). As far as a I understand, it's some kind of LINQ for python.
You may also try to convert your JSON to XML and then use Xquery language to get data from it - XQuery library under Python