I a trying to expose an API (if that's the correct way to say it). I am using Quart, a python library made out of Flask and this is what my code looks like:
async def capture_post_request(request_json):
for item in request_json:
callbackidd = item['callbackid']
print(callbackidd)
#app.route('/start_work/', methods=['POST'])
async def start_work():
content_type = request.headers.get('content-type')
if (content_type == 'application/json'):
request_json = await request.get_json()
loop = asyncio.get_event_loop()
loop.create_task(capture_post_request(request_json))
body = "Async Job Started"
return body
else:
return 'Content-Type not supported!'
My schema looks like that:
[
{
"callbackid": "dd",
"itemid": "234r",
"input": [
{
"type": "thistype",
"uri": "www.uri.com"
}
],
"destination": {
"type": "thattype",
"uri": "www.urino2.com"
}
},
{
"statusCode": "202"
}
]
So far what I am getting is this error:
line 11, in capture_post_request
callbackidd = item['callbackid']
KeyError: 'callbackid'
I've tried so many stackoverflow posts to see how to iterate through my list of dicts but nothing worked. At one point in my start_work function I was using the get_data(as_text=True) method but still no results. In fact with the last method (or attr) I got:
TypeError: string indices must be integers
Any help on how to access those values is greatly appreciated. Cheers.
Your schema indicates there are two items in the request_json. The first indeed has the callbackid, the 2nd only has statusCode.
Debugging this should be easy:
async def capture_post_request(request_json):
for item in request_json:
print(item)
callbackidd = item.get('callbackid')
print(callbackidd) # will be None in case of the 2nd 'item'
This will print two dicts:
{
"callbackid": "dd",
"itemid": "234r",
"input": [
{
"type": "thistype",
"uri": "www.uri.com"
}
],
"destination": {
"type": "thattype",
"uri": "www.urino2.com"
}
}
And the 2nd, the cause of your KeyError:
{
"statusCode": "202"
}
I included the 'fix' of sorts already:
callbackidd = item.get('callbackid')
This will default to None if the key isn't in the dict.
Hopefully this will get you further!
Edit
How to work with only the dict containing your key? There are two options.
First, using filter. Something like this:
def has_callbackid(dict_to_test):
return 'callbackid' in dict_to_test
list_with_only_list_callbackid_items = list(filter(has_callbackid, request_json))
# Still a list at this point! With dicts which have the `callbackid` key
Filter accepts some arguments:
Function to call to determine if the value being tested should be filtered out or not.
The iterable you want to filter
Could also use a 'lambda function', but it's a bit evil. But serves the purpose just as well:
list_with_only_list_callbackid_items = list(filter(lambda x: 'callbackid' in x, request_json))
# Still a list at this point! With dict(s) which have the `callbackid` key
Option 2, simply loop over the result and only grab the one you want to use.
found_item = None # default
for item in request_json:
if 'callbackid' in item:
found_item = item
break # found what we're looking for, stop now
# Do stuff with the found_item from this point.
Related
Situation
I want to make a function that makes me free to give a full dictionary path parameter, and get back the value or node I need, without doing it node by node.
Code
This is the function. Obviously, as is now, it throws TypeError: unhashable type: 'list'. But it's only for getting the idea.
def get_section(api_data, section):
if "/" in section:
section = section.split("/")
return api_data.json()[section]
return api_data.json()[section]
Example
JSON
{
"component": {
"name": "gino",
"measures": [
{
"value": "12",
},
{
"value": "14"
}
]
},
"metrics": {
...
}
}
Expectation
analyses = get_section(analyses_data, "component/measures") # Returns measures node
analyses = get_section(analyses_data, "component/name") # Returns 'gino'
analyses = get_section(analyses_data, "component/measures/value") # Returns error, because it's ambigous
Request
How can I do it?
Edits
Added examples for clarity
A cool solution could be:
def get_section(api_data, section):
return [api_data := api_data[sec] for sec in section.split("/")][-1]
So if you execute it with:
analyses_data = {
"analyses": {
"dates": {
"xyz": "abc"
}
}
}
print(get_section(analyses_data, "analyses/dates/xyz")) # Returns: abc
Or since you are accessing a json using a custom method:
print(get_section(analyses_data.json(), "analyses/dates/xyz")) # Returns: abc
This works because the := operator in python is a variable assignment that returns the assigned value, so it loops all the parts of the section string by reassigning the api_data variable to the result of accessing that key and storing the result of every assignment in a list. Then with the [-1] at the end it returns the last assignment that corresponds to the last accessed key (a.k.a the last accessed dictionary level).
Appreciate if you could help me for the best way to transform a result into json as below.
We have a result like below, where we are getting an information on the employees and the companies. In the result, somehow, we are getting a enum like T, but not for all the properties.
[ {
"T.id":"Employee_11",
"T.category":"Employee",
"node_id":["11"]
},
{
"T.id":"Company_12",
"T.category":"Company",
"node_id":["12"],
"employeecount":800
},
{
"T.id":"id~Employee_11_to_Company_12",
"T.category":"WorksIn",
},
{
"T.id":"Employee_13",
"T.category":"Employee",
"node_id":["13"]
},
{
"T.id":"Parent_Company_14",
"T.category":"ParentCompany",
"node_id":["14"],
"employeecount":900,
"childcompany":"Company_12"
},
{
"T.id":"id~Employee_13_to_Parent_Company_14",
"T.category":"Contractorin",
}]
We need to transform this result into a different structure and grouping based on the category, if category in Employee, Company and ParentCompany, then it should be under the node_properties object, else, should be in the edge_properties. And also, apart from the common properties(property_id, property_category and node), different properties to be added if the category is company and parent company. There are few more logic also where we have to get the from and to properties of the edge object based on the 'to' . the expected response is,
"node_properties":[
{
"property_id":"Employee_11",
"property_category":"Employee",
"node":{node_id: "11"}
},
{
"property_id":"Company_12",
"property_category":"Company",
"node":{node_id: "12"},
"employeecount":800
},
{
"property_id":"Employee_13",
"property_category":"Employee",
"node":{node_id: "13"}
},
{
"property_id":"Company_14",
"property_category":"ParentCompany",
"node":{node_id: "14"},
"employeecount":900,
"childcompany":"Company_12"
}
],
"edge_properties":[
{
"from":"Employee_11",
"to":"Company_12",
"property_id":"Employee_11_to_Company_12",
},
{
"from":"Employee_13",
"to":"Parent_Company_14",
"property_id":"Employee_13_to_Parent_Company_14",
}
]
In java, we have used the enhanced for loop, switch etc. How we can write the code in the python to get the structure as above from the initial result structure. ( I am new to python), thank you in advance.
Regards
Here is a method that I quickly made, you can adjust it to your requirements. You can use regex or your own function to get the IDs of the edge_properties then assign it to an object like the way I did. I am not so sure of your full requirements but if that list that you gave is all the categories then this will be sufficient.
def transform(input_list):
node_properties = []
edge_properties = []
for input_obj in input_list:
# print(obj)
new_obj = {}
if input_obj['T.category'] == 'Employee' or input_obj['T.category'] == 'Company' or input_obj['T.category'] == 'ParentCompany':
new_obj['property_id'] = input_obj['T.id']
new_obj['property_category'] = input_obj['T.category']
new_obj['node'] = {input_obj['node_id'][0]}
if "employeecount" in input_obj:
new_obj['employeecount'] = input_obj['employeecount']
if "childcompany" in input_obj:
new_obj['childcompany'] = input_obj['childcompany']
node_properties.append(new_obj)
else: # You can do elif == to as well based on your requirements if there are other outliers
# You can use regex or whichever method here to split the string and add the values like above
edge_properties.append(new_obj)
return [node_properties, edge_properties]
I'm trying to find a cogent way to check to see whether certain keys exist in a dictionary and use those to build a new one.
Here is my example json:
"dmarc": {
"record": "v=DMARC1; p=none; rua=mailto:dmarc.spc#test.domain; adkim=s; aspf=s",
"valid": true,
"location": "test.domain",
"warnings": [
"DMARC record at root of test.domain has no effect"
],
"tags": {
"v": {
"value": "DMARC1",
"explicit": true
},
"p": {
"value": "none",
"explicit": true
},
"rua": {
"value": [
{
"scheme": "mailto",
"address": "ssc.dmarc.spc#canada.ca",
"size_limit": null
}
],
"explicit": true
},
"adkim": {
"value": "s",
"explicit": true
},
"aspf": {
"value": "s",
"explicit": true
},
"fo": {
"value": [
"0"
],
"explicit": false
},
"pct": {
"value": 100,
"explicit": false
},
"rf": {
"value": [
"afrf"
],
"explicit": false
},
"ri": {
"value": 86400,
"explicit": false
},
"sp": {
"value": "none",
"explicit": false
}
}
}
}
What I'm specifically looking to do, is pull record, valid, location, tags-p, tags-sp, and tags-pct in a programmatic way, instead of doing a bunch of try/excepts. For example, to get valid, I do:
try:
res_dict['valid'] = jsonData['valid']
except KeyError:
res_dict['valid'] = None
Now, this is easy enough to loop/repeat for top level key/values, but how would I accomplish this for the nested key/values?
No, you don't need a try-except block for the same. You can check if the key exists using:
if jsonData.get("valid"):
res_dict["valid"] = jsonData.get("valid")
The .get("key") method returns the value for the given key, if present in the dictionary. If not, then it will return None (if get() is used with only one argument).
If you want it to return something else if it doesn't find the key then suppose:
jsonData.get("valid", "invalid_something_else")
One way of handling this is by taking advantage of the fact that the result of dict.keys can be treated as a set. See the following code.
my_keys = {'record', 'valid', 'location'} # you can add more here
new_dict = {}
available_keys = my_keys & jsonData.keys()
for key in available_keys:
new_dict[key] = jsonData[key]
Above, we define the keys we are interested in within the my_keys set. We then get the available keys by taking the intersection of the keys in the dictionary and the keys we are interested in. This, in effect, only gets the keys that we are interested in that are also defined in the dictionary. Finally, we just iterate through the available_keys and build the new dictionary.
However, this does not set keys to None if they do not exist in the input dictionary. For that, it may be best to use the get method as mentioned in other answers, like so:
my_keys = ['record', 'valid', 'location'] # you can add more here
new_dict = {}
for key in my_keys:
new_dict[key] = jsonData.get(key)
The get method allows us to attempt to get the value for a key in the dictionary. If that key is not defined, it returns None. You can also change the returned default by adding an extra argument to the get method like so new_dict[key] = jsonData.get(key, "some other default value")
Simple: instead of dict['key'] use
dict.get('key', {}) for all nodes that are not leaves, and
dict.get('key', DEFAULT) for leaves, where DEFAULT is whatever you need.
If you omit DEFAULT and 'key' is absent, you get None. See the docs.
E.g.:
jsonData.get('record', "") # empty string if no 'record' key
jsonData.get('valid', False) # False if no 'valid' key
jsonData.get('location') # None if no 'location'
jsonData.get('tags', {}).get('p') # None if no 'tags' and/or no 'p'
jsonData.get('tags', {}).get('p', {}) # {} if no 'tags' and/or no 'p'
jsonData.get('tags', {}).get('p', {}).get('explicit', False) # and so on
The above presumes that you don't traverse lists (JSON arrays). If you do, you can still use
dict.get('key', [])
but if you have to dive deeper from there, you will probably have to loop over list items.
I have created a Flask app and wanted to test it. In a single endpoint, I would like to post a multipart request, which includes a file and a complex JSON object. I thought at first of using werkzeug EnvironBuilder for this task, as it seems to provide a quite automated approach, handling content types, etc. My snippet of code for preparing the request is the following:
# client is an instance of FlaskClient produced using a pytest fixture and the method test client
def _post(endpoint, file_path=None, serialized_message=None):
with open(file_path, 'rb') as fin:
fil = io.BytesIO(fin.read())
file_name = file_path.split(os.sep)[-1]
builder = EnvironBuilder(path='/' + endpoint,
method='POST',
data=json.loads(
serialized_message),
content_type="application/json")
builder.files[file_name] = fil
result = client.open(builder, buffered=True)
return result
This failed with the following error:
def _add_file_from_data(self, key, value):
"""Called in the EnvironBuilder to add files from the data dict."""
if isinstance(value, tuple):
self.files.add_file(key, *value)
elif isinstance(value, dict):
from warnings import warn
warn(DeprecationWarning('it\'s no longer possible to pass dicts '
'as `data`. Use tuples or FileStorage '
'objects instead'), stacklevel=2)
value = dict(value)
mimetype = value.pop('mimetype', None)
if mimetype is not None:
value['content_type'] = mimetype
> self.files.add_file(key, **value)
E TypeError: add_file() got an unexpected keyword argument 'globalServiceOptionId'
With the globalServiceOptionId being a key of a nested dictionary in the dictionary I am posting. I have some thoughts over bypassing this problem, with converting to string jsons the inner dictionaries, but I would like something more concrete as an answer, as I do not want the representation of the request to be changed inside and outside of testing. Thank you.
Update 1
The form of the passwed dictionary doesn't really matter, as long as it has nested dictionaries inside it. This json is given in this example:
{
"attachments": [],
"Ids": [],
"globalServiceOptions": [{
"globalServiceOptionId": {
"id": 2,
"agentServiceId": {
"id": 2
},
"serviceOptionName": "Time",
"value": "T_last",
"required": false,
"defaultValue": "T_last",
"description": "UTC Timestamp",
"serviceOptionType": "TIME"
},
"name": "Time",
"value": null
}]
}
Update 2
I tested another snippet:
def _post(endpoint, file_path=None, serialized_message=None):
with open(file_path, 'rb') as fin:
fil = io.BytesIO(fin.read())
files = {
'file': (file_path, fil, 'application/octet-stream')
}
for key, item in json.loads(serialized_message).items():
files[key] = (None, json.dumps(item), 'application/json')
builder = EnvironBuilder(path='/' + endpoint,
method='POST', data=files,
)
result = client.open(builder, buffered=True)
return result
Although this runs without errors, Flask recognizes (as expected) the incoming jsons as files, which again requires different handling during testing and normal running.
I ran into a similar issue, and what ended up working for me was changing the data approach to exclude nested dicts. Taking your sample JSON, doing the following should allow it to clear the EnvironBuilder:
data_json = {
"attachments": [],
"Ids": [],
"globalServiceOptions": [json.dumps({ # Dump all nested JSON to a string representation
"globalServiceOptionId": {
"id": 2,
"agentServiceId": {
"id": 2
},
"serviceOptionName": "Time",
"value": "T_last",
"required": false,
"defaultValue": "T_last",
"description": "UTC Timestamp",
"serviceOptionType": "TIME"
},
"name": "Time",
"value": null
})
]
}
builder = EnvironBuilder(path='/' + endpoint,
method='POST',
data=data_json,
content_type="application/json")
Taking the approach above still allowed the nested dict/JSON to be passed appropriately while clearing the werkzeug limitation.
Given this json string, how can I pull out the value of id if code equals 4003?
error_json = '''{
'error': {
'meta': {
'code': 4003,
'message': 'Tracking already exists.',
'type': 'BadRequest'
},
'data': {
'tracking': {
'slug': 'fedex',
'tracking_number': '442783308929',
'id': '5b59ea69a9335baf0b5befcf',
'created_at': '2018-07-26T15:36:09+00:00'
}
}
}
}'''
I can't assume that anything other than the error element exists at the beginning, so the meta and code elements may or may not be there. The data, tracking, and id may or may not be there either.
The question is how to extract the value of id if the elements are all there. If any of the elements are missing, the value of id should be None
A python dictionary has a get(key, default) method that supports returning a default value if a key is not found. You can chain empty dictionaries to reach nested elements.
# use get method to access keys without raising exceptions
# see https://docs.quantifiedcode.com/python-anti-patterns/correctness/not_using_get_to_return_a_default_value_from_a_dictionary.html
code = error_json.get('error', {}).get('meta', {}).get('code', None)
if code == 4003:
# change id with _id to avoid overriding builtin methods
# see https://docs.quantifiedcode.com/python-anti-patterns/correctness/assigning_to_builtin.html
_id = error_json.get('error', {}).get('data', {}).get('tracking', {}).get('id', None)
Now, given a string that looks like a JSON you can parse it into a dictionary using json.loads(), as shown in Parse JSON in Python
I would try this:
import json
error_json = '''{
"error": {
"meta": {
"code": 4003,
"message": "Tracking already exists.",
"type": "BadRequest"
},
"data": {
"tracking": {
"slug": "fedex",
"tracking_number": "442783308929",
"id": "5b59ea69a9335baf0b5befcf",
"created_at": "2018-07-26T15:36:09+00:00"
}
}
}
}'''
parsed_json = json.loads(error_json)
try:
if parsed_json["error"]["meta"]["code"] == int(parsed_json["error"]["meta"]["code"]):
print(str(parsed_json["error"]["data"]["tracking"]["id"]))
except:
print("no soup for you")
Output:
5b59ea69a9335baf0b5befcf
A lot of python seems to be it's better to ask for forgiveness instead of permission. You could look up to see if that key in the dictionary is there, but really it's easier to just try. I'm specifically doing a check to make sure that code is an int, but you could change it around any way you'd like. If it can be other things you'd have to adjust it. There are several different solutions to this, it's really whatever you feel the most comfortable doing and maintaining.
You can add a check for key errors to the whole operation:
def get_id(j)
try:
if j['error']['meta’]['code'] == 4003:
return j['error']['data']['tracking']['id']
except KeyError:
pass
return None
If any element is missing, this function will quietly return None. If all the required elements are present, it will return the required ID. The only time it could really fail is if one of the intermediate keys does not refer to a dictionary. That could potentially result in a TypeError.