I have the following JSON data the I need to read in Flask:
{
"NewList":[
{
"key" : "myvalue1",
"value" : "value1"
},
{
"key" : "myvalue2",
"value" : "value2"
},
{
"key" : "myvalu3",
"value" : "value4"
}
]
}
And I'm having trouble doing so. The code I currently have is as follows:
#app.route('/dataread', methods=['GET', 'POST'])
def dataread():
if(request.json):
myvalue1 = request.json['NewList']['myvalue1']
return str(myvalue1)
else:
return 'nothing'
But it isn't working. I'm getting the following error:
KeyError: 'NewList'
I know my syntax must be wrong but I can't figure how to fix it. I'm sorry for such a newb question. Please can you help.
Thanks.
There's a few things going in your example that aren't correct.
The json is actually interpreted as :
{ "NewList":[
{
"key1" : "value1",
"key2" : "value2"
},
{
"key1" : "value1",
"key2" : "value2"
}
]}
So in your example myvalue1 is a value not a key.
The main error however is that in flask request.json() only returns the whole json input, you can't select certain elements from the json. Also request.json is deprecated so it now should be request.get_json()
So the final solution given your input data would be something like
{
"NewList":[
{
"key" : "myvalue1",
"value" : "value1"
},
{
"key" : "myvalue2",
"value" : "value2"
},
{
"key" : "myvalu3",
"value" : "value4"
}
]
}
#app.route('/dataread', methods=['GET', 'POST'])
def dataread():
if(request.data):
jdata = request.get_json()
return str(jdata['Newlist'][0]['key'])
else:
return 'nothing'
The [0] being the first object in the array that's the value of Newlist.
Unless you know it's always going to be array element 0 you'll get dud data, plus you're returning the value of the "key". Given that input data I suspect you probably want something more like.
if(request.data):
jdata = request.get_json()
for j in jdata['Newlist']:
if j['key'] == 'myvalue1':
return str(j['value'])
return 'not found'
else:
return 'nothing'
I wasn't sure which part of this you were having an issue with so I did it long hand. It's obviously not ideal but should be easily understandable.But I would echo the comment above that you should start by printing exactly what you got in. May not be a complete match.
The dictionary is a set of lists of dictionaries so you end up walking the set for each one.
dict = {
"NewList":[
{
"key" : "myvalue1",
"value" : "value1"
},
{
"key" : "myvalue2",
"value" : "value2"
},
{
"key" : "myvalu3",
"value" : "value4"
}
]
}
for firstkey, big_list in dict.items():
print('print dict: ' + str(firstkey))
for pair in big_list:
print('print sets in dict: ' + str(pair))
nextdict = pair
for nextkey, small_list in nextdict.items():
print('print each: ' + str(nextkey)+ '->' + str(small_list))
#address each one
print('pull just data: ' + str(nextdict[nextkey]))
"""
results
print dict: NewList
print sets in dict: {'key': 'myvalue1', 'value': 'value1'}
print each: key->myvalue1
pull just data: myvalue1
print each: value->value1
pull just data: value1
print sets in dict: {'key': 'myvalue2', 'value': 'value2'}
print each: key->myvalue2
pull just data: myvalue2
print each: value->value2
pull just data: value2
print sets in dict: {'key': 'myvalu3', 'value': 'value4'}
print each: key->myvalu3
pull just data: myvalu3
print each: value->value4
pull just data: value4
"""
The value of newList is a list. You have to access an element of this list first before reading off "key" and "value".
# print first item in the list
print(request.json['NewList'][0]['value']
# print them all
for item in request.json['NewList']
print(item['key'])
print(item['value'])
Related
I have a structure like this:
{
"id" : 1,
"user" : "somebody",
"players" : [
{
"name" : "lala",
"surname" : "baba",
"player_place" : "1",
"start_num" : "123",
"results" : {
"1" : { ... }
"2" : { ... },
...
}
},
...
]
}
I am pretty new to MongoDB and I just cannot figure out how to extract results for a specific user (in this case "somebody", but there are many other users and each has an array of players and each player has many results) for a specific player with start_num.
I am using pymongo and this is the code I came up with:
record = collection.find(
{'user' : name}, {'players' : {'$elemMatch' : {'start_num' : start_num}}, '_id' : False}
)
This extracts players with specific player for a given user. That is good, but now I need to get specific result from results, something like this:
{ 'results' : { '2' : { ... } } }.
I tried:
record = collection.find(
{'user' : name}, {'players' : {'$elemMatch' : {'start_num' : start_num}}, 'results' : result_num, '_id' : False}
)
but that, of course, doesn't work. I could just turn that to list in Python and extract what I need, but I would like to do that with query in Mongo.
Also, what would I need to do to replace specific result in results for specific player for specific user? Let's say I have a new result with key 2 and I want to replace existing result that has key 2. Can I do it with same query as for find() (just replacing method find with method replace or find_and_replace)?
You can replace a specific result and the syntax for that should be something like this,
assuming you want to replace the result with key 1,
collection.updateOne({
"user": name,
"players.start_num": start_num
},
{ $set: { "players.$.results.1" : new_result }})
I have a dictionary which contains the following json elements.
myjsonDictionary = \
{
"Teams": {
"TeamA": {
"#oid": "123.0.0.1",
"dataRequestList": {
"state": {
"#default": "0",
"#oid": "2"
}
},
"TeamSub": {
"#oid": "3",
"dataRequestList": {
"state": {
"#default": "0",
"#oid": "2"
}
}
}
},
# ....many nested layers
}
}
I have the following issue and am currently very confused on how to solve this problem.
I want to be able to parse this dictionary and get the concatenation of the "#oid" value and the respective "#oid" when I request the "key" such as "TeamA" or "TeamSub".
I have a function which takes in the gettheiDLevelConcatoid(myjsonDictionary, key).
I can call this function like this:
gettheiDLevelConcatoid(myjsonDictionary, key) where "key" is like "TeamA"
And the expected output should be "123.0.0.1.2". Note the 2 appended to the 123.0.0.1.
gettheiDLevelConcatoid(myjsonDictionary, key) where "key" is like TeamSub
Output is "123.0.0.1.3.2". Note the "3.2" added to the "123.0.0.1".
My current implementation:
def gettheiDLevelConcatoid(myjsonDictionary, key)
for item in myjsonDictionary:
if (item == key):
#not sure what to do
I am so lost on how to implement a generic method or approach for this.
With recursive traversal for specific keys:
def get_team_idlvel_oid_pair(d, search_key):
for k, v in d.items():
if k.startswith('Team'):
if k == search_key:
return '{}{}.{}'.format(d['#oid'] + '.' if '#oid' in d else '',
v['#oid'], v['dataRequestList']['state']['#oid'])
elif any(k.startswith('Team') for k_ in v):
return get_team_idlvel_oid_pair(v, search_key)
print(get_team_idlvel_oid_pair(myjsonDictionary['Teams'], 'TeamA'))
print(get_team_idlvel_oid_pair(myjsonDictionary['Teams'], 'TeamSub'))
Sample output:
123.0.0.1.2
123.0.0.1.3.2
I am attempting to parse some JSON that I am receiving from a RESTful API, but I am having trouble accessing the data in Python because it appears that there is an empty property name.
A sample of the JSON returned:
{
"extractorData" : {
"url" : "RetreivedDataURL",
"resourceId" : "e38e1a7dd8f23dffbc77baf2d14ee500",
"data" : [ {
"group" : [ {
"CaseNumber" : [ {
"text" : "PO-1994-1350",
"href" : "http://www.referenceURL.net"
} ],
"DateFiled" : [ {
"text" : "03/11/1994"
} ],
"CaseDescription" : [ {
"text" : "Mary v. JONES"
} ],
"FoundParty" : [ {
"text" : "Lastname, MARY BETH (Plaintiff)"
} ]
}, {
"CaseNumber" : [ {
"text" : "NP-1998-2194",
"href" : "http://www.referenceURL.net"
}, {
"text" : "FD-1998-2310",
"href" : "http://www.referenceURL.net"
} ],
"DateFiled" : [ {
"text" : "08/13/1993"
}, {
"text" : "06/02/1998"
} ],
"CaseDescription" : [ {
"text" : "IN RE: NOTARY PUBLIC VS REDACTED"
}, {
"text" : "REDACTED"
} ],
"FoundParty" : [ {
"text" : "Lastname, MARY H (Plaintiff)"
}, {
"text" : "Lastname, MARY BETH (Defendant)"
} ]
} ]
} ]
And the Python code I am attempting to use
import requests
import json
FirstName = raw_input("Please Enter First name: ")
LastName = raw_input("Please Enter Last Name: ")
with requests.Session() as c:
url = ('https://www.requestURL.net/?name={}&lastname={}').format(LastName, FirstName)
page = c.get(url)
data = page.content
theJSON = json.loads(data)
def myprint(d):
stack = d.items()
while stack:
k, v = stack.pop()
if isinstance(v, dict):
stack.extend(v.iteritems())
else:
print("%s: %s" % (k, v))
print myprint(theJSON["extractorData"]["data"]["group"])
I get the error:
TypeError: list indices must be integers, not str
I am new to parsing Python and more than simple python in general so excuse my ignorance. But what leads me to believe that it is an empty property is that when I use a tool to view the JSON visually online, I get empty brackets, Like so:
Any help parsing this data into text would be of great help.
EDIT: Now I am able to reference a certain node with this code:
for d in group:
print group[0]['CaseNumber'][0]["text"]
But now how can I iterate over all the dictionaries listed in the group property to list all the nodes labeled "CaseNumber" because it should exist in every one of them. e.g
print group[0]['CaseNumber'][0]["text"]
then
for d in group:
print group[1]['CaseNumber'][0]["text"]
and so on and so forth. Perhaps incrementing some sort of integer until it reaches the end? I am not quite sure.
If you look at json carefully the data key that you are accessing is actually a list, but data['group'] is trying to access it as if it were a dictionary, which is raising the TypeError.
To minify your json it is something like this
{
"extractorData": {
"url": "string",
"resourceId": "string",
"data": [{
"group": []
}]
}
}
So if you want to access group, you should first retrieve data which is a list.
data = sample['extractorData']['data']
then you can iterate over data and get group within it
for d in data:
group = d['group']
I hope this clarifies things a bit for you.
I'm using Flask-Eve to provide an API for my data. I would like to insert my records using Eve, so that I get a _created attribute and the other Eve-added attributes.
Two of my fields are dicts, and one is a list. When I try to insert that to Eve the structure seems to get flattened, losing some information. Trying to tell Eve about the dict & list elements gives me an error on POST, saying those fields need to be dicts and lists, but they already are! Please can someone help me & tell me what I'm doing wrong?
My Eve conf looked like this:
'myendpoint': { 'allow_unknown': True,
'schema': { 'JobTitle': { 'type': 'string',
'required': True,
'empty': False,
'minlength': 3,
'maxlength': 99 },
'JobDescription': { 'type': 'string',
'required': True,
'empty': False,
'minlength': 32,
'maxlength': 102400 },
},
},
But when I POST the following structure using requests:
{
"_id" : ObjectId("56e840686dbf9a5fe069220e"),
"Salary" : {
"OtherPay" : "On Application"
},
"ContactPhone" : "xx",
"JobTypeCodeList" : [
"Public Sector",
"Other"
],
"CompanyName" : "Scc",
"url" : "xx",
"JobTitle" : "xxx",
"WebAdID" : "TA7494725_1_1",
"JobDescription" : "xxxx",
"JobLocation" : {
"DisplayCity" : "BRIDGWATER",
"City" : "BRIDGWATER",
"StateProvince" : "Somerset",
"Country" : "UK",
"PostalCode" : "TA6"
},
"CustomField1" : "Permanent",
"CustomField3" : "FTJOBUKNCSG",
"WebAdManagerEmail" : "xxxx",
"JobType" : "Full",
"ProductID" : "JCPRI0UK"
}
The post line looks like this:
resp = requests.post(url, data = job)
It gets 'flattened' and loses the information from the dicts and list:
{
"_id" : ObjectId("56e83f5a6dbf9a6395ea559d"),
"Salary" : "OtherPay",
"_updated" : ISODate("2016-03-15T16:59:06Z"),
"ContactPhone" : "xx",
"JobTypeCodeList" : "Public Sector",
"CompanyName" : "Scc",
"url" : "xxx",
"JobTitle" : "xx",
"WebAdID" : "TA7494725_1_1",
"JobDescription" : "xxx",
"JobLocation" : "DisplayCity",
"CustomField1" : "Permanent",
"_created" : ISODate("2016-03-15T16:59:06Z"),
"CustomField3" : "FTJOBUKNCSG",
"_etag" : "55d8d394141652f5dc2892a900aa450403a63d10",
"JobType" : "Full",
"ProductID" : "JCPRI0UK"
}
I've tried updating my schema to say some are dicts and lists:
'JobTypeCodeList': { 'type': 'list'},
'Salary': { 'type': 'dict'},
'JobLocation': { 'type': 'dict'},
But then when I POST in the new record I get an error saying
{u'Salary': u'must be of dict type', u'JobTypeCodeList': u'must be of list type', u'JobLocation': u'must be of dict type'},
I've verified before the POST that type(job.Salary) == dict etc, so I'm not sure how to resolve this. While I can POST the record directly into MongoDB ok, bypassing Eve, I'd prefer to use Eve if possible.
In case this is useful to anyone else, I ended up working around this issue by posting a flat structure into Eve, and then using the on_insert and on_update events to loop through the keys and construct objects (and lists) from them.
It's a bit convoluted but it does the trick and now that it's in place it's fairly transparent to use. My objects added to MongoDB through Eve now have embedded lists and hashes, but they also get the handy Eve attributes like _created and _updated, while the POST and PATCH requests also get validated through Eve's normal schema.
The only really awkward thing is that on_insert and on_update send slightly different arguments, so there's a lot of repetition in the code below which I haven't yet refactored out.
Any characters can be used as flags: I'm using two underscores to indicate key/values which should end up as a single object, and two ampersands for values which should be split into a list. The structure I'm posting in now looks like this:
"Salary__OtherPay" : "On Application"
"ContactPhone" : "xx",
"JobTypeCodeList" : "Public Sector&&Other",
"CompanyName" : "Scc",
"url" : "xx",
"JobTitle" : "xxx",
"WebAdID" : "TA7494725_1_1",
"JobDescription" : "xxxx",
"JobLocation__DisplayCity" : "BRIDGWATER",
"JobLocation__City" : "BRIDGWATER",
"JobLocation__StateProvince" : "Somerset",
"JobLocation__Country" : "UK",
"JobLocation__PostalCode" : "TA6"
"CustomField1" : "Permanent",
"CustomField3" : "FTJOBUKNCSG",
"WebAdManagerEmail" : "xxxx",
"JobType" : "Full",
"ProductID" : "JCPRI0UK"
And my Eve schema has been updated accordingly to validate the values of those new key names. Then in the backend I've defined the function below which checks the incoming keys/values and converts them into objects/lists, and also deletes the original __ and && data:
import re
def flat_to_complex(items=None, orig=None):
if type(items) is dict: # inserts of new objects
if True: # just to force indentation
objects = {} # hash-based container for each object
lists = {} # hash-based container for each list
for key,value in items.items():
has_object_wildcard = re.search(r'^([^_]+)__', key, re.IGNORECASE)
if bool(has_object_wildcard):
objects[has_object_wildcard.group(1)] = None
elif bool(re.search(r'&&', unicode(value))):
lists[key] = str(value).split('&&')
for list_name, this_list in lists.items():
items[list_name] = this_list
for obj_name in objects:
this_obj = {}
for key,value in items.items():
if key.startswith('{s}__'.format(s=obj_name)):
match = re.search(r'__(.+)$', key)
this_obj[match.group(1)] = value
del(items[key])
objects[obj_name] = this_obj
for obj_name, this_obj in objects.items():
items[obj_name] = this_obj
elif type(items) is list: # updates to existing objects
for idx in range(len(items)):
if type(items[idx]) is dict:
objects = {} # hash-based container for each object
lists = {} # hash-based container for each list
for key,value in items[idx].items():
has_object_wildcard = re.search(r'^([^_]+)__', key, re.IGNORECASE)
if bool(has_object_wildcard):
objects[has_object_wildcard.group(1)] = None
elif bool(re.search(r'&&', unicode(value))):
lists[key] = str(value).split('&&')
for list_name, this_list in lists.items():
items[idx][list_name] = this_list
for obj_name in objects:
this_obj = {}
for key,value in items[idx].items():
if key.startswith('{s}__'.format(s=obj_name)):
match = re.search(r'__(.+)$', key)
this_obj[match.group(1)] = value
del(items[idx][key])
objects[obj_name] = this_obj
for obj_name, this_obj in objects.items():
items[idx][obj_name] = this_obj
And then I just tell Eve to run that function on inserts and updates to that collection:
app.on_insert_myendpoint += flat_to_complex
app.on_update_myendpoint += flat_to_complex
This achieves what I needed and the resulting record in Mongo is the same as the one from the question above (with _created and _updated attributes). It's obviously not ideal but it gets there, and it's fairly easy to work with once it's in place.
I had a problem on converting dictionaries to strings which has recursive features.
I had a map of routing such as the following;
urls = {
'/' : 'BaseController.hello',
'/api' : {
'/auth' : {
'/me' : 'ApiController.hello',
'/login' : {
'/guest' : 'ApiController.guest_login',
'/member': 'ApiController.member_login'
}
}
}
}
What I need to do is to generate a dictionary from that into the following;
url_map = {
'/' : 'BaseController.hello',
'/api/auth/me' : 'ApiController.hello',
'/api/auth/login/guest' : 'ApiController.guest_login',
'/api/auth/login/member': 'ApiController.member_login',
}
This feature is called route grouping but I haven't been able to write a function to generate that. Any ideas ?
You can recursively do it like this
def flatten(current_dict, current_key, result_dict):
# For every key in the dictionary
for key in current_dict:
# If the value is of type `dict`, then recurse with the value
if isinstance(current_dict[key], dict):
flatten(current_dict[key], current_key + key, result_dict)
# Otherwise, add the element to the result
else:
result_dict[current_key + key] = current_dict[key]
return result_dict
print flatten(urls, "", {})
Output
{
'/api/auth/me': 'ApiController.hello',
'/api/auth/login/guest': 'ApiController.guest_login',
'/': 'BaseController.hello',
'/api/auth/login/member': 'ApiController.member_login'
}