I am using Python to go through a Firebase DB to return an array of objects, then pick one at random and return its values. I've been using a small test JSON DB that I manually build and import into Firebase. When I do this, the DB's child node's are 0, 1, 2 etc... Using the code below - I can iterate through the data and grab what I need.
I've been building a CMS that will let me input data directly into Firebase (instead of importing a local JSON doc) using the push() method.
Accordingly, the child nodes become obfuscated timestamps that look like this: K036VOR90fh8sd80, KO698fhs7Hf8sfds etc...
Now when I attempt to for-loop through the nodes, I get the following error at ln9 caption=....:
TypeError: string indices must be integers
I'm assuming this is happening because the child nodes are strings now. Since I have to use the CMS - how do I now loop through these nodes?
Code:
if 'Item' in intent['slots']:
chosen_item = intent['slots']['Item']['value']
result = firebase.get(chosen_item, None)
if result:
item_object = []
for item in result:
item_object.append(item)
random_item = (random.choice(item_object))
caption = random_item['caption']
audio_src = random_item['audioPath']
Here is an approximation of what the Firebase looks like:
{
"1001" : {
"-K036VOR90fh8sd80EQ" : {
"audioPath" : "https://s3.amazonaws.com/bucket-output/audio/audio_0_1001.mp3",
"caption" : "Record 0 from 1001"
},
"-KO698fhs7Hf8sfdsWJS" : {
"audioPath" : "https://s3.amazonaws.com/bucket-output/audio/audio_1_1001.mp3",
"caption" : "Record 1 from 1001"
}
},
"2001" : {
"-KOFsPBMKVtwHSOfiDJ" : {
"audioPath" : "https://s3.amazonaws.com/bucket-output/audio/audio_0_2001.mp3",
"caption" : "Record 0 from 2001"
},
"-KOFsQvwgF9icSIolU" : {
"audioPath" : "https://s3.amazonaws.com/bucket-output/audio/audio_1_2001.mp3",
"caption" : "Record 1 from 2001"
}
}
}
What needs to be done is to strip the dictionary result of the parent nodes using a for-loop and the .items() Python method - through the key-values (k,v) and .append the values (v).
This strips the result of the parent Firebase dictionary keys i.e -KOFsQvwgF9icSIolU.
if 'Item' in intent['slots']:
chosen_item = intent['slots']['Item']['value']
result = firebase.get(chosen_item, None)
if result:
item_object = []
for k,v in result.items():
item_object.append(v)
random_item = (random.choice(item_object))
caption = random_item['caption']
audio_src = random_item['audioPath']
Related
I have a strategic issue of writing a program doing a job.
I have CSV files like:
Column1 Column 2
------- ----------
parent1 [child1, child2, child3]
parent2 [child4, child5, child6]
child1 [child7, child8]
child5 [child10, child33]
... ...
It is unknown how deep each element of those lists will be extended and I want to loop through them.
Code:
def make_parentClass(self):
for i in self.csv_rows_list:
self.parentClassList.append(parentClass(i))
# after first Parent
for i in self.parentClassList:
if i.children !=[]:
for child in i.children:
for z in self.parentClassList:
if str(child) == str(z.node_parent):
i.node_children.append(z)
self.parentClassList.remove(z)
class parentClass():
node_children = []
def __init__(self, the_list):
self.node_parent = the_list[0]
self.children = the_list[1]
The above code might be a solution if I will find a way to iterate. Let me see if you like the question and makes sense now.
Output:
My aim is to build up a treeview through another language but first I need to make this output in JSON format. So the output expected to be something like:
{
paren1:{'child1':{'child7':{}, 'child8':{}},
'child2': {},
'child3': {},
},
parent2: {
'child4':{},
'child5': {
'child10':{},
'child33':{}
},
'child6':{}
}
}
I would recommend a solution using two dictionaries. One nested one with the actually data structure you plan to convert to JSON, and one flat one that will let you actually find the keys. Since everything is a reference in Python, you can make sure that both dictionaries have the exact same values. Carefully modifying the flat dictionary will build your structure for you.
The following code assumes that you have already managed to split each line into a string parent and list children, containing values form the two columns.
json_dict = {}
flat_dict = {}
for parent, children in file_iterator():
if parent in flat_dict:
value = flat_dict[parent]
else:
value = {}
flat_dict[parent] = json_dict[parent] = value
for child in children:
flat_dict[child] = value[child] = {}
Running this produces json_dict like this:
{
'parent1': {
'child1': {
'child7': {},
'child8': {}
},
'child2': {},
'child3': {}
},
'parent2': {
'child4': {},
'child5': {
'child10': {},
'child33': {}
},
'child6': {}
}
}
Here is an IDEOne link to play with.
I need to export dict to yaml with pyyaml without the "-"
export_dash_dict = {}
export_dash_dict["dashboards"] = []
for dashboard in dashboards_to_export:
single_dashboard = {}
single_dashboard[dashboard.title] = {}
single_dashboard[dashboard.title]["owner"] = dashboard.owner.username
single_dashboard[dashboard.title]["description"] = dashboard.description
export_dash_dict["dashboards"].append(single_dashboard)
final_yaml = yaml.dump(export_dash_dict, default_flow_style=False,default_style=None)
This is how pyyaml export my dict:
dashboards:
- Dashboard title 1:
description: First
owner: username1
- Dashboard title 2:
description: Second
owner: username2
Using an online parsed, this is what I get:
{
"dashboards": [
{
"Dashboard title 1": {
"owner": "username1",
"description": "First"
}
},
{
"Dashboard title 2": {
"owner": "username2",
"description": "Second"
}
}
]
}
But I need something like this:
dashboards:
Dashboard title 1:
description: First
owner: username1
Dashboard title 2:
description: Second
owner: username2
with the online parser:
{
"dashboards": {
"Dashboard title 2": {
"owner": "username2",
"description": "Second"
},
"Dashboard title 1": {
"owner": "username1",
"description": "First"
}
}
}
In this way I can avoid a entire level dept of data when I use yaml.load
Im using: default_flow_style=False parameters but I can't find a way to avoid "-"
Is that a standard of YAML?
The yaml dump is correctly exporting your data structure. You have a dictionary (export_dash_dict) with one key and one value. The one key is dashboards and its value is a list with two elements. The two elements are dictionaries (whose contents we won't go into here, but they have two key-value pairs).
The data structure you say that you would like to export is not a dictionary with one key/value pair where the value is a list, but rather a dictionary with two key-value pairs. The first key is to be Dashboard title 1 and its value is to be a dictionary (with two key-value pairs that we won't detail here). The second key is to be Dashboard title 2 and its value is to be a dictionary (with, again, two key-value pairs that we won't detail).
If you want yaml.dump to write such a data structure, you need to construct that data structure.
Thanks to torek, this is the correct way to build the dict:
export_dash_dict = {}
export_dash_dict["dashboards"] = {}
for dashboard in dashboards_to_export:
export_dash_dict["dashboards"][dashboard.title] = {}
export_dash_dict["dashboards"][dashboard.title]["owner"] = dashboard.owner.username
export_dash_dict["dashboards"][dashboard.title]["description"] = dashboard.description
final_yaml = yaml.dump(export_dash_dict, default_flow_style=False,default_style=None)
Giving data organized in JSON format (code example bellow) how can we get the path of keys and sub-keys associated with a given value?
i.e.
Giving an input "23314" we need to return a list with:
Fanerozoico, Cenozoico, Quaternario, Pleistocenico, Superior.
Since data is a json file, using python and json lib we had decoded it:
import json
def decode_crono(crono_file):
with open(crono_file) as json_file:
data = json.load(json_file)
Now on we do not know how to treat it in a way to get what we need.
We can access keys like this:
k = data["Fanerozoico"]["Cenozoico"]["Quaternario "]["Pleistocenico "].keys()
or values like this:
v= data["Fanerozoico"]["Cenozoico"]["Quaternario "]["Pleistocenico "]["Superior"].values()
but this is still far from what we need.
{
"Fanerozoico": {
"id": "20000",
"Cenozoico": {
"id": "23000",
"Quaternario": {
"id": "23300",
"Pleistocenico": {
"id": "23310",
"Superior": {
"id": "23314"
},
"Medio": {
"id": "23313"
},
"Calabriano": {
"id": "23312"
},
"Gelasiano": {
"id": "23311"
}
}
}
}
}
}
It's a little hard to understand exactly what you are after here, but it seems like for some reason you have a bunch of nested json and you want to search it for an id and return a list that represents the path down the json nesting. If so, the quick and easy path is to recurse on the dictionary (that you got from json.load) and collect the keys as you go. When you find an 'id' key that matches the id you are searching for you are done. Here is some code that does that:
def all_keys(search_dict, key_id):
def _all_keys(search_dict, key_id, keys=None):
if not keys:
keys = []
for i in search_dict:
if search_dict[i] == key_id:
return keys + [i]
if isinstance(search_dict[i], dict):
potential_keys = _all_keys(search_dict[i], key_id, keys + [i])
if 'id' in potential_keys:
keys = potential_keys
break
return keys
return _all_keys(search_dict, key_id)[:-1]
The reason for the nested function is to strip off the 'id' key that would otherwise be on the end of the list.
This is really just to give you an idea of what a solution might look like. Beware the python recursion limit!
Based on the assumption that you need the full dictionary path until a key named id has a particular value, here's a recursive solution that iterates the whole dict. Bear in mind that:
The code is not optimized at all
For huge json objects it might yield StackOverflow :)
It will stop at first encountered value found (in theory there shouldn't be more than 1 if the json is semantically correct)
The code:
import json
from types import DictType
SEARCH_KEY_NAME = "id"
FOUND_FLAG = ()
CRONO_FILE = "a.jsn"
def decode_crono(crono_file):
with open(crono_file) as json_file:
return json.load(json_file)
def traverse_dict(dict_obj, value):
for key in dict_obj:
key_obj = dict_obj[key]
if key == SEARCH_KEY_NAME and key_obj == value:
return FOUND_FLAG
elif isinstance(key_obj, DictType):
inner = traverse_dict(key_obj, value)
if inner is not None:
return (key,) + inner
return None
if __name__ == "__main__":
value = "23314"
json_dict = decode_crono(CRONO_FILE)
result = traverse_dict(json_dict, value)
print result
I'm using Flask-Eve to provide an API for my data. I would like to insert my records using Eve, so that I get a _created attribute and the other Eve-added attributes.
Two of my fields are dicts, and one is a list. When I try to insert that to Eve the structure seems to get flattened, losing some information. Trying to tell Eve about the dict & list elements gives me an error on POST, saying those fields need to be dicts and lists, but they already are! Please can someone help me & tell me what I'm doing wrong?
My Eve conf looked like this:
'myendpoint': { 'allow_unknown': True,
'schema': { 'JobTitle': { 'type': 'string',
'required': True,
'empty': False,
'minlength': 3,
'maxlength': 99 },
'JobDescription': { 'type': 'string',
'required': True,
'empty': False,
'minlength': 32,
'maxlength': 102400 },
},
},
But when I POST the following structure using requests:
{
"_id" : ObjectId("56e840686dbf9a5fe069220e"),
"Salary" : {
"OtherPay" : "On Application"
},
"ContactPhone" : "xx",
"JobTypeCodeList" : [
"Public Sector",
"Other"
],
"CompanyName" : "Scc",
"url" : "xx",
"JobTitle" : "xxx",
"WebAdID" : "TA7494725_1_1",
"JobDescription" : "xxxx",
"JobLocation" : {
"DisplayCity" : "BRIDGWATER",
"City" : "BRIDGWATER",
"StateProvince" : "Somerset",
"Country" : "UK",
"PostalCode" : "TA6"
},
"CustomField1" : "Permanent",
"CustomField3" : "FTJOBUKNCSG",
"WebAdManagerEmail" : "xxxx",
"JobType" : "Full",
"ProductID" : "JCPRI0UK"
}
The post line looks like this:
resp = requests.post(url, data = job)
It gets 'flattened' and loses the information from the dicts and list:
{
"_id" : ObjectId("56e83f5a6dbf9a6395ea559d"),
"Salary" : "OtherPay",
"_updated" : ISODate("2016-03-15T16:59:06Z"),
"ContactPhone" : "xx",
"JobTypeCodeList" : "Public Sector",
"CompanyName" : "Scc",
"url" : "xxx",
"JobTitle" : "xx",
"WebAdID" : "TA7494725_1_1",
"JobDescription" : "xxx",
"JobLocation" : "DisplayCity",
"CustomField1" : "Permanent",
"_created" : ISODate("2016-03-15T16:59:06Z"),
"CustomField3" : "FTJOBUKNCSG",
"_etag" : "55d8d394141652f5dc2892a900aa450403a63d10",
"JobType" : "Full",
"ProductID" : "JCPRI0UK"
}
I've tried updating my schema to say some are dicts and lists:
'JobTypeCodeList': { 'type': 'list'},
'Salary': { 'type': 'dict'},
'JobLocation': { 'type': 'dict'},
But then when I POST in the new record I get an error saying
{u'Salary': u'must be of dict type', u'JobTypeCodeList': u'must be of list type', u'JobLocation': u'must be of dict type'},
I've verified before the POST that type(job.Salary) == dict etc, so I'm not sure how to resolve this. While I can POST the record directly into MongoDB ok, bypassing Eve, I'd prefer to use Eve if possible.
In case this is useful to anyone else, I ended up working around this issue by posting a flat structure into Eve, and then using the on_insert and on_update events to loop through the keys and construct objects (and lists) from them.
It's a bit convoluted but it does the trick and now that it's in place it's fairly transparent to use. My objects added to MongoDB through Eve now have embedded lists and hashes, but they also get the handy Eve attributes like _created and _updated, while the POST and PATCH requests also get validated through Eve's normal schema.
The only really awkward thing is that on_insert and on_update send slightly different arguments, so there's a lot of repetition in the code below which I haven't yet refactored out.
Any characters can be used as flags: I'm using two underscores to indicate key/values which should end up as a single object, and two ampersands for values which should be split into a list. The structure I'm posting in now looks like this:
"Salary__OtherPay" : "On Application"
"ContactPhone" : "xx",
"JobTypeCodeList" : "Public Sector&&Other",
"CompanyName" : "Scc",
"url" : "xx",
"JobTitle" : "xxx",
"WebAdID" : "TA7494725_1_1",
"JobDescription" : "xxxx",
"JobLocation__DisplayCity" : "BRIDGWATER",
"JobLocation__City" : "BRIDGWATER",
"JobLocation__StateProvince" : "Somerset",
"JobLocation__Country" : "UK",
"JobLocation__PostalCode" : "TA6"
"CustomField1" : "Permanent",
"CustomField3" : "FTJOBUKNCSG",
"WebAdManagerEmail" : "xxxx",
"JobType" : "Full",
"ProductID" : "JCPRI0UK"
And my Eve schema has been updated accordingly to validate the values of those new key names. Then in the backend I've defined the function below which checks the incoming keys/values and converts them into objects/lists, and also deletes the original __ and && data:
import re
def flat_to_complex(items=None, orig=None):
if type(items) is dict: # inserts of new objects
if True: # just to force indentation
objects = {} # hash-based container for each object
lists = {} # hash-based container for each list
for key,value in items.items():
has_object_wildcard = re.search(r'^([^_]+)__', key, re.IGNORECASE)
if bool(has_object_wildcard):
objects[has_object_wildcard.group(1)] = None
elif bool(re.search(r'&&', unicode(value))):
lists[key] = str(value).split('&&')
for list_name, this_list in lists.items():
items[list_name] = this_list
for obj_name in objects:
this_obj = {}
for key,value in items.items():
if key.startswith('{s}__'.format(s=obj_name)):
match = re.search(r'__(.+)$', key)
this_obj[match.group(1)] = value
del(items[key])
objects[obj_name] = this_obj
for obj_name, this_obj in objects.items():
items[obj_name] = this_obj
elif type(items) is list: # updates to existing objects
for idx in range(len(items)):
if type(items[idx]) is dict:
objects = {} # hash-based container for each object
lists = {} # hash-based container for each list
for key,value in items[idx].items():
has_object_wildcard = re.search(r'^([^_]+)__', key, re.IGNORECASE)
if bool(has_object_wildcard):
objects[has_object_wildcard.group(1)] = None
elif bool(re.search(r'&&', unicode(value))):
lists[key] = str(value).split('&&')
for list_name, this_list in lists.items():
items[idx][list_name] = this_list
for obj_name in objects:
this_obj = {}
for key,value in items[idx].items():
if key.startswith('{s}__'.format(s=obj_name)):
match = re.search(r'__(.+)$', key)
this_obj[match.group(1)] = value
del(items[idx][key])
objects[obj_name] = this_obj
for obj_name, this_obj in objects.items():
items[idx][obj_name] = this_obj
And then I just tell Eve to run that function on inserts and updates to that collection:
app.on_insert_myendpoint += flat_to_complex
app.on_update_myendpoint += flat_to_complex
This achieves what I needed and the resulting record in Mongo is the same as the one from the question above (with _created and _updated attributes). It's obviously not ideal but it gets there, and it's fairly easy to work with once it's in place.
As part of a for-in loop in python using pymongo I want to add some nested documents/objects inside a linktype field which is to be within a links field:
neither the links field or linktype field exists before the first such entries are to be added.
What is the commands to do this ?
Here is an item before adding links:
item = {
"_id" : ObjectId("5067c26b9d595266e25e825a"),
"name": "a Name"
}
And after adding one link of type typeA:
toType = "typeA"
to_link = {"_id" : ObjectId("5067c26b9d595266e25e825b"), "property":"value"}
{
"_id" : ObjectId("5067c26b9d595266e25e825a"),
"name": "a Name",
"links" : {
"typeA":{
{"_id" : ObjectId("5067c26b9d595266e25e825b"), "property":"value"}
}
}
}
I have tried:
db.collection.update({"name":"a Name"},{{"links":{"$addToSet":{toType:to_link}}})
which doesnt work.
If I just use:
db.collection.update({"name":"a Name"},{ {"$addToSet":{toType:to_link}} )
that works but that is not what i want.
$addToSet is for adding to an array. To add a new property to an existing embedded object you need to use the $set operator and dot notation as:
db.collection.update({name: 'a name'}, {$set: {'links.' + toType: to_link}})