I am somewhat new to the google app engine models system(and all database models for that matter), and I am trying to figure out how make a model have a collection of model instances as one of the properties. Some how I feel that there has to be a better way to model the data I have.
So here is the example. I'm writing a journal webapp. I want to allow users to have multible journals as defined by the model Journals
#Represents a collection of journals
class Journals(db.Model):
user = db.UserProperty()
journals = db.ListProperty(int) #there has to be a better way of keeping track of journals then by a list of their id
In the property journals I keep a list of the id's of the Journal model(shown below).
class Journal(db.Model):
user = db.UserProperty()
name = db.StringProperty()
date_created = db.DateTimeProperty(auto_now_add=True)
last_modified = db.DateTimeProperty(auto_now=True)
journal_item = db.ListProperty(int)
In the Journal model I store a list of id's of JournalItem instances as a list of id's
class JournalItem(db.Model):
user = db.UserProperty()
name = db.StringProperty()
date_created = db.DateTimeProperty(auto_now_add=True)
last_modified = db.DateTimeProperty(auto_now=True)
content = db.StringProperty(multiline=True)
So I guess the question really is how should I model data with this structure(I omitted some properties to simplify it)(shown in json).
"Journals": {
"Journal": {
"user": "GAE user object",
"name": "journal 1",
"JournalItems": {
"JournalItem": {
"name": "entry 1",
"content": "some example content"
},
"JournalItem": {
"name": "entry 2",
"content": "some example content"
},
"JournalItem": {
"name": "entry n",
"content": "some example content"
}
}
}
"Journal": {
"user": "GAE user object",
"name": "journal 2",
"JournalItems": {
"JournalItem": {
"name": "entry 1",
"content": "some example content"
},
"JournalItem": {
"name": "entry 2",
"content": "some example content"
},
"JournalItem": {
"name": "entry n",
"content": "some example content"
}
}
}
}
I hope that wasn't to long-winded to be annoying. Any help on the matter would be greatly appreciated.
That's pretty much the way to do it, except I would use a ListProperty(db.Key) instead of int. Then you can do db.get(userjournals.journals) to get all the user's journals once you have the Journals instance belonging to that user (note that to avoid confusion, you should probably call that something like UserJournals).
Remember that GAE is a non-relational datastore, and sometimes you have to think a bit differently from how you would using SQL.
You can follow this guide which describes in an elegant fashion how you can model your entities relationships on Google Appengine.
The reversed approach you can take is to create a reference for example of a Journal in the JournalItem entity and assign a collection_name to it.
class JournalItem:
Journal = db.ReferencePoperty(Journal, collection_name='journal_items')
name = db....
content = db...
The you will be able to iterate through the JournalItems of the Journal in this fashion:
for item in journal.journal_items
logging.info(item.content)
Related
I'm quite new to marshmallow but my question refers to the issue of handling dict-like objects. There are no workable examples in the Marshmallow documentation. I came across with a simple example here in stack overflow Original question and this is the original code for the answer suppose this should be quite simple
from marshmallow import Schema, fields, post_load, pprint
class UserSchema(Schema):
name = fields.String()
email = fields.Email()
friends = fields.List(fields.String())
class AddressBookSchema(Schema):
contacts =fields.Dict(keys=fields.String(),values=fields.Nested(UserSchema))
#post_load
def trans_friends(self, item):
for name in item['contacts']:
item['contacts'][name]['friends'] = [item['contacts'][n] for n in item['contacts'][name]['friends']]
data = """
{"contacts": {
"Steve": {
"name": "Steve",
"email": "steve#example.com",
"friends": ["Mike"]
},
"Mike": {
"name": "Mike",
"email": "mike#example.com",
"friends": []
}
}
}
"""
deserialized_data = AddressBookSchema().loads(data)
pprint(deserialized_data)
However, when I run the code I get the following NoneType value
`None`
The input hasn't been marshalled.
I'm using the latest beta version of marshmallow 3.0.0b20. I can't find a way to make this work even it looks so simple. The information seems to indicate that nested dictionaries are being worked by the framework.
Currently I'm working in a cataloging application for flask where I'm receiving JSON messages where I can't really specify the schema beforehand. My specific problem is the following:
data = """
{"book": {
"title": {
"english": "Don Quixiote",
"spanish": "Don Quijote"
},
"author": {
"first_name": "Miguel",
"last_name": "Cervantes de Saavedra"
}
},
"book": {
"title": {
"english": "20000 Leagues Under The Sea",
"french": "20000 Lieues Sous Le Mer",
"japanese": "海の下で20000リーグ",
"spanish": "20000 Leguas Bajo El Mar",
"german": "20000 Meilen unter dem Meeresspiegel",
"russian": "20000 лиг под водой"
},
"author": {
"first_name": "Jules",
"last_name": "Verne"
}
}
}
This is just toy data but exemplifies that the keys in the dictionaries are not fixed, they change in number and text.
So the questions are why am I getting the validation error in a simple already worked example and if it's possible to use the marshmallow framework to validate my data,
Thanks
There are two issues in your code.
The first is the indentation of the post_load decorator. You introduced it when copying the code here, but you don't have it in the code you're running, otherwise you wouldn't get None.
The second is due to a documented change in marshmallow 3. pre/post_load/dump functions are expected to return the value rather than mutate it.
Here's a working version. I also reworked the decorator:
from marshmallow import Schema, fields, post_load, pprint
class UserSchema(Schema):
name = fields.String()
email = fields.Email()
friends = fields.List(fields.String())
class AddressBookSchema(Schema):
contacts = fields.Dict(keys=fields.String(),values=fields.Nested(UserSchema))
#post_load
def trans_friends(self, item):
for contact in item['contacts'].values():
contact['friends'] = [item['contacts'][n] for n in contact['friends']]
return item
data = """
{
"contacts": {
"Steve": {
"name": "Steve",
"email": "steve#example.com",
"friends": ["Mike"]
},
"Mike": {
"name": "Mike",
"email": "mike#example.com",
"friends": []
}
}
}
"""
deserialized_data = AddressBookSchema().loads(data)
pprint(deserialized_data)
And finally, the Dict in marshmallow 2 doesn't have key/value validation feature, so it will just silently ignore the keys and values argument and perform no validation.
I am using the python library marketo-rest-api to pull data from Marketo. I am just pulling one day to attempt to connect the dots from activities to campaigns. I am making the following calls:
print('Getting Campaigns')
with open(marketoCampaignsFile,'w') as fcamp:
campaigns = mc.execute(method='get_multiple_campaigns', id=None, name=None, programName=None, workspaceName=None, batchSize=None)
for campaign in campaigns:
jsonString = json.dumps(campaign)
fcamp.write(jsonString)
fcamp.close()
print('Getting Activities...')
activitiesFile = 'c:\\users\\mark\\marketocsv\\emailActivities.2016-07-26.json'
with open(activitiesFile,'w',newline='') as fopen:
for activities in mc.execute(method='get_lead_activities_yield', activityTypeIds=['6','7','8','9','10'], nextPageToken=None, sinceDatetime='2016-07-26', untilDatetime='2016-07-27', batchSize=None, listId=None, leadIds=None):
for item in activities:
jsonString = json.dumps(item)
fopen.write(jsonString+'\n')
fopen.close()
What I have found is that the campaign IDs in the activities file do not match any of the campaign IDs in the campaign file. Does anyone know why this might be? I need campaign attributes in order to filter the specific activities that I need. Thanks.
The activity types that you are downloading don't include the Campaign ID, they provide the Email ID instead.
So Jep was right. I did finally find the EmailID. It's called the primaryAttributeValueId. You can link this back to the EmailID provided by Marketo. I never did find the campaignID but I can get to the campaign through the email. Here's the full JSON from one of the requests:
{
"primaryAttributeValue": "2016-07-Email-To-Customers",
"activityDate": "2016-07-26T19:05:41Z",
"attributes": [{
"value": "0",
"name": "Choice Number"
},
{
"value": "43182",
"name": "Step ID"
}
],
"primaryAttributeValueId": 17030,
"leadId": 115345,
"id": 393962103,
"activityTypeId": 7,
"campaignId": 15937
}
I have the following two classes in my app.models and i'm using the wagtail APIs to get the data as json
class AuthorMeta(Page):
author=models.OneToOneField(User)
city = models.ForeignKey('Cities', related_name='related_author')
class Cities(Page):
name = models.CharField(max_length=30)
So, when I try /api/v1/pages/?type=dashboard.AuthorMeta&fields=title,city, it returns the following data:
{
"meta": {
"total_count": 1
},
"pages": [
{
"id": 11,
"meta": {
"type": "dashboard.AuthorMeta",
"detail_url": "http://localhost:8000/api/v1/pages/11/"
},
"title": "Suneet Choudhary",
"city": {
"id": 10,
"meta": {
"type": "dashboard.Cities",
"detail_url": "http://localhost:8000/api/v1/pages/10/"
}
}
}
]
}
In the city field, it returns the id and meta of the city. How can I get the name of the city in the response here, without making an extra query? :/
I couldn't find any solution in the Documentation. Am I missing something?
Use Django model property to return through the ForeignKey:
class AuthorMeta(Page):
author=models.OneToOneField(User)
city = models.ForeignKey('Cities', related_name='related_author')
city_name = property(get_city_name)
def get_city_name(self):
return self.city.name
Check Term Property to better understand the concept
In case you have the foreign key in a Streamfield, e.g. a PageChooserBlock, you can customize the api response by overwriting the get_api_representation of a block, as described in the example as provided here:
class CustomPageChooserBlock(blocks.PageChooserBlock):
""" Customize the api response. """
def get_api_representation(self, value, context=None):
""" Return the url path instead of the id. """
return value.url_path
I have a schema that using MongoEngine that looks like this
class User(db.Document)
email = db.EmailField(unique=True)
class QueueElement(db.EmbeddedDocument):
accepts = db.ListField(db.ReferenceField('Resource'))
user = db.ReferenceField(User)
class Resource(db.Document):
name = db.StringField(max_length=255, required=True)
current_queue_element = db.EmbeddedDocumentField('QueueElement')
class Queue(db.EmbeddedDocument):
name = db.StringField(max_length=255, required=True)
resources = db.ListField(db.ReferenceField(Resource))
queue_elements = db.ListField(db.EmbeddedDocumentField('QueueElement'))
class Room(db.Document):
name = db.StringField(max_length=255, required=True)
queues = db.ListField(db.EmbeddedDocumentField('Queue'))
and I would like to return a JSON object of a Room object that would include the information about its queues (together with the referenced resources), and the nested queue_elements ( together with their referenced "accepts" references, and user references)
However, when I want to return a Room with its relationships dereferenced:
room = Room.objects(slug=slug).select_related()
if (room):
return ast.literal_eval(room.to_json())
abort(404)
I don't get any dereferencing. I get:
{
"_cls":"Room",
"_id":{
"$oid":"552ab000605cd92f22347d79"
},
"created_at":{
"$date":1428842482049
},
"name":"second",
"queues":[
{
"created_at":{
"$date":1428842781490
},
"name":"myQueue",
"queue_elements":[
{
"accepts":[
{
"$oid":"552aafb3605cd92f22347d78"
},
{
"$oid":"552aafb3605cd92f22347d78"
},
{
"$oid":"552ab1f8605cd92f22347d7a"
}
],
"created_at":{
"$date":1428849389503
},
"user":{
"$oid":"552ac8c7605cd92f22347d7b"
}
}
],
"resources":[
{
"$oid":"552aafb3605cd92f22347d78"
},
{
"$oid":"552aafb3605cd92f22347d78"
},
{
"$oid":"552ab1f8605cd92f22347d7a"
}
]
}
],
"slug":"secondslug"
}
even though I'm using the select_related() function. I believe this is because MongoEngine may not follow references on embedded documents. Note, I can actually dereference in the python if I do something like this:
room = Room.objects(slug=slug).first().queues[0].queue_elements[0].accepts[0]
return ast.literal_eval(room.to_json())
which yields
{
"_id":{
"$oid":"552aafb3605cd92f22347d78"
},
"created_at":{
"$date":1428842849393
},
"name":"myRes"
}
which is clearly the dereferenced Resource document.
Is there a way I can follow references on embedded documents? Or is this coming up because I'm following a bad pattern, and should be finding a different way to store this information in MongoDB (or indeed, switch to a Relational DB) ? Thanks!
I want to store locations in google's datastore. Each entry shall have got 'sys'-fields, which shall contain information set by the datastore.
I've got the class model below and the WebService JSON request/response looks ok, but I have to set the values manually. It looks like auto_current_user_add, auto_now_add, auto_current_user and auto_now does not trigger.
from google.appengine.ext import ndb
from endpoints_proto_datastore.ndb import EndpointsModel
class Created(EndpointsModel):
by = ndb.UserProperty(auto_current_user_add=True)
on = ndb.DateTimeProperty(auto_now_add=True)
class Updated(EndpointsModel):
by = ndb.UserProperty(auto_current_user=True)
on = ndb.DateTimeProperty(auto_now=True)
class Sys(EndpointsModel):
created = ndb.StructuredProperty(Created)
updated = ndb.StructuredProperty(Updated)
class Location(EndpointsModel):
name = ndb.StringProperty(required=True)
description = ndb.TextProperty()
address = ndb.StringProperty()
sys = ndb.StructuredProperty(Sys)
When I submit a create request (location.put()) I get the following response:
{
"id": "4020001",
"name": "asdf"
}
When I set it manually using:
location.sys = Sys(created=Created(on=datetime.datetime.now(),
by=current_user),
updated=Updated(on=datetime.datetime.now(),
by=current_user))
location.put()
I get the expected result:
{
"id": "4020002",
"name": "asdf",
"sys": {
"created": {
"by": {
"auth_domain": "gmail.com",
"email": "decurgia#XYZ"
},
"on": "2015-01-27T16:05:41.465497"
},
"updated": {
"by": {
"auth_domain": "gmail.com",
"email": "decurgia#XYZ"
},
"on": "2015-01-27T16:05:41.465577"
}
}
}
How can I get those fields (sys.created.on, sys.created.by, sys.updated.on, sys.updated.by) automatically set?
In my limited work with StructuredProperty, I found it to be slower and more difficult to use than simply inserting the properties directly into the model. NDB seems to store those properties separately and perform a "join" when retrieving them. My recommendation is to use a "flat" model:
class Location(EndpointsModel):
name = ndb.StringProperty(required=True)
description = ndb.TextProperty()
address = ndb.StringProperty()
created_by = ndb.UserProperty(auto_current_user_add=True)
created_on = ndb.DateTimeProperty(auto_now_add=True)
updated_by = ndb.UserProperty(auto_current_user=True)
updated_on = ndb.DateTimeProperty(auto_now=True)
This should cause the auto_ properties to be triggered automatically.