A user accesses his contacts on his mobile device. I want to send back to the server all the phone numbers (say 250), and then query for any User entities that have matching phone numbers.
A user has a phone field which is indexed. So I do User.query(User.phone.IN(phone_list)), but I just looked at AppStats, and is this damn expensive. It cost me 250 reads for this one operation, and this is something I expect a user to do often.
What are some alternatives? I suppose I can set the User entity's id value to be his phone number (i.e when creating a user I'd do user = User(id = phone_number)), and then get directly by keys via ndb.get_multi(phones), but I also want to perform this same query with emails too.
Any ideas?
You could create a PhoneUser model like so:
from google.appengine.ext import ndb
class PhoneUser(ndb.Model):
number = ndb.StringProperty()
user = ndb.KeyProperty()
class User(ndb.Model):
pass
u = User()
u.put()
p = PhoneUser(id='123-456-7890', number='123-456-7890', user=u.key)
p.put()
u2 = User()
u2.put()
p2 = PhoneUser(id='555-555-5555', number='555-555-5555', user=u2.key)
result = ndb.get_multi([ndb.Key(PhoneUser, '123-456-7890'), ndb.Key(PhoneUser, '555-555-5555')])
I think that would work in this situation. You would just have to add/delete your PhoneUser model whenever you update your User. You can do this using post hooks: https://developers.google.com/appengine/docs/python/ndb/modelclass#Model__post_delete_hook
I misunderstood part of your problem, I thought you were issuing a query that was giving you 250 entities.
I see what the problem is now, you're issuing an IN query with a list of 250 phone numbers, behind the scenes, the datastore is actually doing 250 individual queries, which is why you're getting 250 read ops.
I can't think of a way to avoid this. I'd recommend avoiding searching on long lists of phone numbers. This seems like something you'd need to do only once, the first time the user logs in using that phone. Try to find some way to store the results and avoid the query again.
there is no efficient way to do an IN query.
so instead avoid it all together.
how?
invert the query, instead of finding all people that belong to this guys phone list.
try
finding all people that have this users phoneid in their list.
this however is not without some extra cost.
the phonelist for each user much be stored and indexed.
class User(ndb.Model):
phoneList = ndb.PropertyList()
phone_id= ndb.StringProperty()
select from where User.phoneList = :this_phone_number
Related
I have a collection with multiple Course documents. On it, i have an array of course_users like the following example:
Like this, I can have multiple courses in the same collection.
My purpose is to do something with any given User id, but first, I need to know if this user exists on any of the courses of the collection, for example, if User with an ID of 123 is on Course A OR Course B, I should return True. If user with an ID 456 is not on any Course I should return False.
Is there a way to make this query in as few steps as possible using mongoengine in flask?
My idea was to use the $ operator like the following:
def userIsInCourse(user_id):
course_students = Course.objects(students__S__in=[user_id])
print(course_students)
Thinking that Course.objects will retrieve every course filtered with students__S__in=user_id_ but I'm failing.
Sorry for the vague question.
I ended up modifying my query.
Instead of trying to use the $ operator, I modified my query to use a raw query:
def userIsInCourse(user_id):
course_students = Course.objects(__raw__={'students': {'$elemMatch': {'student_id': user_id}}})
if course_students:
return True
else:
return False
It retrieves the whole Courses array, and looks in each Course's students array of objects, then filters using the given user's id. If the user exists on the course, returns true.
With this approach it was simpler than i thought.
I'm having an issue here that I want clarification with, see, I'm making a program that does analysis of data.
I wanted to query data from different users, the data is numerical by the way, whenever I'd get subject marks from a user, I want the system to return the name of the user who has those marks.
It was all working fine until I tried querying the users with the same marks, and all I could get was an error
analyzer.models.ResultsALevel.MultipleObjectsReturned: get() returned more than one
ResultsALevel -- it returned 4!
So I was trying to find a way to still query and return the name of users with that subject mark, even if they have the same marks.
I believe that should be possible since the users have different id's and stuff, help would be much appreciated!
Here's my views.py
biology_marks = []
for student in ResultsALevel.objects.all():
biology_marks.append(student.advanced_biology_1)
value_1_biology =
ResultsALevel.objects.get(advanced_biology_1=biology_marks[-1]).student_name.last_name
value_2_biology =
ResultsALevel.objects.get(advanced_biology_1=biology_marks[-2]).student_name.last_name
value_3_biology =
ResultsALevel.objects.get(advanced_biology_1=biology_marks[-3]).student_name.last_name
value_4_biology =
ResultsALevel.objects.get(advanced_biology_1=biology_marks[-4]).student_name.last_name
ObjectManager.get() is used to retrieve single instances, usually selected by a primary key field.
Using ObjectManager.get() on a field where multiple data exists which matches the query, an error is returned (MultipeObjectsReturned)
Use ObjectManager.filter() in stead.
I am creating a dynamic form in html template to obtain the currency pair that user select. The form looks like this:
The user can dynamically add and remove currency. I used a Jquery to generate different currency pairs name of currency_n for the nth currency.
My best question is how to store those into datastore.
If user only create one currency pair (i.e. The form is not dynamic), the get.request should be
def post(self):
user_currency_1_1 = self.request.get('currency_1_1')
user_currency_1_2 = self.request.get('currency_1_2')
I didnt know how many currency pairs does customer submit, how to get the correct value? Also, how to design the datastore structure to store those dynamic content?
class currency(db.Model):
user = db.EmailProperty(required = True)
currency = db.StringProperty(required = True)
I hesitate to even answer, as you have not responded to answers in your last question. We are not here to write your code for you. But, you are fortunate that SO users are generous with their time.
Following up on my last answer:
from google.appengine.ext import ndb
class UserCurrencies(ndb.Model):
user_email = ndb.StringProperty()
currencies = ndb.JsonProperty(indexed=False)
The model for my Resource class is as follows:
class Resource(ndb.Model):
name = ndb.StringProperty()
availability = ndb.StructuredProperty(Availability, repeated=True)
tags = ndb.StringProperty(repeated=True)
owner = ndb.StringProperty()
id = ndb.StringProperty(indexed=True, required=True)
lastReservedTime = ndb.DateTimeProperty(auto_now_add=False)
startString = ndb.StringProperty()
endString = ndb.StringProperty()
I want to extract records where the owner is equal to a certain string.
I have tried the below query. It does not give an error but does not return any result either.
Resource.query(Resource.owner== 'abc#xyz.com').fetch()
As per my understanding if a column has duplicate values it shouldn't be indexed and that is why owner is not indexed. Please correct me if I am wrong.
Can someone help me figure out how to achieve a where clause kind of functionality?
Any help is appreciated! Thanks!
Just tried this. It worked first time. Either you have no Resource entities with an owner of "abc#xyz.com", or the owner property was not indexed when the entities were put (which can happen if you had indexed=False at the time the entities were put).
My test:
Resource(id='1', owner='abc#xyz.com').put()
Resource(id='2', owner='abc#xyz.com').put()
resources = Resource.query(Resource.owner == 'abc#xyz.com').fetch()
assert len(resources) == 2
Also, your comment:
As per my understanding if a column has duplicate values it shouldn't
be indexed and that is why owner is not indexed. Please correct me if
I am wrong.
Your wrong!
Firstly, there is no concept of a 'column' in a datastore model, so I will I assume you mean 'Property'.
Next, to clarify what you mean by "if a column property has duplicate values":
I assume you mean 'multiple entities created from the same model with the same value for a specific property', in your case 'owner'. This has no effect on indexing, each entity will be indexed as expected.
Or maybe you mean 'a single entity with a property that allows multiple values (ie a list)', which also does not prevent indexing. In this case, the entity will be indexed multiple times, once for each item in the list.
To further elaborate, most properties (ie ones that accept primitive types such as string, int, float etc) are indexed automatically, unless you add the attribute indexed=False to the Property constructor. In fact, the only time you really need to worry about indexing is when you need to perform more complex queries, which involve querying against more that 1 property (and even then, by default, the app engine dev server will auto create the indexes for you in your local index.yaml file), or using inequality filters.
Please read the docs for more detail.
Hope this helps!
Say you the table Users (one): id and activities (many): type, created_on , how do you get all users who's latest activity.type is 'message'?
Heres the query I have so far:
User.objects.filter(activity__type='message').annotate(Max('activity__created_on'))
But its not working since:
User.objects.filter(activity__type='message').annotate(Max('activity__created_on'))[0].activity.latest('created_on').type
is not equal to 'message'.
I believe it does not work because when you filter(activity__type='message') Django is going through all the activities for that User and discarding the User if it has an activity with type other than message, it's not discarding the activities.
If you're using PostgreSQL you can use distinct and get fields from the User model, not actual instances of them though, but that might be enough for you:
Activity.objects.order_by('user__username', 'created_on')\
.distinct('user__username')\
.filter(type='message')\
.values('user__username')
If you are not using PostgreSQL or need full User instances you'll need to do it manually in Python, something like:
users_with_message_activity = []
for u in User.objects.all():
if u.activities_set.latest('created_on').type == 'message':
users_with_message_activity.append(u)
Which is not ideal since it means more queries.
Hope that helps.