Develop Reference

Chaining methods with lazy execution in Python

I was studying Django when i found they are chaining their query methods like Post.objects.filter(pk=1).filter(title='first').filter(author='me') to construct a query without actually executing it, and only execute the query when we try to access and work with its result.
From there i got interested to know how they are doing this so i can apply the same approach in my work, so for instance i can have something like
Writing code like myProduct.discount('10%').discountLimit('100$').tax('10$').shipping('20$') will only evaluate when i try work with it.
Build custom DB manager for non Django apps where i can chain my query methods and execute the query automatically only when i try to access its result (or at least when chaining ends). So i can end with something like
#doesn't hit the DB
myPost = Post.objects.select(...).where(...).where(...).limit(...)
#only hit the DB on usage
print(myPost.title)
So my QUESTION is, how can i do so?
Approaches that i thought of but i don't like
I can implement an .execute() method to perform the actual execution, calling it at the tail of the chain or whenever desired Post.objects.select(x).where(y).offset(z).execute()
I can insert a delay within each of the query builder methods to make sure it is the last in chain
class Post:
def where(self,...):
me = now()
self.lastCall = me
#process the inputs here
self.query += "WHERE ..."
self.lazyExecute(me)
return self
def lazyExecute(self,identifier):
delay(5000)
if self.lastCall = identifier
self.executeQuery()
else:
pass

Django method same object not saving

This is something I have been trying to solve for 3 days now and I just can't get my head around it why this is not working.
I have a method that creates a new version of an Object. It used to work, that you would pass in the sou obj. and this would be the source from which a new version is created. You can also pass in a destination, which is not really important in this example. Now I wanted to add locking to this method as we want to add multiple users. So I want to be sure that I always have the most current object from which I create a new one. So I added a line that would just get the newest object. If there is no newer object in the database it would be the same anyway.
def createRevision(request, what, sou, destination=None, ignore = [], **args):
...
if "initial" not in args.keys():
source = get_object_or_404(BaseItem, ppk=sou.ppk, project=sou.project, current=True)
print "------------"
print source == sou
print "------------"
# This outputs True
else:
source = sou
further down in the method I do something like
source.current = False
source.save()
Basically the idea is that I pass in BaseItem and if I don't specify the "initial" keyword then I get the current item from that project with the same ppk (Which is a special random pk in conduction with current). I do this just to be on the save side, that I really have the most current object. And if it is the initial version I just use that one, as there can not be another version.
So now the problem is, that everything works fine if I use sou in this method. I can save it etc .. but as soon as I use source and initial is not in the args it just doesn't save it. The print statement tells me they are the same. Everything I print after the save tells me it has been saved but it just doesn't do it.
source.current = False
source.save()
print "SAVED !!!!"
print source.pk
print source.current
rofl = get_object_or_404(BaseItem, pk=source.pk, project=sou.project)
print rofl.pk
print source.current
outputs the same pk and the same current value but somehow it is not properly saved. As soon as I look into django admin or do a select current = True.
I really don't know what to do anymore.
Why does it work without a problem if I pass in the object into the method but starts to fail when I get the exact same object in the method?
Of course I call the method with the same object:
x = get_object_or_404(BaseItem, ppk=sou.ppk, project=sou.project, current=True)
createRevision(request, "", x)

Thank you pztrick for the hint with the caches. I finally solved it. So the problem was that I was doing:
x = get_object_or_404(BaseItem, ppk=sou.ppk, project=sou.project, current=True)
createRevision(request, "", x)
# .... loads of lines of code
unlock(x)
unlock is a method I wrote that just sets a timestamp so I know no other user is editing it. So now the problem was that I was saving x in createRevision with all the correct data but of course unlock(x) still had a reference to an "old" not updated object and of course was saving it again. Hence it was overwriting my changes in createRevision.
Thank you again to everyone who helped with this.

I think you may be running afoul of model manager caching which is intended to limit database queries. However, by invoking the .all() method on the model manager you force it to hit the databse again.
So, try this: Replace your argument from the BaseItem class to the model manager's .all() QuerySet:
source = get_object_or_404(BaseItem.objects.all(), ppk=sou.ppk, project=sou.project, current=True)
# ...
rofl = get_object_or_404(BaseItem.objects.all(), pk=source.pk, project=sou.project)
get_object_or_404 supports mode classes, model managers, or QuerySets as the first parameter so this is valid.

Python django how to properly test if a queryset returned a result

I haven't had a very thorough training in python and sometimes don't know the correct way of doing things. One of these things is testing if my resultQuery returned a result or not. I find myself doing this a lot:
try:
user = User.objects.get(email=email)
except DoesNotExist:
user = User()
I don't know about python but try catches in other languages are supposed to be for exceptions and not for normal program flow. How would you do this with an if else?
I guess I want something similar to:
if request.GET.get('email','') is not None:
email = request.GET['email'])

Your exception example is most commonly the preferred way to do things. Wikipedia has quite a nice description about this:
Python style calls for the use of exceptions whenever an error
condition might arise. Rather than testing for access to a file or
resource before actually using it, it is conventional in Python to
just go ahead and try to use it, catching the exception if access is
rejected.
Exceptions are often used as an alternative to the if-block
(...). A commonly-invoked motto is EAFP, or "It is Easier to Ask
for Forgiveness than Permission."
Exceptions aren't necessarily costly in Python. Again, quoting the Wikipedia example:
if hasattr(spam, 'eggs'):
ham = spam.eggs
else:
handle_error()
... versus:
try:
ham = spam.eggs
except AttributeError:
handle_error()
These two code samples have the same effect, although there will be performance differences. When spam has the attribute eggs, the EAFP sample will run faster. When spam does not have the attribute eggs (the "exceptional" case), the EAFP sample will run slower.
Specifically for Django, I would use the exception example. It is the standard way to do things in that framework, and following standards is never a bad thing :).

You can use .filter() instead of .get(), and also make use of django's optimized queryset evaluation.
You can do
qs = User.objects.filter(email=email)
if qs :
user = qs[0]
...

I agree that catching the exception is the normal pattern here. If you do need to check for whether or not a queryset returns results, the exists method is one way to do so that I don't think has been mentioned yet. The docs run through the various performance implications, most notably that if you know the queryset will be evaluated it's better to just check the boolean interpretation of the queryset.

Django defines a class EmptyQuerySet that can be used for special implementations when strict isinstance(...) testing is necessary.
A straightforward way is as follows:
try:
user = User.objects.get(email=email)
if user:
# ...
else:
# ...
except DoesNotExist:
user = User()
Django provides a get method to directly retrieve one single Model instance or will raise an Exception. In this case the normal way is to check for Exceptions such as DoesNotExist
To retrieve entire rows from a database query as you normally would, use the user.objects.filter(...) method. The returned QuerySet instance provides you with .count(), .exists() methods.

Idiomatic/fast Django ORM check for existence on mysql/postgres

If I want to check for the existence and if possible retrieve an object, which of the following methods is faster? More idiomatic? And why? If not either of the two examples I list, how else would one go about doing this?
if Object.objects.get(**kwargs).exists():
my_object = Object.objects.get(**kwargs)
my_object = Object.objects.filter(**kwargs)
if my_object:
my_object = my_object[0]
If relevant, I care about mysql and postgres for this.

Why not do this in a try/except block to avoid the multiple queries / query then an if?
try:
obj = Object.objects.get(**kwargs)
except Object.DoesNotExist:
pass
Just add your else logic under the except.

django provides a pretty good overview of exists
Using your first example it will do the query two times, according to the documentation:
if some_queryset has not yet been evaluated, but you
know that it will be at some point, then using some_queryset.exists()
will do more overall work (one query for the existence check plus an
extra one to later retrieve the results) than simply using
bool(some_queryset), which retrieves the results and then checks if
any were returned.
So if you're going to be using the object, after checking for existance, the docs suggest just using it and forcing evaluation 1 time using
if my_object:
pass

Multiple 'endings' to a function - return an object, return a string, raise an exception?

This one's a structure design problem, I guess. Back for some advice.
To start: I'm writing a module. Hence the effort of making it as usable to potential developers as possible.
Inside an object (let's call it Swoosh) I have a method which, when called, may result in either success (a new object is returned -- for insight: it's an httplib.HTTPResponse) or failure (surprising, isn't it?).
I'm having trouble deciding how to handle failures. There are two main cases here:
user supplied data that was incorrect
data was okay, but user interaction will be needed () - I need to pass back to the user a string that he or she will need to use in some way.
In (1) I decided to raise ValueError() with an appropriate description.
In (2), as I need to actually pass a str back to the user.. I'm not sure about whether it would be best to just return a string and leave it to the user to check what the function returned (httplib.HTTPResponse or str) or raise a custom exception? Is passing data through raising exceptions a good idea? I don't think I've seen this done anywhere, but on the other hand - I haven't seen much.
What would you, as a developer, expect from an object/function like this?
Or perhaps you find the whole design ridiculous - let me know, I'll happily learn.

As much as I like the approach of handling both cases with specifically-typed exceptions, I'm going to offer a different approach in case it helps: callbacks.
Callbacks tend to work better if you're already using an asynchronous framework like Twisted, but that's not their only place. So you might have a method that takes a function for each outcome, like this:
def do_request(on_success, on_interaction_needed, on_failure):
"""
Submits the swoosh request, and awaits a response.
If no user interaction is needed, calls on_success with a
httplib.HTTPResponse object.
If user interaction is needed, on_interaction_needed is
called with a single string parameter.
If the request failed, a ValueError is passed to on_failure
"""
response = sumbit_request()
if response.is_fine():
on_success(response)
elif response.is_partial()
on_interaction_needed(response.message)
else:
on_failure(ValueError(response.message))
Being Python, there are a million ways to do this. You might not like passing an exception to a function, so you maybe just take a callback for the user input scenario. Also, you might pass the callbacks in to the Swoosh initialiser instead.
But there are drawbacks to this too, such as:
Carelessness may result in spaghetti code
You're allowing your caller to inject logic into your function (eg. exceptions raised in the callback will propagate out of Swoosh)
My example here is simple, your actual function might not be
As usual, careful consideration and good documentation should avoid these problems. In theory.

I think raising an exception may actually be a pretty good idea in this case. Squashing multiple signals into a single return value of a function isn't ideal in Python, due to duck typing. It's not very Pythonic; every time you need to do something like:
result = some_function(...)
if isinstance(result, TypeA):
do_something(result)
elif isinstance(result, TypeB):
do_something_else(result)
you should be thinking about whether it's really the best design (as you're doing).
In this case, if you implement a custom exception, then the code that calls your function can just treat the returned value as a HTTPResponse. Any path where the function is unable to return something its caller can treat that way is handled by throwing an exception.
Likewise, the code that catches the exception and prompts the user with the message doesn't have to worry about the exact type of the thing its getting. It just knows that it's been explicitly instructed (by the exception) to show something to the user.

If the user interaction case means the calling code has to show a prompt, get some input and them pass control back to your function, it might be ugly trying to handle that with an exception. Eg,
try:
Swoosh.method()
except UserInteraction, ex:
# do some user interaction stuff
# pass it back to Swoosh.method()?
# did Swoosh need to save some state from the last call?
except ValueError:
pass # whatever
If this user interaction is a normal part of the control flow, it might be cleaner to pass a user-interaction function into your method in the first place - then it can return a result to the Swoosh code. For example:
# in Swoosh
def method(self, userinteractor):
if more_info_needed:
more_info = userinteractor.prompt("more info")
...
ui = MyUserInteractor(self) # or other state
Swoosh.method(ui)

You can return a tuple of (httplib.HTTPResponse, str) with the str being optionally None.
Definitely raise an exception for 1).
If you don't like returning a tuple, you can also create a "response object" i.e. an instance of a new class ( lets say SomethingResponse ) that encapsulates the HTTPResponse with optional messages to the end-user( in the simplest case, just a str).