Develop Reference

How to avoid brackets in SQL around Django custom database function call?

A short intoduction to the problem...
PostgreSQL has very neat array fields (int array, string array) and functions for them like UNNEST and ANY.
These fields are supported by Django (I am using djorm_pgarray for that), but functions are not natively supported.
One could use .extra(), but Django 1.8 introduced a new concept of database functions.
Let me provide a most primitive example of what I am basicly doing with all these. A Dealer has a list of makes that it supports. A Vehicle has a make and is linked to a dealer. But it happens that Vehicle's make does not match Dealer's make list, that is inevitable.
MAKE_CHOICES = [('honda', 'Honda'), ...]
class Dealer(models.Model):
make_list = TextArrayField(choices=MAKE_CHOICES)
class Vehicle(models.Model):
dealer = models.ForeignKey(Dealer, null=True, blank=True)
make = models.CharField(max_length=255, choices=MAKE_CHOICES, blank=True)
Having a database of dealers and makes, I want to count all vehicles for which the vehicle's make and its dealer's make list do match. That's how I do it avoiding .extra().
from django.db.models import functions
class SelectUnnest(functions.Func):
function = 'SELECT UNNEST'
...
Vehicle.objects.filter(
make__in=SelectUnnest('dealer__make_list')
).count()
Resulting SQL:
SELECT COUNT(*) AS "__count" FROM "myapp_vehicle"
INNER JOIN "myapp_dealer"
ON ( "myapp_vehicle"."dealer_id" = "myapp_dealer"."id" )
WHERE "myapp_vehicle"."make"
IN (SELECT UNNEST("myapp_dealer"."make_list"))
And it works, and much faster than a traditional M2M approach we could use in Django. BUT, for this task, UNNEST is not a very good solution: ANY is much faster. Let's try it.
class Any(functions.Func):
function = 'ANY'
...
Vehicle.objects.filter(
make=Any('dealer__make_list')
).count()
It generates the following SQL:
SELECT COUNT(*) AS "__count" FROM "myapp_vehicle"
INNER JOIN "myapp_dealer"
ON ( "myapp_vehicle"."dealer_id" = "myapp_dealer"."id" )
WHERE "myapp_vehicle"."make" =
(ANY("myapp_dealer"."make_list"))
And it fails, because braces around ANY are bogus. If you remove them, it runs in the psql console with no problems, and fast.
So my question.
Is there any way to remove these braces? I could not find anything about that in Django documentation.
If not, - maybe there are other ways to rephrase this query?
P. S. I think that an extensive library of database functions for different backends would be very helpful for database-heavy Django apps.
Of course, most of these will not be portable. But you typically do not often migrate such a project from one database backend to another. In our example, using array fields and PostGIS we are stuck to PostgreSQL and do not intend to move.
Is anybody developing such a thing?
P. P. S. One might say that, in this case, we should be using a separate table for makes and intarray instead of string array, that is correct and will be done, but nature of problem does not change.
UPDATE.
TextArrayField is defined at djorm_pgarray. At the linked source file, you can see how it works.
The value is list of text strings. In Python, it is represented as a list. Example: ['honda', 'mazda', 'anything else'].
Here is what is said about it in the database.
=# select id, make from appname_tablename limit 3;
id | make
---+----------------------
58 | {vw}
76 | {lexus,scion,toyota}
39 | {chevrolet}
And underlying PostgreSQL field type is text[].

I've managed to get (more or less) what you need using following:
from django.db.models.lookups import BuiltinLookup
from django.db.models.fields import Field
class Any(BuiltinLookup):
lookup_name = 'any'
def get_rhs_op(self, connection, rhs):
return " = ANY(%s)" % (rhs,)
Field.register_lookup(Any)
and query:
Vehicle.objects.filter(make__any=F('dealer__make_list')).count()
as result:
SELECT COUNT(*) AS "__count" FROM "zz_vehicle"
INNER JOIN "zz_dealer" ON ("zz_vehicle"."dealer_id" = "zz_dealer"."id")
WHERE "zz_vehicle"."make" = ANY(("zz_dealer"."make_list"))
btw. instead djorm_pgarray and TextArrayField you can use native django:
make_list = ArrayField(models.CharField(max_length=200), blank=True)
(to simplify your dependencies)

How to output logarithm of some calculated value using Django, MySQL

I want to create the following query in Django:
select field1, count(field1), log(count(field1)) from object_table
where parent_id = 12345
group by field1;
I've implemented field1, count(field1) and group by field1 by following:
from django.db.models import Count
Object.objects.filter(
parent = 12345
).values_list(
'field1'
).annotate(
count=Count('field1')
)
However if I add something like this
.extra(
select={'_log':'log(count)'}
)
it doesn't affect my results. Could you give me a clue what am I doing wrong? How to implement log(count(field)) within Django?
PS, I'm using Django 1.9.
Thanks in advance!

Note that some databases don't natively support logarithm function (e.g. sqlite). This is probably an operation that should be done in your Python code instead of the database query.
import math
for obj in object_list:
# use math.log() for natural logarithm
obj._log = math.log10(obj.count)
If you are certain you can rely on a database function and you want to use the database to perform the computation, you can use raw queries. For example, postgres has the log function implemented:
query = """\
select count(field1), log(count(field1)) as logvalue
from myapp_mymodel
group by field1"""
queryset = MyModel.objects.raw(query)
for obj in queryset:
print(obj.logvalue)

How do I make a db relationship with secondary backref work in both directions in Flask SQLAlchemy?

I created a User model and Kid model. In the kid model I have:
parents = db.relationship('User', secondary=kids_users,
backref=db.backref('kids', lazy='dynamic'))
The secondary table definition looks like this:
kids_users = db.Table('kids_users',
db.Column('kid_id', db.Integer(), db.ForeignKey('kid.id')),
db.Column('user_id',db.Integer(), db.ForeignKey('user.id')))
And my User model does not contain anything related to this since this code does everything.
Here's the situation: when I query some_kid.parents it gives me back a nice array:
[<User u'someuser#yahoo.com'>]
But when I query the other way around some_user.kids it gives me back some sql query:
SELECT kid.id AS kid_id, kid.first_name AS kid_first_name, kid.middle_name AS kid_middle_name, kid.last_name AS kid_last_name, kid.dob AS kid_dob, kid.profile_pic_small AS kid_profile_pic_small, kid.profile_pic_smallish AS kid_profile_pic_smallish, kid.profile_pic_med AS kid_profile_pic_med, kid.profile_pic_large AS kid_profile_pic_large, kid.gender AS kid_gender, kid.current_group AS kid_current_group, kid.status AS kid_status, kid.status_time AS kid_status_time, kid.potty_trained AS kid_potty_trained, kid.pending AS kid_pending, kid.scholarship AS kid_scholarship, kid.govt AS kid_govt, kid.extra_credit AS kid_extra_credit, kid.hold AS kid_hold, kid.school_id AS kid_school_id, kid.between_schedule_id AS kid_between_schedule_id, kid.one_for_all_schedule_id AS kid_one_for_all_schedule_id, kid.hourly_schedule_id AS kid_hourly_schedule_id, kid.calendar_schedule_id AS kid_calendar_schedule_id, kid.code AS kid_code FROM kid, kids_users
I am trying to populate forms with SomeForm(request.data, obj=some_user), and it never populates the kids select field, because it doesn't return an array.
How do I get it to work nicely in both directions?

You've used lazy='dynamic', which makes the relationship return another query rather than a collection. To actually get the collection, call some_kid.parents.all(). The dynamic query has the methods append(instance) and remove(instance) to manage the collection. Typically, lazy='dynamic' is used for very large collections where you will mostly filter it further before loading any related instances.
Usually, you just need lazy='select' (the default) which instructs SQLAlchemy to load the collection with an automatic select on access. Then it will behave as a collection like you expect.

Two way table and django ORM

Consider two django models 'User' and 'BoardGame', the latter has a ManyToMany field 'vote' defined with a custom through table:
class Vote(models.Model):
user = models.ForeignKey(User)
boardgame = models.ForeignKey(BoardGame)
vote = models.IntegerField()
I need to print a two way table having users names on the top, boardgames names on the left column and votes in the middle.
Is there a way to obtain this using django? (Remember that a user might not have voted every single boardgame.)
UPDATE: MORE DETAILS
1) Clearly this can be work out using some lines of python (which probably would result in many queries to the database), but I'm more interested in discovering if there is something directly implemented in django that could do the work. After all a ManyToMany field is nothing but a two way table (in this case with some data associated).
2) A possible 'solution' would be a FULL OUTER JOIN using a raw query, but, again, I am looking for something built-in inside django.
3) More specifically I'm using Class Based View and I was wondering if there exists an appropriate query to associate to queryset parameter of ListView.

Assuming:
class User(models.Model):
...
class BoardGame(models.Model):
users = models.ManyToManyField(User, through='Vote')
...
class Vote(models.Model):
user = models.ForeignKey(User)
boardgame = models.ForeignKey(BoardGame)
vote = models.IntegerField()
would work like this:
from django.db import connections, reset_queries
reset_queries()
users = User.objects.all().prefetch_related('vote_set')
table = []
table.append([''] + list(users))
for board_game in BoardGame.objects.all().prefetch_related('vote_set'):
row = [board_game]
for user in users:
for vote in user.vote_set.all():
if vote in board_game.vote_set.all():
row.append(vote)
break
else:
row.append('')
table.append(row)
len(connection.queries) # Returns always 4
This is not the solution you wanted, but it shows a way to get the table from the database with only 4 queries no matter how many objects you have.

I don't think there is anything in the Django core or generic Class Based Views that will render tables for you, but try django-tables2

Django: How to filter Users that belong to a specific group

I'm looking to narrow a query set for a form field that has a foreignkey to the User's table down to the group that a user belongs to.
The groups have been previously associated by me. The model might have something like the following:
myuser = models.ForeignKey(User)
And my ModelForm is very bare bones:
class MyForm(ModelForm):
class Meta:
model = MyModel
So when I instantiate the form I do something like this in my views.py:
form = MyForm()
Now my question is, how can I take the myuser field, and filter it so only users of group 'foo' show up.. something like:
form.fields["myuser"].queryset = ???
The query in SQL looks like this:
mysql> SELECT * from auth_user INNER JOIN auth_user_groups ON auth_user.id = auth_user_groups.user_id INNER JOIN auth_group ON auth_group.id = auth_user_groups.group_id WHERE auth_group.name = 'client';
I'd like to avoid using raw SQL though. Is it possible to do so?

You'll want to use Django's convention for joining across relationships to join to the group table in your query set.
Firstly, I recommend giving your relationship a related_name. This makes the code more readable than what Django generates by default.
class Group(models.Model):
myuser = models.ForeignKey(User, related_name='groups')
If you want only a single group, you can join across that relationship and compare the name field using either of these methods:
form.fields['myuser'].queryset = User.objects.filter(
groups__name='foo')
form.fields['myuser'].queryset = User.objects.filter(
groups__name__in=['foo'])
If you want to qualify multiple groups, use the in clause:
form.fields['myuser'].queryset = User.objects.filter(
groups__name__in=['foo', 'bar'])
If you want to quickly see the generated SQL, you can do this:
qs = User.objects.filter(groups__name='foo')
print qs.query

This is a really old question, but for those googling the answer to this (like I did), please know that the accepted answer is no longer 100% correct. A user can belong to multiple groups, so to correctly check if a user is in some group, you should do:
qs = User.objects.filter(groups__name__in=['foo'])
Of course, if you want to check for multiple groups, you can add those to the list:
qs = User.objects.filter(groups__name__in=['foo', 'bar'])