Develop Reference

Annotate causes ProgrammingError: must appear in the GROUP BY clause or be used in an aggregate function

I have something like this, that represents my model:
class Model(DjangoModel):
created_at = DateTimeField(null=True)
updated_at = DateTimeField(null=True)
...
class Vat(Model):
creator = models.ForeignKey(User, default=None)
name = models.CharField(max_length=32, unique=True)
disabled = models.BooleanField(default=False)
...
class Customer(SomeInheritedModel):
name = models.CharField(max_length=50)
...
class Invoice(Model):
customer = models.ForeignKey(Customer, related_name='invoices')
creditted_by = models.OnetoOneField(User, related_name='creditee')
...
class Account(Model):
customer = models.ForeignKey(Customer, related_name='accounts')
...
class Product(Model):
name = models.CharField(max_length=50)
...
class License(Model):
vat = models.ForeignKey('Vat', null=True) # Null for imported
imported = models.BooleanField(default=False)
account = models.ForeignKey(Account)
product = models.ForeignKey(Product)
maintenance = models.ManyToManyField('Maintenance', related_name="maintenances")
class Maintenance(Model):
invoice = models.ForeignField(Invoice)
start_date = models.DateTimeField(null=True)
expiration_date = models.DateTimeField(null=True)
Then I am trying to generate a report on expired licenses, making sure to get only the latest Maintenance object sold to this license.
# maintenances__invoice__* filters credited invoices, which we want to exclude
report.licenses = License.objects.filter(
Q(imported=True, expiry_date__range=(report.start_date, report.end_date)),
maintenances__invoice__creditted_by__isnull=True,
maintenances__invoice__creditee__isnull=True
)\
.distinct()\
.annotate(max_exp_date=Max('maintenances__end'))\
.filter(max_exp_date__range=(report.start_date, report.end_date))
However I get the following error:
ProgrammingError at /report-menu/create/
column "v3_vat.created_at" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: ...ummary", "v3_agreement"."file_id", "v3_vat"."id", "v3_vat"."...
^
Request Method: POST
Request URL: http://localhost:8000/report-menu/create/?report_type=5
Django Version: 1.10.6
Postgree Version: 9.5
Exception Type: ProgrammingError
Exception Value:
column "v3_vat.created_at" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: ...ummary", "v3_agreement"."file_id", "v3_vat"."id", "v3_vat"."...
^
The weird thing: I have two test and a production environment, but this error only happens in production. I have no problem whatsoever with the other two test environments. I know it has something to do with the database, because I have made a backup, restored it in one of the test environments and the same error popped. But what if they are supposed to be identical?
And why field v3_vat.created_at? I am not even mentioning this in my annotate function.

TLDR; use only
Whenever you call only() it replaces the set of fields to load
immediately. The method’s name is mnemonic: only those fields are
loaded immediately; the remainder are deferred. Thus, successive calls
to only() result in only the final fields being considered:
Django tries to be database agnostic but it isn't easy, because mysql and sqlite in particular have a lot of non standard behavior. One of them centers around aggregation. Whereas postgresql is a lot closer to standards compliance. This is best explained in the mysql 5.7 manual
SQL92 and earlier does not permit queries for which the select list,
HAVING condition, or ORDER BY list refer to nonaggregated columns that
are neither named in the GROUP BY clause nor are functionally
dependent on (uniquely determined by) GROUP BY columns.
The problem here is that your query selects all the columns (as is the default with django) but some of them do not have meaningfull values when you use an aggregation. For example you have 100 students in 10 classrooms and each of them have a grade. You select the count of students in each class with an aggregate. You want to show the grade as well. But whose grade do you show?
Thus use the only function limit the columns that are selected by django.

Create ORM object from dict and add to session

Let's say I have a User model with attributes id, name, email and a relationship languages.
Is it possible to create a User instance from existing data that behaves like I would have queried it with dbsession.query(User).get(42)?
What I mean in particular is that I want that an access to user.languages creates a subquery and populates the attribute.
Here a code example:
I have a class User:
class User(Base):
id = Column(Integer, primary_key=True)
name = Column(String(64))
email = Column(String(64))
languages = relationship('Language', secondary='user_languages')
I already have a lot of users stored in my DB.
And I know that I have, for example, this user in my DB:
user_dict = {
'id': 23,
'name': 'foo',
'email': 'foo#bar',
}
So I have all the attributes but the relations.
Now I want to make a sqlalchemy User instance
and kind of register it in sqlalchemy's system
so I can get the languages if needed.
user = User(**user_dict)
# Now I can access the id, name email attributes
assert user.id == 23
# but since sqlalchemy thinks it's a new model it doesn't
# lazy load any relationships
assert len(user.languages) == 0
# I want here that the languages for the user with id 23 appear
# So I want that `user` is the same as when I would have done
user_from_db = DBSession.query(User).get(23)
assert user == user_from_db
The use-case is that I have a big model with lots of complex
relationships but 90% of the time I don't need the data from those.
So I only want to cache the direct attributes plus what else I need
and then load those from the cache like above and be able to
use the sqlalchemy model like I would have queried it from the db.

From the sqlalchemy mailing list:
# to make it look like it was freshly loaded from the db
from sqlalchemy.orm.session import make_transient_to_detached
make_transient_to_detached(user)
# merge instance in session without emitting sql
user = DBSession.merge(user, load=False)
This answer was extracted from the question

Specifying Table to Connect to with Python Peewee

I have read the Peewee MySQL API documentation and this question; however, what I do not understand is how to connect to a specified table in a db using Peewee. Essentially all I'm trying to do is connect to a a table called Persons in a db called as_schema, set up some sort of basic object-relational mapping, and print out all entries' aNum column values.
My table Persons that I'm trying to read from has the following columns:
varchar called aNum
bool called access
bool called ajar
bool called ebr
date called weekof
My code consists of the following:
import peewee
from peewee import *
db = MySQLDatabase('as_schema', user='root',passwd='')#this connection path works perfectly, tried it using the standard MySQLdb.connect
class Person(Model):
class Meta:
database = db
class User(Person):
aNum = CharField()
Person.create_table()
person = User(aNum = 'a549189')
person.save();
for person in Person:
print person.aNum
The error I'm getting is:

class Person(Model):
class Meta:
database = db
db_table = 'Persons' # Add this.
In the docs, you can find a list of supported meta options:
http://docs.peewee-orm.com/en/latest/peewee/models.html#model-options-and-table-metadata

Update many-to-many relationships using Flask, SQLAlchemy and WTForms?

I want to create a form for many-to-many relations using Flask, SQLAlchemy and WTForms that represents these models:
personaddress = db.Table('personaddress',
db.Column('person', db.Integer, db.ForeignKey('person.id')),
db.Column('address', db.Integer, db.ForeignKey('address.id'))
)
class Person(db.Model):
__tablename__ = "person"
id = db.Column(Integer, primary_key=True)
name = db.Column(String, nullable=False)
addresses = db.relationship('Address', secondary=personaddress, backref=db.backref('person', lazy='dynamic'))
class Address(db.Model):
__tablename__ = "address"
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String, nullable=False)
Requirements
Now I want to create a single page that contains forms to achieve the following:
add/edit/delete a person
add/edit/delete a address
add/edit/delete a relation between a person and an address
Important requirement: Using QuerySelectField, I can choose existing addresses for a person. But I want to add new addresses in the same form.
I've played around with model_form for the main models and subforms using FormField for the junction table but I just can't figure out how to update everything including the foreign key relations. The page should have a single submit button for all forms and subforms displayed.
Questions
How are the above requirements typically implemented in Flask?
Is this many-to-many scenario something that Django can handle more easily through its admin interface?

I have also encountered something similar earlier. I tried to solve it by using model_form, but it doesn't quite solve the problem of adding new entries dynamically, and I was having a hard time using it when dealing with relations.
Using the QuerySelectField in WTForms will only help you populating eg. an < select > with id, value pairs corresponding to the existing addresses. But it still renders to a regular html form in the template.
By using some sort of multiselect with the possibility to dynamically add new options in the frontend you can send additional addresses in the same form. The the endpoint will take care of creating new Addresses if they don't exist in the db.
The WTForm form would be:
from app import db
class PersonAddressForm(Form):
id = HiddenField('id')
name = StringField('Name')
addresses = QuerySelectField('Addresses',
query_factory=lambda: db.session.query(Address),
get_pk=lambda a: a.id, get_label=lambda a: a.name)
# Custom validate
def validate(self):
# ... custom validation
return True
And the route something like:
# ... this will be used to create and update a user
#route('create/<userid>', methods=["GET"])
def get_user_form(userid):
# ... Get the Person
user = Person()
if userid:
# ... if userid supplied, use existing Person object
user = Person.query.get(userid)
# ... Populate the form
person_form = PersonAddressForm(obj=user)
# ... return form
return render_template('somepage.html', form=person_form)
#route('create/<userid>', methods=["POST"])
def post_person_form(userid):
person_form = PersonAddressForm(request.form)
if person_form.validate():
# ... Get db object
person = db.session.query(Person).get(form.id)
# ... Add changes to the object from the form
person_form.populate_obj(obj=person_address)
# ... Get addresses
addresses = form.addresses.raw_data
# ... loop over and add to person
for address in addresses:
# Add or create an address
actual_address = db.session.query(Address).get(address.id)
# ... check if address is existing
if not actual_address:
# ... if address not existing, create new one
actual_address = Address(address.name)
db.session.add(actual_address)
# ... Append new or created address to person
person.addresses.append(actual_address)
# ... save changes to the db
db.session.commit()
# ... Update/Create complete
return redirect(url_for('get_users'))
else:
# ... form not valid, notify user
# ...
This will handle edit/create user and create Address. As well as create the relation between. To make it also support delete Address, change
person.addresses.append(actual_address)
to
person.addresses = list_of_actual_addresses
and change this in the person model (cascade='delete-orphan')
addresses = db.relationship('Address', secondary=personaddress, cascade='delete-orphan' backref=db.backref('person', lazy='dynamic'))
This will make the form update the entire address relation each time and the cascade will delete orphaned addresses. So the entire addresses list for a person would be updated each time the form is submitted.
When dealing with WTForms in templates i highly recommend using macros if you don't already. You would have to rewrite it to some degree, but check this out.
Hope this helps

How to query datastore when using ReferenceProperty?

I am trying to understand the 1-to-many relationships in datastore; but I fail to understand how query and update the record of a user when the model includes ReferenceProperty. Say I have this model:
class User(db.Model):
userEmail = db.StringProperty()
userScore = db.IntegerProperty(default=0)
class Comment(db.Model):
user = db.ReferenceProperty(User, collection_name="comments")
comment = db.StringProperty()
class Venue(db.Model):
user = db.ReferenceProperty(User, collection_name="venues")
venue = db.StringProperty()
If I understand correctly, the same user, uniquely identified by userEmail can have many comments and may be associated with many venues (restaurants etc.)
Now, let's say the user az#example.com is already in the database and he submits a new entry.
Based on this answer I do something like:
q = User.all()
q.filter("userEmail =", az#example.com)
results = q.fetch(1)
newEntry = results[0]
But I am not clear what this does! What I want to do is to update comment and venue fields which are under class Comment and class Venue.
Can you help me understand how this works? Thanks.

The snippet you posted is doing this (see comments):
q = User.all() # prepare User table for querying
q.filter("userEmail =", "az#example.com") # apply filter, email lookup
- this is a simple where clause
results = q.fetch(1) # execute the query, apply limit 1
the_user = results[0] # the results is a list of objects, grab the first one
After this code the_user will be an object that corresponds to the user record with email "az#example.com". Seing you've set up your reference properties, you can access its comments and venues with the_user.comments and the_user.venues. Some venue of these can be modified, say like this:
some_venue = the_user.venues[0] # the first from the list
some_venue.venue = 'At DC. square'
db.put(some_venue) # the entry will be updated
I suggest that you make a general sweep of the gae documentation that has very good examples, you will find it very helpful:
http://code.google.com/appengine/docs/python/overview.html
** UPDATE **: For adding new venue to user, simply create new venue and assign the queried user object as the venue's user attribute:
new_venue = Venue(venue='Jeferson memorial', user=the_user) # careful with the quoting
db.put(new_venue)

To get all Comments for a given user, filter the user property using the key of the user:
comments = Comment.all().filter("user =", user.key()).fetch(50)
So you could first lookup the user by the email, and then search comments or venues using its key.