Develop Reference

peewee : how to filter data in a table based on values across rows in another table

Considering following peewee backed python code, how to filter data in a table based on values across rows in another table
e.g.
if I want to get a note in Note table having all java, lambda, generics tags in customtags
#!/usr/bin/env python3
import peewee
import datetime
db = peewee.SqliteDatabase('test.db')
class Note(peewee.Model):
id = peewee.AutoField()
name = peewee.CharField()
text = peewee.CharField()
created = peewee.DateField(default=datetime.date.today)
class Meta:
database = db
db_table = 'notes'
class CustomTag(peewee.Model):
id = peewee.AutoField()
note = peewee.ForeignKeyField(Note, backref='notes')
tag = peewee.CharField()
class Meta:
database = db
db_table = 'customtags'
indexes = ((("note_id", "tag"), True),)
Note.drop_table()
CustomTag.drop_table()
Note.create_table()
CustomTag.create_table()
note1 = Note.create(name="note1",text='Java 8 lambda with generics')
note1.save()
CustomTag.insert(note=note1, tag='java').on_conflict_ignore(True)
CustomTag.insert(note=note1, tag='lambda').on_conflict_ignore(True)
CustomTag.insert(note=note1, tag='generics').on_conflict_ignore(True)
note2 = Note.create(name="note2",text='Core Java concepts',
created=datetime.date(2018, 10, 20))
note2.save()
CustomTag.insert(note=note2, tag='java').on_conflict_ignore(True)
note3 = Note.create(name="note3",text='Java collection framework',
created=datetime.date(2018, 10, 22))
note3.save()
CustomTag.insert(note=note3, tag='java').on_conflict_ignore(True)
note4 = Note.create(name="note4",text='Java NIO packageJava nio package')
note4.save()
CustomTag.insert(note=note4, tag='java').on_conflict_ignore(True)
notes = Note.select().join(CustomTag, peewee.JOIN.LEFT_OUTER,on=(Note.id == CustomTag.note_id)).order_by(Note.name)
for note in notes:
print('{} with text {} on {}'.format(note.name, note.text, note.created))
I am really not having idea how to modify my code the get the data above mentioned, I know that issue is in following code
notes = Note.select().join(CustomTag, peewee.JOIN.LEFT_OUTER,on=(Note.id == CustomTag.note_id)).order_by(Note.name)

e.g. if I want to get a note in Note table having all java, lambda, generics tags in customtags
tags = ['foo', 'bar', 'baz']
query = (Note
.select()
.join(CustomTag)
.where(CustomTag.tag.in_(tags))
.group_by(Note)
.having(fn.COUNT(CustomTag.id) == len(tags)))

It seems execute() is missing on your insert statements. it should be corrected as of each line...
CustomTag.insert(note=note1, tag='java').on_conflict_ignore(True).execute()
I've included a bit improved version of coleifer answer here, results includes list of tags as well...
from peewee import fn
filter_tags = ['java', 'lambda', 'generics']
notes = (Note
.select(Note, fn.GROUP_CONCAT(CustomTag.tag,',').alias('tags'))
.join(CustomTag)
.where(CustomTag.tag.in_(filter_tags))
.group_by(Note)
.having(fn.COUNT(CustomTag.id) >= len(filter_tags)))
for note in notes:
print(f'name : {note.name} text: {note.text} tags: {note.tags} created : {note.created}')

Django-tables2 - can't I use [A('argument')] inside the "text" parameter?

I'm trying to make this table with a clickable field which changes the boolean for the entry to its opposite value. It works, but I want an alternative text as "False" or "True" does not look nice, and the users are mainly Norwegian.
def bool_to_norwegian(boolean):
if boolean:
return "Ja"
else:
return "Nei"
class OrderTable(tables.Table):
id = tables.LinkColumn('admin_detail', args=[A('id')])
name = tables.Column()
address = tables.Column()
order = tables.Column()
order_placed_at = tables.DateTimeColumn()
order_delivery_at = tables.DateColumn()
price = tables.Column()
comment = tables.Column()
sent = tables.LinkColumn('status_sent', args=[A('id')])
paid = tables.LinkColumn('status_paid', args=[A('id')], text=[A('paid')])
class Meta:
attrs = {'class': 'order-table'}
If you look under the "paid" entry I am testing this right now, why can't I access the data with the same accessor as I do in the args? If I change the args to args=[A('paid')] and look at the link, it does indeed have the correct data on it. The model names are the same as the ones in this table, and "paid" and "sent" are BooleanFields.
This is kind of what I ultimately want:
text=bool_to_norwegian([A('paid')])
Here is what I send to the table:
orders = Order.objects.order_by("-order_delivery_at")
orders = orders.values()
table = OrderTable(orders)
RequestConfig(request).configure(table)

The text argument expects a callable that accepts a record, and returns a text value. You are passing it a list (which it will just ignore), and your function is expecting a boolean instead of a record. There is also no need for using accessors here.
Something like this should work:
def bool_to_norwegian(record):
if record.paid:
return "Ja"
else:
return "Nei"
Then in your column:
paid = tables.LinkColumn('status_paid', text=bool_to_norwegian)
(Note, it is not clear from your question where the data is coming from - is paid a boolean? You may need to adjust this to fit).
As an aside, the way you are passing args to your columns is weird (it seems the documentation also recommends this, but I don't understand why - it's very confusing). A more standard approach would be:
id = tables.LinkColumn('admin_detail', A('id'))
or using named arguments:
id = tables.LinkColumn('admin_detail', accessor=A('id'))

Django wildcard query

I have the following logic in my application:
provider = request.POST.get('provider', '*')
order_items = OrderItem.objects.filter(provider=provider)
Is there a wildcard I can use in django, such that if no provider is found in the POST request, it will return all objects for it?
In other words, is there a way to accomplish this?
if request.POST.get('provider'):
order_items = OrderItem.objects.filter(provider=provider)
else:
order_items = OrderItem.objects.all()

There is no direct wildcard parameter, so what you have is perfectly acceptable. Code readability also counts, so even though you might end up with more code, it may be more maintainable.
You could chain the queryset like this:
provider = request.POST.get('provider')
order_items = OrderItem.objects.all()
if provider is not None:
order_items = order_items.filter(provider=provider)
Or you could set up an empty dictionary of kwargs for a filter() call like this, but it's less readable in my opinion:
provider = request.POST.get('provider')
kwargs = {}
if provider is not None:
kwargs['provider'] = provider
order_items = OrderItem.objects.filter(**kwargs)
This could be turned into a function call like this:
def all_or_filter_args(request, item):
"""Return dictionary of arguments for filter() if item is specified in request."""
value = request.get(item)
if value is None:
return {}
return { item : value }
and then the following one-liner used for queries:
order_items = OrderItem.objects.filter(**all_or_filter_args(request, 'provider'))
But again, I don't find this as readable as the example you've given.
Another approach is to write a custom manager which allows you to use your own functions for filtering. With a custom manager you could implement code to allow queries like this, where you provide a function for all_or_filtered to apply the correct filtering:
order_items = OrderItem.objects.all_or_filtered('provider', request.POST.get('provider'))

How about this...
parameter = request.POST.get('provider', '%%')
order_items = OrderItem.objects.raw("SELECT * FROM %s WHERE provider LIKE '%s'" % (OrderItem._meta.db_table, parameter))

How would you inherit from and override the django model classes to create a listOfStringsField?

I want to create a new type of field for django models that is basically a ListOfStrings. So in your model code you would have the following:
models.py:
from django.db import models
class ListOfStringsField(???):
???
class myDjangoModelClass():
myName = models.CharField(max_length=64)
myFriends = ListOfStringsField() #
other.py:
myclass = myDjangoModelClass()
myclass.myName = "bob"
myclass.myFriends = ["me", "myself", "and I"]
myclass.save()
id = myclass.id
loadedmyclass = myDjangoModelClass.objects.filter(id__exact=id)
myFriendsList = loadedclass.myFriends
# myFriendsList is a list and should equal ["me", "myself", "and I"]
How would you go about writing this field type, with the following stipulations?
We don't want to do create a field which just crams all the strings together and separates them with a token in one field like this. It is a good solution in some cases, but we want to keep the string data normalized so tools other than django can query the data.
The field should automatically create any secondary tables needed to store the string data.
The secondary table should ideally have only one copy of each unique string. This is optional, but would be nice to have.
Looking in the Django code it looks like I would want to do something similar to what ForeignKey is doing, but the documentation is sparse.
This leads to the following questions:
Can this be done?
Has it been done (and if so where)?
Is there any documentation on Django about how to extend and override their model classes, specifically their relationship classes? I have not seen a lot of documentation on that aspect of their code, but there is this.
This is comes from this question.

There's some very good documentation on creating custom fields here.
However, I think you're overthinking this. It sounds like you actually just want a standard foreign key, but with the additional ability to retrieve all the elements as a single list. So the easiest thing would be to just use a ForeignKey, and define a get_myfield_as_list method on the model:
class Friends(model.Model):
name = models.CharField(max_length=100)
my_items = models.ForeignKey(MyModel)
class MyModel(models.Model):
...
def get_my_friends_as_list(self):
return ', '.join(self.friends_set.values_list('name', flat=True))
Now calling get_my_friends_as_list() on an instance of MyModel will return you a list of strings, as required.

What you have described sounds to me really similar to the tags.
So, why not using django tagging?
It works like a charm, you can install it independently from your application and its API is quite easy to use.

I also think you're going about this the wrong way. Trying to make a Django field create an ancillary database table is almost certainly the wrong approach. It would be very difficult to do, and would likely confuse third party developers if you are trying to make your solution generally useful.
If you're trying to store a denormalized blob of data in a single column, I'd take an approach similar to the one you linked to, serializing the Python data structure and storing it in a TextField. If you want tools other than Django to be able to operate on the data then you can serialize to JSON (or some other format that has wide language support):
from django.db import models
from django.utils import simplejson
class JSONDataField(models.TextField):
__metaclass__ = models.SubfieldBase
def to_python(self, value):
if value is None:
return None
if not isinstance(value, basestring):
return value
return simplejson.loads(value)
def get_db_prep_save(self, value):
if value is None:
return None
return simplejson.dumps(value)
If you just want a django Manager-like descriptor that lets you operate on a list of strings associated with a model then you can manually create a join table and use a descriptor to manage the relationship. It's not exactly what you need, but this code should get you started.

Thanks for all those that answered. Even if I didn't use your answer directly the examples and links got me going in the right direction.
I am not sure if this is production ready, but it appears to be working in all my tests so far.
class ListValueDescriptor(object):
def __init__(self, lvd_parent, lvd_model_name, lvd_value_type, lvd_unique, **kwargs):
"""
This descriptor object acts like a django field, but it will accept
a list of values, instead a single value.
For example:
# define our model
class Person(models.Model):
name = models.CharField(max_length=120)
friends = ListValueDescriptor("Person", "Friend", "CharField", True, max_length=120)
# Later in the code we can do this
p = Person("John")
p.save() # we have to have an id
p.friends = ["Jerry", "Jimmy", "Jamail"]
...
p = Person.objects.get(name="John")
friends = p.friends
# and now friends is a list.
lvd_parent - The name of our parent class
lvd_model_name - The name of our new model
lvd_value_type - The value type of the value in our new model
This has to be the name of one of the valid django
model field types such as 'CharField', 'FloatField',
or a valid custom field name.
lvd_unique - Set this to true if you want the values in the list to
be unique in the table they are stored in. For
example if you are storing a list of strings and
the strings are always "foo", "bar", and "baz", your
data table would only have those three strings listed in
it in the database.
kwargs - These are passed to the value field.
"""
self.related_set_name = lvd_model_name.lower() + "_set"
self.model_name = lvd_model_name
self.parent = lvd_parent
self.unique = lvd_unique
# only set this to true if they have not already set it.
# this helps speed up the searchs when unique is true.
kwargs['db_index'] = kwargs.get('db_index', True)
filter = ["lvd_parent", "lvd_model_name", "lvd_value_type", "lvd_unique"]
evalStr = """class %s (models.Model):\n""" % (self.model_name)
evalStr += """ value = models.%s(""" % (lvd_value_type)
evalStr += self._params_from_kwargs(filter, **kwargs)
evalStr += ")\n"
if self.unique:
evalStr += """ parent = models.ManyToManyField('%s')\n""" % (self.parent)
else:
evalStr += """ parent = models.ForeignKey('%s')\n""" % (self.parent)
evalStr += "\n"
evalStr += """self.innerClass = %s\n""" % (self.model_name)
print evalStr
exec (evalStr) # build the inner class
def __get__(self, instance, owner):
value_set = instance.__getattribute__(self.related_set_name)
l = []
for x in value_set.all():
l.append(x.value)
return l
def __set__(self, instance, values):
value_set = instance.__getattribute__(self.related_set_name)
for x in values:
value_set.add(self._get_or_create_value(x))
def __delete__(self, instance):
pass # I should probably try and do something here.
def _get_or_create_value(self, x):
if self.unique:
# Try and find an existing value
try:
return self.innerClass.objects.get(value=x)
except django.core.exceptions.ObjectDoesNotExist:
pass
v = self.innerClass(value=x)
v.save() # we have to save to create the id.
return v
def _params_from_kwargs(self, filter, **kwargs):
"""Given a dictionary of arguments, build a string which
represents it as a parameter list, and filter out any
keywords in filter."""
params = ""
for key in kwargs:
if key not in filter:
value = kwargs[key]
params += "%s=%s, " % (key, value.__repr__())
return params[:-2] # chop off the last ', '
class Person(models.Model):
name = models.CharField(max_length=120)
friends = ListValueDescriptor("Person", "Friend", "CharField", True, max_length=120)
Ultimately I think this would still be better if it were pushed deeper into the django code and worked more like the ManyToManyField or the ForeignKey.

I think what you want is a custom model field.

Python SQLAlchemy/Elixer Question

I am trying to define a SQLAlchemy/Elixer model that can describe the following relationship. I have an SSP table, which has multiple Foreign Keys to the POC table. I've defined the ManyToOne relationships correctly within the SSP object (allowing me to SSP.get(1).action.first_name correctly). What I would also like to add is the other side of this relationship, where I can perform something like POC.get(1).csa and return a list of SSP objects in which this POC is defined as the idPOCCSA.
I know this would be best for a polymorphic association but I really can not change the DB schema at all (creating a new poc2ssp table with a column for type of association).
class POC(Entity):
using_options(tablename = 'poc', autoload = True)
# These two line visually display my "issue":
# csa = OneToMany('SSP')
# action = OneToMany('SSP')
class SSP(Entity):
'''
Many to One Relationships:
- csa: ssp.idPOCCSA = poc.id
- action: ssp.idPOCAction = poc.id
- super: ssp.idSuper = poc.id
'''
using_options(tablename = 'spp', autoload = True)
csa = ManyToOne('POC', colname = 'idPOCCSA')
action = ManyToOne('POC', colname = 'idPOCAction')
super = ManyToOne('POC', colname = 'idPOCSuper')
Any ideas to accomplish this? The Elixer FAQ has a good example utilizing the primaryjoin and foreign_keys parameters but I can't find them in the documentation. I was kind of hoping OneToMany() just supported a colname parameter like ManyToOne() does. Something a bit less verbose.

Try the following:
class POC(Entity):
# ...
#declare the one-to-many relationships
csas = OneToMany('SSP')
actions = OneToMany('SSP')
# ...
class SSP(Entity):
# ...
#Tell Elixir how to disambiguate POC/SSP relationships by specifying
#the inverse explicitly.
csa = ManyToOne('POC', colname = 'idPOCCSA', inverse='csas')
action = ManyToOne('POC', colname = 'idPOCAction', inverse='actions')
# ...