Develop Reference

tag search on postgres via sqlalchemy

I'm trying to query a tags column (currently modelled as a character varying array). I would like to find any rows in which the tags column contains the query string as a left anchored substring, and would like to do so using sqlalchemy. My research has led me to learn about different ways of optimizing text search, but several lookup message still require usage of 'unnest'. I am open to changing the column from character varying array to something else (or having a separate, related table for tags), but am also curious on using unnest in sqlalchemy.
eg.
SELECT * FROM batches, UNNEST(tags) t WHERE t like 'poe%';
works and will find a row where tags column is ['math', 'poetry'].
I haven't found the right way to use unnest in sqlalchemy's python orm. Any help appreciated.

Creating Dynamic Select Query in SqlAlchemy

I have researched this topic and have decided just to ask here since I can't seem to find anything. I'll explain below:
Context: Flask Application with a form the client fills out and posts to the server. The form inputs are used to create a query and return data.
I am using SQLalchemy currently to construct the query from scratch. At this point, I have successfully connected to my existing Redshift database and can query properly but I cannot figure out how to dynamically construct a simple Select x, y, z statement based on the user's form inputs.
The main problem being that Query() can't take in a python list of columns. It seems you must specify each column like table.c.column1 which doesn't work well with a dynamic query since I don't know what columns I want until the user submits the form.
My 2 ideas so far:
Loop through all column names and use Query.add_columns(table.c['colname'])
Use select([col1, col2, ...]) instead of Query()
Use load_columns() to load only specific columns in a table to query. Unfortunately seems to only work with model objects and not reflected tables unless I am mistaken
Both of these seem backwards to me as they do not really accomplish my goal effectively.

SQLAlchemy is quite flexible, so both 1 and 2 get the job done. If you've no need for ORM functionality, then perhaps #2 is more natural. If the user were to pass a list of column names such as
columns = request.args.getlist('columns')
you could then create your select() quite easily with a bunch of column() constructs:
stmt = select([column(c) for c in columns]).\
select_from(some_table)
or if you have the table at hand, like you hint in the question:
stmt = select([table.c[c] for c in columns])
and then all that is left is to execute your statement:
results = db.session.execute(stmt).fetchall()

Getting certain elements from DB in Django

I have a simple Python/Django class:
class myModel(models.Model):
date = models.DateTimeField()
value = models.IntegerField()
and I want to get two elements from my database. First is the newest element and the second is newest positive element. So I can do this like this:
myModel.objects.all().order_by('-date')[:1][0]
myModel.objects.filter(value__gte = 0).order_by('-date')[:1][0]
Note those [:1][0] at the end - this is because I want to get maximum use of database sql engine. The thing is that I still need two queries and I want to combine it into a single one (something like [:2] at the end which will produce the result I want). I know about Django's Q, but can't figure out how to use it in this context. Maybe some raw sql? I'm waiting for ideas. :)

This looks like premature optimisation to me. Is two queries instead of one really so bad? At the moment, anyone who knows the Django ORM can understand your two queries. After you've replaced it with some funky raw SQL, that might not be the case.
You should use [0] instead of [:1][0]. Django knows how to slice querysets efficiently -- both queries will result in the exact same SQL.

This doesn't fully answer your question, but you can get rid of those [:1][0] and order_by by using latest QuerySet method, it will return the latest element in the QuerySet using the argument provided as a date field.

Searching across multiple tables (best practices)

I have property management application consisting of tables:
tenants
landlords
units
properties
vendors-contacts
Basically I want one search field to search them all rather than having to select which category I am searching. Would this be an acceptable solution (technology wise?)
Will searching across 5 tables be OK in the long run and not bog down the server? What's the best way of accomplishing this?
Using PostgreSQL

Why not create a view which is a union of the tables which aggregates the columns you want to search on into one, and then search on that aggregated column?
You could do something like this:
select 'tenants:' + ltrim(str(t.Id)), <shared fields> from Tenants as t union
select 'landlords:' + ltrim(str(l.Id)), <shared fields> from Tenants as l union
...
This requires some logic to be embedded from the client querying; it has to know how to fabricate the key that it's looking for in order to search on a single field.
That said, it's probably better if you just have a separate column which contains a "type" value (e.g. landlord, tenant) and then filter on both the type and the ID, as it will be computationally less expensive (and can be optimized better).

You want to use the built-in full text search or a separate product like Lucene. This is optimised for unstructured searches over heterogeneous data.
Also, don't forget that normal indices cannot be used for something LIKE '%...%'. Using a full text search engine will also be able to do efficient substring searches.

I would suggest using a specialized full-text indexing tool like Lucene for this. It will probably be easier to get up and running, and the result is faster and more featureful too. Postgres full text indexes will be useful if you also need structured search capability on top of this or transactionality of your search index is important.
If you do want to implement this in the database, something like the following scheme might work, assuming you use surrogate keys:
for each searchable table create a view that has the primary key column of that table, the name of the table and a concatenation of all the searchable fields in that table.
create a functional GIN or GiST index on the underlying over the to_tsvector() of the exact same concatenation.
create a UNION ALL over all the views to create the searchable view.
After that you can do the searches like this:
SELECT id, table_name, ts_rank_cd(body, query) AS rank
FROM search_view, to_tsquery('search&words') query
WHERE query ## body
ORDER BY rank DESC
LIMIT 10;

You should be fine, and there's really no other good (easy) way to do this. Just make sure the fields you are searching on are properly indexed though.

How to query filter in django without multiple occurrences

I have 2 models:
ParentModel: 'just' sits there
ChildModel: has a foreign key to ParentModel
ParentModel.objects.filter(childmodel__in=ChildModel.objects.all()) gives multiple occurrences of ParentModel.
How do I query all ParentModels that have at least one ChildModel that's referring to it? And without multiple occurrences...

You almost got it right...
ParentModel.objects.filter(childmodel__in=ChildModel.objects.all()).distinct()

You might want to avoid using childmodel__in=ChildModel.objects.all() if the number of ChildModel objects is large. This will generate SQL with all ChildModel id's enumerated in a list, possibly creating a huge SQL query.
If you can use Django 1.1 with aggregation support, you could do something like:
ParentModel.objects.annotate(num_children=Count('child')).filter(num_children__gte=1)
which should generate better SQL.