Develop Reference

Peewee group_concat/case issue

I'm trying to use peewee to fetch and format some data coming from a sqlite database using GROUP_CONCAT and Case. But I'm facing an issue with those functions.
First let start with what I want to achieve:
I simplified my table structure to better point the problem: 1 simple table with two columns: name (Char), is_controlled (Boolean).
This SQL request compute the desired result:
SELECT
SUM(is_controlled),
GROUP_CONCAT(CASE WHEN is_controlled = 1 THEN name ELSE NULL END, ':') as controlled,
GROUP_CONCAT(CASE WHEN is_controlled = 0 THEN name ELSE NULL END, ':') as not_controlled
FROM component;
Output (which is what I expect to have with peewee):
2
comp1:comp3
comp2
Here is an script allowing to test my problem:
from peewee import *
db = SqliteDatabase('test.db')
class Component(Model):
name = CharField()
is_controlled = BooleanField()
class Meta:
database = db
raw_data = [
{'name': 'comp1', 'is_controlled': True},
{'name': 'comp2', 'is_controlled': False},
{'name': 'comp3', 'is_controlled': True},
]
db.connect()
# Populate database
db.create_tables([Component])
for item in raw_data:
Component.get_or_create(**item)
res = Component.select(
fn.Sum(Component.is_controlled).alias('controlled_count'),
fn.GROUP_CONCAT(Case(None, [((Component.is_controlled == True), Component.name)], None), ':').alias('controlled'),
fn.GROUP_CONCAT(Case(None, [((Component.is_controlled == False), Component.name)], None), ':').alias('not_controlled')
)
print res[0].controlled_count
print res[0].controlled
print res[0].not_controlled
db.close()
As you can see, the data structure is simple (I simplified at maximum the example). The ouput is:
2
:comp3:
:
I inspected the SQL query generated by peewee (using res.sql()) and it looks like that:
sql = 'SELECT Sum("t1"."is_controlled") AS "controlled_count", GROUP_CONCAT(CASE WHEN ("t1"."is_controlled" = ?) THEN ? END, "t1"."name") AS "controlled", GROUP_CONCAT(CASE WHEN ("t1"."is_controlled" = ?) THEN ? END, "t1"."name") AS "not_controlled" FROM "component" AS "t1"'
params = [True, ':', False, ':']
We can see that the ELSE NULL part is missing from the peewee generated SQL request. I have tried several things, like adapting the parameters given to the Case function, but I can not get it to work correctly.
How can correctly use peewee to have the same result as using SQL ?
(I'm using python 2.7.15 with peewee 3.6.4 ans sqlite 3.19.4)

The Case function's signature provides a clue:
def Case(predicate, expression_tuples, default=None):
Inside the code, it checks:
if default is not None:
clauses.extend((SQL('ELSE'), default))
So, when you're passing None it's indistinguishable from the "empty/unspecified" case, and Peewee ignores it.
As a workaround you could instead specify SQL('NULL') as the default value. Or you could use an empty string, although I'm not sure whether you're relying on some behavior of group-concat with nulls, so that may not work?

Filter query by linked object key in SQLAlchemy

Judging by the title this would be the exact same question, but I can't see how any of the answers are applicable to my use case:
I have two classes and a relationship between them:
treatment_association = Table('tr_association', Base.metadata,
Column('chronic_treatments_id', Integer, ForeignKey('chronic_treatments.code')),
Column('animals_id', Integer, ForeignKey('animals.id'))
)
class ChronicTreatment(Base):
__tablename__ = "chronic_treatments"
code = Column(String, primary_key=True)
class Animal(Base):
__tablename__ = "animals"
treatment = relationship("ChronicTreatment", secondary=treatment_association, backref="animals")
I would like to be able to select only the animals which have undergon a treatment which has the code "X". I tried quite a few approaches.
This one fails with an AttributeError:
sql_query = session.query(Animal.treatment).filter(Animal.treatment.code == "chrFlu")
for item in sql_query:
pass
mystring = str(session.query(Animal))
And this one happily returns a list of unfiltered animals:
sql_query = session.query(Animal.treatment).filter(ChronicTreatment.code == "chrFlu")
for item in sql_query:
pass
mystring = str(session.query(Animal))
The closest thing to the example from the aforementioned thread I could put together:
subq = session.query(Animal.id).subquery()
sql_query = session.query(ChronicTreatment).join((subq, subq.c.treatment_id=="chrFlu"))
for item in sql_query:
pass
mystring = str(session.query(Animal))
mydf = pd.read_sql_query(mystring,engine)
Also fails with an AttributeError.
Can you hel me sort this list?

First, there are two issues with table definitions:
1) In the treatment_association you have Integer column pointing to chronic_treatments.code while the code is String column.
I think it's just better to have an integer id in the chronic_treatments, so you don't duplicate the string code in another table and also have a chance to add more fields to chronic_treatments later.
Update: not exactly correct, you still can add more fields, but it will be more complex to change your 'code' if you decide to rename it.
2) In the Animal model you have a relation named treatment. This is confusing because you have many-to-many relation, it should be plural - treatments.
After fixing the above two, it should be clearer why your queries did not work.
This one (I replaced treatment with treatments:
sql_query = session.query(Animal.treatments).filter(
Animal.treatments.code == "chrFlu")
The Animal.treatments represents a many-to-many relation, it is not an SQL Alchemy mode, so you can't pass it to the query nor use in a filter.
Next one can't work for the same reason (you pass Animal.treatments into the query.
The last one is closer, you actually need join to get your results.
I think it is easier to understand the query as SQL (and you anyway need to know SQL to be able to use sqlalchemy):
animals = session.query(Animal).from_statement(text(
"""
select distinct animals.* from animals
left join tr_association assoc on assoc.animals_id = animals.id
left join chronic_treatments on chronic_treatments.id = assoc.chronic_treatments_id
where chronic_treatments.code = :code
""")
).params(code='chrFlu')
It will select animals and join chronic_treatments through the tr_association and filter the result by code.
Having this it is easy to rewrite it using SQL-less syntax:
sql_query = session.query(Animal).join(Animal.treatments).filter(
ChronicTreatment.code == "chrFlu")
That will return what you want - a list of animals who have related chronic treatment with given code.

SQLAlchemy WHERE IN single value (raw SQL)

I'm having trouble with SQLAlchemy when doing a raw SQL which checks against multiple values.
my_sess.execute(
"SELECT * FROM table WHERE `key`='rating' AND uid IN :uids",
params=dict(uids=some_list)
).fetchall()
There are 2 scenarios for this query, one that works and one that doesn't. If some_list = [1], it throws me an SQL error that I have a syntax error near ). But if some_list = [1, 2], the query executes successfully.
Any reason why this would happen?

No, SQL parameters only ever deal with scalar values. You'll have to generate the SQL here; if you need raw SQL, use:
statement = "SELECT * FROM table WHERE `key`='rating' AND uid IN ({})".format(
', '.join([':i{}'.format(i) for i in range(len(some_list))]))
my_sess.execute(
statement,
params={'i{}'.format(i): v for i, v in enumerate(some_list)})
).fetchall()
e.g. generate enough parameters to hold all values in some_list with string formatting, then generate matching parameters to fill them.
Better still would be to use a literal_column() object to do all the generating for you:
from sqlalchemy.sql import literal_column
uid_in = literal_column('uid').in_(some_list)
statement = "SELECT * FROM able WHERE `key`='rating' AND {}".format(uid_in)
my_sess.execute(
statement,
params={'uid_{}'.format(i): v for i, v in enumerate(some_list)})
).fetchall()
but then you perhaps could just generate the whole statement using the `sqlalchemy.sql.expression module, as this would make supporting multiple database dialects much easier.
Moreover, the uid_in object already holds references to the right values for the bind parameters; instead of turning it into a string as we do with the str.format() action above, SQLAlchemy would have the actual object plus the associated parameters and you would no longer have to generate the params dictionary either.
The following should work:
from sqlalchemy.sql import table, literal_column, select
tbl = table('table')
key_clause = literal_column('key') == 'rating'
uid_clause = literal_column('uid').in_(some_list)
my_sess.execute(select('*', key_clause & uid_clause, [tbl]))
where the sqlalchemy.sql.select() takes a column spec (here hard-coded to *), a where clause (generated from the two clauses with & to generate a SQL AND clause) and a list of selectables; here your one sqlalchemy.sql.table() value.
Quick demo:
>>> from sqlalchemy.sql import table, literal_column, select
>>> some_list = ['foo', 'bar']
>>> tbl = table('table')
>>> key_clause = literal_column('key') == 'rating'
>>> uid_clause = literal_column('uid').in_(some_list)
>>> print select('*', key_clause & uid_clause, [tbl])
SELECT *
FROM "table"
WHERE key = :key_1 AND uid IN (:uid_1, :uid_2)
but the actual object tree generated from all this contains the actual values for the bind parameters too, so my_sess.execute() can access these directly.

How to pass the name of a column as a parameter in SQLAlchemy Core?

I have an sqlalchemy core bulk update query that I need to programmatically pass the name of the column that is to be updated.
The function looks as below with comments on each variable:
def update_columns(table_name, pids, column_to_update):
'''
1. table_name: a string denoting the name of the table to be updated
2. pid: a list of primary ids
3. column_to_update: a string representing the name of the column that will be flagged. Sometimes the name can be is_processed or is_active and several more other columns. I thus need to pass the name as a parameter.
'''
for pid in pids:
COL_DICT_UPDATE = {}
COL_DICT_UPDATE['b_id'] = pid
COL_DICT_UPDATE['b_column_to_update'] = True
COL_LIST_UPDATE.append(COL_DICT_UPDATE)
tbl = Table(table_name, meta, autoload=True, autoload_with=Engine)
trans = CONN.begin()
stmt = tbl.update().where(tbl.c.id == bindparam('b_id')).values(tbl.c.column_to_update==bindparam('b_column_to_update'))
trans.commit()
The table parameter gets accepted and works fine.
The column_to_update doesn't work when passed as a parameter. It fails with the error raise AttributeError(key) AttributeError: column_to_mark. If I however hard code the column name, the query runs.
How can I pass the name of the column_to_update for SQLAlchemy to recognize it?
EDIT: Final Script
Thanks to #Paulo, the final script looks like this:
def update_columns(table_name, pids, column_to_update):
for pid in pids:
COL_DICT_UPDATE = {}
COL_DICT_UPDATE['b_id'] = pid
COL_DICT_UPDATE['b_column_to_update'] = True
COL_LIST_UPDATE.append(COL_DICT_UPDATE)
tbl = Table(table_name, meta, autoload=True, autoload_with=Engine)
trans = CONN.begin()
stmt = tbl.update().where(
tbl.c.id == bindparam('b_id')
).values(**{column_to_update: bindparam('b_column_to_update')})
CONN.execute(stmt, COL_LIST_UPDATE)
trans.commit()

I'm not sure if I understood what you want, and your code looks very different from what I consider idiomatic sqlalchemy (I'm not criticizing, just commenting we probably use orthogonal code styles).
If you want to pass a literal column as a parameter use:
from sqlalchemy.sql import literal_column
...
tbl.update().where(
tbl.c.id == bindparam('b_id')
).values({
tbl.c.column_to_update: literal_column('b_column_to_update')
})
If you want to set the right side of the expression dynamically, use:
tbl.update().where(
tbl.c.id == bindparam('b_id')
).values({
getattr(tbl.c, 'column_to_update'): bindparam('b_column_to_update')
})
If none of this is not what you want, comment on the answer or improve your question and I will try to help.
[update]
The values method uses named arguments like .values(column_to_update=value) where column_to_update is the actual column name, not a variable holding the column name. Example:
stmt = users.update().\
where(users.c.id==5).\
values(id=-5)
Note that where uses the comparison operator == while values uses the attribution operator = instead - the former uses the column object in a Boolean expression and the latter uses the column name as a keyword argument binding.
If you need it to be dynamic, use the **kwargs notation: .values(**{'column_to_update': value})
But probably you want to use the values argument instead of the values method.

There is also another simple way:
tbl.c[column_name_here]

How to construct a slightly more complex filter using "or_" or "and_" in SQLAlchemy

I'm trying to do a very simple search from a list of terms
terms = ['term1', 'term2', 'term3']
How do I programmatically go through the list of terms and construct the conditions from the list of terms so that I can make the query using filter and or_ or _and?
query.filter(or_(#something constructed from terms))

If you have a list of terms and want to find rows where a field matches one of them, then you could use the in_() method:
terms = ['term1', 'term2', 'term3']
query.filter(Cls.field.in_(terms))
If you want to do something more complex, then or_() and and_() take ClauseElement objects as parameters. ClauseElement and its subclasses basically represent the SQL AST of your query. Typically, you create clause elements by invoking a comparison operator on Column or InstrumentedAttribute objects:
# Create the clause element
clause = (users_table.columns['name'] == "something")
# you can also use the shorthand users_table.c.name
# The clause is a binary expression ...
print(type(clause))
# <class 'sqlalchemy.sql.expression._BinaryExpression'>
# ... that compares a column for equality with a bound value.
print(type(clause.left), clause.operator, type(clause.right))
# <class 'sqlalchemy.schema.Column'>, <built-in function eq>,
# <class 'sqlalchemy.sql.expression._BindParamClause'>
# str() compiles it to SQL
print(str(clause))
# users.name = ?
# You can also do that with ORM attributes
clause = (User.name == "something")
print(str(clause))
# users.name = ?
You can handle clause elements representing your conditions like any Python objects, put them into lists, compose them into other clause elements, etc. So you can do something like this:
# Collect the separate conditions to a list
conditions = []
for term in terms:
conditions.append(User.name == term)
# Combine them with or to a BooleanClauseList
condition = or_(*conditions)
# Can now use the clause element as a predicate in queries
query = query.filter(condition)
# or to view the SQL fragment
print(str(condition))
# users.name = ? OR users.name = ? OR users.name = ?

Assuming that your terms variable contains valid SQL statement fragments, you can simply pass terms preceded by an asterisk to or_ or and_:
>>> from sqlalchemy.sql import and_, or_
>>> terms = ["name='spam'", "email='spam#eggs.com'"]
>>> print or_(*terms)
name='spam' OR email='spam#eggs.com'
>>> print and_(*terms)
name='spam' AND email='spam#eggs.com'
Note that this assumes that terms contains only valid and properly escaped SQL fragments, so this is potentially unsafe if a malicious user can access terms somehow.
Instead of building SQL fragments yourself, you should let SQLAlchemy build parameterised SQL queries using other methods from sqlalchemy.sql. I don't know whether you have prepared Table objects for your tables or not; if so, assume that you have a variable called users which is an instance of Table and it describes your users table in the database. Then you can do the following:
from sqlalchemy.sql import select, or_, and_
terms = [users.c.name == 'spam', users.c.email == 'spam#eggs.com']
query = select([users], and_(*terms))
for row in conn.execute(query):
# do whatever you want here
Here, users.c.name == 'spam' will create an sqlalchemy.sql.expression._BinaryExpression object that records that this is a binary equality relation between the name column of the users table and a string literal that contains spam. When you convert this object to a string, you will get an SQL fragment like users.name = :1, where :1 is a placeholder for the parameter. The _BinaryExpression object also remembers the binding of :1 to 'spam', but it won't insert it until the SQL query is executed. When it is inserted, the database engine will make sure that it is properly escaped. Suggested reading: SQLAlchemy's operator paradigm
If you only have the database table but you don't have a users variable that describes the table, you can create it yourself:
from sqlalchemy import Table, MetaData, Column, String, Boolean
metadata = MetaData()
users = Table('users', metadata,
Column('id', Integer, primary_key=True),
Column('name', String),
Column('email', String),
Column('active', Integer)
)
Alternatively, you can use autoloading which queries the database engine for the structure of the database and builds users automatically; obviously this is more time-consuming:
users = Table('users', metadata, autoload=True)

I had the same issue in"SQLAlchemy: an efficient/better select by primary keys?":
terms = ['one', 'two', 'three']
clauses = or_( * [Table.field == x for x in terms] )
query = Session.query(Table).filter(clauses)

You can use the "Conjunctions" documentation to combine conditions And, Or and not.