Develop Reference

Peewee group_concat/case issue

I'm trying to use peewee to fetch and format some data coming from a sqlite database using GROUP_CONCAT and Case. But I'm facing an issue with those functions.
First let start with what I want to achieve:
I simplified my table structure to better point the problem: 1 simple table with two columns: name (Char), is_controlled (Boolean).
This SQL request compute the desired result:
SELECT
SUM(is_controlled),
GROUP_CONCAT(CASE WHEN is_controlled = 1 THEN name ELSE NULL END, ':') as controlled,
GROUP_CONCAT(CASE WHEN is_controlled = 0 THEN name ELSE NULL END, ':') as not_controlled
FROM component;
Output (which is what I expect to have with peewee):
2
comp1:comp3
comp2
Here is an script allowing to test my problem:
from peewee import *
db = SqliteDatabase('test.db')
class Component(Model):
name = CharField()
is_controlled = BooleanField()
class Meta:
database = db
raw_data = [
{'name': 'comp1', 'is_controlled': True},
{'name': 'comp2', 'is_controlled': False},
{'name': 'comp3', 'is_controlled': True},
]
db.connect()
# Populate database
db.create_tables([Component])
for item in raw_data:
Component.get_or_create(**item)
res = Component.select(
fn.Sum(Component.is_controlled).alias('controlled_count'),
fn.GROUP_CONCAT(Case(None, [((Component.is_controlled == True), Component.name)], None), ':').alias('controlled'),
fn.GROUP_CONCAT(Case(None, [((Component.is_controlled == False), Component.name)], None), ':').alias('not_controlled')
)
print res[0].controlled_count
print res[0].controlled
print res[0].not_controlled
db.close()
As you can see, the data structure is simple (I simplified at maximum the example). The ouput is:
2
:comp3:
:
I inspected the SQL query generated by peewee (using res.sql()) and it looks like that:
sql = 'SELECT Sum("t1"."is_controlled") AS "controlled_count", GROUP_CONCAT(CASE WHEN ("t1"."is_controlled" = ?) THEN ? END, "t1"."name") AS "controlled", GROUP_CONCAT(CASE WHEN ("t1"."is_controlled" = ?) THEN ? END, "t1"."name") AS "not_controlled" FROM "component" AS "t1"'
params = [True, ':', False, ':']
We can see that the ELSE NULL part is missing from the peewee generated SQL request. I have tried several things, like adapting the parameters given to the Case function, but I can not get it to work correctly.
How can correctly use peewee to have the same result as using SQL ?
(I'm using python 2.7.15 with peewee 3.6.4 ans sqlite 3.19.4)

The Case function's signature provides a clue:
def Case(predicate, expression_tuples, default=None):
Inside the code, it checks:
if default is not None:
clauses.extend((SQL('ELSE'), default))
So, when you're passing None it's indistinguishable from the "empty/unspecified" case, and Peewee ignores it.
As a workaround you could instead specify SQL('NULL') as the default value. Or you could use an empty string, although I'm not sure whether you're relying on some behavior of group-concat with nulls, so that may not work?

How to execute a raw query with returning in sqlalchemy

I have a table Ticket that has the id (autoincremental), ticket_number (trigger that reads a sequence), value and date in Oracle. And I want to do the following:
INSERT INTO TICKET (value, date) values (100, TO_DATE('07-29-2015', 'mm-dd-yyyy')) returning ticket_number into :ticket_number;
I need to do this in raw SQL, but I don't know how to get the value in sqlalchemy. is it possible?

I've tried this with a toy table in Postgres and it works, I think should be equivalent in Oracle, please let me know.
In [15]:
result = session.connection().execute("insert into usertable values('m6', 'kk2', 'Chile') returning username")
for r in result:
print r
(u'm6',)
Hope it helps.
EDIT for Oracle: the one way to do it that I've found is not very elegant. It would be using the raw connection underneath SQLAlchemy connection, something like:
In [15]:
from sqlalchemy.sql import text
import cx_Oracle
​
cur = session.connection().connection.cursor()
out = cur.var(cx_Oracle.STRING)
par = { "u" : out } ​
cur.prepare("insert into usertable values('m34', 'kk2', 'Chile') returning username into :u")
cur.execute(None, par)
​
print(out)
print(type(out))
print(out.getvalue())
​
​
<cx_Oracle.STRING with value 'm34'>
<type 'cx_Oracle.STRING'>
m34
Unfortunately, I don't think there is a way to create a cx_oracle variable instance, it is just not available in the api, see docs.
Then, there is no way to avoid creating the cursor, even if it works when you delegate more to SQLAlchemy:
In [28]:
from sqlalchemy.sql import text
import cx_Oracle
​
cur = session.connection().connection.cursor()
out = cur.var(cx_Oracle.STRING)
par = { "u" : out }
​
q = text("insert into usertable values('m43', 'kk2', 'Chile') returning username into :u")
result = session.connection().execute(q, par)
print(par["u"])
print(out)
type(out)
​<cx_Oracle.STRING with value 'm43'>
<cx_Oracle.STRING with value 'm43'>
Out[28]:
cx_Oracle.STRING
Of course, you should close the cursor in this second case (in the first one, oracle closes it). The point that there is no way to create an instance like out = cx_Oracle.STRING()
As I say, it is not very elegant, but I don't think there is a way to create an equivalent variable in SQLAlchemy. It is something that the code handles internally. I would just go for the raw connection-cursor.
Hope it helps.
EDIT2: In the code above, added out.getvalue() as suggested. Thanks!

You should be able to use the execute command.
Something like this:
raw_SQL = "INSERT INTO TICKET (value, date) values (100, TO_DATE('07-29-2015', 'mm-dd-yyyy')) returning ticket_number into :ticket_number;"
connection = engine.connect()
result = connection.execute(raw_SQL)

How to write multi column in clause with sqlalchemy

Please suggest is there way to write query multi-column in clause using SQLAlchemy?
Here is example of the actual query:
SELECT url FROM pages WHERE (url_crc, url) IN ((2752937066, 'http://members.aye.net/~gharris/blog/'), (3799762538, 'http://www.coxandforkum.com/'));
I have a table that has two columns primary key and I'm hoping to avoid adding one more key just to be used as an index.
PS I'm using mysql DB.
Update: This query will be used for batch processing - so I would need to put few hundreds pairs into the in clause. With IN clause approach I hope to know fixed limit of how many pairs I can stick into one query. Like Oracle has 1000 enum limit by default.
Using AND/OR combination might be limited by the length of the query in chars. Which would be variable and less predictable.

Assuming that you have your model defined in Page, here's an example using tuple_:
keys = [
(2752937066, 'http://members.aye.net/~gharris/blog/'),
(3799762538, 'http://www.coxandforkum.com/')
]
select([
Page.url
]).select_from(
Page
).where(
tuple_(Page.url_crc, Page.url).in_(keys)
)
Or, using the query API:
session.query(Page.url).filter(tuple_(Page.url_crc, Page.url).in_(keys))

I do not think this is currently possible in sqlalchemy, and not all RDMBS support this.
You can always transform this to a OR(AND...) condition though:
filter_rows = [
(2752937066, 'http://members.aye.net/~gharris/blog/'),
(3799762538, 'http://www.coxandforkum.com/'),
]
qry = session.query(Page)
qry = qry.filter(or_(*(and_(Page.url_crc == crc, Page.url == url) for crc, url in filter_rows)))
print qry
should produce something like (for SQLite):
SELECT pages.id AS pages_id, pages.url_crc AS pages_url_crc, pages.url AS pages_url
FROM pages
WHERE pages.url_crc = ? AND pages.url = ? OR pages.url_crc = ? AND pages.url = ?
-- (2752937066L, 'http://members.aye.net/~gharris/blog/', 3799762538L, 'http://www.coxandforkum.com/')
Alternatively, you can combine two columns into just one:
filter_rows = [
(2752937066, 'http://members.aye.net/~gharris/blog/'),
(3799762538, 'http://www.coxandforkum.com/'),
]
qry = session.query(Page)
qry = qry.filter((func.cast(Page.url_crc, String) + '|' + Page.url).in_(["{}|{}".format(*_frow) for _frow in filter_rows]))
print qry
which produces the below (for SQLite), so you can use IN:
SELECT pages.id AS pages_id, pages.url_crc AS pages_url_crc, pages.url AS pages_url
FROM pages
WHERE (CAST(pages.url_crc AS VARCHAR) || ? || pages.url) IN (?, ?)
-- ('|', '2752937066|http://members.aye.net/~gharris/blog/', '3799762538|http://www.coxandforkum.com/')

I ended up using the test() based solution: generated "(a,b) in ((:a1, :b1), (:a2,:b2), ...)" with named bind vars and generating dictionary with bind vars' values.
params = {}
for counter, r in enumerate(records):
a_param = "a%s" % counter
params[a_param] = r['a']
b_param = "b%s" % counter
params[b_param] = r['b']
pair_text = "(:%s,:%s)" % (a_param, b_param)
enum_pairs.append(pair_text)
multicol_in_enumeration = ','.join(enum_pairs)
multicol_in_clause = text(
" (a,b) in (" + multicol_in_enumeration + ")")
q = session.query(Table.id, Table.a,
Table.b).filter(multicol_in_clause).params(params)
Another option I thought about using mysql upserts but this would make whole included even less portable for the other db engine then using multicolumn in clause.
Update SQLAlchemy has sqlalchemy.sql.expression.tuple_(*clauses, **kw) construct that can be used for the same purpose. (I haven't tried it yet)

Usage of "aliased" in SQLAlchemy ORM

From the SQLAlchemy ORM Tutorial:
You can control the names using the label() construct for scalar attributes and aliased for class constructs:
>>> from sqlalchemy.orm import aliased
>>> user_alias = aliased(User, name='user_alias')
>>> for row in session.query(user_alias, user_alias.name.label('name_label')).all():
... print row.user_alias, row.name_label
This seems to be a lot more typing and a lot less readable than the plain class-instrumented descriptors:
>>> for row in session.query(User, User.name).all():
... print row.User, row.name
But it must exist for a reason. How should it be used? What are some good use cases?

aliased() or alias() are used whenever you need to use the SELECT ... FROM my_table my_table_alias ... construct in SQL, mostly when using the same table more than once in a query (self-joins, with or without extra tables). You also need to alias subqueries in certain cases.
There's an example in the documentation: http://www.sqlalchemy.org/docs/orm/query.html?highlight=aliased#sqlalchemy.orm.util.AliasedClass

As #jd. said,mostly when using the same table more than once in a query.
Example:
dict_code_type, dict_code_status = aliased(DictCode), aliased(DictCode)
query = Device.query \
.join(dict_code_type, dict_code_type.codeValue == Device.deviceType) \
.join(dict_code_status, dict_code_status.codeValue == Device.status) \
.with_entities(Device.id, Device.deviceName, Device.status,
Device.deviceID, Device.deviceUsername, Device.token,
dict_code_type.codeLabel.label('deviceTypeLabel'),
dict_code_status.codeLabel.label('statusLabel'), Device.createAt, Device.authType) \
.filter(and_(dict_code_type.code == 'deviceType', dict_code_status.code == 'status'))

How do I do a not equal in Django queryset filtering?

In Django model QuerySets, I see that there is a __gt and __lt for comparative values, but is there a __ne or != (not equals)? I want to filter out using a not equals. For example, for
Model:
bool a;
int x;
I want to do
results = Model.objects.exclude(a=True, x!=5)
The != is not correct syntax. I also tried __ne.
I ended up using:
results = Model.objects.exclude(a=True, x__lt=5).exclude(a=True, x__gt=5)

You can use Q objects for this. They can be negated with the ~ operator and combined much like normal Python expressions:
from myapp.models import Entry
from django.db.models import Q
Entry.objects.filter(~Q(id=3))
will return all entries except the one(s) with 3 as their ID:
[<Entry: Entry object>, <Entry: Entry object>, <Entry: Entry object>, ...]

Your query appears to have a double negative, you want to exclude all rows where x is not 5, so in other words you want to include all rows where x is 5. I believe this will do the trick:
results = Model.objects.filter(x=5).exclude(a=True)
To answer your specific question, there is no "not equal to" field lookup but that's probably because Django has both filter and exclude methods available so you can always just switch the logic around to get the desired result.

the field=value syntax in queries is a shorthand for field__exact=value. That is to say that Django puts query operators on query fields in the identifiers. Django supports the following operators:
exact
iexact
contains
icontains
in
gt
gte
lt
lte
startswith
istartswith
endswith
iendswith
range
date
year
iso_year
month
day
week
week_day
iso_week_day
quarter
time
hour
minute
second
isnull
regex
iregex
I'm sure by combining these with the Q objects as Dave Vogt suggests and using filter() or exclude() as Jason Baker suggests you'll get exactly what you need for just about any possible query.

There are three options:
Chain exclude and filter
results = Model.objects.exclude(a=True).filter(x=5)
Use Q() objects and the ~ operator
from django.db.models import Q
object_list = QuerySet.filter(~Q(a=True), x=5)
Register a custom lookup function
from django.db.models import Lookup
from django.db.models import Field
#Field.register_lookup
class NotEqual(Lookup):
lookup_name = 'ne'
def as_sql(self, compiler, connection):
lhs, lhs_params = self.process_lhs(compiler, connection)
rhs, rhs_params = self.process_rhs(compiler, connection)
params = lhs_params + rhs_params
return '%s <> %s' % (lhs, rhs), params
Which can the be used as usual:
results = Model.objects.exclude(a=True, x__ne=5)

It's easy to create a custom lookup, there's an __ne lookup example in Django's official documentation.
You need to create the lookup itself first:
from django.db.models import Lookup
class NotEqual(Lookup):
lookup_name = 'ne'
def as_sql(self, compiler, connection):
lhs, lhs_params = self.process_lhs(compiler, connection)
rhs, rhs_params = self.process_rhs(compiler, connection)
params = lhs_params + rhs_params
return '%s <> %s' % (lhs, rhs), params
Then you need to register it:
from django.db.models import Field
Field.register_lookup(NotEqual)
And now you can use the __ne lookup in your queries like this:
results = Model.objects.exclude(a=True, x__ne=5)

While you can filter Models with =, __gt, __gte, __lt, __lte, you cannot use ne or !=. However, you can achieve better filtering using the Q object.
You can avoid chaining QuerySet.filter() and QuerySet.exclude(), and use this:
from django.db.models import Q
object_list = QuerySet.filter(~Q(field='not wanted'), field='wanted')

Pending design decision. Meanwhile, use exclude()
The Django issue tracker has the remarkable entry #5763,
titled "Queryset doesn't have a "not equal" filter operator".
It is remarkable because (as of April 2016) it was
"opened 9 years ago" (in the Django stone age),
"closed 4 years ago", and
"last changed 5 months ago".
Read through the discussion, it is interesting.
Basically, some people argue __ne should be added
while others say exclude() is clearer and hence __ne
should not be added.
(I agree with the former, because the latter argument is
roughly equivalent to saying Python should not have != because
it has == and not already...)

Using exclude and filter
results = Model.objects.filter(x=5).exclude(a=true)

You should use filter and exclude like this
results = Model.objects.exclude(a=true).filter(x=5)

This will give your desired result.
from django.db.models import Q
results = Model.objects.exclude(Q(a=True) & ~Q(x=5))
for not equal you can use ~ on an equal query. obviously, Q can be used to reach the equal query.

What you are looking for are all objects that have either a=false or x=5. In Django, | serves as OR operator between querysets:
results = Model.objects.filter(a=false)|Model.objects.filter(x=5)

Django-model-values (disclosure: author) provides an implementation of the NotEqual lookup, as in this answer. It also provides syntactic support for it:
from model_values import F
Model.objects.exclude(F.x != 5, a=True)

results = Model.objects.filter(a = True).exclude(x = 5) Generetes this sql: select * from tablex where a != 0 and x !=5The sql depends on how your True/False field is represented, and the database engine. The django code is all you need though.

The last bit of code will exclude all objects where x!=5 and a is True. Try this:
results = Model.objects.filter(a=False, x=5)
Remember, the = sign in the above line is assigning False to the parameter a and the number 5 to the parameter x. It's not checking for equality. Thus, there isn't really any way to use the != symbol in a query call.

This should work
results = Model.objects.filter(x=5).exclude(a=True)

Watch out for lots of incorrect answers to this question!
Gerard's logic is correct, though it will return a list rather than a queryset (which might not matter).
If you need a queryset, use Q:
from django.db.models import Q
results = Model.objects.filter(Q(a=false) | Q(x=5))

If we need to exclude/negate based on the sub queryset we can use,
Conditional filter:
When a conditional expression returns a boolean value, it is possible to use it directly in filters. Here non_unique_account_type returns a boolean value. But, still, we can use it in the filter.
>>> non_unique_account_type = Client.objects.filter(
... account_type=OuterRef('account_type'),
... ).exclude(pk=OuterRef('pk')).values('pk')
>>> Client.objects.filter(~Exists(non_unique_account_type))
In the SQL terms, it evaluates to:
SELECT * FROM client c0
WHERE NOT EXISTS (
SELECT c1.id
FROM client c1
WHERE c1.account_type = c0.account_type AND NOT c1.id = c0.id
)