Develop Reference

Convert SQL to peewee

How to convert this to peewee query?
_, storage, spaceLeft = db.execute('''
SELECT session, storage, storage - count(questID) FROM Hypos
INNER JOIN Quests ON Hypos.hypoID = Quests.hypoID
WHERE Hypos.hypoID = ?1 AND session = ?2
GROUP BY session
UNION SELECT NULL, max(storage), storage FROM Hypos
WHERE Hypos.hypoID = ?1
ORDER BY session DESC LIMIT 1
''', (body.hypo_id, session)
).fetchone()
I don't understand how to create UNION SELECT in peewee. My attempt:
_, storage, spaceLeft = (db_models.Hypos
.select(db_models.Quests.session, db_models.Hypos.storage, db_models.Hypos.storage-fn.COUNT(db_models.Quests.questID))
.join(db_models.Quests, on=(db_models.Hypos.hypoID == db_models.Quests.hypoID), attr='quests')
.where(db_models.Hypos.hypoID==body.hypo_id, db_models.Quests.session==session)
.group_by(db_models.Quests.session)
) | (db_models.Hypos.select(None, fn.MAX(db_models.Hypos.storage), db_models.Hypos.storage)
.where(db_models.Hypos.hypoID==body.hypo_id)
.order_by(db_models.Quests.session.desc()).limit(1)
)
I got :
1st ORDER BY term does not match any column in the result set
without order_by problem is not enough values to unpack (expected 3, got 2)

Try something like this:
lhs = (Hypo
.select(Quest.session, Hypo.storage, Hypo.storage - fn.COUNT(Quest.questID))
.join(Quest)
.where(
(Hypo.hypoID == hypo_id) &
(Quest.session == session))
.group_by(Quest.session))
rhs = (Hypo
.select(Value(None), fn.MAX(Hypo.storage), Hypo.storage)
.where(Hypo.hypoID == hypo_id))
union = (lhs | rhs).order_by(SQL('1').desc()).limit(1)
_, storage, spaceLeft = union.scalar(as_tuple=True)
The resulting SQL:
SELECT "t1"."session", "t2"."storage", ("t2"."storage" - COUNT("t1"."questID"))
FROM "hypo" AS "t2"
INNER JOIN "quest" AS "t1" ON ("t1"."hypo_id" = "t2"."hypoID")
WHERE (("t2"."hypoID" = ?) AND ("t1"."session" = ?))
GROUP BY "t1"."session"
UNION
SELECT ?, MAX("t3"."storage"), "t3"."storage"
FROM "hypo" AS "t3"
WHERE ("t3"."hypoID" = ?)
ORDER BY 1 DESC LIMIT ?

SqlAlchemy: Sub-Select with a Dual EXISTS (OR) and an additional boolean check

In SqlAlchemy I need to implement the following subquery, which runs fine in PostgreSQL. It is an OR condition consisting of 2 EXISTS, plus an additional AND block. That whole column results in a True/False boolean value.
SELECT
...
...
,
(SELECT
(
EXISTS (SELECT id from participating_ic_t pi1 where pi1.id = agreement_t_1.participating_ic_id
AND pi1.ic_nihsac = substr(ned_person_t_2.nihsac, 1, 3))
OR EXISTS (SELECT id from external_people_t ep1 where ep1.participating_ic_id = agreement_t_1.participating_ic_id
AND ep1.uniqueidentifier = ned_person_t_2.uniqueidentifier)
)
AND ned_person_t_2.current_flag = 'Y' and ned_person_t_2.inactive_date is null and ned_person_t_2.organizationalstat = 'EMPLOYEE'
) as ACTIVE_APPROVER1,
First of all, if I omit the additional AND block, the following OR-EXISTS by itself works OK:
subq1 = db_session.query(ParticipatingIcT.id).filter((ParticipatingIcT.id == agreement.participating_ic_id),
(ParticipatingIcT.ic_nihsac == func.substr(approver.nihsac, 1, 3)))
.subquery()
subq2 = db_session.query(ExternalPeopleT.id).filter((ExternalPeopleT.participating_ic_id == agreement.participating_ic_id),
(ExternalPeopleT.uniqueidentifier == approver.uniqueidentifier))
.subquery()
subqMain = db_session.query(or_(exists(subq1), exists(subq2))
.subquery()
# ...
# Now we will select from subqMain.
agreements = (db_session.query(
..,
subqMain
But the problem starts when I introduce the final AND block. Conceptually, the final result should be the following:
subqMain = db_session.query(and_(
or_(exists(subq1), exists(subq2)),
approver.current_flag == 'Y',
approver.inactive_date == None,
approver.organizationalstat == 'EMPLOYEE'))
.subquery()
But this actually emits " .. FROM APPROVER_T" right in the sub-query, whereas it should be linked to the FROM APPROVER_T of the main query at the very end. I need to avoid adding the .. FROM [table] which happens as soon as I specify the and_(..). I don't understand why it's doing that. All subqueries are specifically marked as subquery(). The alias approver is defined at the very top as approver = aliased(NedPersonT).

exists1 = db_session.query(
ParticipatingIcT.id
).filter(
(ParticipatingIcT.id == agreement.participating_ic_id),
(ParticipatingIcT.ic_nihsac == func.substr(approver.nihsac, 1, 3))
).exists() # use exists here instead of subquery
exists2 = db_session.query(
ExternalPeopleT.id
).filter(
(ExternalPeopleT.participating_ic_id == agreement.participating_ic_id),
(ExternalPeopleT.uniqueidentifier == approver.uniqueidentifier)
).exists() # use exists again instead of subquery
subqMain = db_session.query(and_(
or_(exists1, exists2),
approver.current_flag == 'Y',
approver.inactive_date == None,
approver.organizationalstat == 'EMPLOYEE')).subquery()
# ...
# Now we will select from subqMain.
agreements = (db_session.query(
OtherModel,
subqMain))
sqlalchemy.orm.Query.exists
Half Baked Example
I also use aliased here because I don't quite understand the method of operations concerning wrapping sub queries in parentheses. It seems to work without it but the RAW sql is confusing to read. This is toy example I made to try and see if the SQL produced was invalid.
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True, autoincrement=True)
name = Column(String, nullable=False)
hobbies = relationship('Hobby', backref='user')
class Hobby(Base):
__tablename__ = 'hobbies'
id = Column(Integer, primary_key=True, autoincrement=True)
user_id = Column(Integer, ForeignKey('users.id'), nullable = False)
name = Column(String, nullable=False)
Base.metadata.create_all(engine, checkfirst=True)
with Session(engine) as session:
user1 = User(name='user1')
session.add(user1)
session.add_all([
Hobby(name='biking', user=user1),
Hobby(name='running', user=user1),
Hobby(name='eating', user=user1),
])
user2 = User(name='user2')
session.add(user2)
session.add(Hobby(name='biking', user=user2))
session.commit()
nested_user = aliased(User)
subq1 = session.query(Hobby.id).filter(nested_user.id ==Hobby.user_id, Hobby.name.like('bik%'))
subq2 = session.query(Hobby.id).filter(nested_user.id ==Hobby.user_id, Hobby.name.like('eat%'))
subqmain = session.query(nested_user).filter(or_(subq1.exists(), subq2.exists()), nested_user.id > 0).subquery()
q = session.query(User).select_from(User).join(subqmain, User.id == subqmain.c.id)
print (q)
print ([user.name for user in q.all()])
SELECT users.id AS users_id, users.name AS users_name
FROM users JOIN (SELECT users_1.id AS id, users_1.name AS name
FROM users AS users_1
WHERE ((EXISTS (SELECT 1
FROM hobbies
WHERE users_1.id = hobbies.user_id AND hobbies.name LIKE %(name_1)s)) OR (EXISTS (SELECT 1
FROM hobbies
WHERE users_1.id = hobbies.user_id AND hobbies.name LIKE %(name_2)s))) AND users_1.id > %(id_1)s) AS anon_1 ON users.id = anon_1.id
['user1', 'user2']

Something wasn't working with that Sub-Select so I rewrote it as a CASE statement instead. Now it's working. Also, I had to join on a couple of extra tables in the main query to facilitate this new CASE.
casePrimaryApprover = case(
[
(and_(
(agreement.approving_official_id.is_not(None)),
(approver.current_flag == 'Y'),
(approver.inactive_date.is_(None)),
(approver.organizationalstat == 'EMPLOYEE'),
or_(
(participatingIc.ic_nihsac == func.substr(approver.nihsac, 1, 3)),
(externalPeoplePrimaryApprover.id.is_not(None))
)
), 'Y')
],
else_ = 'N'
)
# Main Query
agreements = (db_session.query(
agreement.id,
...
casePrimaryApprover
...
.join(participatingIc, agreement.participating_ic_id == participatingIc.id)
.outerjoin(externalPeoplePrimaryApprover, approver.uniqueidentifier == externalPeoplePrimaryApprover.uniqueidentifier)

Building a psycopg2 query using a list of the column names to fetch

A rather simple question but for which we surprisingly didn't found a solution.
Here is my current code, for executing a simple SQL query on a PostgreSQL database from Python 3.6.9 using psycopg2 ('2.9.1 (dt dec pq3 ext lo64)'):
import psycopg2
myid = 100
fields = ('p.id', 'p.name', 'p.type', 'p.price', 'p.warehouse', 'p.location', )
sql_query = ("SELECT " + ', '.join(fields) + " FROM product p "
"INNER JOIN owner o ON p.id = o.product_id "
"WHERE p.id = {} AND (o.dateof_purchase IS NOT NULL "
"OR o.state = 'checked_out' );"
).format(myid)
try:
with psycopg2.connect(**DB_PARAMS) as conn:
with conn.cursor(cursor_factory=DictCursor) as curs:
curs.execute(sql_query, )
row = curs.fetchone()
except psycopg2.Error as error:
raise ValueError(f"ERR: something went wrong with the query :\n{sql_query}") from None
We're more and more thinking that this is... not good. (awfully bad to be honest).
Therefore, we're trying to use a modern f-string notation:
sql_query = (f"""SELECT {fields} FROM product p
INNER JOIN owner o ON p.id = o.product_id
WHERE p.id = {myid} AND (o.dateof_purchase IS NOT NULL
OR o.state = 'checked_out' );""")
But then, the query looks like:
SELECT ('p.id', 'p.name', 'p.type', 'p.price', 'p.warehouse', 'p.location', ) FROM ...;
which is not valid in PSQL because 1. of the brackets, and 2. of the single quoted column names.
We'd like to figure out a way to get rid of these.
In between, we went back to the doc and remembered this:
https://www.psycopg.org/docs/usage.html
Ooops! So we refactored it this way:
sql_query = (f"""SELECT %s FROM product p
INNER JOIN owner o ON p.id = o.product_id
WHERE p.id = %s AND (o.dateof_purchase IS NOT NULL
OR o.state = 'checked_out' );""")
try:
with psycopg2.connect(**DB_PARAMS) as conn:
with conn.cursor(cursor_factory=DictCursor) as curs:
# passing a tuple as it only accept one more argument after the query!
curs.execute(sql_query, (fields, myid))
row = curs.fetchone()
and mogrify() says:
"SELECT ('p.id', 'p.name', 'p.type', 'p.price', 'p.warehouse', 'p.location', ) FROM ...;"
here again, the brackets and the single quotes are causing troubles, but no error is actually raised.
The only thing is that row evaluates to this strange result:
['('p.id', 'p.name', 'p.type', 'p.price', 'p.warehouse', 'p.location', )']
So, how could we cleverly and dynamically build a psycopg2 query using a list of parameters for column names without neglecting the security?
(A trick could be to fetch all columns and filter them out after... but there are too many columns, some with quiet large amount of data that we don't need, that's why we want to run a query using a precisely defined selection of columns, which may get dynamically extended by some function, otherwise we would have hard-coded these column names of course).
OS: Ubuntu 18.04
PostgreSQL: 13.3 (Debian 13.3-1.pgdg100+1)

The '%s' insertion will try to turn every argument into an SQL string, as #AdamKG pointed out. Instead, you can use the psycopg2.sql module will allow you to insert identifiers into queries, not just strings:
from psycopg2 import sql
fields = ('id', 'name', 'type', 'price', 'warehouse', 'location', )
sql_query = sql.SQL(
"""SELECT {} FROM product p
INNER JOIN owner o ON p.id = o.product_id
WHERE p.id = %s AND (o.dateof_purchase IS NOT NULL
OR o.state = 'checked_out' );""")
try:
with psycopg2.connect(**DB_PARAMS) as conn:
with conn.cursor(cursor_factory=DictCursor) as curs:
# passing a tuple as it only accept one more argument after the query!
curs.execute(sql_query.format(*[sql.Identifier(field) for field in fields]), (*fields, myid))
row = curs.fetchone()

I finally found a solution. It makes use of map to use a list or a tuple of column names and sql.Literal to use a given id, this is maybe cleaner:
conn = psycopg2.connect(**DB_PARAMS)
myid = 100
# using the simple column identifiers
fields_1 = ('id', 'name', 'type', 'price', 'warehouse', 'location',)
# using the dot notation with the table alias 'p' as the prefix:
fields_2 = ('p.id', 'p.name', 'p.type', 'p.price', 'p.warehouse', 'p.location',)
sql_query_1 = sql.SQL("""
SELECT {f} FROM product p
INNER JOIN owner o ON p.id = o.product_id
WHERE p.id = {j} AND (o.dateof_purchase IS NOT NULL
OR o.state = 'checked_out' );"""
).format(
f = sql.SQL(',').join(map(sql.Identifier, fields_1)),
j = sql.Literal(myid)
)
sql_query_2 = sql.SQL("""
SELECT {f} FROM product p
INNER JOIN owner o ON p.id = o.product_id
WHERE p.id = {j} AND (o.dateof_purchase IS NOT NULL
OR o.state = 'checked_out' );"""
).format(
f = sql.SQL(',').join(map(sql.SQL, fields_2)), # use sql.SQL!
j = sql.Literal(myid)
)
sql_query_2b = sql.SQL("""
SELECT {f} FROM product p
INNER JOIN owner o ON p.id = o.product_id
WHERE p.id = {j} AND (o.dateof_purchase IS NOT NULL
OR o.state = 'checked_out' );"""
).format(
f = sql.SQL(',').join(map(sql.Identifier, fields_2)), # DON'T use sql.Identifier!
j = sql.Literal(myid)
)
# VALID SQL QUERY:
print(sql_query_1.as_string(conn))
# will print:
# SELECT "id","name","type","price","warehouse","location" FROM product p
# INNER JOIN owner o ON p.id = o.product_id
# WHERE p.id = 100 AND (o.dateof_purchase IS NOT NULL
# OR o.state = 'checked_out' );
# VALID SQL QUERY:
print(sql_query_2.as_string(conn))
# will print:
# SELECT p.id,p.name,p.type,p.price,p.warehouse,p.location FROM product p
# INNER JOIN owner o ON p.id = o.product_id
# WHERE p.id = 100 AND (o.dateof_purchase IS NOT NULL
# OR o.state = 'checked_out' );
# /!\ INVALID SQL QUERY /!\:
print(sql_query_2b.as_string(conn))
# will print:
# SELECT "p.id","p.name","p.type","p.price","p.warehouse","p.location" FROM product p
# INNER JOIN owner o ON p.id = o.product_id
# WHERE p.id = 100 AND (o.dateof_purchase IS NOT NULL
# OR o.state = 'checked_out' );
But because of that:
simple columns names are evaluated correctly when in double quotes, eg. id is equivalent to "id", name is equivalent to "name" for PostgreSQL,
column name, when prefixed with the dot notation using the table alias or identifier, e.g. p.id or product.id instead of just id or "id" will miserably fail with the following error:
UndefinedColumn: column "p.id" does not exist
LINE 1: SELECT "p.id","p.type","p.price","p.warehouse","p.location",...
^
HINT: Perhaps you meant to reference the column "p.id".

Python MYSQL multiple statement execution issue

Suppose I have this multiple-process Python-MySql query:
self.calculateLeadTime = ("""SET #lastDate = (SELECT sessionDate FROM stock
WHERE product = (%s)
ORDER BY stocksessionID DESC LIMIT 1);
SET #secondLastDate = (SELECT sessionDate FROM stock WHERE product = (%s)
ORDER BY stocksessionID DESC LIMIT 1, 1);
SET #leadTime = (SELECT DATEDIFF(#lastDate, #secondLastDate));
SET #lastStockSessionID = (SELECT stocksessionID
FROM stock WHERE product = (%s) ORDER BY stocksessionID DESC LIMIT 1);
UPDATE stock SET leadTime = (#leadTime)
WHERE stocksessionID = #lastStockSessionID;""", (self.productID.get(), self.productID.get(), self.productID.get()))
self.query = self.cur.execute(self.calculateLeadTime, multi=True)
for self.cur in self.results:
print('cursor:', self.cur)
if self.cur.with_rows:
print('result:', self.cur.fetchall())
self.cn.commit()
I am subject to the error:
stmt = operation.encode(self._connection.python_charset)
AttributeError: 'tuple' object has no attribute 'encode'
I have read the MySql Python documentation regarding multi=True when executing multiple SQL statements via Python. However, my implementation does not work. Any ideas?

Currently, you are passing a tuple as first argument in the cursor.execute call when it expects a single scalar query string in first argument and tuple/list of parameters in second argument:
self.calculateLeadTime = """SET #lastDate = (SELECT sessionDate
FROM stock
WHERE product = (%s)
ORDER BY stocksessionID DESC LIMIT 1);
SET #secondLastDate = (SELECT sessionDate
FROM stock WHERE product = (%s)
ORDER BY stocksessionID DESC LIMIT 1, 1);
SET #leadTime = (SELECT DATEDIFF(#lastDate, #secondLastDate));
SET #lastStockSessionID = (SELECT stocksessionID
FROM stock WHERE product = (%s)
ORDER BY stocksessionID DESC LIMIT 1);
UPDATE stock
SET leadTime = (#leadTime)
WHERE stocksessionID = #lastStockSessionID;
"""
self.query = self.cur.execute(self.calculateLeadTime,
params=(self.productID.get(), self.productID.get(), self.productID.get()),
multi=True)
By the way, MySQL supports JOIN in UPDATE statements for a single statement. Run below with two parameters:
UPDATE stock s
INNER JOIN
(SELECT product, MAX(stocksessionID) AS MaxID
FROM stock
WHERE product = %s
GROUP BY product
) agg_id
ON s.stocksessionID = agg_id.MaxID
INNER JOIN
(SELECT product, MAX(sessionDate) As MaxDate
FROM stock
GROUP BY product) max_dt
ON max_dt.product = s.product
INNER JOIN
(SELECT product, MAX(sessionDate) As SecMaxDate
FROM stock
WHERE sessionDate < ( SELECT MAX(sessionDate)
FROM stock
WHERE product = %s )
GROUP BY product
) sec_max_dt
ON max_dt.product = max_dt.product
SET leadTime = DATEDIFF(MaxDate, SecMaxDate);

I am unsure, if this will work or counts even as multiple queries, because Multi = true is for multiple Resultsets from SELECTS.
You must add teh data for the %s
self.query = self.cur.execute(self.calculateLeadTime,('product','product','product') , multi=True)

How to limit list in python to show N records for each unique row?

I am trying to limit mysql output of query to show only Top N records for each genre. This is my code :
def selectTopNactors(n):
# Create a new connection
con=connection()
# Create a cursor on the connection
cur=con.cursor()
#execute query
int(n)
sql ="""SELECT g.genre_name, a.actor_id,COUNT(mg.genre_id) as num_mov
FROM actor as a, role as r,movie as m,genre as g, movie_has_genre as mg
WHERE a.actor_id = r.actor_id AND m.movie_id = r.movie_id
AND m.movie_id = mg.movie_id AND g.genre_id = mg.genre_id
AND (g.genre_id, m.movie_id) IN (SELECT g.genre_id, m.movie_id
FROM movie as m, genre as g, movie_has_genre as mg
WHERE m.movie_id = mg.movie_id AND mg.genre_id = g.genre_id
ORDER BY g.genre_id)
GROUP BY g.genre_name, a.actor_id
ORDER BY g.genre_name, COUNT(*) desc """
cur.execute(sql)
results = cur.fetchall()
listab = []
listac = []
for row in results:
lista = []
lista.append(row[0])
lista.append(row[1])
lista.append(row[2])
listab = tuple(lista)
listac.append(listab)
head = ("genreName","actorId","numberOfMovies")
listac.insert(0,head)
print (n)
con.commit()
return listac
And the list which is returning it is huge (6000+) records, So I want to show only N records for each genre.
returned list is here

In version of MySQL before 8.0, we can emulate analytic functions using user-defined variables in carefully crafted queries. Note that we are depending on behavior of user-defined variables that is not guaranteed (documented in the MySQL Reference Manual).
SELECT #rn := IF(c.genre_name=#prev_genre,#rn+1,1) AS rn
, #prev_genre := c.genre_name AS genre_name
, c.actor_id AS actor_id
, c.num_mov AS num_mov
FROM ( SELECT #prev_genre := NULL, #rn := 0 ) i
CROSS
JOIN ( SELECT g.genre_name
, a.actor_id
, COUNT(1) AS num_mov
FROM actor a
JOIN role r
ON r.actor_id = a.actor_id
JOIN movie m
ON m.movie_id = r.movie_id
JOIN movie_has_genre mg
ON mg.movie_id = m.movie_id
JOIN genre g
ON g.genre_id = mg.genre_id
GROUP
BY g.genre_name
, a.actor_id
ORDER
BY g.genre_name
, COUNT(1) DESC
, a.actor_id
) c
ORDER
BY c.genre_name
, c.num_mov DESC
, c.actor_id
HAVING rn <= 4
The literal 4 at the end of the query represents the value N in the question.
In MySQL 8.0, we can use the newly introduced analytic functions, to obtain an equivalent result:
SELECT ROW_NUMBER() OVER(PARTITION BY c.genre_name ORDER BY c.num_mov DESC, c.actor_id)
AS rn
, c.genre_name AS genre_name
, c.actor_id AS actor_id
, c.num_mov AS num_mov
FROM ( SELECT g.genre_name
, a.actor_id
, COUNT(1) AS num_mov
FROM actor a
JOIN role r
ON r.actor_id = a.actor_id
JOIN movie m
ON m.movie_id = r.movie_id
JOIN movie_has_genre mg
ON mg.movie_id = m.movie_id
JOIN genre g
ON g.genre_id = mg.genre_id
GROUP
BY g.genre_name
, a.actor_id
ORDER
BY g.genre_name
, COUNT(1) DESC
, a.actor_id
) c
ORDER
BY c.genre_name
, c.num_mov DESC
, c.actor_id
HAVING rn <= 4