I am trying to perform a SELECT query with an IN() clause, and have sqlalchemy perform the
parameter escaping for me. I am using pyodbc as my database connector.
This is the code I have written so far:
tables = ['table1', 'table2', ... ]
sql = "SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME IN(:tables)"
result = session.execute(sql, {"tables": tables})
Unfortunatenly this fails with an error:
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('Invalid parameter type. param-index=0 param-type=list', 'HY105')
Is there any way I can have sqlalchemy escape the whole list of parameters and join them with ,
without manually adding a :tableX placeholder for each item of the list?
Try something like this....
DECLARE #string Varchar(100) = 'Table1,table2,table3'
declare #xml xml
set #xml = N'<root><r>' + replace(#string,',','</r><r>') + '</r></root>'
SELECT * FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME IN( select
r.value('.','varchar(max)') as item
from #xml.nodes('//root/r') as records(r)
)
For good reasons it is not possible to expand a list of arguments as you wish.
If you really would like to create a raw SQL query, then you can just enumerate over your list and dynamically create the query:
vals = {"param{}".format(i): table for i, table in enumerate(tables)}
keys = ", ".join([":{}".format(k) for k in vals])
sql = "SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME IN ({keys})".format(keys=keys)
result = session.execute(sql, vals)
for tbl in result:
print(tbl)
But you could ask sqlalchemy to do this for you. Here we make a fake mapping of the INFORMATION_SCHEMA.tables view, and query it using sqlalchemy toolset:
# definition (done just once)
class ISTable(Base):
__tablename__ = 'tables'
__table_args__ = {'schema': 'INFORMATION_SCHEMA'}
_fake_id = Column(Integer, primary_key=True)
table_catalog = Column(String)
table_schema = Column(String)
table_name = Column(String)
table_type = Column(String)
# actual usage
result = session.query(
ISTable.table_catalog, ISTable.table_schema,
ISTable.table_name, ISTable.table_type,
).filter(
ISTable.table_name.in_(tables))
for tbl in result:
print(tbl)
One gotcha: you cannot query for the whole mapped class (like this query(ISTable)) because the primary_key does not exist and an exception will be raised. But querying only columns we can about (as shown above) is good enough for the purpose.
Related
I want to convert this sql query to SQLALCHEMY:
SELECT * FROM dbcloud.client_feedback as a
join (select distinct(max(submitted_on)) sub,pb_channel_id pb, mail_thread_id mail from client_feedback group by pb_channel_id, mail_thread_id) as b
where (a.submitted_on = b.sub and a.pb_channel_id = b.pb) or ( a.submitted_on = b.sub and a.mail_thread_id = b.mail )
I can't find as keyword in SQLALCHEMY
I think that what you may be looking for is .label(name).
Assuming you have a model
class MyModel(db.Model):
id = db.Column(primary_key=True)
name = db.Column()
here is an example of how .label(name) can be used
query = db.session.query(MyModel.name.label('a'))
will produce the SQL
SELECT my_model.name as a FROM my_model
I am trying to run a SQL "SELECT" query in Postgres from Python using Psycopg2. I am trying to compose the query string as below, but getting error message, using psycopg2 version 2.9.
from psycopg2 import sql
tablename = "mytab"
schema = "public"
query = sql.SQL("SELECT table_name from information_schema.tables where table_name = {tablename} and table_schema = {schema};")
query = query.format(tablename=sql.Identifier(tablename), schema=sql.Identifier(schema))
cursor.execute(query)
result = cursor.fetchone()[0]
Error:
psycopg2.error.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block
Can someone please help. Thanks.
In the (a bit strange) query
select table_name
from information_schema.tables
where table_name = 'mytab'
and table_schema = 'public';
'mytab' and 'public' are literals, not identifiers. For comparison, mytab is an identifier here:
select *
from mytab;
Thus your format statement should look like this:
query = query.format(tablename=sql.Literal(tablename), schema=sql.Literal(schema))
Note that the quoted error message is somewhat misleading as it is about executing a query other than what is shown in the question.
Since this query is only dealing with dynamic values it can be simplified to:
import psycopg2
con = psycopg2.connect(<params>)
cursor = con.cursor()
tablename = "mytab"
schema = "public"
# Regular placeholders
query = """SELECT
table_name
from
information_schema.tables
where
table_name = %s and table_schema = %s"""
cursor.execute(query, [tablename, schema])
result = cursor.fetchone()[0]
# Named placeholders
query = """SELECT
table_name
from
information_schema.tables
where
table_name = %(table)s and table_schema = %(schema)s"""
cursor.execute(query, {"table": tablename, "schema": schema})
result = cursor.fetchone()[0]
I have two tables, ProjectData and Label, like this.
class ProjectData(db.Model):
__tablename__ = "project_data"
id = db.Column(db.Integer, primary_key=True)
class Label(db.Model):
__tablename__ = "labels"
id = db.Column(db.Integer, primary_key=True)
data_id = db.Column(db.Integer, db.ForeignKey('project_data.id'))
What I want to do is select all records from ProjectData that are not represented in Label - basically the opposite of a join, or a right outer join, which is not a feature SQLAlchemy offers.
I have tried to do it like this, but it doesn't work.
db.session.query(ProjectData).select_from(Label).outerjoin(
ProjectData
).all()
Finding records in one table with no match in another is known as an anti-join.
You can do this with a NOT EXISTS query:
from sqlalchemy.sql import exists
stmt = exists().where(Label.data_id == ProjectData.id)
q = db.session.query(ProjectData).filter(~stmt)
which generates this SQL:
SELECT project_data.id AS project_data_id
FROM project_data
WHERE NOT (
EXISTS (
SELECT *
FROM labels
WHERE labels.data_id = project_data.id
)
)
Or by doing a LEFT JOIN and filtering for null ids in the other table:
q = (db.session.query(ProjectData)
.outerjoin(Label, ProjectData.id == Label.data_id)
.filter(Label.id == None)
)
which generates this SQL:
SELECT project_data.id AS project_data_id
FROM project_data
LEFT OUTER JOIN labels ON project_data.id = labels.data_id
WHERE labels.id IS NULL
If you know your desired SQL statement to run, you can utilize the 'text' function from sqlalchemy in order to execute a complex query
https://docs.sqlalchemy.org/en/13/core/sqlelement.html
from sqlalchemy import text
t = text("SELECT * "
"FROM users "
"where user_id=:user_id "
).params(user_id=user_id)
results = db.session.query(t)
I have an SQL query that inserts a row into a table. One of the columns is assigned a value returned from sub-query that aggregates multiple values using string_agg. How can this be written using SQLAlchemy ORM?
The postgres query
INSERT INTO blobs (_id, data) VALUES (nextval('blobs_id_seq'),
(SELECT string_agg(blobs.data, NULL ORDER BY blobs._id DESC)
FROM blobs where blobs._id IN (1,2) ) );
The table ORM
class Blob(Base):
__tablename__ = 'blobs'
_id = Column(Integer, Sequence('blobs_id_seq'), primary_key=True, unique=True)
data = Column(LargeBinary)
Base.metadata.create_all(engine)
db = Session(engine)
db.add(Blob(data=b'aa'))
db.add(Blob(data=b'bb'))
db.add(Blob(data=b'cc'))
db.commit()
Invalid ORM insert
# Should yield a "Blob" with data = b"bbaa"
db.add(Blob(data=db.query(func.string_agg(Blob.data, None))
.filter( Blob._id.in_([1, 2]) )
.order_by( Blob._id.desc() ) ))
It turns out there is a dialect specific helper aggregate_order_by that renders a query that outputs the expected results.
db.add(Blob(data=db.query(func.string_agg(Blob.data,
aggregate_order_by(None, Blob._id.desc())))
.filter( Blob._id.in_([1, 2]) )))
I use sqlalchemy to make changes to a table in SQL Server database, and would like to get back number of affected rows.
I know there is .rowcount attribute to ResultProxy, but as, for example, this answer is demonstrating .rowcount is not necessarily the same as number of affected rows.
SQL Server uses ##ROWCOUNT to access number of affected rows from the previous statement execution.
Is there a way to modify an sqlalchemy expression that uses insert / update statement to end with SELECT ##ROWCOUNT?
For example, given:
from sqlalchemy import Table, Column, Integer, String, MetaData, create_engine
url = 'mssql+pyodbc://dsn'
engine = create_engine(url)
metadata = MetaData()
users = Table('users', metadata,
Column('id', Integer, primary_key=True),
Column('name', String),
Column('fullname', String),
)
ins = users.insert().values(name='jack', fullname='Jack Jones')
upd1 = users.update().values(fullname='Jack Doe').where(users.c.name == 'jack')
upd2 = users.update().values(fullname='Jack Doe').where(users.c.name == 'jack')
I could prepend SELECT ##ROWCOUNT to an update statement:
sel = select([text('##ROWCOUNT')])
sql1 = sel.suffix_with(upd2)
print(sql1.compile(engine, compile_kwargs={"literal_binds": True}))
Yielding "wrong" query:
SELECT ##ROWCOUNT UPDATE users SET fullname='Jack Doe' WHERE users.name = 'jack'
Trying to do the "right" thing:
sql2 = upd2.suffix_with(sel)
Raises AttributeError since 'Update' object has no attribute 'suffix_with'.
So is there a way to get desired sql query:
UPDATE users SET fullname='Jack Doe' WHERE users.name = 'jack';
SELECT ##ROWCOUNT
Using sql expression language without fully textual constructs.