Develop Reference

Why Does SQLAlchemy Label Columns in Query

When I make a query in SQLAlchemy, I noticed that the queries use the AS keyword for each column. It sets the alias_name = column_name for every column.
For example, if I run the command print(session.query(DefaultLog)), it returns:
Note: DefaultLog is my table object.
SELECT default_log.id AS default_log_id, default_log.msg AS default_log_msg, default_log.logger_time AS default_log_logger_time, default_log.logger_line AS default_log_logger_line, default_log.logger_filepath AS default_log_logger_filepath, default_log.level AS default_log_level, default_log.logger_name AS default_log_logger_name, default_log.logger_method AS default_log_logger_method, default_log.hostname AS default_log_hostname
FROM default_log
Why does it use an alias = original name? Is there some way I can disable this behavior?
Thank you in advance!

Query.statement:
The full SELECT statement represented by this Query.
The statement by default will not have disambiguating labels applied
to the construct unless with_labels(True) is called first.
Using this model:
class DefaultLog(Base):
id = sa.Column(sa.Integer, primary_key=True)
msg = sa.Column(sa.String(128))
logger_time = sa.Column(sa.DateTime)
logger_line = sa.Column(sa.Integer)
print(session.query(DefaultLog).statement) shows:
SELECT defaultlog.id, defaultlog.msg, defaultlog.logger_time, defaultlog.logger_line
FROM defaultlog
print(session.query(DefaultLog).with_labels().statement) shows:
SELECT defaultlog.id AS defaultlog_id, defaultlog.msg AS defaultlog_msg, defaultlog.logger_time AS defaultlog_logger_time, defaultlog.logger_line AS defaultlog_logger_line
FROM defaultlog
You asked:
Why does it use an alias = original name?
From Query.with_labels docs:
...this is commonly used to disambiguate columns from multiple tables which have the same name.
So if you want to issue a single query that calls upon multiple tables, there is nothing stopping those tables having columns that share the same name.
Is there some way I can disable this behavior?
Also from the Query.with_labels docs:
When the Query actually issues SQL to load rows, it always uses column
labeling.
All of the methods that retrieve rows (get(), one(), one_or_none(), all() and iterating over the Query) route through the Query.__iter__() method:
def __iter__(self):
context = self._compile_context()
context.statement.use_labels = True
if self._autoflush and not self._populate_existing:
self.session._autoflush()
return self._execute_and_instances(context)
... where this line hard codes the label usage: context.statement.use_labels = True. So it is "baked in" and can't be disabled.
You can execute the statement without labels:
session.execute(session.query(DefaultLog).statement)
... but that takes the ORM out of the equation.

It is possible to hack sqlachemy Query class to not add labels. But one must be aware that this will breaks when a table is used twice in the query. For example, self join or join thought another table.
from sqlalchemy.orm import Query
class MyQuery(Query):
def __iter__(self):
"""Patch to disable auto labels"""
context = self._compile_context(labels=False)
context.statement.use_labels = False
if self._autoflush and not self._populate_existing:
self.session._autoflush()
return self._execute_and_instances(context)
And then use it according to mtth answer
sessionmaker(bind=engine, query_cls=MyQuery)

Printing an SQLAlchemy query is tricky and produced not human-friendly output. Not only columns but also bind params are in an odd place.
Here's how to do it correctly:
qry = session.query(SomeTable)
compiled = qry.statement.compile(dialect=session.bind.dialect, compile_kwargs={"literal_binds": True})
print(compiled)
Here's how to fix it for all your future work:
from sqlalchemy.orm import Query
class MyQuery(Query):
def __str__(self):
dialect = self.session.bind.dialect
compiled = self.statement.compile(dialect=dialect, compile_kwargs={"literal_binds": True})
return str(compiled)
To use:
session = sessionmaker(bind=engine, query_cls=MyQuery)()

Python Pony ORM Insert multiple values at once

I'm trying to insert multiple values into my postgres database using Pony ORM. My current approach is very inefficient:
from pony.orm import *
db = Database()
class Names(db.Entity):
first_name = Optional(str)
last_name = Optional(str)
family = [["Peter", "Mueller"], ["Paul", "Meyer"], ...]
#db_session
def populate_names(name_list)
for name in name_list:
db.insert("Names", first_name=name[0], last_name=name[1])
if __name__ == "__main__":
db.bind(provider='postgres', user='', password='', host='', database='')
db.generate_mappings(create_tables=True)
populate_names(family)
This is just a short example but the structure of the input is the same:
a list of lists.
I'm extracting the data from several xml files and insert one "file" at a time.
Does anyone has an idea on how to put several rows of data into one insert query in Pony ORM?

Pony doesn't provide something special for this, you can use execute_values from psycopg2.extras. Get connection object from db to use it.
from psycopg2.extras import execute_values
...
names = [
('はると', '一温'),
('りく', '俐空'),
('はる', '和晴'),
('ひなた', '向日'),
('ゆうと', '佑篤')
]
#db_session
def populate_persons(names):
sql = 'insert into Person(first_name, last_name) values %s'
con = db.get_connection()
cur = con.cursor()
execute_values(cur, sql, names)
populate_persons(names)
execute_values is in Fast execution helpers list so I think that iе should be the most efficient way.

Currently I'm experimenting with PonyORM for a future project and also came to the conclusion you provided.
The only way on how to insert data in a bulky way is:
# assuming data has this structure:
# [['foo','bar','bazooka'],...]
#db_session
def insert_bulk_array(field1, field2, field3):
MyClass(field1=field1, field2=field2, field3=field3)
# assuming the data is:
# {'field1':'foo','field2':'bar','field3':'bazooka'}
#db_session
def insert_bulk_dict(data)
MyClass(**data)
But from my point of view this is still somehow handy, specially when your data comes as JSON.

There is an open issue in the issue tracker of PonyORM which asks for exactly this feature.
I recommend to vote for it.

executing a raw sql query from sqlalchemy on postgresql

I have a raw sql query which is:
select distinct(user_id) from details_table where event_id in (29,10) and user_id in (7,11,24,45) and epoch_timestamp >= 1433116800 and epoch_timestamp <= 1506816000;
which in psql returns:
user_id
---------
7
24
(2 rows)
Now when i run this raw sql query via sqlalchemy I'm getting a sqlalchemy.engine.result.ResultProxy object in response and not the result as above. The code i'm using right now is as follows:
from flask import current_app
sql_query = text(select distinct(user_id) from details_table where event_id in (29,10) and user_id in (7,24) and epoch_timestamp >= 1433116800 and epoch_timestamp <= 1506816000;)
filtering_users = db.get_engine(current_app, bind='<my_binding>')\
.execute(sql_query)
print(type(filtering_users))
# <class 'sqlalchemy.engine.result.ResultProxy'>
print(filtering_users)
# <sqlalchemy.engine.result.ResultProxy object at 0x7fde74469550>
I used the reference from here but unlike the solution there I'm getting a ResultProxy object.
What am I doing wrong here? My end goal is to get the list of users returned from executing the raw sql-query, stored into a list.

As explained is the SQLAlchemy documentation, the .execute() method returns only a proxy on which you'll have to iterate (or apply any aggregation method) to view the actual result of the query. Apparently, in your case, what you want is the .fetchall() method.
If you try something like this:
from sqlalchemy import create_engine
engine = create_engine('/path/to/your/db...')
connection = engine.connect()
my_query = 'SELECT * FROM my_table'
results = connection.execute(my_query).fetchall()
the results variable will be a list of all the items that the query fetches.
Hope this helps!

How to execute raw SQL in Flask-SQLAlchemy app

How do you execute raw SQL in SQLAlchemy?
I have a python web app that runs on flask and interfaces to the database through SQLAlchemy.
I need a way to run the raw SQL. The query involves multiple table joins along with Inline views.
I've tried:
connection = db.session.connection()
connection.execute( <sql here> )
But I keep getting gateway errors.

Have you tried:
result = db.engine.execute("<sql here>")
or:
from sqlalchemy import text
sql = text('select name from penguins')
result = db.engine.execute(sql)
names = [row[0] for row in result]
print names
Note that db.engine.execute() is "connectionless", which is deprecated in SQLAlchemy 2.0.

SQL Alchemy session objects have their own execute method:
result = db.session.execute('SELECT * FROM my_table WHERE my_column = :val', {'val': 5})
All your application queries should be going through a session object, whether they're raw SQL or not. This ensures that the queries are properly managed by a transaction, which allows multiple queries in the same request to be committed or rolled back as a single unit. Going outside the transaction using the engine or the connection puts you at much greater risk of subtle, possibly hard to detect bugs that can leave you with corrupted data. Each request should be associated with only one transaction, and using db.session will ensure this is the case for your application.
Also take note that execute is designed for parameterized queries. Use parameters, like :val in the example, for any inputs to the query to protect yourself from SQL injection attacks. You can provide the value for these parameters by passing a dict as the second argument, where each key is the name of the parameter as it appears in the query. The exact syntax of the parameter itself may be different depending on your database, but all of the major relational databases support them in some form.
Assuming it's a SELECT query, this will return an iterable of RowProxy objects.
You can access individual columns with a variety of techniques:
for r in result:
print(r[0]) # Access by positional index
print(r['my_column']) # Access by column name as a string
r_dict = dict(r.items()) # convert to dict keyed by column names
Personally, I prefer to convert the results into namedtuples:
from collections import namedtuple
Record = namedtuple('Record', result.keys())
records = [Record(*r) for r in result.fetchall()]
for r in records:
print(r.my_column)
print(r)
If you're not using the Flask-SQLAlchemy extension, you can still easily use a session:
import sqlalchemy
from sqlalchemy.orm import sessionmaker, scoped_session
engine = sqlalchemy.create_engine('my connection string')
Session = scoped_session(sessionmaker(bind=engine))
s = Session()
result = s.execute('SELECT * FROM my_table WHERE my_column = :val', {'val': 5})

docs: SQL Expression Language Tutorial - Using Text
example:
from sqlalchemy.sql import text
connection = engine.connect()
# recommended
cmd = 'select * from Employees where EmployeeGroup = :group'
employeeGroup = 'Staff'
employees = connection.execute(text(cmd), group = employeeGroup)
# or - wee more difficult to interpret the command
employeeGroup = 'Staff'
employees = connection.execute(
text('select * from Employees where EmployeeGroup = :group'),
group = employeeGroup)
# or - notice the requirement to quote 'Staff'
employees = connection.execute(
text("select * from Employees where EmployeeGroup = 'Staff'"))
for employee in employees: logger.debug(employee)
# output
(0, 'Tim', 'Gurra', 'Staff', '991-509-9284')
(1, 'Jim', 'Carey', 'Staff', '832-252-1910')
(2, 'Lee', 'Asher', 'Staff', '897-747-1564')
(3, 'Ben', 'Hayes', 'Staff', '584-255-2631')

You can get the results of SELECT SQL queries using from_statement() and text() as shown here. You don't have to deal with tuples this way. As an example for a class User having the table name users you can try,
from sqlalchemy.sql import text
user = session.query(User).from_statement(
text("""SELECT * FROM users where name=:name""")
).params(name="ed").all()
return user

For SQLAlchemy ≥ 1.4
Starting in SQLAlchemy 1.4, connectionless or implicit execution has been deprecated, i.e.
db.engine.execute(...) # DEPRECATED
as well as bare strings as queries.
The new API requires an explicit connection, e.g.
from sqlalchemy import text
with db.engine.connect() as connection:
result = connection.execute(text("SELECT * FROM ..."))
for row in result:
# ...
Similarly, it’s encouraged to use an existing Session if one is available:
result = session.execute(sqlalchemy.text("SELECT * FROM ..."))
or using parameters:
session.execute(sqlalchemy.text("SELECT * FROM a_table WHERE a_column = :val"),
{'val': 5})
See "Connectionless Execution, Implicit Execution" in the documentation for more details.

result = db.engine.execute(text("<sql here>"))
executes the <sql here> but doesn't commit it unless you're on autocommit mode. So, inserts and updates wouldn't reflect in the database.
To commit after the changes, do
result = db.engine.execute(text("<sql here>").execution_options(autocommit=True))

This is a simplified answer of how to run SQL query from Flask Shell
First, map your module (if your module/app is manage.py in the principal folder and you are in a UNIX Operating system), run:
export FLASK_APP=manage
Run Flask shell
flask shell
Import what we need::
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy(app)
from sqlalchemy import text
Run your query:
result = db.engine.execute(text("<sql here>").execution_options(autocommit=True))
This use the currently database connection which has the application.

Flask-SQLAlchemy v: 3.0.x / SQLAlchemy v: 1.4
users = db.session.execute(db.select(User).order_by(User.title.desc()).limit(150)).scalars()
So basically for the latest stable version of the flask-sqlalchemy specifically the documentation suggests using the session.execute() method in conjunction with the db.select(Object).

Have you tried using connection.execute(text( <sql here> ), <bind params here> ) and bind parameters as described in the docs? This can help solve many parameter formatting and performance problems. Maybe the gateway error is a timeout? Bind parameters tend to make complex queries execute substantially faster.

If you want to avoid tuples, another way is by calling the first, one or all methods:
query = db.engine.execute("SELECT * FROM blogs "
"WHERE id = 1 ")
assert query.first().name == "Welcome to my blog"

How to get inserted_primary_key from db.engine.connect().execute call

I'm using:
CPython 2.7.3,
Flask==0.10.1
Flask-SQLAlchemy==0.16
psycopg2==2.5.1
and
postgresql-9.2
Trying to get PK from insert call with alchemy.
Getting engine like so:
app = Flask(__name__)
app.config.from_envvar('SOME_VAR')
app.wsgi_app = ProxyFix(app.wsgi_app) # Fix for old proxyes
db = SQLAlchemy(app)
And executing insert query in app:
from sqlalchemy import text, exc
def query():
return db.engine.connect().execute(text('''
insert into test...'''), kw)
rv = query()
But trying access inserted_primary_key property, get:
InvalidRequestError: Statement is not an insert() expression construct.
How to enable implicit_returning in my case, reading the docs doesn't help?

You can use the RETURNING clause and handle this yourself:
INSERT INTO test (...) VALUES (...) RETURNING id
Then you can retrieve the id as you normally retrieve values from queries.
Note that this works on Postgres, but does not work on other db engines like MySQL or sqlite.
I don't think there is a db agnostic way to do this within SQLAlchemy without using the ORM functionality.

Is there any reason you do text query instead of normal sqlalchemy insert()? If you're using sqlalchemy it will probably be much easier for you to rephrase your query into:
from sqlalchemy import text, exc, insert
# in values you can put dictionary of keyvalue pairs
# key is the name of the column, value the value to insert
con = db.engine.connect()
ins = tablename.insert().values(users="frank")
res = con.execute(ins)
res.inserted_primary_key
[1]
This way sqlalchemy will do the binding for you.

You can use lastrowid
rv = query()
rv.lastrowid