Develop Reference

Insert query not getting executed from sqlalchemy with parameters [duplicate]

How can I call stored procedures of sql server with sqlAlchemy?

Engines and Connections have an execute() method you can use for arbitrary sql statements, and so do Sessions. For example:
results = sess.execute('myproc ?, ?', [param1, param2])
You can use outparam() to create output parameters if you need to (or for bind parameters use bindparam() with the isoutparam=True option)

context: I use flask-sqlalchemy with MySQL and without ORM-mapping. Usually, I use:
# in the init method
_db = SqlAlchemy(app)
#... somewhere in my code ...
_db.session.execute(query)
Calling stored procedures is not supported out of the box: the callproc is not generic, but specific to the mysql connector.
For stored procedures without out params, it is possible to execute a query like
_db.session.execute(sqlalchemy.text("CALL my_proc(:param)"), param='something')
as usual. Things get more complicated when you have out params...
One way to use out params is to access the underlying connector is through engine.raw_connection(). For example:
conn = _db.engine.raw_connection()
# do the call. The actual parameter does not matter, could be ['lala'] as well
results = conn.cursor().callproc('my_proc_with_one_out_param', [0])
conn.close() # commit
print(results) # will print (<out param result>)
This is nice since we are able to access the out parameter, BUT this connection is not managed by the flask session. This means that it won't be committed/aborted as with the other managed queries... (problematic only if your procedure has side-effect).
Finally, I ended up doing this:
# do the call and store the result in a local mysql variabl
# the name does not matter, as long as it is prefixed by #
_db.session.execute('CALL my_proc_with_one_out_param(#out)')
# do another query to get back the result
result = _db.session.execute('SELECT #out').fetchone()
The result will be a tuple with one value: the out param. This is not ideal, but the least dangerous: if another query fails during the session, the procedure call will be aborted (rollback) as well.

Just execute procedure object created with func:
from sqlalchemy import create_engine, func
from sqlalchemy.orm import sessionmaker
engine = create_engine('sqlite://', echo=True)
print engine.execute(func.upper('abc')).scalar() # Using engine
session = sessionmaker(bind=engine)()
print session.execute(func.upper('abc')).scalar() # Using session

The easiest way to call a stored procedure in MySQL using SQLAlchemy is by using callproc method of Engine.raw_connection(). call_proc will require the procedure name and parameters required for the stored procedure being called.
def call_procedure(function_name, params):
connection = cloudsql.Engine.raw_connection()
try:
cursor = connection.cursor()
cursor.callproc(function_name, params)
results = list(cursor.fetchall())
cursor.close()
connection.commit()
return results
finally:
connection.close()

Supposing you already have session created with sessionmaker(), you can use following function:
def exec_procedure(session, proc_name, params):
sql_params = ",".join(["#{0}={1}".format(name, value) for name, value in params.items()])
sql_string = """
DECLARE #return_value int;
EXEC #return_value = [dbo].[{proc_name}] {params};
SELECT 'Return Value' = #return_value;
""".format(proc_name=proc_name, params=sql_params)
return session.execute(sql_string).fetchall()
Now you can execute your stored procedure 'MyProc' with parameters simply like that:
params = {
'Foo': foo_value,
'Bar': bar_value
}
exec_procedure(session, 'MyProc', params)

Out of desperate need for a project of mine, I wrote a function that handles Stored Procedure calls.
Here you go:
import sqlalchemy as sql
def execute_db_store_procedure(database, types, sql_store_procedure, *sp_args):
""" Execute the store procedure and return the response table.
Attention: No injection checking!!!
Does work with the CALL syntax as of yet (TODO: other databases).
Attributes:
database -- the database
types -- tuple of strings of SQLAlchemy type names.
Each type describes the type of the argument
with the same number.
List: http://docs.sqlalchemy.org/en/rel_0_7/core/types.html
sql_store_procudure -- string of the stored procedure to be executed
sp_args -- arguments passed to the stored procedure
"""
if not len(types) == len(sp_args):
raise ValueError("types tuple must be the length of the sp args.")
# Construch the type list for the given types
# See
# http://docs.sqlalchemy.org/en/latest/core/sqlelement.html?highlight=expression.text#sqlalchemy.sql.expression.text
# sp_args (and their types) are numbered from 0 to len(sp_args)-1
type_list = [sql.sql.expression.bindparam(
str(no), type_=getattr(sql.types, typ)())
for no, typ in zip(range(len(types)), types)]
try:
# Adapts to the number of arguments given to the function
sp_call = sql.text("CALL `%s`(%s)" % (
sql_store_procedure,
", ".join([":%s" % n for n in range(len(sp_args))])),
bindparams=type_list
)
#raise ValueError("%s\n%s" % (sp_call, type_list))
with database.engine.begin() as connection:
return connection.execute(
sp_call,
# Don't do this at home, kids...
**dict((str(no), arg)
for (no, arg) in zip(range(len(sp_args)), sp_args)))
except sql.exc.DatabaseError:
raise
It works with the CALL syntax, so MySQL should work as expected. MSSQL uses EXEC instead of call and a little differennt syntax, I guess. So making it server agnostic is up to you but shouldn’t be too hard.

Another workaround:
query = f'call Procedure ("{#param1}", "{#param2}", "{#param3}")'
sqlEngine = sqlalchemy.create_engine(jdbc)
conn = sqlEngine.connect()
df = pd.read_sql(query,conn,index_col=None)

I had a stored procedure for postgresql with following signature -
CREATE OR REPLACE PROCEDURE inc_run_count(
_host text,
_org text,
_repo text,
_rule_ids text[]
)
After quite a few error and trial, I found this is how to call the procedure from python3.
def update_db_rule_count(rule_ids: List[str], host: str, org: str, repo: str):
param_dict = {"host": host, "org": org, "repo": repo, "rule_ids": f'{{ {",".join(rule_ids)} }}'}
with AnalyticsSession() as analytics_db:
analytics_db.execute('call inc_run_count(:host, :org, :repo, :rule_ids)', param_dict)
analytics_db.commit()

SQLAlchemy does not update/expire model instances with external changes

Recently I came across strange behavior of SQLAlchemy regarding refreshing/populating model instances with the the changes that were made outside of the current session. I created the following minimal working example and was able to reproduce problem with it.
from time import sleep
from sqlalchemy import orm, create_engine, Column, BigInteger, Integer
from sqlalchemy.ext.declarative import declarative_base
DATABASE_URI = "postgresql://{user}:{password}#{host}:{port}/{name}".format(
user="postgres",
password="postgres",
host="127.0.0.1",
name="so_sqlalchemy",
port="5432",
)
class SQLAlchemy:
def __init__(self, db_url, autocommit=False, autoflush=True):
self.engine = create_engine(db_url)
self.session = None
self.autocommit = autocommit
self.autoflush = autoflush
def connect(self):
session_maker = orm.sessionmaker(
bind=self.engine,
autocommit=self.autocommit,
autoflush=self.autoflush,
expire_on_commit=True
)
self.session = orm.scoped_session(session_maker)
def disconnect(self):
self.session.flush()
self.session.close()
self.session.remove()
self.session = None
BaseModel = declarative_base()
class TestModel(BaseModel):
__tablename__ = "test_models"
id = Column(BigInteger, primary_key=True, nullable=False)
field = Column(Integer, nullable=False)
def loop(db):
while True:
with db.session.begin():
t = db.session.query(TestModel).with_for_update().get(1)
if t is None:
print("No entry in db, creating...")
t = TestModel(id=1, field=0)
db.session.add(t)
db.session.flush()
print(f"t.field value is {t.field}")
t.field += 1
print(f"t.field value before flush is {t.field}")
db.session.flush()
print(f"t.field value after flush is {t.field}")
print(f"t.field value after transaction is {t.field}")
print("Sleeping for 2 seconds.")
sleep(2.0)
def main():
db = SQLAlchemy(DATABASE_URI, autocommit=True, autoflush=True)
db.connect()
try:
loop(db)
except KeyboardInterrupt:
print("Canceled")
if __name__ == '__main__':
main()
My requirements.txt file looks like this:
alembic==1.0.10
psycopg2-binary==2.8.2
sqlalchemy==1.3.3
If I run the script (I use Python 3.7.3 on my laptop running Ubuntu 16.04), it will nicely increment a value every two seconds as expected:
t.field value is 0
t.field value before flush is 1
t.field value after flush is 1
t.field value after transaction is 1
Sleeping for 2 seconds.
t.field value is 1
t.field value before flush is 2
t.field value after flush is 2
t.field value after transaction is 2
Sleeping for 2 seconds.
...
Now at some point I open postgres database shell and begin another transaction:
so_sqlalchemy=# BEGIN;
BEGIN
so_sqlalchemy=# UPDATE test_models SET field=100 WHERE id=1;
UPDATE 1
so_sqlalchemy=# COMMIT;
COMMIT
As soon as I press Enter after the UPDATE query, the script blocks as expected, as I'm issuing SELECT ... FOR UPDATE query there. However, when I commit the transaction in the database shell, script continues from the previous value (say, 27) and does not detect that external transaction has changed the value of field in database to 100.
My question is, why does this happen at all? There are several factors that seem to contradict the current behavior:
I'm using expire_on_commit setting set to True, which seems to imply that every model instance that has been used in transaction will be marked as expired after the transaction has been committed. (Quoting documentation, "When True, all instances will be fully expired after each commit(), so that all attribute/object access subsequent to a completed transaction will load from the most recent database state.").
I'm not accessing some old model instance but rather issue completely new query every time. As far as I understand, this should lead to direct query to the database and not access cached instance. I can confirm that this is indeed the case if I turn sqlalchemy debug log on.
The quick and dirty fix for this problem is to call db.session.expire_all() right after the transaction has begun, but this seems very inelegant and counter-intuitive. I would be very glad to understand what's wrong with the way I'm working with sqlalchemy here.

I ran into a very similar situation with MySQL. I needed to "see" changes to the table that were coming from external sources in the middle of my code's database operations. I ended up having to set autocommit=True in my session call and use the begin() / commit() methods of the session to "see" data that was updated externally.
The SQLAlchemy docs say this is a legacy configuration:
Warning
“autocommit” mode is a legacy mode of use and should not be considered for new projects.
but also say in the next paragraph:
Modern usage of “autocommit mode” tends to be for framework integrations that wish to control specifically when the “begin” state occurs
So it doesn't seem to be clear which statement is correct.

Close SQLAlchemy connection

I have the following function in python:
def add_odm_object(obj, table_name, primary_key, unique_column):
db = create_engine('mysql+pymysql://root:#127.0.0.1/mydb')
metadata = MetaData(db)
t = Table(table_name, metadata, autoload=True)
s = t.select(t.c[unique_column] == obj[unique_column])
rs = s.execute()
r = rs.fetchone()
if not r:
i = t.insert()
i_res = i.execute(obj)
v_id = i_res.inserted_primary_key[0]
return v_id
else:
return r[primary_key]
This function looks if the object obj is in the database, and if it is not found, it saves it to the DB. Now, I have a problem. I call the above function in a loop many times. And after few hundred times, I get an error: user root has exceeded the max_user_connections resource (current value: 30) I tried to search for answers and for example the question: How to close sqlalchemy connection in MySQL recommends creating a conn = db.connect() object where dbis the engine and calling conn.close() after my query is completed.
But, where should I open and close the connection in my code? I am not working with the connection directly, but I'm using the Table() and MetaData functions in my code.

The engine is an expensive-to-create factory for database connections. Your application should call create_engine() exactly once per database server.
Similarly, the MetaData and Table objects describe a fixed schema object within a known database. These are also configurational constructs that in most cases are created once, just like classes, in a module.
In this case, your function seems to want to load up tables dynamically, which is fine; the MetaData object acts as a registry, which has the convenience feature that it will give you back an existing table if it already exists.
Within a Python function and especially within a loop, for best performance you typically want to refer to a single database connection only.
Taking these things into account, your module might look like:
# module level variable. can be initialized later,
# but generally just want to create this once.
db = create_engine('mysql+pymysql://root:#127.0.0.1/mydb')
# module level MetaData collection.
metadata = MetaData()
def add_odm_object(obj, table_name, primary_key, unique_column):
with db.begin() as connection:
# will load table_name exactly once, then store it persistently
# within the above MetaData
t = Table(table_name, metadata, autoload=True, autoload_with=conn)
s = t.select(t.c[unique_column] == obj[unique_column])
rs = connection.execute(s)
r = rs.fetchone()
if not r:
i_res = connection.execute(t.insert(), some_col=obj)
v_id = i_res.inserted_primary_key[0]
return v_id
else:
return r[primary_key]

Why do I get SQLAlchemy nested rollback error?

I got an error as follows in my code of python (which collects twitter statuses and store in database).
sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back by a nested rollback() call. To begin a new transaction, issue Session.rollback() first.
I want to know what is the problem, why does it occur, and how can I solve it.
I have no idea about nested rollback. Is there any simple example which occurs nested rollback?

The problem was solved.
The point, in this case, is that rollback is not executed until we call rollback explicitly, so when we include commit(), we should write it in a try statement, and write rollback() in the exception statement (in most case) as written in https://docs.sqlalchemy.org/en/13/faq/sessions.html#this-session-s-transaction-has-been-rolled-back-due-to-a-previous-exception-during-flush-or-similar
And, here is the correct code example. I quoted this from the link above.
try:
<use session>
session.commit()
except:
session.rollback()
raise
finally:
session.close() # optional, depends on use case

As identified by #fbessho above, this is indeed the correct pattern:
try:
<use session>
session.commit()
except:
session.rollback()
However, there are some subtleties that can derail the error handling.
In this example (an imaginary unique constraint violation), the rollback does not occur:
class Thing1(Base):
id = Column(BigInteger, primary_key=True)
class Thing2(Base):
id = Column(BigInteger, primary_key=True)
def do_something(s: session, thing_1: Thing1, duplicate_id):
# imagine this violates a unique constraint on Thing2
thing_2 = Thing2(id=duplicate_id)
s.add(thing_2)
try:
# the exception will occur when the commit statement is executed
s.commit()
except Exception as ex:
# this will log details of the exception
logger.error(f"{ex.__class__.__name__}: {ex}")
# referencing thing_1.id will raise a second exception
logger.error(f"Commit failed. Thing1 id was {thing_1.id}.")
s.rollback()
This second Exception occurs even though thing_1 has nothing to do with the failed insert. Merely referencing thing_1 raises a second Exception which prevents the rollback from being executed.
Solution 1
This requires a little more overhead, but will always work.
def do_something_1(s: session, thing_1: Thing1, duplicate_id):
# create a reference that does not rely on the data object
id_for_thing = thing_1.id
# imagine this violates a unique constraint on Thing2
thing_2 = Thing2(id=duplicate_id)
s.add(thing_2)
try:
# the exception will occur when the commit statement is executed
s.commit()
except Exception as ex:
logger.error(f"{ex.__class__.__name__}: {ex}")
# no direct reference to thing_1
logger.error(f"Commit failed. Thing1 id was {id_for_thing}.")
s.rollback()
Solution 2
This will work as long as thing_1 is not affected by the rollback.
def do_something_2(s: session, thing_1: Thing1, duplicate_id):
# imagine this violates a unique constraint on Thing2
thing_2 = Thing2(id=duplicate_id)
s.add(thing_2)
try:
# the exception will occur when the commit statement is executed
s.commit()
except Exception as ex:
logger.error(f"{ex.__class__.__name__}: {ex}")
s.rollback()
# thing_1.id can be referenced after rollback
logger.error(f"Commit failed. Thing1 id was {thing_1.id}.")

Python call constructor in a member function

Let's take for example this class, which is extending MySQLDB's connection object.
class DBHandler(mysql.connections.Connection):
def __init__(self,cursor=None):
if cursor == None:
cursor = 'DictCursor'
super(DBHandler,self).__init__(host = db_host,
user = db_user,
passwd = db_pass,
db = db,
cursorclass=getattr(mysql.cursors, cursor))
def getall(self,q,params=None):
try:
cur = self.cursor()
cur.execute(q,params)
res = cur.fetchall()
return res
except mysql.OperationalError:
#this is the line in question
pass
def execute(self,q,params):
cur = self.cursor()
cur.execute(q,params)
self.commit()
return cur.lastrowid
This thing is largely a convenience to get simpler access to common required queries.
On the line marked with the comment, is it possible in Python to recall the object constructor, even though this is a member function? I use this example to illustrate because it would effectively reestablish the connection in the event it is dropped on timeout before a query is run.
I'm aware of MySQLdb's ping() method, this is really just a question of capability. In python, Is it possible to call a constructor from within a member function called on an instance to re-initialize that instance? Thanks!

Yes, you can, since it would be preferable to extract your initialization code in another method (a def init(self):).
This is because __init__ is not really the constructor of the object, it is more the "initializer" of your instance, the real constructor is the __new__ method, that is responsible of the instance creation.