How can I call stored procedures of sql server with sqlAlchemy?
Engines and Connections have an execute() method you can use for arbitrary sql statements, and so do Sessions. For example:
results = sess.execute('myproc ?, ?', [param1, param2])
You can use outparam() to create output parameters if you need to (or for bind parameters use bindparam() with the isoutparam=True option)
context: I use flask-sqlalchemy with MySQL and without ORM-mapping. Usually, I use:
# in the init method
_db = SqlAlchemy(app)
#... somewhere in my code ...
_db.session.execute(query)
Calling stored procedures is not supported out of the box: the callproc is not generic, but specific to the mysql connector.
For stored procedures without out params, it is possible to execute a query like
_db.session.execute(sqlalchemy.text("CALL my_proc(:param)"), param='something')
as usual. Things get more complicated when you have out params...
One way to use out params is to access the underlying connector is through engine.raw_connection(). For example:
conn = _db.engine.raw_connection()
# do the call. The actual parameter does not matter, could be ['lala'] as well
results = conn.cursor().callproc('my_proc_with_one_out_param', [0])
conn.close() # commit
print(results) # will print (<out param result>)
This is nice since we are able to access the out parameter, BUT this connection is not managed by the flask session. This means that it won't be committed/aborted as with the other managed queries... (problematic only if your procedure has side-effect).
Finally, I ended up doing this:
# do the call and store the result in a local mysql variabl
# the name does not matter, as long as it is prefixed by #
_db.session.execute('CALL my_proc_with_one_out_param(#out)')
# do another query to get back the result
result = _db.session.execute('SELECT #out').fetchone()
The result will be a tuple with one value: the out param. This is not ideal, but the least dangerous: if another query fails during the session, the procedure call will be aborted (rollback) as well.
Just execute procedure object created with func:
from sqlalchemy import create_engine, func
from sqlalchemy.orm import sessionmaker
engine = create_engine('sqlite://', echo=True)
print engine.execute(func.upper('abc')).scalar() # Using engine
session = sessionmaker(bind=engine)()
print session.execute(func.upper('abc')).scalar() # Using session
The easiest way to call a stored procedure in MySQL using SQLAlchemy is by using callproc method of Engine.raw_connection(). call_proc will require the procedure name and parameters required for the stored procedure being called.
def call_procedure(function_name, params):
connection = cloudsql.Engine.raw_connection()
try:
cursor = connection.cursor()
cursor.callproc(function_name, params)
results = list(cursor.fetchall())
cursor.close()
connection.commit()
return results
finally:
connection.close()
Supposing you already have session created with sessionmaker(), you can use following function:
def exec_procedure(session, proc_name, params):
sql_params = ",".join(["#{0}={1}".format(name, value) for name, value in params.items()])
sql_string = """
DECLARE #return_value int;
EXEC #return_value = [dbo].[{proc_name}] {params};
SELECT 'Return Value' = #return_value;
""".format(proc_name=proc_name, params=sql_params)
return session.execute(sql_string).fetchall()
Now you can execute your stored procedure 'MyProc' with parameters simply like that:
params = {
'Foo': foo_value,
'Bar': bar_value
}
exec_procedure(session, 'MyProc', params)
Out of desperate need for a project of mine, I wrote a function that handles Stored Procedure calls.
Here you go:
import sqlalchemy as sql
def execute_db_store_procedure(database, types, sql_store_procedure, *sp_args):
""" Execute the store procedure and return the response table.
Attention: No injection checking!!!
Does work with the CALL syntax as of yet (TODO: other databases).
Attributes:
database -- the database
types -- tuple of strings of SQLAlchemy type names.
Each type describes the type of the argument
with the same number.
List: http://docs.sqlalchemy.org/en/rel_0_7/core/types.html
sql_store_procudure -- string of the stored procedure to be executed
sp_args -- arguments passed to the stored procedure
"""
if not len(types) == len(sp_args):
raise ValueError("types tuple must be the length of the sp args.")
# Construch the type list for the given types
# See
# http://docs.sqlalchemy.org/en/latest/core/sqlelement.html?highlight=expression.text#sqlalchemy.sql.expression.text
# sp_args (and their types) are numbered from 0 to len(sp_args)-1
type_list = [sql.sql.expression.bindparam(
str(no), type_=getattr(sql.types, typ)())
for no, typ in zip(range(len(types)), types)]
try:
# Adapts to the number of arguments given to the function
sp_call = sql.text("CALL `%s`(%s)" % (
sql_store_procedure,
", ".join([":%s" % n for n in range(len(sp_args))])),
bindparams=type_list
)
#raise ValueError("%s\n%s" % (sp_call, type_list))
with database.engine.begin() as connection:
return connection.execute(
sp_call,
# Don't do this at home, kids...
**dict((str(no), arg)
for (no, arg) in zip(range(len(sp_args)), sp_args)))
except sql.exc.DatabaseError:
raise
It works with the CALL syntax, so MySQL should work as expected. MSSQL uses EXEC instead of call and a little differennt syntax, I guess. So making it server agnostic is up to you but shouldn’t be too hard.
Another workaround:
query = f'call Procedure ("{#param1}", "{#param2}", "{#param3}")'
sqlEngine = sqlalchemy.create_engine(jdbc)
conn = sqlEngine.connect()
df = pd.read_sql(query,conn,index_col=None)
I had a stored procedure for postgresql with following signature -
CREATE OR REPLACE PROCEDURE inc_run_count(
_host text,
_org text,
_repo text,
_rule_ids text[]
)
After quite a few error and trial, I found this is how to call the procedure from python3.
def update_db_rule_count(rule_ids: List[str], host: str, org: str, repo: str):
param_dict = {"host": host, "org": org, "repo": repo, "rule_ids": f'{{ {",".join(rule_ids)} }}'}
with AnalyticsSession() as analytics_db:
analytics_db.execute('call inc_run_count(:host, :org, :repo, :rule_ids)', param_dict)
analytics_db.commit()
I'm running into an error when I try to instantiate an object from a cursor in SQLite and I've exhausted my research and couldn't find a solution.
Premise: I cannot use SqlAlchemy or anything of that sorts.
Assumption: The database (SQLite) works, it contains a table named table_cars, and the table is populated with data in its single column: name.
So, I have a class lets say:
class Car():
def __init__(self, name):
self.name = name
#classmethod
def from_cursor(cls, c):
car = cls(c(0))
# this line breaks when called from the function below.
And I also have a db module, with the following function:
def get_cars_from_db():
sql = 'SELECT * FROM table_cars;'
conn = get_conn()
cur = conn.cursor()
cur.execute(sql)
data = cur.fetchall()
# at this point, if i print the cursor, I can see all data, so far so good.
cars = [Car.from_cursor(c) for c in data]
# the line above causes the code to break
return cars
The code breaks with the following error:
TypeError: 'tuple' object is not callable
What am I doing wrong here?
You can use cls(c[0]) or cls(*c) to unpack tuple to function arguments.
It's also worth to specify an exact order of your columns in query.
select name from table_cars
I'd like to access a field called timeCreated on a mapped object. The field is instantiated via a utcnow() FunctionElement (taken from this example).
After doing a merge or add call, committing, then closing the session, I've noticed that I get the below error when accessing the field. I have expire_on_commit set to False.
sqlalchemy.orm.exc.DetachedInstanceError: Instance <User at 0x102046710> is not bound to a Session; attribute refresh operation cannot proceed (Background on this error at: http://sqlalche.me/e/13/bhk3)
Example code:
def write(obj):
with sessionScope() as session:
obj.timeCreated = utcnow()
ret = session.merge(obj)
return ret
user = User(name='Totoro')
savedUser = write(user)
# Error occurs when accessing timeCreated
print(savedUser.timeCreated)
SessionScope() is taken from these docs, it's defined as:
sessionFactory = sessionmaker(bind=engine)
#contextmanager
def sessionScope():
try:
session = sessionFactory()
yield session
session.commit()
except Exception as e:
session.rollback()
raise
finally:
session.close()
return
Is there a reason why timeCreated is not resolved after commit()? If, after committing, but before closing, I access timeCreated, then subsequent accesses after close still work.
Is there a way to "eager" load this type of column?
The problem is that when timeCreated is assigned to the result of calling utcnow, SQLAlchemy doesn't assign the result from the server immediately; instead a placeholder value is assigned, as we can see in the debugger:
(Pdb) obj.__dict__
{..., 'timeCreated': <__main__.utcnow at 0x7f237d0a90a0; utcnow object>}
When the session is closed, this placeholder value is expired:
(Pdb) sa.inspect(savedUser).expired_attributes
{'timeCreated'}
So, as stated at the end of the question, the value of timeCreated must be loaded before the session is closed to prevent a DetachedInstanceError if it is accessed later.
Based on the documentation for Fetching Server-Generated Defaults (case 1), this can be done by by setting timeCreated's server_default attribute to a FetchedValue, and setting eager_defaults to True in the mapper args.
Here is an example model (tested on Mariadb and Postgresql):
from sqlalchemy.schema import FetchedValue
class User(Base):
...
timeCreated = Column(DateTime, server_default=FetchedValue())
__mapper_args__ = {'eager_defaults': True}
For what it's worth, setting server_default=utcnow() in combination with the mapper args would work just as well, and avoid the need to set timeCreated explicitly; but perhaps the OP has their own reason for doing this.
I'm newer to OOP in Python, and have been trying for awhile to use my database class database within another class.
How can I do so?
class database(object):
def connect_db(self):
try:
import sqlite3 as sqli
connection = sqli.connect('pw.db')
cur = connection.cursor()
except:
print("There was an error connecting to the database.")
I've been trying to like this, but it doesnt work:
import db_helper as dbh
class account_settings(object):
def create_account(self):
setting_username = input('\nUsername?\n')
setting_password = input('\nPassword?\n')
cur = db.connect_db()
with cur:
cur.execute('''
CREATE TABLE IF NOT EXISTS my_passwords(
id INTEGER PRIMARY KEY AUTOINCREMENT NULL,
password text UNIQUE
)
''')
try:
cur.execute('INSERT INTO my_passwords(password) VALUES(?)', (self.create_password(),) )
except:
print("Error occurred trying to insert password into db. Please retry")
c = account_settings()
c.create_account()
new error:
File "settings.py", line 30, in <module>
c.create_account()
File "settings.py", line 15, in create_account
with cur:
AttributeError: __exit__
You need to learn about variable scope. db.connect_db() creates a cursor connection with the name cur, but does not do anything with it; when that method finishes, the object is destroyed. In particular, it never makes it back to the create_account method.
There is a simple way to solve this: return the object back to the method, and use it there.
def connect_db(self):
...
cur = connection.cursor()
return cur
...
def create_account(self):
cur = db.connect_db()
with cur:
or even beter:
with db.connect_db()
Note that really, neither of these should be classes. Classes in Python are really only useful when you're keeping some kind of state, which isn't happening here. connect_db and create_account should just be standalone functions in their respective modules.
I have this error when i perform the following task,
results = db1.executeSelectCommand(siteSql, (),)
TypeError: unbound method executeSelectCommand() must be called with dbConnn instance as first argument (got str instance instead)
My code is as follows:
class dbConnn:
db_con = None
execfile("/Users/usera/Documents/workspace/testing/src/db/db_config.py")
def executeSelectCommand(self,sql,ip):
#psycopg connection here.
I use this class here:
from db import dbConnections
db1 = dbConnections.dbConnn
siteSql = 'select post_content from post_content_ss order by RANDOM() limit 500' #order by year,month ASC'
results = db1.executeSelectCommand(siteSql, (),)
In windows, there don't seem to have a problem with this? God, it must be really elementary but I can't find it.
db1 = dbConnections.dbConnn
Here you assign the class dbConn to the variable db1. You probably wanted to create a new instance instead:
db1 = dbConnections.dbConnn()