I am using SQLAlchemy 0.9.7 over Postgres with psyopg2 as the driver.
I have a stray transaction that isn't being closed properly, and in order to debug it, I would like to log all of the operations being sent to the database.
The psycopg2.extras.LoggingConnection looks like it provides the functionality I need, but I can't see how I might persuade SQLAlchemy to use this feature of the dialect.
Is this possible via SQLAlchemy?
You could pass custom connection factory to SQLAlchemy engine:
def _connection_factory(*args, **kwargs):
connection = psycopg2.extras.LoggingConnection(*args, **kwargs)
connection.initialize(open('sql.log', 'a'))
return connection
db_engine = create_engine(conn_string,
connect_args={ "connection_factory": _connection_factory })
Alternatively, you could implement a custom cursor class (see psycopg2.extras.LoggingCursor for example), and pass it in a similar way:
connect_args={ "cursor_factory": MyCursor }
It isn't a direct answer to my own question, but a workaround: similar functionality can be obtained by turning on query logging at the SQLAlchemy layer, rather than the Psycopg2 layer:
Related
The pandas package have a method called .to_sql that help to insert the current data frame on to the database.
.to_sql doc:
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html
The second parameter is con
sqlalchemy.engine.(Engine or Connection) or sqlite3.Connection
Is it possible to generate the SQL query without passing a database connection?
We actually cannot print the query without a database connection, but we can use sqlalchemy create_mock_engine method and pass "memory" as the database URI to trick pandas, e.g:
from sqlalchemy import create_mock_engine, Metadata
def dump(sql, *multiparams, **params):
print(sql.compile(dialect=engine.dialect))
engine = create_mock_engine("sqlite://:memory:", echo=True)
Metadata.create_all(engine, checkfirst=False)
frame.to_sql(engine)
Currently I have a query similar to the below in flask sqlalchemy:
from sqlalchemy.sql import func
models = (
Model.query
.join(ModelTwo)
.filter(Model.finish_time >= func.from_unixtime(ModelTwo.start_date))
.all()
)
This works fine with MySQL which I am running in production, however when I run tests against the method using an in-memory SqlLite database it fails because from_unixtime is not a SqlLite function.
Aside from the running tests on the same database as production as closely as possible issue and the fact that I have two different ways of representing data in the database, is there a database agnostic method in SqlAlchemy for handling the conversion of dates to unix timestamps and vice-versa?
For anyone else interested in this, I found a way to create custom functions in SqlAlchemy based on the SQL dialect being used. As such the below achieves what I need:
from sqlalchemy.sql import expression
from sqlalchemy.ext.compiler import compiles
class convert_timestamp_to_date(expression.FunctionElement):
name = 'convert_timestamp_to_date'
#compiles(convert_timestamp_to_date)
def mysql_convert_timestamp_to_date(element, compiler, **kwargs):
return 'from_unixtime({})'.format(compiler.process(element.clauses))
#compiles(convert_timestamp_to_date, 'sqlite')
def sqlite_convert_timestamp_to_date(element, compiler, **kwargs):
return 'datetime({}, "unixepoch")'.format(compiler.process(element.clauses))
The query above can now be re-written as such:
models = (
Model.query
.join(ModelTwo)
.filter(Model.finish_time >= convert_timestamp_to_date(ModelTwo.start_date))
.all()
)
As the title states I need some help with Python and MySQL. I am currently studying Python further and I am focusing hard on using Python and MySQL for database design, development, administration and applications.
I am familiar with MySQL and somewhat familiar with Python. Currently I am working on object orientated programming and I am trying my hand at setting up a database connection inside of a database class and then using the class to Create, Update, Delete and Read data.
I have created a new Python object:
import pymysql as MySQL
class Database(object):
Host = "127.0.0.1"
Database = "****"
user = "****"
password = "****"
#staticmethod
def initialize():
currentdb = MySQL.connect(Database.Host, Database.user, Database.password, Database.Database)
cursor = currentdb.cursor()
#staticmethod
def insert(Table, DataDict):
placeholders = ", ".join(["%s"] * len(DataDict))
columns = ", ".join(DataDict.keys())
sql = "INSERT INTO %s (%s) VALUES (%s)"%(Table, columns, placeholders)
cursor.execute(sql, DataDict.values())
I want to know, how do I work with the cursor inside of a object? I don't know if my current approach is even close to how it should be handled, I am really not sure.
Can the cursor be initialized in this way, and then used further in the object as I intend on doing in the above extract?
Any help would be highly appreciated.
The right way to work with cursors is like this:
import contextlib
def doSomething():
with contextlib.closing(database.cursor()) as cursor:
cursor.execute(...)
# At the end of the `with` statement, cursor is closed
Do not keep a cursor open for too long. Keeping a connection open for a long time, as you do, is fine. Also, read on transaction control.
If you're doing more than a handful of DB operations, consider using a library like SQLAlchemy or Pony ORM.
import contextlib
def doSomething():
with contextlib.closing(database.cursor()) as cursor:
cursor.execute(...)
library for db SQLAlchemy or Pony ORM.
Have you considered using SQLAlchemy? This gives you a mapping between Python classes and MySQL (or any other RDBMS) tables. I've recently been using it on a fairly hefty real-world project and it seems to do the job fairly well and is easy enough to learn.
Check out the following code. I added the content in your initialize() to the standard python class init method and made the database be initialized with different types of parameters:
import pymysql as MySQL
class Database(object):
def __init__(self, host, db, user, pw):
self.currentdb = MySQL.connect(Database.host, user, pw, db)
def insert(self, Table, DataDict):
placeholders = ", ".join(["%s"] * len(DataDict))
columns = ", ".join(DataDict.keys())
sql = "INSERT INTO %s (%s) VALUES (%s)"%(Table, columns, placeholders)
with self.currentdb.cursor() as db_cursor:
db_cursor.execute(sql, DataDict.values())
Once you are here, then you can initialize a Database object as below and insert data:
my_db = Database(host="127.0.0.1", user="****", pw="****", db="****")
my_db.insert('table_name', data_dict)
Please note, I haven't changed your code, only presenting an organization based on your initial post that could work.
PostgreSQL supports specifying Date Formats using the DateStyle Property as mentioned here,
http://www.postgresql.org/docs/current/interactive/runtime-config-client.html#GUC-DATESTYLE. (link was originally to 8.3 version of docs).
I could not find any SQLAlchemy ORM documentation reference on to how to define this property. Is it possible to do it?
SQLAlchemy makes use of the DBAPI, usually psycopg2, to marshal date values to and from python datetime objects - you can then format/parse any way you want using standard python techniques. So no database-side date formatting features are needed.
If you do want to set this variable, you can just execute PG's SET statement:
conn = engine.connect()
conn.execute("SET DateStyle='somestring'")
# work with conn
to make this global to all connections:
from sqlalchemy import event
from sqlalchemy.engine import Engine
#event.listens_for(Engine, "connect")
def connect(dbapi_connection, connection_record):
cursor = dbapi_connection.cursor()
cursor.execute("SET DateStyle='somestring'")
cursor.close()
I would like to modify some database data as part of an alembic upgrade.
I thought I could just add any code in the upgrade of my migration, but the following fails:
def upgrade():
### commands auto generated by Alembic - please adjust! ###
op.add_column('smsdelivery', sa.Column('sms_message_part_id', sa.Integer(), sa.ForeignKey('smsmessagepart.id'), nullable=True))
### end Alembic commands ###
from volunteer.models import DBSession, SmsDelivery, SmsMessagePart
for sms_delivery in DBSession.query(SmsDelivery).all():
message_part = DBSession.query(SmsMessagePart).filter(SmsMessagePart.message_id == sms_delivery.message_id).first()
if message_part is not None:
sms_delivery.sms_message_part = message_part
with the following error:
sqlalchemy.exc.UnboundExecutionError: Could not locate a bind configured on mapper Mapper|SmsDelivery|smsdelivery, SQL expression or this Session
I am not really understanding this error. How can I fix this or is doing operations like this not a possibility?
It is difficult to understand what exactly you are trying to achieve from the code excerpt your provided. But I'll try to guess. So the following answer will be based on my guess.
Line 4 - you import things (DBSession, SmsDelivery, SmsMessagePart) form your modules and then you are trying to operate with these objects like you do in your application.
The error shows that SmsDelivery is a mapper object - so it is pointing to some table. mapper objects should bind to valid sqlalchemy connection.
Which tells me that you skipped initialization of DB objects (connection and binding this connection to mapper objects) like you normally do in your application code.
DBSession looks like SQLAlchemy session object - it should have connection bind too.
Alembic already has connection ready and open - for making changes to db schema you are requesting with op.* methods.
So there should be way to get this connection.
According to Alembic manual op.get_bind() will return current Connection bind:
For full interaction with a connected database, use the “bind” available from the context:
from alembic import op
connection = op.get_bind()
So you may use this connection to run your queries into db.
PS. I would assume you wanted to perform some modifications to data in your table. You may try to formulate this modification into one update query. Alembic has special method for executing such changes - so you would not need to deal with connection.
alembic.operations.Operations.execute
execute(sql, execution_options=None)
Execute the given SQL using the current migration context.
In a SQL script context, the statement is emitted directly to the output stream. There is no return result, however, as this function is oriented towards generating a change script that can run in “offline” mode.
Parameters: sql – Any legal SQLAlchemy expression, including:
a string a sqlalchemy.sql.expression.text() construct.
a sqlalchemy.sql.expression.insert() construct.
a sqlalchemy.sql.expression.update(),
sqlalchemy.sql.expression.insert(), or
sqlalchemy.sql.expression.delete() construct. Pretty much anything
that’s “executable” as described in SQL Expression Language Tutorial.
Its worth noting that if you do this, you probably want to freeze a copy of your orm model inside the migration, like this:
class MyType(Base):
__tablename__ = 'existing_table'
__table_args__ = {'extend_existing': True}
id = Column(Integer, ...)
..
def upgrade():
Base.metadata.bind = op.get_bind()
for item in Session.query(MyType).all():
...
Otherwise you'll inevitably end up in a situation where you orm model changes and previous migrations no longer work.
Particularly note that you want to extend Base, not the base type itself (app.models.MyType) because your type might go away as some point, and once again, your migrations will fail.
You need to import Base also and then
Base.metatada.bind = op.get_bind()
and after this you can use your models like always without errors.