I'm experiencing an issue with sqlalchemy where an update to a record in one session is not reflected in a second session even after committing and refreshing the object.
To demonstrate, consider this (complete) example:
import logging
from sqlalchemy import create_engine, Column, Boolean, Integer
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
logging.basicConfig(level=logging.INFO)
logging.getLogger("sqlalchemy.engine").setLevel(logging.INFO)
# works with this
#engine = create_engine("sqlite://")
# fails with this
engine = create_engine("mysql+mysqldb://{user}:{pass}#{host}:{port}/{database}?charset=utf8mb4".format(**DB_SETTINGS))
Session = sessionmaker(bind=engine)
Base = declarative_base()
class Foo(Base):
__tablename__ = "foo"
id = Column(Integer, primary_key=True, autoincrement=True)
flag = Column(Boolean)
def __repr__(self):
return "Foo(id={0.id}, flag={0.flag})".format(self)
# create the table
Base.metadata.create_all(engine)
# add a row
session = Session()
foo = Foo(id=1, flag=False)
session.add(foo)
session.commit()
# fetch the row in a different session
session2 = Session()
foo2 = session2.query(Foo).filter_by(id=1).one()
logging.info("SESSION2: Got {0}".format(foo2))
# update the row in first session and commit
foo.flag = True
session.commit()
# refresh the row in second session
logging.info("SESSION2: Refreshing...")
session2.refresh(foo2)
logging.info("SESSION2: After refresh: {0}".format(foo2))
# does "flag" come back as True?
When I run this against with the mysql+mysqldb:// engine to connect to my remote MySQL instance, the change to foo.flag is not reflected in session2.
But if I uncomment the line that creates an engine using a simple sqlite:// in-memory database, the change to foo.flag is reflected in session2.
What is it about my MySQL server configuration could cause an UPDATE command in one session followed immediately by a SELECT query in another session to return different data?
Related
I'm trying to use a session with autocommit=true to create a row in a table, and it does not seem to be working. The row is not saved to the table.
import os
import sqlalchemy
from sqlalchemy import Table
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy import Column, create_engine, String
db_hostname = os.environ['DB_HOSTNAME']
db_username = os.environ['DB_USERNAME']
db_password = os.environ['DB_PASSWORD']
db_servicename = os.environ['DB_SERVICENAME']
engine_string = f"postgresql://{db_username}:{db_password}#{db_hostname}:5432/{db_servicename}"
engine = create_engine(engine_string, isolation_level='REPEATABLE READ',
poolclass=sqlalchemy.pool.NullPool
)
base = declarative_base()
class Testing(base):
__tablename__ = 'testing'
name = Column(String, primary_key=True)
comment = Column(String)
base.metadata.create_all(engine)
S1 = sessionmaker(engine)
with S1() as session:
test1 = Testing(name="Jimmy", comment="test1")
session.add(test1)
session.commit()
S2 = sessionmaker(engine, autocommit=True)
with S2() as session:
test2 = Testing(name="Johnny", comment="test2")
session.add(test2)
In this code, the first row with name="Jimmy" and an explicit session.commit() is saved to the table.
But the second row with name="Johnny" is not saved. Specifying autocommit=True on the session appears to have no effect.
What is the cause?
If you enable the SQLALCHEMY_WARN_20=1 environment variable you will see
RemovedIn20Warning: The Session.autocommit parameter is deprecated and will be removed in SQLAlchemy version 2.0. …
The "2.0 way" to accomplish that "autocommit" behaviour is to do
S2 = sessionmaker(engine)
with S2() as session, session.begin():
test2 = Testing(name="Johnny", comment="test2")
session.add(test2)
# no explicit session.commit() required
The changes will automatically be committed when the context manager (with block) exits, provided that no errors have occurred.
I want to maintain audit log for database changes (if any update happens then the previous state to be inserted in log table
below is the code i have written
from sqlalchemy import create_engine
from sqlalchemy import event
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, DateTime
from datetime import datetime
from sqlalchemy.orm import sessionmaker
engine = create_engine('mysql+pymysql://root:p4ssw0rd#0.0.0.0:33061/error_details')
Base = declarative_base()
Base.metadata.create_all(engine)
Session = sessionmaker()
Session.configure(bind=engine)
session = Session()
class User(Base):
__tablename__ = "users"
id = Column(Integer, primary_key=True)
name = Column(String(10))
def __repr__(self):
return f"User(name={self.name}, id={self.id})"
class User_Log(Base):
__tablename__ = "users_log"
id = Column(Integer, primary_key=True)
name = Column(String(10))
modified_on = Column(DateTime, default=datetime.utcnow)
action = Column(String(10))
#event.listens_for(User, 'before_update')
def update_user_log(mapper, connection, target):
print("func() called ")
session.query(User).filter(User.id == 2).update({User.name: "ankit--"})
session.commit()
function update_user_log is not getting called
Note - I want to perform the database audit logs in sqlalchemy only
Update and deletes with arbitrary where clauses must be intercepted with the SessionEvents.do_orm_execute_handler. This because
The ORM-enabled UPDATE and DELETE features bypass ORM unit-of-work automation in favor being able to emit a single UPDATE or DELETE statement that matches multiple rows at once without complexity.*
You can intercept such updates with a handler like this:
#event.listens_for(Session, 'do_orm_execute')
def update_user_log(orm_execute_state):
if orm_execute_state.is_update:
print("func() called ")
* Quote taken from the second warning block at the end of the previously linked section.
It's not terribly clear in the docs, but merge() will cause the before_update event to fire, see Session API docs
I am having trouble writing tables to a postgres database using SQLAlchemy ORM and Python scripts.
I know the problem has something to do with incorrect Session imports because when I place all the code below into a single file, the script creates the table without trouble.
However, when I break the script up into multiple files (necessary for this project), I receive the error "psycopg2.errors.UndefinedTable: relation "user" does not exist".
I have read many posts here on SO, tried reorganising my files, the function call order, changing from non-scoped to scoped sessions, eliminating and adding Base.metadata.create_all(bind=engine) in various spots, changed how the sessions are organised and created in base.py, as well as other things, but the script still errors and I'm not sure which code sequence is out of order.
The code currently looks like:
base.py:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import scoped_session, sessionmaker
# SQLAlchemy requires creating an Engine to interact with our database.
engine = create_engine('postgresql://user:pass#localhost:5432/testdb', echo=True)
# Create a configured ORM 'Session' factory to get a new Session bound to this engine
#_SessionFactory = sessionmaker(bind=engine)
# Use scoped session
db_session = scoped_session(
sessionmaker(
bind=engine,
autocommit=False,
autoflush=False
)
)
# Create a Base class for our classes definitions
Base = declarative_base()
models.py
from sqlalchemy import Column, DateTime, Integer, Text
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
email = Column(Text, nullable=False, unique=True)
name = Column(Text)
date_last_seen = Column(DateTime(timezone=True))
def __init__(self, email, name, date_last_seen):
self.email = email
self.name = name
self.date_last_seen = date_last_seen
inserts.py
from datetime import date
from base import db_session, engine, Base
from models import User
def init_db():
# Generate database schema based on our definitions in model.py
Base.metadata.create_all(bind=engine)
# Extract a new session from the session factory
#session = _SessionFactory()
# Create instance of the User class
alice = User('alice#throughthelooking.glass', 'Alice', date(1865, 11, 26))
# Use the current session to persist data
db_session.add(alice)
# Commit current session to database and close session
db_session.commit()
db_session.close()
print('Initialized the db')
return
if __name__ == '__main__':
init_db()
Thank you for any insight you're able to offer!
I registered an event at updating User.name for apply some rules after or just before update.
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy import event
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('sqlite:///:memory:', echo=True)
Session = sessionmaker(bind=engine) # engine part
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String)
fullname = Column(String)
password = Column(String)
rule = Column(String)
Base.metadata.create_all(engine)
session = Session()
u1 = User(id=1, name="hello world", fullname="hello kitty")
u2 = User(id=2, name="hello world2", fullname="hello kitty2")
session.add(u1)
session.add(u2)
#event.listens_for(User, 'before_update')
def User_before_update(target, value, initiator):
print ":::::received before insert event for target"
#event.listens_for(User.name, 'set')
def name_set(target, value, old_value, initiator):
print ":::::set before insert event for target"
### option A
user = session.query(User).get(2)
user.name = u"wawamsma"
session.merge(user)
### option B
session.query(User).filter(User.id == 2).update({User.name: u"eenimenee"})
option A
Every things works fine, but I have some update method using many fields to sqlalchemy.
option B
Updated, but do not trigger both print func.
So I wonder, is this the Wrong way to register event or wrong way to do the update?
To receive events when using the query.update() method, you need to use the after_bulk_update event. As this operation does not deal with individual objects and instead emits an UPDATE statement directly, the actual objects in memory which may have been affected here are not locally available; you may need to query for them. The context will have a matched_objects attribute if "synchronize_session" is set to "evaluate".
I have the following mapping (straight from SA examples):
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String)
fullname = Column(String)
password = Column(String)
I'm working with a MySql DB and the table has an innoDB engine.
I have a single record in my table:
1|'user1'|'user1 test'|'password'
I've opened a session with the following code:
from sqlalchemy.orm.session import sessionmaker
from sqlalchemy.engine import create_engine
from sqlalchemy.orm.scoping import scoped_session
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
db_engine = create_engine('mysql://...#localhost/test_db?charset=utf8',echo=False,pool_recycle=1800)
session_factory = sessionmaker(bind=db_engine,autocommit=False,autoflush=False)
session_maker = scoped_session(session_factory)
session = session_maker()
user_1 = session.query(User).filter(User.id==1).one()
user_1.name # This prints: u'user1'
Now, when I change the record's name in the DB to 'user1_change' and commit it and then refresh the object like this:
session.refresh(user_1)
user_1.name # This still prints: u'user1' and not u'user1_change'
It still prints: u'user1' and not u'user1_change'.
What am I missing (or setting up wrong) here?
Thanks!
From the docs:
Note that a highly isolated transaction will return the same values as were previously read in that same transaction, regardless of changes in database state outside of that transaction
SQLAlchemy uses a transactional unit of work model, wherein each transaction is assumed to be internally consistent. A session is an interface on top of a transaction. Since a transaction is assumed to be internally consistent, SQLAlchemy will only (well, not quite, but for ease of explanation...) retrieve a given piece of data from the database and update the state of the associated objects once per transaction. Since you already queried for the object in the same session transaction, SQLAlchemy will not update the data in that object from the database again within that transaction scope. If you want to poll the database, you'll need to do it with a fresh transaction each time.
session.refresh() didn't work for me either. Even though I saw a low-level SELECT the object was not updated after the refresh.
This answer https://stackoverflow.com/a/11121788/562267 hints to doing an actual commit/rollback to reset the session, and that worked for me:
user_1 = session.query(User).filter(User.id==1).one()
user_1.name # This prints: u'user1'
# update the database from another client here
session.commit()
user_1 = session.query(User).filter(User.id==1).one()
user_1.name # Should be updated now.
Did you try with "expire" as described in the official doc:
http://docs.sqlalchemy.org/en/rel_0_8/orm/session.html#refreshing-expiring
# expire objects obj1, obj2, attributes will be reloaded
# on the next access:
session.expire(user_1)
session.refresh(user_1)
Using expire on a object results in a reload that will occur upon next access.
Merge the session.
u = session.query(User).get(id)
u.name = 'user1_changed'
u = session.merge(u)
This will update the database and return the newer object.