I am using SQLAlchemy together with Zope transactions. My object is something like:
class MyItem(DeclarativeBase):
# ....
id = Column(Integer, primary_key=True)
# ....
then when I create a new object, and commit transaction, I would like to obtain genarated id, for example:
mi = MyItem()
DBSession.add(mi)
transaction.commit()
print mi.id # currently object is expired (not persistent with this session)
Is there some easy way to achieve this behaviour?
You can just use the default session pattern; add the object, and then flush to save the object to the database. Do not commit the transaction, let Zope handle that:
mi = MyItem()
DBSession.add(mi)
DBSession.flush()
print mi.id
Your object will be added to the database in the current transaction, and thus the primary key will be known.
Related
When using SQLAlchemy I would like the foreign key fields to be filled in on the Python object when I pass in a related object. For example, assume you have network devices with ports, and assume that the device has a composite primary key in the database.
If I already have a reference to a "Device" instance and want to create a new "Port" instance linked to that device without knowing if it already exists in the database I would use the merge operation in SA. However, only setting the device attribute on the port instance is insufficient. The fields of the composite foreign key will not be propagated to the port instance and SA will be unable to determine the existence of the row in the database and unconditionally issue an INSERT statement instead of an UPDATE.
The following code examples demonstrate the issue. They should be run as one .py file so we have the same in-memory SQLite instance! They have only been split for readability.
Model Definition
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Unicode, ForeignKeyConstraint, create_engine
from sqlalchemy.orm import sessionmaker, relation
from textwrap import dedent
Base = declarative_base()
class Device(Base):
__tablename__ = 'device'
hostname = Column(Unicode, primary_key=True)
scope = Column(Unicode, primary_key=True)
poll_ip = Column(Unicode, primary_key=True)
notes = Column(Unicode)
ports = relation('Port', backref='device')
class Port(Base):
__tablename__ = 'port'
__table_args__ = (
ForeignKeyConstraint(
['hostname', 'scope', 'poll_ip'],
['device.hostname', 'device.scope', 'device.poll_ip'],
onupdate='CASCADE', ondelete='CASCADE'
),
)
hostname = Column(Unicode, primary_key=True)
scope = Column(Unicode, primary_key=True)
poll_ip = Column(Unicode, primary_key=True)
name = Column(Unicode, primary_key=True)
engine = create_engine('sqlite://', echo=True)
Base.metadata.bind = engine
Base.metadata.create_all()
Session = sessionmaker(bind=engine)
The model defines a Device class with a composite PK with three fields. The Port class references Device through a composite FK on those three columns. Device also has a relationship to Port which will use that FK.
Using the model
First, we add a new device and port. As we're using an in-memory SQLite DB, these will be the only two entries in the DB. And by inserting one device into the database we have something in the device
table that we expect to be loaded on the subsequent merge in session "sess2"
sess1 = Session()
d1 = Device(hostname='d1', scope='s1', poll_ip='pi1')
p1 = Port(device=d1, name='port1')
sess1.add(d1)
sess1.commit()
sess1.close()
Working example
This block works, but it is not written in a way I would expect it to behave. More precisely, the instance "d1" is instantiated with "hostname", "scope" and "poll_ip", and that instance is passed to the "Port" instance "p2". I would expect that "p2" would "receive" those 3 values through the foreign key. But it doesn't. I am forced to manually assign the values to "p2" before calling "merge". If the values are not assigned, SA does not find the identity and tries to run an "INSERT" query for "p2" which will conflict with the already existing instance.
sess2 = Session()
d1 = Device(hostname='d1', scope='s1', poll_ip='pi1')
p2 = Port(device=d1, name='port1')
p2.hostname=d1.hostname
p2.poll_ip=d1.poll_ip
p2.scope = d1.scope
p2 = sess2.merge(p2)
sess2.commit()
sess2.close()
Broken example (but expecting it to work)
This block shows how I would expect it to work. I would expect that assigning a value to "device" when creating the Port instance should be enough.
sess3 = Session()
d1 = Device(hostname='d1', scope='s1', poll_ip='pi1')
p2 = Port(device=d1, name='port1')
p2 = sess3.merge(p2)
sess3.commit()
sess3.close()
How can I make this last block work?
The FK of the child object isn't updated until you issue a flush() either explicitly or through a commit(). I think the reason for this is that if the parent object of a relationship is also a new instance with an auto-increment PK, SQLAlchemy needs to get the PK from the database before it can update the FK on the child object (but I stand to be corrected!).
According to the docs, a merge():
examines the primary key of the instance. If it’s present, it attempts
to locate that instance in the local identity map. If the load=True
flag is left at its default, it also checks the database for this
primary key if not located locally.
If the given instance has no primary key, or if no instance can be
found with the primary key given, a new instance is created.
As you are merging before flushing, there is incomplete PK data on your p2 instance and so this line p2 = sess3.merge(p2) returns a new Port instance with the same attribute values as the p2 you previously created, that is tracked by the session. Then, sess3.commit() finally issues the flush where the FK data is populated onto p2 and then the integrity error is raised when it tries to write to the port table. Although, inserting a sess3.flush() will only raise the integrity error earlier, not avoid it.
Something like this would work:
def existing_or_new(sess, kls, **kwargs):
inst = sess.query(kls).filter_by(**kwargs).one_or_none()
if not inst:
inst = kls(**kwargs)
return inst
id_data = dict(hostname='d1', scope='s1', poll_ip='pi1')
sess3 = Session()
d1 = Device(**id_data)
p2 = existing_or_new(sess3, Port, name='port1', **id_data)
d1.ports.append(p2)
sess3.commit()
sess3.close()
This question has more thorough examples of existing_or_new style functions for SQLAlchemy.
I am new in sqlalchemy. I want to do add and update in single transaction for same model.code snippet is below. Application throwing error like 'Session' object has no attribute 'update'
current_date = datetime.datetime.now()
try:
session = Session()
user = UserProvision()
user.username = admin["username"]
user.password= admin["password"]
user.client_id= admin["client_id"]
user.fname= admin["fname"]
user.lname= admin['lname']
user.phone= admin['phone']
session.add(user)
session.flush()
user_id = user.user_id
user.name = admin["fname"]+" "+admin["lname"]
user.setCreated_by=user_id
user.setModified_by=user_id
user.setCreated_name=admin["fname"]+" "+admin["lname"]
user.setModified_name=admin["fname"]+" "+admin["lname"]
user.setLast_reset_date=current_date
user.setLast_reset_by = current_date
session.update(user)
session.flush()
session.commit()
except Exception as ex:
print ex.__str__()
finally:
session.close()
When you've added the model object to the session its state is already tracked for changes. There's no need to explicitly mark it as updated, and as you've noted there is no such method Session.update(). Simply remove that line and your code should work as expected.
The tracking is achieved through instrumentation of model class attributes:
The SQLAlchemy mapping process, among other things, adds database-enabled descriptors to a mapped class which each represent a particular database column or relationship to a related class.
In other words when your model class is constructed the Column attributes will be replaced with InstrumentedAttribute descriptor instances that among other things keep track of changes to the value.
Note that there's no need to manually flush just before Session.commit():
Flush pending changes and commit the current transaction.
I am working on finding a way in SQLAlchemy events to call an external API upon an attribute gets updated and persisted into the database. Here is my context:
An User model with an attribute named birthday. When an instance of User model gets updated and saved, I want to call to an external API to update this user's birthday accordingly.
I've tried Attribute Events, however, it generates too many hits and there is no way to guarantee that the set/remove attribute event would get persisted eventually (auto commit is set to False and transaction gets rolled back when errors occurred.)
Session Events would not work either because it requires a Session/SessionFactory as a parameter and there are just so many places in the code based that sessions have been used.
I have been looking at all the possible SQLAlchemy ORM event hooks in the official documentation but I couldn't find any one of them satisfy my requirement.
I wonder if anyone else has any insight into how to implement this kind of combination event trigger in SQLAlchemy. Thanks.
You can do this by combining multiple events. The specific events you need to use depend on your particular application, but the basic idea is this:
[InstanceEvents.load] when an instance is loaded, note down the fact that it was loaded and not added to the session later (we only want to save the initial state if the instance was loaded)
[AttributeEvents.set/append/remove] when an attribute changes, note down the fact that it was changed, and, if necessary, what it was changed from (these first two steps are optional if you don't need the initial state)
[SessionEvents.before_flush] when a flush happens, note down which instances are actually being saved
[SessionEvents.before_commit] before a commit completes, note down the current state of the instance (because you may not have access to it anymore after the commit)
[SessionEvents.after_commit] after a commit completes, fire off the custom event handler and clear the instances that you saved
An interesting challenge is the ordering of the events. If you do a session.commit() without doing a session.flush(), you'll notice that the before_commit event fires before the before_flush event, which is different from the scenario where you do a session.flush() before session.commit(). The solution is to call session.flush() in your before_commit call to force the ordering. This is probably not 100% kosher, but it works for me in production.
Here's a (simple) diagram of the ordering of events:
begin
load
(save initial state)
set attribute
...
flush
set attribute
...
flush
...
(save modified state)
commit
(fire off "object saved and changed" event)
Complete Example
from itertools import chain
from weakref import WeakKeyDictionary, WeakSet
from sqlalchemy import Column, String, Integer, create_engine
from sqlalchemy import event
from sqlalchemy.orm import sessionmaker, object_session
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
engine = create_engine("sqlite://")
Session = sessionmaker(bind=engine)
class User(Base):
__tablename__ = "users"
id = Column(Integer, primary_key=True)
birthday = Column(String)
#event.listens_for(User.birthday, "set", active_history=True)
def _record_initial_state(target, value, old, initiator):
session = object_session(target)
if session is None:
return
if target not in session.info.get("loaded_instances", set()):
return
initial_state = session.info.setdefault("initial_state", WeakKeyDictionary())
# this is where you save the entire object's state, not necessarily just the birthday attribute
initial_state.setdefault(target, old)
#event.listens_for(User, "load")
def _record_loaded_instances_on_load(target, context):
session = object_session(target)
loaded_instances = session.info.setdefault("loaded_instances", WeakSet())
loaded_instances.add(target)
#event.listens_for(Session, "before_flush")
def track_instances_before_flush(session, context, instances):
modified_instances = session.info.setdefault("modified_instances", WeakSet())
for obj in chain(session.new, session.dirty):
if session.is_modified(obj) and isinstance(obj, User):
modified_instances.add(obj)
#event.listens_for(Session, "before_commit")
def set_pending_changes_before_commit(session):
session.flush() # IMPORTANT
initial_state = session.info.get("initial_state", {})
modified_instances = session.info.get("modified_instances", set())
del session.info["modified_instances"]
pending_changes = session.info["pending_changes"] = []
for obj in modified_instances:
initial = initial_state.get(obj)
current = obj.birthday
pending_changes.append({
"initial": initial,
"current": current,
})
initial_state[obj] = current
#event.listens_for(Session, "after_commit")
def after_commit(session):
pending_changes = session.info.get("pending_changes", {})
del session.info["pending_changes"]
for changes in pending_changes:
print(changes) # this is where you would fire your custom event
loaded_instances = session.info["loaded_instances"] = WeakSet()
for v in session.identity_map.values():
if isinstance(v, User):
loaded_instances.add(v)
def main():
engine = create_engine("sqlite://", echo=False)
Base.metadata.create_all(bind=engine)
session = Session(bind=engine)
user = User(birthday="foo")
session.add(user)
user.birthday = "bar"
session.flush()
user.birthday = "baz"
session.commit() # prints: {"initial": None, "current": "baz"}
user.birthday = "foobar"
session.commit() # prints: {"initial": "baz", "current": "foobar"}
session.close()
if __name__ == "__main__":
main()
As you can see, it's a little complicated and not very ergonomic. It would be nicer if it were integrated into the ORM, but I also understand there may be reasons for not doing so.
Imagine that I have one table in my project with some rows in it.
For example:
# -*- coding: utf-8 -*-
import sqlalchemy as sa
from app import db
class Article(db.Model):
__tablename__ = 'article'
id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
name = sa.Column(sa.Unicode(255))
content = sa.Column(sa.UnicodeText)
I'm using Flask-SQLAlchemy, so db.session is scoped session object.
I saw in https://github.com/zzzeek/sqlalchemy/blob/master/examples/versioned_history/history_meta.py
but i can't understand how to use it with my existing tables and anymore how to start it. (I get ArgumentError: Session event listen on a scoped_session requires that its creation callable is associated with the Session class. error when I pass db.session in versioned_session func)
From versioning I need the following:
1) query for old versions of object
2) query old versions by date range when they changed
3) revert old state to existing object
4) add additional info to history table when version is creating (for example editor user_id, date_edit, remote_ip)
Please, tell me what are the best practicies for my case and if you can add a little working example for it.
You can work around that error by attaching the event handler to the SignallingSession class[1] instead of the created session object:
from flask.ext.sqlalchemy import SignallingSession
from history_meta import versioned_session, Versioned
# Create your Flask app...
versioned_session(SignallingSession)
db = SQLAlchemy(app)
class Article(Versioned, db.Model):
__tablename__ = 'article'
id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
name = sa.Column(sa.Unicode(255))
content = sa.Column(sa.UnicodeText)
The sample code creates parallel tables with a _history suffix and an additional changed datetime column. Querying for old versions is just a matter of looking in that table.
For managing the extra fields, I would put them on your main table, and they'll automatically be kept track of in the history table.
[1] Note, if you override SQLAlchemy.create_session() to use a different session class, you should adjust the class you pass to versioned_session.
I think the problem is you're running into this bug: https://github.com/mitsuhiko/flask-sqlalchemy/issues/182
One workaround would be to stop using flask-sqlalchemy and configure sqlalchemy yourself.
I have the following mapping (straight from SA examples):
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String)
fullname = Column(String)
password = Column(String)
I'm working with a MySql DB and the table has an innoDB engine.
I have a single record in my table:
1|'user1'|'user1 test'|'password'
I've opened a session with the following code:
from sqlalchemy.orm.session import sessionmaker
from sqlalchemy.engine import create_engine
from sqlalchemy.orm.scoping import scoped_session
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
db_engine = create_engine('mysql://...#localhost/test_db?charset=utf8',echo=False,pool_recycle=1800)
session_factory = sessionmaker(bind=db_engine,autocommit=False,autoflush=False)
session_maker = scoped_session(session_factory)
session = session_maker()
user_1 = session.query(User).filter(User.id==1).one()
user_1.name # This prints: u'user1'
Now, when I change the record's name in the DB to 'user1_change' and commit it and then refresh the object like this:
session.refresh(user_1)
user_1.name # This still prints: u'user1' and not u'user1_change'
It still prints: u'user1' and not u'user1_change'.
What am I missing (or setting up wrong) here?
Thanks!
From the docs:
Note that a highly isolated transaction will return the same values as were previously read in that same transaction, regardless of changes in database state outside of that transaction
SQLAlchemy uses a transactional unit of work model, wherein each transaction is assumed to be internally consistent. A session is an interface on top of a transaction. Since a transaction is assumed to be internally consistent, SQLAlchemy will only (well, not quite, but for ease of explanation...) retrieve a given piece of data from the database and update the state of the associated objects once per transaction. Since you already queried for the object in the same session transaction, SQLAlchemy will not update the data in that object from the database again within that transaction scope. If you want to poll the database, you'll need to do it with a fresh transaction each time.
session.refresh() didn't work for me either. Even though I saw a low-level SELECT the object was not updated after the refresh.
This answer https://stackoverflow.com/a/11121788/562267 hints to doing an actual commit/rollback to reset the session, and that worked for me:
user_1 = session.query(User).filter(User.id==1).one()
user_1.name # This prints: u'user1'
# update the database from another client here
session.commit()
user_1 = session.query(User).filter(User.id==1).one()
user_1.name # Should be updated now.
Did you try with "expire" as described in the official doc:
http://docs.sqlalchemy.org/en/rel_0_8/orm/session.html#refreshing-expiring
# expire objects obj1, obj2, attributes will be reloaded
# on the next access:
session.expire(user_1)
session.refresh(user_1)
Using expire on a object results in a reload that will occur upon next access.
Merge the session.
u = session.query(User).get(id)
u.name = 'user1_changed'
u = session.merge(u)
This will update the database and return the newer object.