I've got two mapped objects, Parent and Child.
class Parent(Base):
__tablename__ = 'parent'
id = ...
name = ...
date_modified = Column(SA_DateTime, default=DateTime.now,
onupdate=DateTime.now, nullable=False)
class Child(Base):
__tablename__ = 'child'
id = ...
name = ...
date_modified = Column(SA_DateTime, default=DateTime.now,
onupdate=DateTime.now, nullable=False)
parent = relationship(Parent, backref='parent')
When a child is updated, I want not only Child.date_modified to be changed, but also Child.parent.date_modified.
I tried to do this like this:
#event.listens_for(Child, 'after_update')
def modified_listener(mapper, connection, target):
if object_session(target).is_modified(target, include_collections=False):
target.parent.date_modified = DateTime.now()
But this doesn't work, because I'm already in a flush and I get something like
SAWarning: Attribute history events accumulated on 1 previously clean instance within inner-flush event handlers have been reset, and will not result in database updates. Consider using set_committed_value() within inner-flush event handlers to avoid this warning.
How can I solve this with SQLAlchemy?
Basic update-parent-when-child-changes using SQLAlchemy events has been covered on this site before here and here, but in your case you're trying to update the parent during the flush, possibly using an update default value from the child, which will be visible after the update, or a new value entirely. Modifying the parent in the event handler is not as straightforward as you might first imagine:
Warning
Mapper-level flush events only allow very limited operations, on attributes local to the row being operated upon only, as well as allowing any SQL to be emitted on the given Connection. Please read fully the notes at Mapper-level Events for guidelines on using these methods; generally, the SessionEvents.before_flush() method should be preferred for general on-flush changes.
As you've noticed, simple
target.parent.date_modified = DateTime.now()
in your event handler warns:
SAWarning: Attribute history events accumulated on 1 previously clean instances within inner-flush event handlers have been reset, and will not result in database updates. Consider using set_committed_value() within inner-flush event handlers to avoid this warning.
set_committed_value() allows setting attributes with no history events, as if the set value was part of the original loaded state.
You've also noticed that receiving a target in an after update event handler does not guarantee that an UPDATE statement was actually emitted:
This method is called for all instances that are marked as “dirty”, even those which have no net changes to their column-based attributes, and for which no UPDATE statement has proceeded.
and
To detect if the column-based attributes on the object have net changes, and therefore resulted in an UPDATE statement, use object_session(instance).is_modified(instance, include_collections=False).
So a solution could be to use the information held in the event target to emit an UPDATE statement on the parent table using the given connection, and then to check if the Parent object is present in the session and set the committed value of it:
from sqlalchemy import event
from sqlalchemy.orm.attributes import set_committed_value
from sqlalchemy.orm.session import object_session
#event.listens_for(Child, 'after_update')
def receive_child_after_update(mapper, connection, target):
session = object_session(target)
if not session.is_modified(target, include_collections=False):
return
new_date_modified = target.date_modified
# Avoid touching the target.parent relationship attribute and
# copy the date_modified value from the child to parent.
# Warning: this will overwrite possible other updates to parent's
# date_modified.
connection.execute(
Parent.__table__.
update().
values(date_modified=new_date_modified).
where(Parent.id == target.parent_id))
parent_key = session.identity_key(Parent, target.parent_id)
try:
the_parent = session.identity_map[parent_key]
except KeyError:
pass
else:
# If the parent object is present in the session, update its
# date_modified attribute **in Python only**, to reflect the
# updated DB state local to this transaction.
set_committed_value(
the_parent, 'date_modified', new_date_modified)
Have you tried:
#event.listens_for(Child, 'after_update')
def modified_listener(mapper, connection, target):
if object_session(target).is_modified(target, include_collections=False):
sa.orm.attributes.set_committed_value(
target.parent,
'date_modified',
DateTime.now())
As per:
http://docs.sqlalchemy.org/en/latest/orm/session_api.html#sqlalchemy.orm.attributes.set_committed_value
Suppose I am using a declarative object like the following:
class MyObject(DeclarativeBase):
...
other_object_id = Column(Integer, ForeignKey('other_object.id'))
other_object = relationship("OtherObject")
...
Suppose I would like to have a convenience method set_other_object_value, which would modify other_object.value, creating an instance of OtherObject and adding it to the session, if necessary:
def set_other_object_value(self, value):
if self.other_object is None:
<create instance of OtherObject and associate with the session>
self.other_object.value = value
The question is: how can I create OtherObject and associate it with the session? A possibly equivalent question: is it possible to access the instance of a Session, to which MyObject was added from within an instance of MyObject?
Use object_session:
from sqlalchemy.orm.session import object_session
# ...
def set_other_object_value(self, value):
if self.other_object is None:
value = OtherObject(...)
self.other_object.value = value
object_session(self).add(value)
But if you have setup you relationships properly, you do not need to add the new value to the session, as sqlalchemy will figure it out and save it to your database on commit anyways.
I am using the SQLAlchemy versioned object example as a reference.
Example: http://docs.sqlalchemy.org/en/rel_0_7/orm/examples.html#versioned-objects
When I update a record I am getting no errors. The case_history table is being created, but the version number is staying at '1' and the case_history table is empty.
(Yes I am aware that I am using 'case' as a class name. Is that bad?)
Here are my code snippets:
models.py:
from history_meta import Versioned, versioned_session
# set up the base class
class Base(object):
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
id = Column(Integer, primary_key = True)
header_row = Column(SmallInteger)
def to_dict(self):
serialized = dict((column_name, getattr(self, column_name))
for column_name in self.__table__.c.keys())
return serialized
Base = declarative_base(cls=Base)
class case(Versioned, Base):
title = Column(String(32))
description = Column(Text)
def __repr__(self):
return self.title
app.py:
engine = create_engine(SQLALCHEMY_DATABASE_URI)
Session = sessionmaker(bind=engine)
versioned_session(Session)
db = Session()
...
#app.route('/<name>/:record', method='POST')
def default(name, record):
myClass = getattr(sys.modules[__name__], name)
db.query(myClass).filter(myClass.id == record).update(request.json)
for u in db.query(case).filter(case.id == record):
print u.version # Version is always 1
db.commit() # I added this just to test versioning.
Any clue as to why the versioning isn't happening?
For others who find their way here...
Remember: even if filter() returns a single object, the update() method is a bulk operation, and acts differently. It is possible the version is only incremented on an event like after_update() which does not trigger on bulk operations.
Read more on caveats for the update() operation here.
An update query will not cause the version to increment even though the data changes. There might be ways to 'listen' for that type of change, but I don't know.
You have to change an attribute of a mapped class:
#Get an instance of the class
myItem = db.query(myClass).get(record)
#Change an attribute
myItem.title="foo"
#Commit if necessary
db.commit()
My question does not really have much to do with sqlalchemy but rather with pure python.
I'd like to control the instantiation of sqlalchemy Model instances. This is a snippet from my code:
class Tag(db.Model):
__tablename__ = 'tags'
query_class = TagQuery
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(), unique=True, nullable=False)
def __init__(self, name):
self.name = name
I want to achieve that whenever an entry is instantiated (Tag('django')) that a new instance should be created only if there is not yet another tag with the name django inside the database. Otherwise, instead of initializing a new object, a reference to the already existent row inside the database should be returned by (Tag('django')).
As of now I am ensuring the uniqueness of tags inside the Post Model:
class Post(db.Model):
# ...
# code code code
# ...
def _set_tags(self, taglist):
"""Associate tags with this entry. The taglist is expected to be already
normalized without duplicates."""
# Remove all previous tags
self._tags = []
for tag_name in taglist:
exists = Tag.query.filter(Tag.name==tag_name).first()
# Only add tags to the database that don't exist yet
# TODO: Put this in the init method of Tag (if possible)
if not exists:
self._tags.append(Tag(tag_name))
else:
self._tags.append(exists)
It does its job but still I'd like to know how to ensure the uniqueness of tags inside the Tag class itself so that I could write the _set_tags method like this:
def _set_tags(self, taglist):
# Remove all previous tags
self._tags = []
for tag_name in taglist:
self._tags.append(Tag(tag_name))
While writing this question and testing I learned that I need to use the __new__ method. This is what I've come up with (it even passes the unit tests and I didn't forget to change the _set_tags method):
class Tag(db.Model):
__tablename__ = 'tags'
query_class = TagQuery
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(), unique=True, nullable=False)
def __new__(cls, *args, **kwargs):
"""Only add tags to the database that don't exist yet. If tag already
exists return a reference to the tag otherwise a new instance"""
exists = Tag.query.filter(Tag.name==args[0]).first() if args else None
if exists:
return exists
else:
return super(Tag, cls).__new__(cls, *args, **kwargs)
What bothers me are two things:
First: I get a warning:
DeprecationWarning: object.__new__() takes no parameters
Second: When I write it like so I get errors (I also tried to rename the paramater name to n but it did not change anything) :
def __new__(cls, name):
"""Only add tags to the database that don't exist yet. If tag already
exists return a reference to the tag otherwise a new instance"""
exists = Tag.query.filter(Tag.name==name).first()
if exists:
return exists
else:
return super(Tag, cls).__new__(cls, name)
Errors (or similar):
TypeError: __new__() takes exactly 2 arguments (1 given)
I hope you can help me!
I use class method for that.
class Tag(Declarative):
...
#classmethod
def get(cls, tag_name):
tag = cls.query.filter(cls.name == tag_name).first()
if not tag:
tag = cls(tag_name)
return tag
And then
def _set_tags(self, taglist):
self._tags = []
for tag_name in taglist:
self._tags.append(Tag.get(tag_name))
As for __new__, you should not confuse it with __init__. It is expected to be called w/out args, so even if your own constructor asks for some, you should not pass them to super/object unless you know that your super needs them. Typical invocation would be:
def __new__(cls, name=None):
tag = cls.query.filter(cls.name == tag_name).first()
if not tag:
tag = object.__new__(cls)
return tag
However this will not work as expected in your case, since it calls __init__ automatically if __new__ returns instance of cls. You would need to use metaclass or add some checks in __init__.
Don't embed this within the class itself.
Option 1. Create a factory that has the pre-existing pool of objects.
tag_pool = {}
def makeTag( name ):
if name not in tag_pool:
tag_pool[name]= Tag(name)
return tag_pool[name]
Life's much simpler.
tag= makeTag( 'django' )
This will create the item if necessary.
Option 2. Define a "get_or_create" version of the makeTag function. This will query the database. If the item is found, return the object. If no item is found, create it, insert it and return it.
Given the OP's latest error msg:
TypeError: __new__() takes exactly 2 arguments (1 given)
it seems that somewhere the class is getting instantiated without the name parameter, i.e. just Tag(). The traceback for that exception should tell you where that "somewhere" is (but we're not shown it, so that's how far as we can go;-).
That being said, I agree with other answers that a factory function (possibly nicely dressed up as a classmethod -- making factories is one of the best uses of classmethod, after all;-) is the way to go, avoiding the complication that __new__ entails (such as forcing __init__ to find out whether the object's already initialized to avoid re-initializing it!-).
I want to get an object from the database if it already exists (based on provided parameters) or create it if it does not.
Django's get_or_create (or source) does this. Is there an equivalent shortcut in SQLAlchemy?
I'm currently writing it out explicitly like this:
def get_or_create_instrument(session, serial_number):
instrument = session.query(Instrument).filter_by(serial_number=serial_number).first()
if instrument:
return instrument
else:
instrument = Instrument(serial_number)
session.add(instrument)
return instrument
Following the solution of #WoLpH, this is the code that worked for me (simple version):
def get_or_create(session, model, **kwargs):
instance = session.query(model).filter_by(**kwargs).first()
if instance:
return instance
else:
instance = model(**kwargs)
session.add(instance)
session.commit()
return instance
With this, I'm able to get_or_create any object of my model.
Suppose my model object is :
class Country(Base):
__tablename__ = 'countries'
id = Column(Integer, primary_key=True)
name = Column(String, unique=True)
To get or create my object I write :
myCountry = get_or_create(session, Country, name=countryName)
That's basically the way to do it, there is no shortcut readily available AFAIK.
You could generalize it ofcourse:
def get_or_create(session, model, defaults=None, **kwargs):
instance = session.query(model).filter_by(**kwargs).one_or_none()
if instance:
return instance, False
else:
params = {k: v for k, v in kwargs.items() if not isinstance(v, ClauseElement)}
params.update(defaults or {})
instance = model(**params)
try:
session.add(instance)
session.commit()
except Exception: # The actual exception depends on the specific database so we catch all exceptions. This is similar to the official documentation: https://docs.sqlalchemy.org/en/latest/orm/session_transaction.html
session.rollback()
instance = session.query(model).filter_by(**kwargs).one()
return instance, False
else:
return instance, True
2020 update (Python 3.9+ ONLY)
Here is a cleaner version with Python 3.9's the new dict union operator (|=)
def get_or_create(session, model, defaults=None, **kwargs):
instance = session.query(model).filter_by(**kwargs).one_or_none()
if instance:
return instance, False
else:
kwargs |= defaults or {}
instance = model(**kwargs)
try:
session.add(instance)
session.commit()
except Exception: # The actual exception depends on the specific database so we catch all exceptions. This is similar to the official documentation: https://docs.sqlalchemy.org/en/latest/orm/session_transaction.html
session.rollback()
instance = session.query(model).filter_by(**kwargs).one()
return instance, False
else:
return instance, True
Note:
Similar to the Django version this will catch duplicate key constraints and similar errors. If your get or create is not guaranteed to return a single result it can still result in race conditions.
To alleviate some of that issue you would need to add another one_or_none() style fetch right after the session.commit(). This still is no 100% guarantee against race conditions unless you also use a with_for_update() or serializable transaction mode.
I've been playing with this problem and have ended up with a fairly robust solution:
def get_one_or_create(session,
model,
create_method='',
create_method_kwargs=None,
**kwargs):
try:
return session.query(model).filter_by(**kwargs).one(), False
except NoResultFound:
kwargs.update(create_method_kwargs or {})
created = getattr(model, create_method, model)(**kwargs)
try:
session.add(created)
session.flush()
return created, True
except IntegrityError:
session.rollback()
return session.query(model).filter_by(**kwargs).one(), False
I just wrote a fairly expansive blog post on all the details, but a few quite ideas of why I used this.
It unpacks to a tuple that tells you if the object existed or not. This can often be useful in your workflow.
The function gives the ability to work with #classmethod decorated creator functions (and attributes specific to them).
The solution protects against Race Conditions when you have more than one process connected to the datastore.
EDIT: I've changed session.commit() to session.flush() as explained in this blog post. Note that these decisions are specific to the datastore used (Postgres in this case).
EDIT 2: I’ve updated using a {} as a default value in the function as this is typical Python gotcha. Thanks for the comment, Nigel! If your curious about this gotcha, check out this StackOverflow question and this blog post.
A modified version of erik's excellent answer
def get_one_or_create(session,
model,
create_method='',
create_method_kwargs=None,
**kwargs):
try:
return session.query(model).filter_by(**kwargs).one(), True
except NoResultFound:
kwargs.update(create_method_kwargs or {})
try:
with session.begin_nested():
created = getattr(model, create_method, model)(**kwargs)
session.add(created)
return created, False
except IntegrityError:
return session.query(model).filter_by(**kwargs).one(), True
Use a nested transaction to only roll back the addition of the new item instead of rolling back everything (See this answer to use nested transactions with SQLite)
Move create_method. If the created object has relations and it is assigned members through those relations, it is automatically added to the session. E.g. create a book, which has user_id and user as corresponding relationship, then doing book.user=<user object> inside of create_method will add book to the session. This means that create_method must be inside with to benefit from an eventual rollback. Note that begin_nested automatically triggers a flush.
Note that if using MySQL, the transaction isolation level must be set to READ COMMITTED rather than REPEATABLE READ for this to work. Django's get_or_create (and here) uses the same stratagem, see also the Django documentation.
This SQLALchemy recipe does the job nice and elegant.
The first thing to do is to define a function that is given a Session to work with, and associates a dictionary with the Session() which keeps track of current unique keys.
def _unique(session, cls, hashfunc, queryfunc, constructor, arg, kw):
cache = getattr(session, '_unique_cache', None)
if cache is None:
session._unique_cache = cache = {}
key = (cls, hashfunc(*arg, **kw))
if key in cache:
return cache[key]
else:
with session.no_autoflush:
q = session.query(cls)
q = queryfunc(q, *arg, **kw)
obj = q.first()
if not obj:
obj = constructor(*arg, **kw)
session.add(obj)
cache[key] = obj
return obj
An example of utilizing this function would be in a mixin:
class UniqueMixin(object):
#classmethod
def unique_hash(cls, *arg, **kw):
raise NotImplementedError()
#classmethod
def unique_filter(cls, query, *arg, **kw):
raise NotImplementedError()
#classmethod
def as_unique(cls, session, *arg, **kw):
return _unique(
session,
cls,
cls.unique_hash,
cls.unique_filter,
cls,
arg, kw
)
And finally creating the unique get_or_create model:
from sqlalchemy import Column, Integer, String, create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
engine = create_engine('sqlite://', echo=True)
Session = sessionmaker(bind=engine)
class Widget(UniqueMixin, Base):
__tablename__ = 'widget'
id = Column(Integer, primary_key=True)
name = Column(String, unique=True, nullable=False)
#classmethod
def unique_hash(cls, name):
return name
#classmethod
def unique_filter(cls, query, name):
return query.filter(Widget.name == name)
Base.metadata.create_all(engine)
session = Session()
w1, w2, w3 = Widget.as_unique(session, name='w1'), \
Widget.as_unique(session, name='w2'), \
Widget.as_unique(session, name='w3')
w1b = Widget.as_unique(session, name='w1')
assert w1 is w1b
assert w2 is not w3
assert w2 is not w1
session.commit()
The recipe goes deeper into the idea and provides different approaches but I've used this one with great success.
The closest semantically is probably:
def get_or_create(model, **kwargs):
"""SqlAlchemy implementation of Django's get_or_create.
"""
session = Session()
instance = session.query(model).filter_by(**kwargs).first()
if instance:
return instance, False
else:
instance = model(**kwargs)
session.add(instance)
session.commit()
return instance, True
not sure how kosher it is to rely on a globally defined Session in sqlalchemy, but the Django version doesn't take a connection so...
The tuple returned contains the instance and a boolean indicating if the instance was created (i.e. it's False if we read the instance from the db).
Django's get_or_create is often used to make sure that global data is available, so I'm committing at the earliest point possible.
I slightly simplified #Kevin. solution to avoid wrapping the whole function in an if/else statement. This way there's only one return, which I find cleaner:
def get_or_create(session, model, **kwargs):
instance = session.query(model).filter_by(**kwargs).first()
if not instance:
instance = model(**kwargs)
session.add(instance)
return instance
There is a Python package that has #erik's solution as well as a version of update_or_create(). https://github.com/enricobarzetti/sqlalchemy_get_or_create
Depending on the isolation level you adopted, none of the above solutions would work.
The best solution I have found is a RAW SQL in the following form:
INSERT INTO table(f1, f2, unique_f3)
SELECT 'v1', 'v2', 'v3'
WHERE NOT EXISTS (SELECT 1 FROM table WHERE f3 = 'v3')
This is transactionally safe whatever the isolation level and the degree of parallelism are.
Beware: in order to make it efficient, it would be wise to have an INDEX for the unique column.
One problem I regularly encounter is when a field has a max length (say, STRING(40)) and you'd like to perform a get or create with a string of large length, the above solutions will fail.
Building off of the above solutions, here's my approach:
from sqlalchemy import Column, String
def get_or_create(self, add=True, flush=True, commit=False, **kwargs):
"""
Get the an entity based on the kwargs or create an entity with those kwargs.
Params:
add: (default True) should the instance be added to the session?
flush: (default True) flush the instance to the session?
commit: (default False) commit the session?
kwargs: key, value pairs of parameters to lookup/create.
Ex: SocialPlatform.get_or_create(**{'name':'facebook'})
returns --> existing record or, will create a new record
---------
NOTE: I like to add this as a classmethod in the base class of my tables, so that
all data models inherit the base class --> functionality is transmitted across
all orm defined models.
"""
# Truncate values if necessary
for key, value in kwargs.items():
# Only use strings
if not isinstance(value, str):
continue
# Only use if it's a column
my_col = getattr(self.__table__.columns, key)
if not isinstance(my_col, Column):
continue
# Skip non strings again here
if not isinstance(my_col.type, String):
continue
# Get the max length
max_len = my_col.type.length
if value and max_len and len(value) > max_len:
# Update the value
value = value[:max_len]
kwargs[key] = value
# -------------------------------------------------
# Make the query...
instance = session.query(self).filter_by(**kwargs).first()
if instance:
return instance
else:
# Max length isn't accounted for here.
# The assumption is that auto-truncation will happen on the child-model
# Or directtly in the db
instance = self(**kwargs)
# You'll usually want to add to the session
if add:
session.add(instance)
# Navigate these with caution
if add and commit:
try:
session.commit()
except IntegrityError:
session.rollback()
elif add and flush:
session.flush()
return instance