SQLalchemy find id and use it to lookup other information - python

I'm making a simple lookup application for Japanese characters (Kanji), where the user can search the database using any of the information available.
My database structure
Kanji:
id
character (A kanji like 頑)
heisig6 (a number indicating the order of showing Kanji)
kanjiorigin (a number indicating the order of showing Kanji)
MeaningEN (1 kanji_id can have multiple entries with different meanings):
kanji_id (FOREIGN KEY(kanji_id) REFERENCES "Kanji" (id)
meaning
User handling
The user can choose to search by 'id', 'character', 'heisig6', 'kanjiorigin' or 'meaning' and it should then return all information in all those fields. (All fields return only 1 result, except meanings, which can return multiple results)
Code, EDIT 4+5: my code with thanks to #ApolloFortyNine and #sqlalchemy on IRC, EDIT 6: join --> outerjoin (otherwise won't find information that has no Origins)
import sqlalchemy as sqla
import sqlalchemy.orm as sqlo
from tableclass import TableKanji, TableMeaningEN, TableMisc, TableOriginKanji # See tableclass.py
# Searches database with argument search method
class SearchDatabase():
def __init__(self):
#self.db_name = "sqlite:///Kanji_story.db"
self.engine = sqla.create_engine("sqlite:///Kanji.db", echo=True)
# Bind the engine to the metadata of the Base class so that the
# declaratives can be accessed through a DBSession instance
tc.sqla_base.metadata.bind = self.engine
# For making sessions to connect to db
self.db_session = sqlo.sessionmaker(bind=self.engine)
def retrieve(self, s_input, s_method):
# s_input: search input
# s_method: search method
print("\nRetrieving results with input: {} and method: {}".format(s_input, s_method))
data = [] # Data to return
# User searches on non-empty string
if s_input:
session = self.db_session()
# Find id in other table than Kanji
if s_method == 'meaning':
s_table = TableMeaningEN # 'MeaningEN'
elif s_method == 'okanji':
s_table = TableOriginKanji # 'OriginKanji'
else:
s_table = TableKanji # 'Kanji'
result = session.query(TableKanji).outerjoin(TableMeaningEN).outerjoin(
(TableOriginKanji, TableKanji.origin_kanji)
).filter(getattr(s_table, s_method) == s_input).all()
print("result: {}".format(result))
for r in result:
print("r: {}".format(r))
meanings = [m.meaning for m in r.meaning_en]
print(meanings)
# TODO transform into origin kanji's
origins = [str(o.okanji_id) for o in r.okanji_id]
print(origins)
data.append({'character': r.character, 'meanings': meanings,
'indexes': [r.id, r.heisig6, r.kanjiorigin], 'origins': origins})
session.close()
if not data:
data = [{'character': 'X', 'meanings': ['invalid', 'search', 'result']}]
return(data)
Question EDIT 4+5
Is this an efficient query?: result = session.query(TableKanji).join(TableMeaningEN).filter(getattr(s_table, s_method) == s_input).all() (The .join statement is necessary, because otherwise e.g. session.query(TableKanji).filter(TableMeaningEN.meaning == 'love').all() returns all the meanings in my database for some reason? So is this either the right query or is my relationship() in my tableclass.py not properly defined?
fixed (see lambda: in tableclass.py) kanji = relationship("TableKanji", foreign_keys=[kanji_id], back_populates="OriginKanji") <-- what is wrong about this? It gives the error:
File "/path/python3.5/site-packages/sqlalchemy/orm/mapper.py", line 1805, in get_property
"Mapper '%s' has no property '%s'" % (self, key))
sqlalchemy.exc.InvalidRequestError: Mapper 'Mapper|TableKanji|Kanji' has no property 'OriginKanji'
Edit 2: tableclass.py (EDIT 3+4+5: updated)
import sqlalchemy as sqla
from sqlalchemy.orm import relationship
import sqlalchemy.ext.declarative as sqld
sqla_base = sqld.declarative_base()
class TableKanji(sqla_base):
__tablename__ = 'Kanji'
id = sqla.Column(sqla.Integer, primary_key=True)
character = sqla.Column(sqla.String, nullable=False)
radical = sqla.Column(sqla.Integer) # Can be defined as Boolean
heisig6 = sqla.Column(sqla.Integer, unique=True, nullable=True)
kanjiorigin = sqla.Column(sqla.Integer, unique=True, nullable=True)
cjk = sqla.Column(sqla.String, unique=True, nullable=True)
meaning_en = relationship("TableMeaningEN", back_populates="kanji") # backref="Kanji")
okanji_id = relationship("TableOriginKanji", foreign_keys=lambda: TableOriginKanji.kanji_id, back_populates="kanji")
class TableMeaningEN(sqla_base):
__tablename__ = 'MeaningEN'
kanji_id = sqla.Column(sqla.Integer, sqla.ForeignKey('Kanji.id'), primary_key=True)
meaning = sqla.Column(sqla.String, primary_key=True)
kanji = relationship("TableKanji", back_populates="meaning_en")
class TableOriginKanji(sqla_base):
__tablename__ = 'OriginKanji'
kanji_id = sqla.Column(sqla.Integer, sqla.ForeignKey('Kanji.id'), primary_key=True)
okanji_id = sqla.Column(sqla.Integer, sqla.ForeignKey('Kanji.id'), primary_key=True)
order = sqla.Column(sqla.Integer)
#okanji = relationship("TableKanji", foreign_keys=[kanji_id], backref="okanji")
kanji = relationship("TableKanji", foreign_keys=[kanji_id], back_populates="okanji_id")

We would really have to be able to see your database schema to give real critique, but assuming no foreign keys, what you said is basically the best you can do.
SQLAlchemy really begins to shine when you have complicated relations going on however. For example, if you properly had foreign keys set, you could do something like the following.
# Assuming kanji is a tc.tableMeaningEN.kanji_id object
kanji_meaning = kanji.meanings
And that would return the meanings for the kanji as an array, without any further queries.
You can go quite deep with relationships, so I'm linking the documentation here. http://docs.sqlalchemy.org/en/latest/orm/relationships.html
EDIT: Actually, you don't need to manually join at all, SQLAlchemy will do it for you.
The case is wrong on your classes, but I'm not sure if SQLAlchemy is case sensitive there or not. If it works, then just move on.
If you query the a table (self.session.query(User).filter(User.username == self.name).first()) you should have an object of the table type (User here).
So in your case, querying the TableKanji table alone will return an object of that type.
kanji_obj = session.query(TableKanji).filter(TableKanji.id == id).first()
# This will return an array of all meaning_ens that match the foreign key
meaning_arr = kanji_obj.meaning_en
# This will return a single meeting, just to show each member of the arr is of type TableMeaningEn
meaning_arr[0].meaning
I have a project made use of some of these features, hope it helps:
https://github.com/ApolloFortyNine/SongSense
Database declaration (with relationships): https://github.com/ApolloFortyNine/SongSense/blob/master/songsense/database.py
Automatic joins: https://github.com/ApolloFortyNine/SongSense/blob/master/songsense/getfriend.py#L134
I really like my database structure, but as for the rest it's pretty awful. Hope it still helps though.

Related

Appending an entry to an existing object (Relationships) : Flask-SQLAlchemy

The main issue is with me trying to use the append function, and getting an SQL integrity error when I call session.commit(), even when the object is not being re-created. Here's my case:
I have a one to many relationship in a Flask SQLAlchemy database.
It's a Tag-Category relationship, tags belong to only one category, and by definition a category has many tags. For the sake of explanation, tags are being used on X objects.
In my Flask models file, I did this:
class Category(db.Model):
__tablename__ = 'category'
id = db.Column(db.Integer, primary_key = True)
category = db.Column(db.Text, unique = True)
tags = db.relationship('Tag', backref = 'category', lazy = 'dynamic')
class Tag(db.Model):
__tablename__ = 'tags'
id = db.Column(db.Integer, primary_key = True)
tag = db.Column(db.Text, unique = True)
category_id = db.Column(db.Integer, db.ForeignKey('category.id'))
This should define a Category class to which I can "append" tags using the session.append(Tag) function.
Categories and tags are unique, I wanna take input from the user, and add the category/tag as the user specifies. In some cases, the user might want to just add a tag to an existing category.
The way I'm taking input is by using a form, from which I receive the category and tag, and I create the objects and add them to the database if they don't exist already, and if they do, I just want to append the tags to the X's or to the categories.
Here's my code:
category = Category(category = request.form['categories']) #Category related to its respective list of tags
tag_list = request.form.getlist('tags') # tags are received as a list
for tag_entity in tag_list:
tag = Tag(tag = tag_entity)
X.tags.append(tag)
# Checking if the category exists or not in the database
list1 = []
if len(db.session.query(Category.tags).filter(Category.category == request.form['categories']).all()) != 0:
list1 = db.session.query(Category.tags).filter(Category.category == request.form['categories']).all()
list1.append(tag)
for entry in list1:
print entry.tag
list1 = [entry.tag for entry in list1]
# Here I should check if the tag already exists as well:
db.session.add(tag)
# Only add category if it doesn't already exist
if len(db.session.query(Category.id).filter(Category.category == request.form['categories']).all()) == 0:
print "I'm here"
db.session.add(Category(category = request.form['categories'], tags = list1))
else:
print "woops im here"
newCat = Category(category=request.form['categories'])
newCat.tags.extend(list1)
db.session.merge(newCat)
db.session.add(X)
db.session.commit()
I know my code is really messy and it looks like it doesn't do what it's supposed to, that's a result of me just failing over and over again.
My main issue was with me trying to call category.tags.append(Tag), then getting an integrity error when session.commit() was being called even when the category was not even re-created.
I hope what I wrote makes sense, any help would be appreciated.

SQLalchemy ORM add to session with dynamic column

I have 2 tables (not the true setup of my database).
tableclass.py:
class TableMeaningEN(sqla_base):
__tablename__ = 'MeaningEN'
id = sqla.Column(sqla.Integer, primary_key=True)
meaning = sqla.Column(sqla.String, primary_key=True)
class TableReadingON(sqla_base):
__tablename__ = 'ReadingON'
id = sqla.Column(sqla.Integer, primary_key=True)
reading = sqla.Column(sqla.String, primary_key=True)
Different column names
As you can see, both have a column id, but TableMeaningEN has meaning and TableReadingON has reading.
Normally (assuming you already have a session) you would add something like this:
session.add(TableMeaningEN(id=1, meaning='test'))
However I want to add dynamically entries to tables, so I have:
import tableclass as tc
for t_name in ['MeaningEN', 'ReadingON']:
session.add(getattr(tc, 'Table{}'.format(t_name))(id=1, ??='test'))
Question
How do I solve the problem that ?? is in one table meaning and in the other reading?
Tried:
columns = sqla.inspect(getattr(tc, 'Table{}'.format(overwrite))).columns.keys()
session.add(getattr(tc, 'Table{}'.format(t_name))(id=1, columns[1]='test'))
I tried this, but however that's not allowed.
If tables are really different by just one attribute, you could create a constructor for both which takes positional arguments as well, such as:
class TableMeaningEN(sqla_base):
def __init__(self, id, meaning):
self.id, self.meaning = id, meaning
# similar __init__ for the other class/table
# then use the following:
session.add(getattr(tc, 'Table{}'.format(t_name))(1, 'test'))
An alternative would be to dynamically create the keyword arguments, assuming there is naming convention:
import tableclass as tc
for t_name in ['MeaningEN', 'ReadingON']:
cls = getattr(tc, 'Table{}'.format(t_name))
fld_name = t_name.lower()[:-2]
kw = {'id': 1, fld_name: 'test'}
session.add(cls(**kw))

sqlalchemy generic foreign key (like in django ORM)

Does sqlalchemy have something like django's GenericForeignKey? And is it right to use generic foreign fields.
My problem is: I have several models (for example, Post, Project, Vacancy, nothing special there) and I want to add comments to each of them. And I want to use only one Comment model. Does it worth to? Or should I use PostComment, ProjectComment etc.? Pros/cons of both ways?
Thanks!
The simplest pattern which I use most often is that you actually have separate Comment tables for each relationship. This may seem frightening at first, but it doesn't incur any additional code versus using any other approach - the tables are created automatically, and the models are referred to using the pattern Post.Comment, Project.Comment, etc. The definition of Comment is maintained in one place. This approach from a referential point of view is the most simple and efficient, as well as the most DBA friendly as different kinds of Comments are kept in their own tables which can be sized individually.
Another pattern to use is a single Comment table, but distinct association tables. This pattern offers the use case that you might want a Comment linked to more than one kind of object at a time (like a Post and a Project at the same time). This pattern is still reasonably efficient.
Thirdly, there's the polymorphic association table. This pattern uses a fixed number of tables to represent the collections and the related class without sacrificing referential integrity. This pattern tries to come the closest to the Django-style "generic foreign key" while still maintaining referential integrity, though it's not as simple as the previous two approaches.
Imitating the pattern used by ROR/Django, where there are no real foreign keys used and rows are matched using application logic, is also possible.
The first three patterns are illustrated in modern form in the SQLAlchemy distribution under examples/generic_associations/.
The ROR/Django pattern, since it gets asked about so often, I will also add to the SQLAlchemy examples, even though I don't like it much. The approach I'm using is not exactly the same as what Django does as they seem to make use of a "contenttypes" table to keep track of types, that seems kind of superfluous to me, but the general idea of an integer column that points to any number of tables based on a discriminator column is present. Here it is:
from sqlalchemy.ext.declarative import declarative_base, declared_attr
from sqlalchemy import create_engine, Integer, Column, \
String, and_
from sqlalchemy.orm import Session, relationship, foreign, remote, backref
from sqlalchemy import event
class Base(object):
"""Base class which provides automated table name
and surrogate primary key column.
"""
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
id = Column(Integer, primary_key=True)
Base = declarative_base(cls=Base)
class Address(Base):
"""The Address class.
This represents all address records in a
single table.
"""
street = Column(String)
city = Column(String)
zip = Column(String)
discriminator = Column(String)
"""Refers to the type of parent."""
parent_id = Column(Integer)
"""Refers to the primary key of the parent.
This could refer to any table.
"""
#property
def parent(self):
"""Provides in-Python access to the "parent" by choosing
the appropriate relationship.
"""
return getattr(self, "parent_%s" % self.discriminator)
def __repr__(self):
return "%s(street=%r, city=%r, zip=%r)" % \
(self.__class__.__name__, self.street,
self.city, self.zip)
class HasAddresses(object):
"""HasAddresses mixin, creates a relationship to
the address_association table for each parent.
"""
#event.listens_for(HasAddresses, "mapper_configured", propagate=True)
def setup_listener(mapper, class_):
name = class_.__name__
discriminator = name.lower()
class_.addresses = relationship(Address,
primaryjoin=and_(
class_.id == foreign(remote(Address.parent_id)),
Address.discriminator == discriminator
),
backref=backref(
"parent_%s" % discriminator,
primaryjoin=remote(class_.id) == foreign(Address.parent_id)
)
)
#event.listens_for(class_.addresses, "append")
def append_address(target, value, initiator):
value.discriminator = discriminator
class Customer(HasAddresses, Base):
name = Column(String)
class Supplier(HasAddresses, Base):
company_name = Column(String)
engine = create_engine('sqlite://', echo=True)
Base.metadata.create_all(engine)
session = Session(engine)
session.add_all([
Customer(
name='customer 1',
addresses=[
Address(
street='123 anywhere street',
city="New York",
zip="10110"),
Address(
street='40 main street',
city="San Francisco",
zip="95732")
]
),
Supplier(
company_name="Ace Hammers",
addresses=[
Address(
street='2569 west elm',
city="Detroit",
zip="56785")
]
),
])
session.commit()
for customer in session.query(Customer):
for address in customer.addresses:
print(address)
print(address.parent)
I know this is probably a terrible way to do this, but it was a quick fix for me.
class GenericRelation(object):
def __init__(self, object_id, object_type):
self.object_id = object_id
self.object_type = object_type
def __composite_values__(self):
return (self.object_id, self.object_type)
class Permission(AbstractBase):
#__abstract__ = True
_object = None
_generic = composite(
GenericRelation,
sql.Column('object_id', data_types.UUID, nullable=False),
sql.Column('object_type', sql.String, nullable=False),
)
permission_type = sql.Column(sql.Integer)
#property
def object(self):
session = object_session(self)
if self._object or not session:
return self._object
else:
object_class = eval(self.object_type)
self._object = session.query(object_class).filter(object_class.id == self.object_id).first()
return self._object
#object.setter
def object(self, value):
self._object = value
self.object_type = value.__class__.__name__
self.object_id = value.id

Reverse mapping from a table to a model in SQLAlchemy

To provide an activity log in my SQLAlchemy-based app, I have a model like this:
class ActivityLog(Base):
__tablename__ = 'activitylog'
id = Column(Integer, primary_key=True)
activity_by_id = Column(Integer, ForeignKey('users.id'), nullable=False)
activity_by = relation(User, primaryjoin=activity_by_id == User.id)
activity_at = Column(DateTime, default=datetime.utcnow, nullable=False)
activity_type = Column(SmallInteger, nullable=False)
target_table = Column(Unicode(20), nullable=False)
target_id = Column(Integer, nullable=False)
target_title = Column(Unicode(255), nullable=False)
The log contains entries for multiple tables, so I can't use ForeignKey relations. Log entries are made like this:
doc = Document(name=u'mydoc', title=u'My Test Document',
created_by=user, edited_by=user)
session.add(doc)
session.flush() # See note below
log = ActivityLog(activity_by=user, activity_type=ACTIVITY_ADD,
target_table=Document.__table__.name, target_id=doc.id,
target_title=doc.title)
session.add(log)
This leaves me with three problems:
I have to flush the session before my doc object gets an id. If I had used a ForeignKey column and a relation mapper, I could have simply called ActivityLog(target=doc) and let SQLAlchemy do the work. Is there any way to work around needing to flush by hand?
The target_table parameter is too verbose. I suppose I could solve this with a target property setter in ActivityLog that automatically retrieves the table name and id from a given instance.
Biggest of all, I'm not sure how to retrieve a model instance from the database. Given an ActivityLog instance log, calling self.session.query(log.target_table).get(log.target_id) does not work, as query() expects a model as parameter.
One workaround appears to be to use polymorphism and derive all my models from a base model which ActivityLog recognises. Something like this:
class Entity(Base):
__tablename__ = 'entities'
id = Column(Integer, primary_key=True)
title = Column(Unicode(255), nullable=False)
edited_at = Column(DateTime, onupdate=datetime.utcnow, nullable=False)
entity_type = Column(Unicode(20), nullable=False)
__mapper_args__ = {'polymorphic_on': entity_type}
class Document(Entity):
__tablename__ = 'documents'
__mapper_args__ = {'polymorphic_identity': 'document'}
body = Column(UnicodeText, nullable=False)
class ActivityLog(Base):
__tablename__ = 'activitylog'
id = Column(Integer, primary_key=True)
...
target_id = Column(Integer, ForeignKey('entities.id'), nullable=False)
target = relation(Entity)
If I do this, ActivityLog(...).target will give me a Document instance when it refers to a Document, but I'm not sure it's worth the overhead of having two tables for everything. Should I go ahead and do it this way?
One way to solve this is polymorphic associations. It should solve all 3 of your issues and also make database foreign key constraints work. See the polymorphic association example in SQLAlchemy source. Mike Bayer has an old blogpost that discusses this in greater detail.
Definitely go through the blogpost and examples Ants linked to. I did not find the explanation confusion, but rather assuming some more experience on the topic.
Few things I can suggest are:
ForeignKeys: in general I agree they are a good thing go have, but I am not sure it is conceptually important in your case: you seem to be using this ActivityLog as an orthogonal cross-cutting concern (AOP); but version with foreign keys would effectively make your business objects aware of the ActivityLog. Another problem with having FK for audit purposes using schema setup you have is that if you allow object deletion, FK requirement will delete all the ActivityLog entries for this object.
Automatic logging: you are doing all this logging manually whenever you create/modify(/delete) the object. With SA you could implement a SessionExtension with before_commit which would do the job for you automatically.
In this way you completely can avoid writing parts like below:
log = ActivityLog(activity_by=user, activity_type=ACTIVITY_ADD,
target_table=Document.__table__.name, target_id=doc.id,
target_title=doc.title)
session.add(log)
EDIT-1: complete sample code added
The code is based on the first non-FK version from http://techspot.zzzeek.org/?p=13.
The choice not to use FK is based on the fact that for audit purposes when the
main object is deleted, it should not cascade to delete all the audit log entries.
Also this keeps auditable objects unaware of the fact they are being audited.
Implementation uses a SA one-to-many relationship. It is possible that some
objects are modified many times, which will result in many audit log entries.
By default SA will load the relationship objects when adding a new entry to the
list. Assuming that during "normal" usage we would like only to add new audit
log entry, we use lazy='noload' flag so that the relation from the main object
will never be loaded. It is loaded when navigated from the other side though,
and also can be loaded from the main object using custom query, which is shown
in the example as well using activitylog_readonly readonly property.
Code (runnable with some tests):
from datetime import datetime
from sqlalchemy import create_engine, Column, Integer, SmallInteger, String, DateTime, ForeignKey, Table, UnicodeText, Unicode, and_
from sqlalchemy.orm import relationship, dynamic_loader, scoped_session, sessionmaker, class_mapper, backref
from sqlalchemy.orm.session import Session
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm.interfaces import SessionExtension
import logging
logging.basicConfig(level=logging.INFO)
_logger = logging.getLogger()
ACTIVITY_ADD = 1
ACTIVITY_MOD = 2
ACTIVITY_DEL = 3
class ActivityLogSessionExtension(SessionExtension):
_logger = logging.getLogger('ActivityLogSessionExtension')
def before_commit(self, session):
self._logger.debug("before_commit: %s", session)
for d in session.new:
self._logger.info("before_commit >> add: %s", d)
if hasattr(d, 'create_activitylog'):
log = d.create_activitylog(ACTIVITY_ADD)
for d in session.dirty:
self._logger.info("before_commit >> mod: %s", d)
if hasattr(d, 'create_activitylog'):
log = d.create_activitylog(ACTIVITY_MOD)
for d in session.deleted:
self._logger.info("before_commit >> del: %s", d)
if hasattr(d, 'create_activitylog'):
log = d.create_activitylog(ACTIVITY_DEL)
# Configure test data SA
engine = create_engine('sqlite:///:memory:', echo=False)
session = scoped_session(sessionmaker(bind=engine, autoflush=False, extension=ActivityLogSessionExtension()))
Base = declarative_base()
Base.query = session.query_property()
class _BaseMixin(object):
""" Just a helper mixin class to set properties on object creation.
Also provides a convenient default __repr__() function, but be aware that
also relationships are printed, which might result in loading relations.
"""
def __init__(self, **kwargs):
for k,v in kwargs.items():
setattr(self, k, v)
def __repr__(self):
return "<%s(%s)>" % (self.__class__.__name__,
', '.join('%s=%r' % (k, self.__dict__[k])
for k in sorted(self.__dict__) if '_sa_' != k[:4] and '_backref_' != k[:9])
)
class User(Base, _BaseMixin):
__tablename__ = u'users'
id = Column(Integer, primary_key=True)
name = Column(String)
class Document(Base, _BaseMixin):
__tablename__ = u'documents'
id = Column(Integer, primary_key=True)
title = Column(Unicode(255), nullable=False)
body = Column(UnicodeText, nullable=False)
class Folder(Base, _BaseMixin):
__tablename__ = u'folders'
id = Column(Integer, primary_key=True)
title = Column(Unicode(255), nullable=False)
comment = Column(UnicodeText, nullable=False)
class ActivityLog(Base, _BaseMixin):
__tablename__ = u'activitylog'
id = Column(Integer, primary_key=True)
activity_by_id = Column(Integer, ForeignKey('users.id'), nullable=False)
activity_by = relationship(User) # #note: no need to specify the primaryjoin
activity_at = Column(DateTime, default=datetime.utcnow, nullable=False)
activity_type = Column(SmallInteger, nullable=False)
target_table = Column(Unicode(20), nullable=False)
target_id = Column(Integer, nullable=False)
target_title = Column(Unicode(255), nullable=False)
# backref relation for auditable
target = property(lambda self: getattr(self, '_backref_%s' % self.target_table))
def _get_user():
""" This method returns the User object for the current user.
#todo: proper implementation required
#hack: currently returns the 'user2'
"""
return session.query(User).filter_by(name='user2').one()
# auditable support function
# based on first non-FK version from http://techspot.zzzeek.org/?p=13
def auditable(cls, name):
def create_activitylog(self, activity_type):
log = ActivityLog(activity_by=_get_user(),
activity_type=activity_type,
target_table=table.name,
target_title=self.title,
)
getattr(self, name).append(log)
return log
mapper = class_mapper(cls)
table = mapper.local_table
cls.create_activitylog = create_activitylog
def _get_activitylog(self):
return Session.object_session(self).query(ActivityLog).with_parent(self).all()
setattr(cls, '%s_readonly' %(name,), property(_get_activitylog))
# no constraints, therefore define constraints in an ad-hoc fashion.
primaryjoin = and_(
list(table.primary_key)[0] == ActivityLog.__table__.c.target_id,
ActivityLog.__table__.c.target_table == table.name
)
foreign_keys = [ActivityLog.__table__.c.target_id]
mapper.add_property(name,
# #note: because we use the relationship, by default all previous
# ActivityLog items will be loaded for an object when new one is
# added. To avoid this, use either dynamic_loader (http://www.sqlalchemy.org/docs/reference/orm/mapping.html#sqlalchemy.orm.dynamic_loader)
# or lazy='noload'. This is the trade-off decision to be made.
# Additional benefit of using lazy='noload' is that one can also
# record DEL operations in the same way as ADD, MOD
relationship(
ActivityLog,
lazy='noload', # important for relationship
primaryjoin=primaryjoin,
foreign_keys=foreign_keys,
backref=backref('_backref_%s' % table.name,
primaryjoin=list(table.primary_key)[0] == ActivityLog.__table__.c.target_id,
foreign_keys=foreign_keys)
)
)
# this will define which classes support the ActivityLog interface
auditable(Document, 'activitylogs')
auditable(Folder, 'activitylogs')
# create db schema
Base.metadata.create_all(engine)
## >>>>> TESTS >>>>>>
# create some basic data first
u1 = User(name='user1')
u2 = User(name='user2')
session.add(u1)
session.add(u2)
session.commit()
session.expunge_all()
# --check--
assert not(_get_user() is None)
##############################
## ADD
##############################
_logger.info('-' * 80)
d1 = Document(title=u'Document-1', body=u'Doc1 some body skipped the body')
# when not using SessionExtension for any reason, this can be called manually
#d1.create_activitylog(ACTIVITY_ADD)
session.add(d1)
session.commit()
f1 = Folder(title=u'Folder-1', comment=u'This folder is empty')
# when not using SessionExtension for any reason, this can be called manually
#f1.create_activitylog(ACTIVITY_ADD)
session.add(f1)
session.commit()
# --check--
session.expunge_all()
logs = session.query(ActivityLog).all()
_logger.debug(logs)
assert len(logs) == 2
assert logs[0].activity_type == ACTIVITY_ADD
assert logs[0].target.title == u'Document-1'
assert logs[0].target.title == logs[0].target_title
assert logs[1].activity_type == ACTIVITY_ADD
assert logs[1].target.title == u'Folder-1'
assert logs[1].target.title == logs[1].target_title
##############################
## MOD(ify)
##############################
_logger.info('-' * 80)
session.expunge_all()
d1 = session.query(Document).filter_by(id=1).one()
assert d1.title == u'Document-1'
assert d1.body == u'Doc1 some body skipped the body'
assert d1.activitylogs == []
d1.title = u'Modified: Document-1'
d1.body = u'Modified: body'
# when not using SessionExtension (or it does not work, this can be called manually)
#d1.create_activitylog(ACTIVITY_MOD)
session.commit()
_logger.debug(d1.activitylogs_readonly)
# --check--
session.expunge_all()
logs = session.query(ActivityLog).all()
assert len(logs)==3
assert logs[2].activity_type == ACTIVITY_MOD
assert logs[2].target.title == u'Modified: Document-1'
assert logs[2].target.title == logs[2].target_title
##############################
## DEL(ete)
##############################
_logger.info('-' * 80)
session.expunge_all()
d1 = session.query(Document).filter_by(id=1).one()
# when not using SessionExtension for any reason, this can be called manually,
#d1.create_activitylog(ACTIVITY_DEL)
session.delete(d1)
session.commit()
session.expunge_all()
# --check--
session.expunge_all()
logs = session.query(ActivityLog).all()
assert len(logs)==4
assert logs[0].target is None
assert logs[2].target is None
assert logs[3].activity_type == ACTIVITY_DEL
assert logs[3].target is None
##############################
## print all activity logs
##############################
_logger.info('=' * 80)
logs = session.query(ActivityLog).all()
for log in logs:
_ = log.target
_logger.info("%s -> %s", log, log.target)
##############################
## navigate from main object
##############################
_logger.info('=' * 80)
session.expunge_all()
f1 = session.query(Folder).filter_by(id=1).one()
_logger.info(f1.activitylogs_readonly)

Python dicts in sqlalchemy

I would like to load/save a dict to/from my sqlite DB, but am having some problems figuring out a simple way to do it. I don't really need to be able to filter, etc., based on the contents so a simple conversion to/from string is fine.
The next-best thing would be foreign keys. Please don't post links to huge examples, my head would explode if I ever set eyes on any those.
The SQLAlchemy PickleType is meant exactly for this.
class SomeEntity(Base):
__tablename__ = 'some_entity'
id = Column(Integer, primary_key=True)
attributes = Column(PickleType)
# Just set the attribute to save it
s = SomeEntity(attributes={'baked': 'beans', 'spam': 'ham'})
session.add(s)
session.commit()
# If mutable=True on PickleType (the default) SQLAlchemy automatically
# notices modifications.
s.attributes['parrot'] = 'dead'
session.commit()
You can change the serialization mechanism by changing out the pickler with something else that has dumps() and loads() methods. The underlying storage mechanism by subclassing PickleType and overriding the impl attritbute:
class TextPickleType(PickleType):
impl = Text
import json
class SomeOtherEntity(Base):
__tablename__ = 'some_other_entity'
id = Column(Integer, primary_key=True)
attributes = Column(TextPickleType(pickler=json))
You can create a custom type by subclassing sqlalchemy.types.TypeDecorator to handle serialization and deserialization to Text.
An implementation might look like
import json
import sqlalchemy
from sqlalchemy.types import TypeDecorator
SIZE = 256
class TextPickleType(TypeDecorator):
impl = sqlalchemy.Text(SIZE)
def process_bind_param(self, value, dialect):
if value is not None:
value = json.dumps(value)
return value
def process_result_value(self, value, dialect):
if value is not None:
value = json.loads(value)
return value
Example usage:
class SomeModel(Base):
__tablename__ = 'the_table'
id = Column(Integer, primary_key=True)
json_field = Column(TextPickleType())
s = SomeModel(json_field={'baked': 'beans', 'spam': 'ham'})
session.add(s)
session.commit()
This is outlined in an example in the SQLAlchemy docs, which also shows how to track mutations of that dictionary.
This approach should work for all versions of Python, whereas simply passing json as the value to the pickler argument of PickleType will not work correctly, as AlexGrönholm points out in his comment on another answer.
SQLAlchemy has a built-in JSON type that you can use:
attributes = Column(JSON)
If you need to map a 1-N relation and map it as dict rather then list, then read Custom Dictionary-Based Collections
But if you mean a field, then what you can do it to have a DB field of type string, which is mapped to your Python object. But on the same python object you provide a property which will be kind-of proxy for this mapped string field of type dict().
Code example (not tested):
class MyObject(object):
# fields (mapped automatically by sqlalchemy using mapper(...)
MyFieldAsString = None
def _get_MyFieldAsDict(self):
if self.MyFieldAsString:
return eval(self.MyFieldAsString)
else:
return {} # be careful with None and empty dict
def _set_MyFieldAsDict(self, value):
if value:
self.MyFieldAsString = str(value)
else:
self.MyFieldAsString = None
MyFieldAsDict = property(_get_MyFieldAsDict, _set_MyFieldAsDict)
You can simply save() method to save dicts in sqlalchemy
For example
class SomeModel(Base):
__tablename__ = 'the_table'
id = Column(Integer, primary_key=True)
baked = Column(String, nullable=True)
spam = Column(String, nullable=True)
s = {'baked': 'beans', 'spam': 'ham'})
SomeModel(**s).save()

Categories

Resources