I have a following table in sqlalchemy:
class FieldType(enum.Enum):
INT_FIELD = 0
FLOAT_FIELD = 1
STRING_FIELD = 2
class EAVTable(Base):
__tablename__ = 'EAVTable'
field_name = Column(Stirng, primary_key=True)
field_type = Column(Enum(FieldType))
int_field = Column(Integer)
float_field = Column(Float)
string_field = Column(String)
This is to model the EAV model which fits my business purpose.
Now to use it easily in the code I have the following hybrid_property.
#hybrid_propderty
def value(self):
if self.field_type == FieldType.INT_FIELD:
return self.int_field
...
#value.setter
def value(self, value):
if type(value) == int:
self.field_type = FieldType.INT_FIELD
self.int_field = value
...
This works fine when I try to get and set the fields in Python code. But I still have a problem:
session.query(EAVTable).filter(EAVTable.value == 123)
This does not work out of the box but I had an idea of using hybrid.expression where we use a case statement:
#value.expression
def value(cls):
return case(
[
(cls.field_type == FieldType.INT_FIELD, cls.int_field),
(cls.field_type == FieldType.FLOAT_FIELD, cls.float_field),
...
]
)
This in theory works, for example, the SQL generated for query session.query(EAVTable.value = 123 looks like:
select * from where case
when field_type = INT_FIELD then int_field
when field_type = FLOAT_FIELD then float_field
when field_type = STRING_FIELD then string_field
end = 123;
Which semantically looks like what I like, but later I find that the case expression requires all the cases have the same type, or they are cast into the same type.
I understand this is a requirement from the SQL language and has nothing to do with sqlachemy, but for more seasoned sqlalchemy user, is there any easy way to do what I want to achieve? Is there a way to walk around this constraint?
You could move the comparison inside the CASE expression using a custom comparator:
from sqlalchemy.ext.hybrid import Comparator
class PolymorphicComparator(Comparator):
def __init__(self, cls):
self.cls = cls
def __clause_element__(self):
# Since SQL doesn't allow polymorphism here, don't bother trying.
raise NotImplementedError(
f"{type(self).__name__} cannot be used as a clause")
def operate(self, op, other):
cls = self.cls
return case(
[
(cls.field_type == field_type, op(field, other))
for field_type, field in [
(FieldType.INT_FIELD, cls.int_field),
(FieldType.FLOAT_FIELD, cls.float_field),
(FieldType.STRING_FIELD, cls.string_field),
]
],
else_=False
)
class EAVTable(Base):
...
# This replaces #value.expression
#value.comparator
def value(cls):
return PolymorphicComparator(cls)
This way the common type is just boolean.
I have a very simple ChoiceString custom column/data type:
class ChoiceString(types.TypeDecorator):
impl = types.String
def __init__(self, choices, **kw):
self.choices = dict(choices)
super(ChoiceString, self).__init__(**kw)
def process_bind_param(self, value, dialect):
return [k for k, v in self.choices.iteritems() if v == value][0]
def process_result_value(self, value, dialect):
return self.choices[value]
And I am iterating over the table columns using a mapper:
from sqlalchemy.inspection import inspect
mapper = inspect(SomeTableClass)
for col in mapper.columns:
print col
# how to check the choice values?
print dir(mapper.columns[col]) # does not show the 'choices' attribute
print dir(inspect(mapper.columns[col])) # does not show the 'choices' attribute
print mapper.columns[col].choices # error
But I am can't seem to access the choices custom attribute of the custom type. I also tried "inspecting" the column directly instead of the class, but that doesn't work either.
So how do we access custom attributes of custom types in sqlalchemy, while inspecting?
You're inspecting the Column objects, not their types. Access the type through the type attribute of a Column object:
In [9]: class SomeTableClass(Base):
...: __tablename__ = 'sometableclass'
...: id = Column(Integer, primary_key=True)
...: choices = Column(ChoiceString({ 'asdf': 'qwer'}))
...:
In [10]: mapper = inspect(SomeTableClass)
In [12]: mapper.columns['choices']
Out[12]: Column('choices', ChoiceString(), table=<sometableclass>)
In [13]: mapper.columns['choices'].type.choices
Out[13]: {'asdf': 'qwer'}
The columns can be accessed through the __table__.columns collection. The type and the underlying python type can be accessed through col.type and cole.type.python_type, e.g:
import enum
from sqlalchemy import Column, Enum, Integer, String, Text
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Gender(enum.Enum):
MALE = "male"
FEMALE = "female"
class Person(Base):
__tablename__ = "table_person"
id = Column(Integer, primary_key=True)
name = Column(String(20))
gender = Column(Enum(Gender))
address = Column(Text)
def test_inspect_columns():
col_id = Person.__table__.columns["id"]
assert isinstance(col_id.type, Integer)
assert col_id.type.python_type is int
col_name = Person.__table__.columns["name"]
assert isinstance(col_name.type, String)
assert col_name.type.python_type is str
col_gender = Person.__table__.columns["gender"]
assert isinstance(col_gender.type, Enum)
assert col_gender.type.python_type is Gender
col_address = Person.__table__.columns["address"]
assert isinstance(col_address.type, Text)
assert col_address.type.python_type is str
I've created some base classes for an application which stores data using SQLAlchemy. One of the Base classes (Content) is polymorphic and have some general fields such as id, title, description, timestamps etc. Subclasses of this class is supposed to add additional fields which are stored in a separate table. I've created a standalone code sample which illustrates the concept better. The example contain the Base classes, some subclasses and some bootstrap code to create a sqlite database. The easiest way to get the example running by pasting the code into 'example.py', creating a virtualenv, installing SQLAlchemy into that virtualenv and using it's interpreter to run the example. The example contain some commented troublesome code, if that code is commented the example should run without errors (atleast it does here).
By uncommenting the commented code the example fails, and I'm not quite sure how to fix this - any help is superwelcome!
Example overview:
It has some base classes (Base and Content).
It has a Task class which extends Content.
A Task may have subtasks, positional ordering should persist.
It has a Project class (commented) which extends Content.
Projects have a due_date and milestones (which is a list of Tasks)
It has a Worklist class (commented) which extends Content.
Worklists belong to an 'employee' and have tasks.
What I'm trying to achieve is having Task work as a standalone class, but additional classes may also have Tasks (such as Project and Worklist). I dont want to end up with several task/related tables, but rather want to utilize Content for this concept and attach Tasks in this 'generic' way.
Example code:
from datetime import datetime
from datetime import timedelta
from sqlalchemy import Column
from sqlalchemy import Integer
from sqlalchemy import Boolean
from sqlalchemy import String
from sqlalchemy import DateTime
from sqlalchemy import Date
from sqlalchemy import Unicode
from sqlalchemy import UnicodeText
from sqlalchemy import ForeignKey
from sqlalchemy import MetaData
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.declarative import declared_attr
from sqlalchemy.ext.orderinglist import ordering_list
from sqlalchemy.orm import Session
from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import relationship
from sqlalchemy.orm import backref
from sqlalchemy.util import classproperty
class Base(object):
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
#property
def columns(self):
return self.__mapper__.columns.keys()
def add(self, **data):
self.update(**data)
db_session.add(self)
db_session.flush()
def delete(self):
db_session.delete(self)
db_session.flush()
def update(self, **data):
"""
Iterate over all columns and set values from data.
"""
for attr in self.columns:
if attr in data and data[attr] is not None:
setattr(self, attr, data[attr])
engine = create_engine('sqlite:///test.db', echo=True)
metadata = MetaData()
db_session = scoped_session(sessionmaker(bind=engine))
Base = declarative_base(cls=Base)
Base.metadata = metadata
Base.query = db_session.query_property()
class Content(Base):
"""
Base class for all content. Includes basic features such as
ownership and timestamps for modification and creation.
"""
#classproperty
def __mapper_args__(cls):
return dict(
polymorphic_on='type',
polymorphic_identity=cls.__name__.lower(),
with_polymorphic='*')
id = Column(Integer(), primary_key=True)
type = Column(String(30), nullable=False)
owner = Column(Unicode(128))
title = Column(Unicode(128))
description = Column(UnicodeText())
creation_date = Column(DateTime(), nullable=False, default=datetime.utcnow)
modification_date = Column(DateTime(), nullable=False, default=datetime.utcnow)
def __init__(self, **data):
self.add(**data)
def update(self, touch=True, **data):
"""
Iterate over all columns and set values from data.
:param touch:
:param data:
:return:
"""
super(Content, self).update(**data)
if touch and 'modification_date' not in data:
self.modification_date = datetime.utcnow()
def __eq__(self, other):
return isinstance(other, Content) and self.id == other.id
def get_content(id):
return Content.query.get(id)
class Task(Content):
id = Column(Integer, ForeignKey(Content.id), primary_key=True)
# content_id = Column(Integer, ForeignKey(Content.id), nullable=True)
done = Column(Boolean, default=False)
position = Column(Integer, default=0)
parent_id = Column(Integer, ForeignKey('task.id'), nullable=True)
tasks = relationship(
'Task',
cascade='all, delete, delete-orphan',
backref=backref('parent', remote_side=id),
foreign_keys='Task.parent_id',
order_by=position,
collection_class=ordering_list('position', reorder_on_append=True)
)
def default_due_date():
return datetime.utcnow() + timedelta(days=60)
# class Project(Content):
#
# id = Column(Integer, ForeignKey(Content.id), primary_key=True)
# due_date = Column(Date, default=default_due_date)
#
# milestones = relationship(
# 'Task',
# cascade='all, delete, delete-orphan',
# backref=backref('content_parent', remote_side=id),
# foreign_keys='Task.content_id',
# collection_class=ordering_list('position', reorder_on_append=True)
# )
#
#
# class Worklist(Content):
#
# id = Column(Integer, ForeignKey(Content.id), primary_key=True)
# employee = Column(Unicode(128), nullable=False)
#
# tasks = relationship(
# 'Task',
# cascade='all, delete, delete-orphan',
# backref=backref('content_parent', remote_side=id),
# foreign_keys='Task.content_id',
# collection_class=ordering_list('position', reorder_on_append=True)
# )
def main():
db_session.registry.clear()
db_session.configure(bind=engine)
metadata.bind = engine
metadata.create_all(engine)
# Test basic operation
task = Task(title=u'Buy milk')
task = get_content(task.id)
# assert Content attributes inherited
assert task.title == u'Buy milk'
assert task.done == False
# add subtasks
task.tasks = [
Task(title=u'Remember to check expiration date'),
Task(title=u'Check bottle is not leaking')
]
# assert that subtasks is added and correctly ordered
task = get_content(task.id)
assert len(task.tasks) == 2
assert [(x.position, x.title) for x in task.tasks] == \
[(0, u'Remember to check expiration date'),
(1, u'Check bottle is not leaking')]
# reorder subtasks
task.tasks.insert(0, task.tasks.pop(1))
task = get_content(task.id)
assert len(task.tasks) == 2
assert [(x.position, x.title) for x in task.tasks] == \
[(0, u'Check bottle is not leaking'),
(1, u'Remember to check expiration date')]
# # Test Project implementation
# project = Project(title=u'My project')
# milestone1 = Task(title=u'Milestone #1', description=u'First milestone')
# milestone2 = Task(title=u'Milestone #2', description=u'Second milestone')
# milestone1.tasks = [Task(title=u'Subtask for Milestone #1'), ]
# milestone2.tasks = [Task(title=u'Subtask #1 for Milestone #2'),
# Task(title=u'Subtask #2 for Milestone #2')]
# project.milestones = [milestone1, milestone2]
# project = get_content(project.id)
# assert project.title == u'My project'
# assert len(project.milestones) == 2
# assert [(x.position, x.title) for x in project.milestones] == \
# [(0, u'Milestone #1'), (1, u'Milestone #2')]
# assert len(Task.query.all()) == 8
# assert isinstance(milestone1.content_parent, Project) == True
#
# # Test Worklist implementation
# worklist = Worklist(title=u'My worklist', employee=u'Torkel Lyng')
# worklist.tasks = [
# Task(title=u'Ask stackoverflow for help'),
# Task(title=u'Learn SQLAlchemy')
# ]
# worklist = get_content(worklist.id)
# assert worklist.title == u'My worklist'
# assert worklist.employee == u'Torkel Lyng'
# assert len(worklist.tasks) == 2
# assert len(Task.query.all()) == 10
# assert isinstance(worklist.tasks[0].content_parent, Worklist) == True
if __name__=='__main__':
main()
I'm sorry for this long example, wanted to supply something that worked standalone. Any help, comments on design or suggestions are greatly appretiated.
I refactored the example a bit and made it somewhat working. Instead of defining the additional ForeignKey on Task (content_id) I added it to the Content class as container_id
from datetime import datetime
from datetime import timedelta
from sqlalchemy import Column
from sqlalchemy import Integer
from sqlalchemy import Boolean
from sqlalchemy import String
from sqlalchemy import DateTime
from sqlalchemy import Date
from sqlalchemy import Unicode
from sqlalchemy import UnicodeText
from sqlalchemy import ForeignKey
from sqlalchemy import MetaData
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.declarative import declared_attr
from sqlalchemy.ext.orderinglist import ordering_list
from sqlalchemy.orm import Session
from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import relationship
from sqlalchemy.orm import backref
from sqlalchemy.util import classproperty
class Base(object):
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
#property
def columns(self):
return self.__mapper__.columns.keys()
def add(self, **data):
self.update(**data)
db_session.add(self)
db_session.flush()
def delete(self):
db_session.delete(self)
db_session.flush()
def update(self, **data):
"""
Iterate over all columns and set values from data.
"""
for attr in self.columns:
if attr in data and data[attr] is not None:
setattr(self, attr, data[attr])
engine = create_engine('sqlite:///test.db', echo=True)
metadata = MetaData()
db_session = scoped_session(sessionmaker(bind=engine))
Base = declarative_base(cls=Base)
Base.metadata = metadata
Base.query = db_session.query_property()
class Content(Base):
"""
Base class for all content. Includes basic features such as
ownership and timestamps for modification and creation.
"""
#classproperty
def __mapper_args__(cls):
return dict(
polymorphic_on='type',
polymorphic_identity=cls.__name__.lower(),
with_polymorphic='*')
id = Column(Integer(), primary_key=True)
container_id = Column(Integer(), ForeignKey('content.id'), nullable=True)
# container = relationship('Content', foreign_keys=[container_id], uselist=False)
type = Column(String(30), nullable=False)
owner = Column(Unicode(128))
title = Column(Unicode(128))
description = Column(UnicodeText())
creation_date = Column(DateTime(), nullable=False, default=datetime.utcnow)
modification_date = Column(DateTime(), nullable=False, default=datetime.utcnow)
def __init__(self, **data):
self.add(**data)
#property
def container(self):
if self.container_id:
return get_content(self.container_id)
return None
def update(self, touch=True, **data):
"""
Iterate over all columns and set values from data.
:param touch:
:param data:
:return:
"""
super(Content, self).update(**data)
if touch and 'modification_date' not in data:
self.modification_date = datetime.utcnow()
def __eq__(self, other):
return isinstance(other, Content) and self.id == other.id
def __repr__(self):
return '<{0} "{1}">'.format(self.__class__.__name__, self.title)
def get_content(id):
return Content.query.get(id)
class Task(Content):
id = Column(Integer, ForeignKey(Content.id), primary_key=True)
done = Column(Boolean, default=False)
position = Column(Integer, default=0)
parent_id = Column(Integer, ForeignKey('task.id'), nullable=True)
tasks = relationship(
'Task',
cascade='all, delete, delete-orphan',
backref=backref('parent', remote_side=id),
foreign_keys='Task.parent_id',
order_by=position,
collection_class=ordering_list('position', reorder_on_append=True)
)
def default_due_date():
return datetime.utcnow() + timedelta(days=60)
class Project(Content):
id = Column(Integer, ForeignKey(Content.id), primary_key=True)
due_date = Column(Date, default=default_due_date)
milestones = relationship(
'Task',
cascade='all, delete, delete-orphan',
foreign_keys='Task.container_id',
collection_class=ordering_list('position', reorder_on_append=True)
)
class Worklist(Content):
id = Column(Integer, ForeignKey(Content.id), primary_key=True)
employee = Column(Unicode(128), nullable=False)
tasks = relationship(
'Task',
cascade='all, delete, delete-orphan',
foreign_keys='Task.container_id',
collection_class=ordering_list('position', reorder_on_append=True)
)
def main():
db_session.registry.clear()
db_session.configure(bind=engine)
metadata.bind = engine
metadata.create_all(engine)
# Test basic operation
task = Task(title=u'Buy milk')
task = get_content(task.id)
# assert Content attributes inherited
assert task.title == u'Buy milk'
assert task.done == False
# add subtasks
task.tasks = [
Task(title=u'Remember to check expiration date'),
Task(title=u'Check bottle is not leaking')
]
# assert that subtasks is added and correctly ordered
task = get_content(task.id)
assert len(task.tasks) == 2
assert [(x.position, x.title) for x in task.tasks] == \
[(0, u'Remember to check expiration date'),
(1, u'Check bottle is not leaking')]
# reorder subtasks
task.tasks.insert(0, task.tasks.pop(1))
task = get_content(task.id)
assert len(task.tasks) == 2
assert [(x.position, x.title) for x in task.tasks] == \
[(0, u'Check bottle is not leaking'),
(1, u'Remember to check expiration date')]
# Test Project implementation
project = Project(title=u'My project')
milestone1 = Task(title=u'Milestone #1', description=u'First milestone')
milestone2 = Task(title=u'Milestone #2', description=u'Second milestone')
milestone1.tasks = [Task(title=u'Subtask for Milestone #1'), ]
milestone2.tasks = [Task(title=u'Subtask #1 for Milestone #2'),
Task(title=u'Subtask #2 for Milestone #2')]
project.milestones = [milestone1, milestone2]
project = get_content(project.id)
assert project.title == u'My project'
assert len(project.milestones) == 2
assert [(x.position, x.title) for x in project.milestones] == \
[(0, u'Milestone #1'), (1, u'Milestone #2')]
assert len(Task.query.all()) == 8
container = milestone1.container
assert isinstance(container, Project) == True
# Test Worklist implementation
worklist = Worklist(title=u'My worklist', employee=u'Torkel Lyng')
worklist.tasks = [
Task(title=u'Ask stackoverflow for help'),
Task(title=u'Learn SQLAlchemy')
]
worklist = get_content(worklist.id)
assert worklist.title == u'My worklist'
assert worklist.employee == u'Torkel Lyng'
assert len(worklist.tasks) == 2
assert len(Task.query.all()) == 10
assert isinstance(worklist.tasks[0].container, Worklist) == True
# Cleanup
task = Task.query.filter_by(title=u'Buy milk').one()
task.delete()
project.delete()
worklist.delete()
assert len(Task.query.all()) == 0
if __name__=='__main__':
main()
The container relationship on the Content-class did not work as expected, it returned None if I did not specify task.container = somecontainer. Instead I opted for a property method instead which returns None or container object. I'll investigate the subject further to perhaps find a more optimal solution. Suggestions or alternative solutions are still very very welcome.
I'm using SQLAlchemy (0.9.4) in my Flask application. There are two tables with soft delete support in application.
class A(SoftDeleteMixin, db.Model):
id = db.Column(db.BigInteger, primary_key=True)
b_id = db.Column(db.BigInteger, db.ForeignKey('b.id'), nullable=False)
b = soft_delete_relationship('B.id', 'A.b_id')
class B(SoftDeleteMixin, db.Model):
id = db.Column(db.BigInteger, primary_key=True)
parent_id = db.Column(db.BigInteger, db.ForeignKey('b.id'))
parent = soft_delete_relationship(remote(id), parent_id, 'B.id', 'B.parent_id')
children = soft_delete_relationship(remote(parent_id), id, 'B.parent_id', 'B.id')
SoftDeleteMixin is based on LimitingQuery (https://bitbucket.org/zzzeek/sqlalchemy/wiki/UsageRecipes/PreFilteredQuery)
from sqlalchemy.orm.query import Query
class NonDeletedQuery(Query):
def get(self, ident):
return Query.get(self.populate_existing(), ident)
def __iter__(self):
return Query.__iter__(self.private())
def from_self(self, *ent):
return Query.from_self(self.private(), *ent)
def private(self):
mzero = self._mapper_zero()
if mzero is not None and hasattr(mzero, 'class_'):
soft_deleted = getattr(mzero.class_, 'soft_deleted', None)
return self.enable_assertions(False).filter(soft_deleted.is_(False)) if soft_deleted else self
else:
return self
And soft_delete_relationship constructs relationship with custom primaryjoin (for join on non-soft_deleted).
def soft_delete_relationship(first, second, *args, **kwargs):
if isinstance(first, str) and isinstance(second, str):
other, other_column = first.split('.')
_this, this_column = second.split('.')
primaryjoin = ' & '.join(['({} == {})'.format(first, second), '{}.soft_deleted.is_(False)'.format(other)])
else:
other, other_column = args[0].split('.')
_this, this_column = args[1].split('.')
primaryjoin = lambda: (first == second) & getattr(second.table.c, 'soft_deleted').is_(False)
kwargs['primaryjoin'] = primaryjoin
return relationship(other, **kwargs)
The problem occurs when I write query with aliased B:
b_parent = aliased(B)
A.query.join(A.b).outerjoin(b_parent, B.parent)
I get following SQL:
SELECT ... FROM a JOIN b ON b.id = a.b_id LEFT OUTER JOIN b AS b_1 ON b_1.id = b.parent_id AND *b*.soft_deleted IS False
But I expect following:
SELECT ... FROM a JOIN b ON b.id = a.b_id LEFT OUTER JOIN b AS b_1 ON b_1.id = b.parent_id AND *b_1*.soft_deleted IS False
When I explicitly write:
A.query.join(A.b).outerjoin(b_parent, (b_parent.id == B.parent_id) & b_parent.soft_deleted.is_(False))
I got right query.
How can I get proper alias to b_1 in query without explicit join condition?
Btw, there was expected SQL in SQLAlchemy 0.7.9.
OK, I figured it out.
getattr(second.table.c, 'soft_deleted') must be also with remote() annotation.
In other words primaryjoin of relationship in B.parent should look like:
(remote(B.id) == B.parent_id) & remote(B.soft_deleted).is_(False)
I have two classes mapped to two tables respectively.
Ex:
Obj 1: ID (PK), KEY (String(20))
Obj 2: ID (PK), obj_1_id (FK), value (String(20))
I would like to be able to perform obj_1.value = *val*, whereby val is stored on the secon'd table's respective column, instead of obj_1.value.value = val`.
How can I create such relationship, spread/mapped to two tables' columns?
What I want is not one-to-one (object HAS object) but rather map a column of an object to a different table.
Following is what I have tried (following the docs) and it does not work as it creates obj1.value.value = .. instead of direct column mapping
What I have tried:
class Obj1(Base):
__tablename__ == ...
id = ..
key = ..
value = relationship("Obj2", uselist=False, backref="obj1")
class Obj2(Base):
__tablename__ == ...
id = .. # PK
obj_1_id = .. # FK
value = ...
Why not just wrap the python property:
class Obj1(Base):
__tablename__ = 'obj1'
id = Column(Integer, primary_key=True)
key = Column(String(20))
_value_rel = relationship("Obj2", uselist=False, backref="obj1")
#property
def value(self):
return self._value_rel and self._value_rel.value
#value.setter
def value(self, value):
if value is None:
self._value_rel = None
elif self._value_rel is None:
self._value_rel = Obj2(value=value)
else:
self._value_rel.value = value