I've been reading various examples from SQLAlchemy documentation for cascade deletes, but nothing I try seems to work. Below is some sample code adapted from that documentation, but using back_populates instead of backref, as I understand that backref is being deprecated.
In the "main" section below, I would expect that deleting the order that "contains" the items would delete the items as well, but that does not happen. Obviously I don't understand something about how to configure these tables... what is it?
# third party imports
from sqlalchemy import Column, ForeignKey, Integer, String, create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship, sessionmaker
from sqlalchemy_utils import create_database, database_exists
Base = declarative_base()
class Order(Base):
__tablename__ = "business_order"
id = Column(Integer, primary_key=True)
name = Column(String(32))
items = relationship(
"Item", back_populates="order", cascade="all, delete, delete-orphan"
)
class Item(Base):
__tablename__ = "business_item"
id = Column(Integer, primary_key=True)
name = Column(String(32))
order_id = Column(Integer, ForeignKey("business_order.id"))
order = relationship("Order", back_populates="items")
def get_session(url="sqlite:///:memory:", create_db=True):
"""Get a SQLAlchemy Session instance for input database URL.
:param url:
SQLAlchemy URL for database, described here:
http://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls.
:param create_db:
Boolean indicating whether to create database from scratch.
:returns:
Sqlalchemy Session instance.
"""
# Create a sqlite in-memory database engine
if not database_exists(url):
if create_db:
create_database(url)
else:
msg = (
"Database does not exist, will not create without "
"create_db turned on."
)
print(msg)
return None
connect_args = {}
engine = create_engine(url, echo=False, connect_args=connect_args)
Base.metadata.create_all(engine)
# create a session object that we can use to insert and
# extract information from the database
Session = sessionmaker(bind=engine, autoflush=False)
session = Session()
return session
if __name__ == "__main__":
sqlite_url = "sqlite:///test_sqlite.db"
session = get_session(sqlite_url)
order = Order(name="order1")
session.add(order)
item = Item(order_id=order.id, name="item1")
session.add(item)
session.commit()
session.delete(order) # should delete items too, right?
session.commit()
orders = session.query(Order).all()
print(len(orders)) # this returns 0 as expected
items = session.query(Item).all()
print(len(items)) # this returns 1, why?
Order has an (implicit) autoincrement PK. When you do
order = Order(name="order1")
session.add(order)
order.id is None. Therefore, when you do
item = Item(order_id=order.id, name="item1")
item.order_id will also be None, so item is actually not associated with order. Therefore, the delete doesn't cascade.
order doesn't get its id until .flush() (or .commit()) is called. So you could either do
order = Order(name="order1")
session.add(order)
session.flush() # !
item = Item(order_id=order.id, name="item1")
session.add(item)
session.commit()
or do
order = Order(name="order1", items=[Item(name="item1")])
session.add(order)
session.commit()
session.delete(order) # should delete items too, right?
session.commit()
orders = session.query(Order).all()
print(len(orders)) # this returns 0 as expected
items = session.query(Item).all()
print(len(items)) # this also returns 0 as expected
Related
I'm trying to persist a One-To-Many self-referential relationship. My table looks something like this:
class Users(Base):
__tablename__ = "users"
id = Column(Integer, primary_key=True, unique=True)
connected_ids = Column(Integer, ForeignKey("users.id"))
connected_with = relationship("Users")
I arrived at this format following this page in the docs for one-to-many and another page describing how to declare self referential relationships. I've also already tried with the following variations:
connected_with = relationship("Users", backref="users")
connected_with = relationship("Users", backref="users", remote_side="users.c.id"")
I can insert the rows, query, commit, etc... but when trying to define a relationship, it fails with the following:
Example One:
u1 = session.get(Users, 1)
u2 = session.get(Users, 2)
u1.connected_ids = [u2.id]
Will raise:
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.DatatypeMismatch) column "connected_ids" is of type integer but expression is of type integer[]
LINE 1: ...users SET last_updated=now(), connected_ids=ARRAY[2911...
Example Two (with connected_with attr):
u1.connected_with = [u2.id]
Will Raise:
AttributeError: 'int' object has no attribute '_sa_instance_state'
Example Three (with the object itself):
u1.connected_ids = [u2]
Will raise:
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'Users'
At this point, my best guess is that the table is not defined the way I expect it to, but I also don't know what is wrong in it.
Any pointers and help will be appreciated.
I'm not sure if adjacency list is the pattern you want. Here is how it could work though. The ForeignKey should be a scalar, ie. single value, not a list. A many side of the relationship can be a list though, here it is children:
import sys
from sqlalchemy import (
create_engine,
Integer,
String,
ForeignKey,
)
from sqlalchemy.schema import (
Column,
)
from sqlalchemy.orm import Session, declarative_base, relationship, backref
username, password, db = sys.argv[1:4]
Base = declarative_base()
engine = create_engine(f"postgresql+psycopg2://{username}:{password}#/{db}", echo=True)
metadata = Base.metadata
class User(Base):
__tablename__ = "users"
id = Column(Integer, primary_key=True)
name = Column(String)
parent_id = Column(Integer, ForeignKey("users.id"), nullable=True)
# OR children = relationship("User", backref=backref("parent", remote_side=id))
parent = relationship("User", back_populates="children", remote_side=id)
children = relationship("User", back_populates="parent", remote_side=parent_id)
metadata.create_all(engine)
with Session(engine) as session, session.begin():
# Create 3 users, 2 connected to root.
root = User(name="root")
a = User(name="a", parent=root)
b = User(name="b", parent=root)
session.add_all([root, a, b])
with Session(engine) as session, session.begin():
# Check that root exists and both children are connected to it.
root = session.query(User).where(User.name == "root").first()
assert len(root.children) == 2
for child in root.children:
assert child.parent == root
with Session(engine) as session, session.begin():
# Add another child to root using the children property with append.
root = session.query(User).where(User.name == "root").first()
root.children.append(User(name="c"))
with Session(engine) as session, session.begin():
# Check that root exists and that there are now 3 children instead of 2.
root = session.query(User).where(User.name == "root").first()
print(root.name)
assert len(root.children) == 3
for child in root.children:
assert child.parent == root
print(child.name)
What I'm trying to do seems simple. I'd like to have a parent Organization which has child Groups. However, one group will be the "main_group". Each Organization should have a main group. Each Group should have a reference to the Organization in which it belongs.
I've been able to create_all and use these Models but in my tests when I do a drop_all, I get a
sqlalchemy.exc.CircularDependencyError: Can't sort tables for DROP; an unresolvable foreign key dependency exists between tables: groups, organizations. Please ensure that the ForeignKey and ForeignKeyConstraint objects involved in the cycle have names so that they can be dropped using DROP CONSTRAINT.
Here is a minimum example to show what I'm trying to do. I've left some commented lines in to show what all I've tried.
from sqlalchemy.orm import relationship
from sqlalchemy.sql.schema import ForeignKey
from sqlalchemy.sql.sqltypes import Boolean, Date, Float
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Integer, ForeignKey, String, Column
Base = declarative_base()
class Organization(Base):
__tablename__ = "organizations"
id = Column(Integer(), primary_key=True)
name = Column(String(100))
### Define one-to-one with Group
main_group_id = Column(Integer, ForeignKey("groups.id"))
main_group = relationship(
"Group",
# uselist=False,
# back_populates="organization",
foreign_keys=[main_group_id],
primaryjoin="Group.id==Organization.main_group_id",
post_update=True
# backref="organization"
)
####
## Defines the one-to-many collection with Group
# groups = relationship(
# "Group",
# uselist=True,
# # foreign_keys="Group.organization_id",
# primaryjoin="Group.id==Organization.main_group_id",
# # back_populates="organization",
# )
class Group(Base):
__tablename__ = "groups"
id = Column(Integer, primary_key=True)
name = Column(String(100))
organization_id = Column(Integer, ForeignKey("organizations.id"), nullable=False)
organization = relationship(
"Organization",
uselist=False,
# backref="groups",
foreign_keys=[organization_id],
primaryjoin="Group.organization_id==Organization.id",
# primaryjoin="Group.id==Organization.main_group_id",
post_update=True,
)
from sqlalchemy import create_engine
from sqlalchemy.orm.session import sessionmaker
conn_string = "sqlite:///example.sql"
engine = create_engine(conn_string)
Base.metadata.create_all(engine) # here we create all tables
Session = sessionmaker(bind=engine)
session = Session()
new_org = Organization(name="my new org")
print(new_org)
session.add(new_org)
org = session.query(Organization).get(1)
new_group = Group(name="my main group", organization=org, organization_id=org.id)
new_org.main_group = new_group
session.commit()
Base.metadata.drop_all(engine)
From The Docs
There is actually an example pretty similar to what you want here, which uses an explicitly named foreign key, fk_favorite_entry:
rows-that-point-to-themselves-mutually-dependent-rows
That doesn't seem to fully solve the drop warning, which maybe seems to be dialect dependent. You can fully solve it with use_alter=True which is explained here:
creating-dropping-foreign-key-constraints-via-alter
Likely Solution
Best case would be to name the atypical constraint and to set use_alter=True on it as well.
class Organization(Base):
# ...
main_group_id = Column(Integer, ForeignKey("groups.id", name="fk_main_group", use_alter=True))
# ...
We have 1 table with a large amount of data and DBA's partitioned it based on a particular parameter. This means I ended up with Employee_TX, Employee_NY kind of table names. Earlier the models.py was simple as in --
class Employee(Base):
__tablename__ = 'Employee'
name = Column...
state = Column...
Now, I don't want to create 50 new classes for the newly partitioned tables as anyways my columns are the same.
Is there a pattern where I can create a single class and then use it in query dynamically? session.query(<Tablename>).filter().all()
Maybe some kind of Factory pattern or something is what I'm looking for.
So far I've tried by running a loop as
for state in ['CA', 'TX', 'NY']:
class Employee(Base):
__qualname__ = __tablename__ = 'Employee_{}'.format(state)
name = Column...
state = Column...
but this doesn't work and I get a warning as - SAWarning: This declarative base already contains a class with the same class name and module name as app_models.employee, and will be replaced in the string-lookup table.
Also it can't find the generated class when I do from app_models import Employee_TX
This is a flask app with PostgreSQL as a backend and sqlalchemy is used as an ORM
Got it by creating a custom function like -
def get_model(state):
DynamicBase = declarative_base(class_registry=dict())
class MyModel(DynamicBase):
__tablename__ = 'Employee_{}'.format(state)
name = Column...
state = Column...
return MyModel
And then from my services.py, I just call with get_model(TX)
Whenever you think of dynamically constructing classes think of type() with 3 arguments (see this answer for a demonstration, and the docs more generally).
In your case, it's just a matter of constructing the classes and keeping a reference to them so you can access them again later.
Here's an example:
from sqlalchemy import Column, Integer, String
from sqlalchemy.engine import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
Base = declarative_base()
# this produces the set of common attributes that each class should have
def attribute_factory():
return dict(
id=Column(Integer, primary_key=True),
name=Column(String, nullable=False),
state=Column(String, nullable=False),
CLASS_VAR=12345678,
)
states = ["CA", "TX", "NY"]
# here we map the state abbreviation to the generated model, notice the templated
# class and table names
model_map = {
state: type(
f"Employee_{state}",
(Base,),
dict(**attribute_factory(), __tablename__=f"Employee_{state}"),
)
for state in states
}
engine = create_engine("sqlite:///", echo=True)
Session = sessionmaker(bind=engine)
Base.metadata.create_all(engine)
if __name__ == "__main__":
# inserts work
s = Session()
for state, model in model_map.items():
s.add(model(name="something", state=state))
s.commit()
s.close()
# queries work
s = Session()
for state, model in model_map.items():
inst = s.query(model).first()
print(inst.state, inst.CLASS_VAR)
This is an edited version of the original question, with a self-contained example.
from sqlalchemy import Column, Integer, String, create_engine, func
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String)
fullname = Column(String)
nickname = Column(String)
engine = create_engine('sqlite://', echo=True)
Session = sessionmaker(bind=engine)
session = Session()
Base.metadata.create_all(engine)
session.add_all([
User(name='wendy', fullname='Wendy Williams'),
User(name='mary', fullname='Mary Contrary'),
User(name='fred', fullname='Fred Flintstone', nickname='freddy')])
session.commit()
items = session.query(
User.id, User.name,
func.coalesce(User.nickname, 'Nicky').label('nickname'))
# Checking that with_entities works on original data set
subset_items = session.query(
User.id, User.name)\
.with_entities(User.name, User.nickname)\
.all()
print(subset_items)
# items will be used elsewhere....
# Wanted: a subset with columns fullname and nickname only
# Result is wrong, it does not use coalesce result
special_items = items\
.with_entities(User.fullname, User.nickname) \
.all()
print(special_items)
How do I use .with_entities to refer to the items? The following fails
.with_entities(items.fullname, items.nickname) \
so did variants with quotes as suggested by #Nick in reply to the original question.
I haven't tested this, but I believe you should be able to use a string here instead of an expression:
.with_entities('pat_no') \
Since there was no reply, I posted this on the community site of SQLAlchemy. The answer is surprisingly awkward:
q = q.with_entities(q.column_descriptions[1]['expr'], q.column_descriptions[2]['expr'])
i've written the following example code to build a search index for the ACL system i am writing. The query in this example resturns all objects that have any of the given ACLs assigned. But i need a query/filter that returns objects that have all ACLs assigned.
Any help is appreciated.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import backref
from sqlalchemy import create_engine
from sqlalchemy import ForeignKey
from sqlalchemy import Integer
from sqlalchemy import String
from sqlalchemy import Column
_db_uri = "sqlite:////tmp/test.sql"
Base = declarative_base()
engine = create_engine(_db_uri, echo=False)
Session = sessionmaker(bind=engine)
class IndexObject(Base):
""" Index object. """
__tablename__ = 'objects'
id = Column(Integer, primary_key=True)
name = Column(String(128), unique=True, nullable=True)
acls = relationship('IndexObjectACL',
cascade = "all,delete",
backref='objects',
lazy='dynamic')
def __repr__(self):
_repr_ =("<IndexObject (name='%s')>" % (self.name))
return _repr_
class IndexObjectACL(Base):
""" Index object ACL. """
__tablename__ = 'acls'
id = Column(Integer, primary_key=True)
value = Column(String(128), nullable=False)
oid = Column(Integer, ForeignKey('objects.id'))
def __repr__(self):
__repr__ = ("<IndexObjectACL (value='%s')>" % (self.value))
return __repr__
object_list = [
"object1",
"object2",
"object3",
]
acl_list = {
"object1" : [
"view",
"edit",
"enable",
"delete",
],
"object2" : [
"view",
"edit",
],
"object3" : [
"enable",
"delete",
],
}
Base.metadata.create_all(engine)
session = Session()
for o in object_list:
acls = []
for acl in acl_list[o]:
a = IndexObjectACL(value=acl)
acls.append(a)
index_object = IndexObject(name=o, acls=acls)
session.add(index_object)
session.commit()
search_acls = [ "enable", "delete" ]
q = session.query(IndexObject)
q = q.join(IndexObject.acls).filter(IndexObjectACL.value.in_(search_acls))
print(q.all())
session.close()
I think this could be an opportunity to use division – in a way. IndexObjectACL divided by SearchAcls should yield IndexObjects that have all the SearchAcls. In other words query for IndexObjects for which no such SearchAcls exist that are not in its IndexObjectACLs:
from sqlalchemy import union, select, literal
# Create an aliased UNION of all the search_acls to query against
search_acls_union = union(*(select([literal(acl).label('acl')])
for acl in search_acls)).alias()
# Query for those IndexObjects where...
# No SearchAcl exists where...
# No IndexObjectACL exists where value == acl AND oid == id
q = session.query(IndexObject).\
filter(~session.query().select_from(search_acls_union).
filter(~IndexObject.acls.any(value=search_acls_union.c.acl)).
exists())
The result of this query is
[<IndexObject (name='object1')>, <IndexObject (name='object3')>]
and if you add
"object4" : [
"enable",
],
"object5" : [
"delete",
],
to your acl_list (and the object names to object_list) for proving that partial matches are not returned, it still returns only objects 1 and 3.
Your original "have any" query could also be rewritten to use a semijoin, or EXISTS in SQL speak:
q = session.query(IndexObject).\
filter(IndexObject.acls.any(
IndexObjectACL.value.in_(search_acls)))
queries = []
acl_q = q.join(IndexObject.acls)
for acl in search_acls:
x = acl_q.filter(IndexObjectACL.value == acl)
queries.append(x)
q = q.intersect(*queries)
I can try to explain it but i am new to sqlalchemy and SQL in general. So i might explain it the wrong way... The join() joins IndexObject and IndexObjectACL tables based on their relationship which results in a new query. This query is used to create a new query for each ACL we want to match using filter(). Finally we use intersect() (SQL INTERSECT) to get all IndexObject that appear in all queries. After some testing it seems like this is a fast way to search objects that have all given ACLs assigned. Its also very pythonic IMHO.