Develop Reference

Two-way foreign keys on different attributes, one-to-one and one-to-many

I have two models: User and ReferralKey, with these requirements:
On creation of User, a ReferralKey is automatically created and added to the DB
ReferralKey keeps track of which users were referred by it
ReferralKey keeps track of which user owns it
As per the answer to this question, the best solution seems to be to create the ReferralKey within the constructor of User. The solution to the other two require foreign keys, and seems really messy—entangling the tables together in such a way that I might as well put them in the same table.
The solution to the first looks like this:
def User(model):
id = Column(BigInteger(), autoincrement=True, primary_key=True)
referral_key = relationship('ReferralKey', uselist=False)
...
def __init__(self):
self.referral_key = ReferralKey()
def ReferralKey(model):
id = Column(BigInteger(), autoincrement=True, primary_key=True)
user_id = Column(BigInteger(), ForeignKey('user.id', ondelete='SET NULL'), nullable=True)
This works as intended, and solves the first and third points. The problem arises when trying to solve the 2nd. This (for some reason) necessitates a new foreign key in User, which necessitates the declaration of a relationship in both User and ReferralKey to (I guess) disambiguate the foreign keys:
def User(model):
id = Column(BigInteger(), autoincrement=True, primary_key=True)
referral_key = relationship('ReferralKey', uselist=False)
referrer_id = Column(BigInteger(), ForeignKey('referral_key.id', ondelete='SET NULL'))
referrer = relationship('ReferralKey', foreign_keys=['referrer_id'], backref='used_by')
...
def __init__(self):
self.referral_key = ReferralKey()
def ReferralKey(model):
__tablename__='referral_key'
id = Column(BigInteger(), autoincrement=True, primary_key=True)
user_id = Column(BigInteger(), ForeignKey('user.id', ondelete='SET NULL'), nullable=True)
user = relationship('User', foreign_keys=['user_id'])
I've tried all different permutations of relationship and ForeignKey, and always get the same error:
sqlalchemy.exc.CircularDependencyError: Can't sort tables for DROP; an unresolvable foreign key dependency exists between tables: referral_key, users. Please ensure that the ForeignKey and ForeignKeyConstraint objects involved in the cycle have names so that they can be dropped using DROP CONSTRAINT.
Ultimately, my problem is that I just don't understand what I'm doing. Why do I need to change the User table at all in order to keep track of things on the ReferralKey table? What purpose does the relationship declaration serve—why is it ambiguous without this declaration? If User has a foreign key referencing ReferralKey and ReferralKey has a foreign key referencing User—and either of these should be set to NULL in case of deletion, why does SQL need more information than that?
Why can't I just have:
def User(model):
id = Column(BigInteger(), autoincrement=True, primary_key=True)
def __init__(self):
ReferralKey(user_id=self.id)
def ReferralKey(model):
__tablename__='referral_key'
id = Column(BigInteger(), autoincrement=True, primary_key=True)
user_id = Column(BigInteger(), ForeignKey('user.id', ondelete='SET NULL'), nullable=True)
used_by = [list of user IDs]
def __init__(self, user_id):
if user_id:
self.user_id == user_id
This feels to me so much cleaner and more intuitive. If I want to add (or remove!) referral keys, I hardly have to worry about adding things to User because it's mostly independent of the functioning of the referral keys. Why do I need to add a column in the user table to keep track of something that I want the ReferralKey to keep track of?
I'm totally ignorant of this, basically. Would anyone mind helping me out?

Try the following.
It relies on the information:
A user creates his own master key on registration.
A user may register a slave key when he registers, but maybe not.
This way the foreign keys remain in one table but you need to differentiate between them..
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
Base= declarative_base()
class User(Base):
__tablename__ = "user"
id = Column(Integer, primary_key=True)
own_referral_id = Column(Integer, ForeignKey('referral_key.id'))
own_referral_key = relationship('ReferralKey', foreign_keys=[own_referral_id], back_populates='owner')
signup_referral_id = Column(Integer, ForeignKey('referral_key.id'))
signup_referral_key = relationship('ReferralKey', foreign_keys=[signup_referral_id], back_populates='signups')
def __init__(self, **kwargs):
self.own_referral_key = ReferralKey()
super().__init__(**kwargs)
class ReferralKey(Base):
__tablename__ = "referral_key"
id = Column(Integer, primary_key=True)
owner = relationship('User', foreign_keys=[User.own_referral_id], back_populates='own_referral_key', uselist=False)
signups = relationship('User', foreign_keys=[User.signup_referral_id], back_populates='signup_referral_key', uselist=True)
e = create_engine("sqlite://")
Base.metadata.create_all(e)
s = Session(e)
u1 = User()
s.add(u1)
s.commit()
u2 = User(signup_referral_id = u1.own_referral_id)
u3 = User(signup_referral_id = u1.own_referral_id)
s.add(u2)
s.add(u3)
s.commit()
print(u1.own_referral_key.signups)

Adding a comments field to database with flask/SQLAlchemy?

I'm working on a flask application that has a database that is set up like this and am using SQLAlchemy. (display_name was renamed to persona_name since that's what the steam api calls it.)
I'm still in the process of learning how work with databases, and what I have now works fine, I can track all the lists a user has made, and I can get suspects from all lists that one user has made. Also, removing a suspect from the suspect table also seems to remove the suspect from all lists it appears on.
This seems to work well, List and Suspect are classes that inherit from db.Model while suspect_list is an auxiliary table that is not itself a class.
The way it is set up I want to be able to have a suspect show up on multiple lists, but now I want to be able to add comments for individual suspects on individual lists.
I'm not sure how to go about this, but I have a few ideas and am not sure if they work or would have potential downsides.
1) Can I add a comment field to suspect_list?
2) Do I make a comment model as a class that inherits from db.Model and then add it to the auxiliary table instead?
3) Should I create a new id field for the suspect table, make it the primary key instead of steam_id , and then add a comment field so there can be duplicate suspects that have differing comments?
I probably could implement 3 but I don't think it is a good idea because duplicates of the same suspect probably is something that should be avoided.
As for 1 and 2 I don't know if they would work and if they did I'm not sure how to properly implement them.
This is the code that I have for my Models.py if it is needed
My question is how would I properly add comments to this database model I have set up?

Rather then an association table, you really need an association object. Here's a working example.
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.secret_key = 'MOO.'
app.config["SQLALCHEMY_DATABASE_URI"] = 'sqlite://' # In memory.
db = SQLAlchemy(app)
class User(db.Model):
__tablename__ = 'user'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String)
def __repr__(self):
return '<User {}>'.format(self.name)
class Suspect(db.Model):
__tablename__ = 'suspect'
id = db.Column(db.Integer, primary_key=True)
steam_name = db.Column(db.String)
def __repr__(self):
return '<Suspect {}>'.format(self.steam_name)
class List(db.Model):
__tablename__ = 'list'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String())
user_id = db.Column(db.Integer, db.ForeignKey('user.id'))
user = db.relationship('User', backref=db.backref('lists'))
suspects = db.relationship('SuspectList', backref='list')
def __repr__(self):
return '<List {}>'.format(self.name)
class SuspectList(db.Model):
__tablename__ = 'suspect_list'
list_id = db.Column(db.Integer, db.ForeignKey('list.id'), primary_key=True)
suspect_id = db.Column(db.Integer, db.ForeignKey('suspect.id'), primary_key=True)
comment = db.Column(db.String)
suspect = db.relationship('Suspect', backref=db.backref("suspect_list"))
def __repr__(self):
return '<SL: %s %s %s>' % (self.list, self.suspect, self.comment)
if __name__ == '__main__':
with app.app_context():
db.create_all()
hacker = Suspect(steam_name='Bob')
cracker = Suspect(steam_name='Alice')
carol = User(name='Carol')
carols_list = List(name='Carols', user=carol)
hacker_suspect = SuspectList(suspect=hacker, comment='RED HAIR')
cracker_suspect = SuspectList(suspect=cracker, comment='RED EYES')
carols_list.suspects.append(hacker_suspect)
carols_list.suspects.append(cracker_suspect)
db.session.add(carols_list) # Trust SQLAlchemy to add all related.
db.session.commit()
## Querying:
my_list = List.query.filter(User.name == 'Carol').first()
for record in my_list.suspects:
print '{} reported {} with comment {}'.format(
record.list.user.name,
record.suspect.steam_name,
record.comment
)
# Carol reported Bob with comment RED HAIR
# Carol reported Alice with comment RED EYES
Two side-notes-- I found it a bit ugly to be use List as a class name, because when you want to query it, your default name would be list = List.query.fi.... which is a bit of a no-go :). Also, what program did you use to generate your ERD?

if i correctly understand, i would just create a new model related to your List Model
class Comments(db.Model):
id = db.Column(db.Integer, primary_key=True)
comment = db.Column(db.String(300))
list_id = db.Column(db.Integer, db.ForeignKey('list.id'))

SQLAlchemy DELETE Error caused by having a both lazy-load AND a dynamic version of the same relationship

Here is some example code:
users_groups = Table('users_groups', Model.metadata,
Column('user_id', Integer, ForeignKey('users.id')),
Column('group_id', Integer, ForeignKey('groups.id'))
)
class User(Model):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
class Group(Model):
__tablename__ = 'groups'
id = Column(Integer, primary_key=True)
users = relationship('User', secondary=users_groups, lazy='select', backref='groups')
users_dynamic = relationship('User', secondary=users_groups, lazy='dynamic')
So what happens here is that if you add a bunch of users to a group like so:
g = Group()
g.users = [User(), User(), User()]
session.add(g)
session.commit()
and then try to delete the group
session.delete(g)
session.commit()
You will get some form of this error:
DELETE statement on table 'users_groups' expected to delete 3 row(s); Only 0 were matched.
Removing the 2nd version of the relationship (the dynamic one in my case) fixes this problem. I am not even sure where to begin in terms of understanding why this is happening. I have been using 2 versions of various relationships in many cases throughout my SQLAlchemy models in order to make it easy to use the most appropriate query-strategy given a situation. This is the first time it has caused an unexpected issue.
Any advice is welcome.

both the Group.users and Group.users_dynamic relationships are attempting to reconcile the fact that the Group is being deleted along with being able to manage the User() objects they refer to; one relationship succeeds while the second one fails, as the rows in the association table were already deleted. The most straightforward solution is to mark all but one of the identical relationships as viewonly:
class Group(Base):
__tablename__ = 'groups'
id = Column(Integer, primary_key=True)
users = relationship('User', secondary=users_groups, lazy='select', backref='groups')
users_dynamic = relationship('User', viewonly=True, secondary=users_groups, lazy='dynamic')
if you're still wanting to have both relationships handle some degree of mutations, you'd need to do this carefully as SQLAlchemy doesn't know how to coordinate among changes in two relationships at the same time, so conflicts like this can continue to happen (like double inserts, etc) if you make equivalent mutations on both relationships. To just take care of the "delete" issue by itself, you can also try setting Group.users_dynamic to passive_deletes=True:
class Group(Base):
__tablename__ = 'groups'
id = Column(Integer, primary_key=True)
users = relationship('User', secondary=users_groups, lazy='select', backref='groups')
users_dynamic = relationship('User', passive_deletes=True, secondary=users_groups, lazy='dynamic')

I just add another simple workaround.
You can delete the collections before deleting the item itself:
>>> for user in group.users:
group.users.remove(user)
>>> db.session.delete(group)
>>> db.session.commit()
Alternatively, you can also set it as an empty list:
>>> group.users = []
>>> db.session.commit()
>>> db.session.delete(group)
>>> db.session.commit()

Sqlalchemy: One to Many relationship combined with Many to Many relationship

I've got a User and Group table with a many to many relationship
_usergroup_table = db.Table('usergroup_table', db.metadata,
db.Column('user_id', db.Integer, db.ForeignKey('user.id')),
db.Column('group_id', db.Integer, db.ForeignKey('group.id')))
class User(db.Model):
"""Handles the usernames, passwords and the login status"""
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(60), nullable=False, unique=True)
class Group(db.Model):
"""Used for unix-style access control."""
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(60), nullable=False)
users = db.relationship('User', secondary=_usergroup_table,
backref='groups')
Now i'd like to add a primary group to the user class. Of course I could just add a group_id column and a relationship to the Group class, but this has drawbacks. I'd like to get all groups when calling User.group, including primary_group. The primary group should always be part of the groups relationship.
Edit:
It seems the way to go is the association object
class User(db.Model, UserMixin):
"""Handles the usernames, passwords and the login status"""
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(60), nullable=False, unique=True)
primary_group = db.relationship(UserGroup,
primaryjoin="and_(User.id==UserGroup.user_id,UserGroup.primary==True)")
class Group(db.Model):
"""Used for unix-style access control."""
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(60), nullable=False)
class UserGroup(db.Model):
user_id = db.Column(db.Integer, db.ForeignKey('user.id'))
group_id = db.Column(db.Integer, db.ForeignKey('group.id'))
active = db.Column(db.Boolean, default=False)
user = db.relationship(User, backref='groups', primaryjoin=(user_id==User.id))
group = db.relationship(Group, backref='users', primaryjoin=(group_id==Group.id))
I could simplify this with the AssociationProxy, but how do I force only a single primary group per user?

The group_id approach you originally thought of has several advantages here to the "boolean flag" approach.
For one thing, it is naturally constrained so that there is only one primary group per user. For another, loading user.primary_group means the ORM can identify this related row by it's primary key, and can look locally in the identity map for it, or emit a simple SELECT by primary key, instead of emitting a query that has a hard-to-index WHERE clause with a boolean inside of it. Yet another is there's no need to get into the association object pattern which simplifies the usage of the association table and allows SQLAlchemy to handle loads and updates from/to this table more efficiently.
Below we use events, including a new version (as of 0.7.7) of #validates that catches "remove" events, to ensure object-level modifications to User.groups and User.primary_group are kept in sync. (If on an older version of 0.7 you can use the attribute "remove" event or the "AttributeExtension.remove" extension method if you're still on 0.6 or earlier). If you wanted to enforce this at the DB level you could possibly use triggers to verify the integrity you're looking for:
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
Base= declarative_base()
_usergroup_table = Table('usergroup_table', Base.metadata,
Column('user_id', Integer, ForeignKey('user.id')),
Column('group_id', Integer, ForeignKey('group.id')))
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String(60), nullable=False, unique=True)
group_id = Column(Integer, ForeignKey('group.id'), nullable=False)
primary_group = relationship("Group")
#validates('primary_group')
def _add_pg(self, key, target):
self.groups.add(target)
return target
#validates('groups', include_removes=True)
def _modify_groups(self, key, target, is_remove):
if is_remove and target is self.primary_group:
del self.primary_group
return target
class Group(Base):
__tablename__ = 'group'
id = Column(Integer, primary_key=True)
name = Column(String(60), nullable=False)
users = relationship('User', secondary=_usergroup_table,
backref=backref('groups', collection_class=set))
e = create_engine("sqlite://", echo=True)
Base.metadata.create_all(e)
s = Session(e)
g1, g2, g3 = Group(name='g1'), Group(name='g2'), Group(name='g3')
u1 = User(name='u1', primary_group=g1)
u1.groups.update([g2, g3])
s.add_all([
g1, g2, g3, u1
])
s.commit()
u1.groups.remove(g1)
assert u1.primary_group is None
u1.primary_group = g2
s.commit()

How about a GroupMemberships model to hold the association, instead of _usergroup_table? A user could have many groups through Group Memberships, and a group membership can hold additional attributes, such as whether a given Group is the associated User's primary group.
EDIT
In order to enforce a limit of one primary group per user, I would use a validation in the User model, such that any attempt to assign more (or fewer) than one primary group would result in an error when the record is saved. I am not aware of a way of achieving the same result relying purely on the database's integrity system. There are any number of ways of coding the validation check - the documentation shows a nice approach using the validates() decorator.

How to set the foreign key to a default value on delete?

How to auto set the product category_id to a default value when category is deleted? For example 1 to point to the first category.
class Product(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(80))
content = db.Column(db.Text(), unique=True)
category_id = db.Column(db.Integer, db.ForeignKey('category.id'))
atime = db.Column(db.DateTime())
def __init__(self, name, content, category_id):
self.name = name
self.content = content
self.category_id = category_id
self.atime = datetime.now()
def __repr__(self):
return '<Product %r>' % self.id
class Category(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(80))
products = db.relationship('Product', backref='category', cascade="all, delete, delete-orphan")
def __init__(self, *args, **kwargs):
if len(kwargs) > 0:
self.name = kwargs['name']
def __repr__(self):
return '<Category %r>' % self.name
I don't want to use cascade to delete them!

There are two things that need to be performed in orchestration here:
Define the Foreign Key with proper referential action
Configure the cascade option of the SA relationship
I think the cleanest way would be to set the category_id to a NULL value when its Category is deleted: SET NULL is one of the possible Referential Actions of the ON DELETE clause, which you can add to your ForeignKey definition:
category_id = db.Column(db.Integer, db.ForeignKey('category.id', ondelete='SET NULL'))
In the same way you can use an option SET DEFAULT, but in this case you need to also configure the default value for the column category_id: category_id = Column(..., server_default=1). Note that implementations of these differ between different RDBMS.
As to the cascade option: You should basically remove the cascade="all, delete, delete-orphan" from your products relationship definition. In fact, you want to ensure that delete, delete-orphan are not there.
Having said that, you really need to test your code to cover different scenarios, as two different deletions of the Category object might produce different results depending on your RDBMS and SA configuration:
# scenario-1: delete in session: SA might set category_id of all chilren Products to None
c1 = session.query(Category).get(1)
session.delete(c1)
session.commit()
# scenario-2: delete without loading an object into the session: SA will perform no additional logic
session.query(Category).filter(Category.id == 2).delete()
session.commit()
Hope all this points you in the right direction. As usual, enabled SQL logging in your test code using echo=True or just by configuring logging module, and you will see what SA is doing to your database. Other changes which which you did not see in the SQL were done by RDBMS itself given your Referential Action.