Develop Reference

Flask Postgresql array not permanently updating

I'm working on a project using Flask and a PostgreSQL database, with SQLAlchemy.
I have Group objects which have a list of User IDs who are members of the group. For some reason, when I try to add an ID to a group, it will not save properly.
If I try members.append(user_id), it doesn't seem to work at all. However, if I try members += [user_id], the id will show up in the view listing all the groups, but if I restart the server, the added value(s) is (are) not there. The initial values, however, are.
Related code:
Adding group to the database initially:
db = SQLAlchemy(app)
# ...
g = Group(request.form['name'], user_id)
db.session.add(g)
db.session.commit()
The Group class:
from flask.ext.sqlalchemy import SQLAlchemy
from sqlalchemy.dialects.postgresql import ARRAY
class Group(db.Model):
__tablename__ = "groups"
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(128))
leader = db.Column(db.Integer)
# list of the members in the group based on user id
members = db.Column(ARRAY(db.Integer))
def __init__(self, name, leader):
self.name = name
self.leader = leader
self.members = [leader]
def __repr__(self):
return "Name: {}, Leader: {}, Members: {}".format(self.name, self.leader, self.members)
def add_user(self, user_id):
self.members += [user_id]
My test function for updating the Group:
def add_2_to_group():
g = Group.query.all()[0]
g.add_user(2)
db.session.commit()
return redirect(url_for('show_groups'))
Thanks for any help!

As you have mentioned, the ARRAY datatype in sqlalchemy is immutable. This means it isn’t possible to add new data into array once it has been initialised.
To solve this, create class MutableList.
from sqlalchemy.ext.mutable import Mutable
class MutableList(Mutable, list):
def append(self, value):
list.append(self, value)
self.changed()
#classmethod
def coerce(cls, key, value):
if not isinstance(value, MutableList):
if isinstance(value, list):
return MutableList(value)
return Mutable.coerce(key, value)
else:
return value
This snippet allows you to extend a list to add mutability to it. So, now you can use the class above to create a mutable array type like:
class Group(db.Model):
...
members = db.Column(MutableList.as_mutable(ARRAY(db.Integer)))
...

You can use the flag_modified function to mark the property as having changed. In this example, you could change your add_user method to:
from sqlalchemy.orm.attributes import flag_modified
# ~~~
def add_user(self, user_id):
self.members += [user_id]
flag_modified(self, 'members')

To anyone in the future: so it turns out that arrays through SQLAlchemy are immutable. So, once they're initialized in the database, they can't change size. There's probably a way to do this, but there are better ways to do what we're trying to do.

This is a hacky solution, but what you can do is:
Store the existing array temporarily
Set the column value to None
Set the column value to the existing temporary array
For example:
g = Group.query.all()[0]
temp_array = g.members
g.members = None
db.session.commit()
db.session.refresh(g)
g.members = temp_array
db.session.commit()

In my case it was solved by using the new reference for storing a object variable and assiging that new created variable in object variable.so, Instead of updating the existing objects variable it will create a new reference address which reflect the changes.
Here in Model,
Table: question
optional_id = sa.Column(sa.ARRAY(sa.Integer), nullable=True)
In views,
option_list=list(question.optional_id if question.optional_id else [])
if option_list:
question.optional_id.clear()
option_list.append(obj.id)
question.optional_id=option_list
else:
question.optional_id=[obj.id]

sqlalchemy generic foreign key (like in django ORM)

Does sqlalchemy have something like django's GenericForeignKey? And is it right to use generic foreign fields.
My problem is: I have several models (for example, Post, Project, Vacancy, nothing special there) and I want to add comments to each of them. And I want to use only one Comment model. Does it worth to? Or should I use PostComment, ProjectComment etc.? Pros/cons of both ways?
Thanks!

The simplest pattern which I use most often is that you actually have separate Comment tables for each relationship. This may seem frightening at first, but it doesn't incur any additional code versus using any other approach - the tables are created automatically, and the models are referred to using the pattern Post.Comment, Project.Comment, etc. The definition of Comment is maintained in one place. This approach from a referential point of view is the most simple and efficient, as well as the most DBA friendly as different kinds of Comments are kept in their own tables which can be sized individually.
Another pattern to use is a single Comment table, but distinct association tables. This pattern offers the use case that you might want a Comment linked to more than one kind of object at a time (like a Post and a Project at the same time). This pattern is still reasonably efficient.
Thirdly, there's the polymorphic association table. This pattern uses a fixed number of tables to represent the collections and the related class without sacrificing referential integrity. This pattern tries to come the closest to the Django-style "generic foreign key" while still maintaining referential integrity, though it's not as simple as the previous two approaches.
Imitating the pattern used by ROR/Django, where there are no real foreign keys used and rows are matched using application logic, is also possible.
The first three patterns are illustrated in modern form in the SQLAlchemy distribution under examples/generic_associations/.
The ROR/Django pattern, since it gets asked about so often, I will also add to the SQLAlchemy examples, even though I don't like it much. The approach I'm using is not exactly the same as what Django does as they seem to make use of a "contenttypes" table to keep track of types, that seems kind of superfluous to me, but the general idea of an integer column that points to any number of tables based on a discriminator column is present. Here it is:
from sqlalchemy.ext.declarative import declarative_base, declared_attr
from sqlalchemy import create_engine, Integer, Column, \
String, and_
from sqlalchemy.orm import Session, relationship, foreign, remote, backref
from sqlalchemy import event
class Base(object):
"""Base class which provides automated table name
and surrogate primary key column.
"""
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
id = Column(Integer, primary_key=True)
Base = declarative_base(cls=Base)
class Address(Base):
"""The Address class.
This represents all address records in a
single table.
"""
street = Column(String)
city = Column(String)
zip = Column(String)
discriminator = Column(String)
"""Refers to the type of parent."""
parent_id = Column(Integer)
"""Refers to the primary key of the parent.
This could refer to any table.
"""
#property
def parent(self):
"""Provides in-Python access to the "parent" by choosing
the appropriate relationship.
"""
return getattr(self, "parent_%s" % self.discriminator)
def __repr__(self):
return "%s(street=%r, city=%r, zip=%r)" % \
(self.__class__.__name__, self.street,
self.city, self.zip)
class HasAddresses(object):
"""HasAddresses mixin, creates a relationship to
the address_association table for each parent.
"""
#event.listens_for(HasAddresses, "mapper_configured", propagate=True)
def setup_listener(mapper, class_):
name = class_.__name__
discriminator = name.lower()
class_.addresses = relationship(Address,
primaryjoin=and_(
class_.id == foreign(remote(Address.parent_id)),
Address.discriminator == discriminator
),
backref=backref(
"parent_%s" % discriminator,
primaryjoin=remote(class_.id) == foreign(Address.parent_id)
)
)
#event.listens_for(class_.addresses, "append")
def append_address(target, value, initiator):
value.discriminator = discriminator
class Customer(HasAddresses, Base):
name = Column(String)
class Supplier(HasAddresses, Base):
company_name = Column(String)
engine = create_engine('sqlite://', echo=True)
Base.metadata.create_all(engine)
session = Session(engine)
session.add_all([
Customer(
name='customer 1',
addresses=[
Address(
street='123 anywhere street',
city="New York",
zip="10110"),
Address(
street='40 main street',
city="San Francisco",
zip="95732")
]
),
Supplier(
company_name="Ace Hammers",
addresses=[
Address(
street='2569 west elm',
city="Detroit",
zip="56785")
]
),
])
session.commit()
for customer in session.query(Customer):
for address in customer.addresses:
print(address)
print(address.parent)

I know this is probably a terrible way to do this, but it was a quick fix for me.
class GenericRelation(object):
def __init__(self, object_id, object_type):
self.object_id = object_id
self.object_type = object_type
def __composite_values__(self):
return (self.object_id, self.object_type)
class Permission(AbstractBase):
#__abstract__ = True
_object = None
_generic = composite(
GenericRelation,
sql.Column('object_id', data_types.UUID, nullable=False),
sql.Column('object_type', sql.String, nullable=False),
)
permission_type = sql.Column(sql.Integer)
#property
def object(self):
session = object_session(self)
if self._object or not session:
return self._object
else:
object_class = eval(self.object_type)
self._object = session.query(object_class).filter(object_class.id == self.object_id).first()
return self._object
#object.setter
def object(self, value):
self._object = value
self.object_type = value.__class__.__name__
self.object_id = value.id

How can I 'index' SQLAlchemy model attributes that are primary keys and relationships

So say I have some classes X, Y and Z using SQLAlchemy declarative syntax to define some simple columns and relationships
Requirements:
At the class level, (X|Y|Z).primary_keys returns a collection of
the respective class' primary keys' (InstrumentedAttribute
objects) I also want (X|Y|Z).relations to reference the class'
relations in the same way
At the instance level, I would like the same attributes to reference
those attributes' instantiated values, whether they've been
populated using my own constructors, individual attributes
setters, or whatever SQLAlchemy does when it retrieves rows from
the db.
So far I have the following.
import collections
import sqlalchemy
import sqlalchemy.ext.declarative
from sqlalchemy import MetaData, Column, Table, ForeignKey, Integer, String, Date, Text
from sqlalchemy.orm import relationship, backref
class IndexedMeta(sqlalchemy.ext.declarative.DeclarativeMeta):
"""Metaclass to initialize some class-level collections on models"""
def __new__(cls, name, bases, defaultdict):
cls.pk_columns = set()
cls.relations = collections.namedtuple('RelationshipItem', 'one many')( set(), set())
return super().__new__(cls, name, bases, defaultdict)
Base = sqlalchemy.ext.declarative.declarative_base(metaclass=IndexedMeta)
def build_class_lens(cls, key, inst):
"""Populates the 'indexes' of primary key and relationship attributes with the attributes' names. Additionally, separates "x to many" relationships from "x to one" relationships and associates "x to one" relathionships with the local-side foreign key column"""
if isinstance(inst.property, sqlalchemy.orm.properties.ColumnProperty):
if inst.property.columns[0].primary_key:
cls.pk_columns.add(inst.key)
elif isinstance(inst.property, sqlalchemy.orm.properties.RelationshipProperty):
if inst.property.direction.name == ('MANYTOONE' or 'ONETOONE'):
local_column = cls.__mapper__.get_property_by_column(inst.property.local_side[0]).key
cls.relations.one.add( (local_column, inst.key) )
else:
cls.relations.many.add(inst.key)
sqlalchemy.event.listen(Base, 'attribute_instrument', build_class_lens)
class Meeting(Base):
__tablename__ = 'meetings'
def __init__(self, memo):
self.memo = memo
id = Column(Integer, primary_key=True)
date = Column(Date)
memo = Column('note', String(60), nullable=True)
category_name = Column('category', String(60), ForeignKey('categories.name'))
category = relationship("Category", backref=backref('meetings'))
topics = relationship("Topic",
secondary=meetings_topics,
backref="meetings")
...
...
Ok, so that gets me by on the class level, though I feel like I am doing silly things with metaclasses, and I get some strange intermittent errors where the 'sqlalchemy' module allegedly isn't recognized in build_class_lens and evals to Nonetype.
I am not quite sure how I should proceed at the instance level.
I've looked into the events interface. I see the ORM event init, but it seems to run prior to the __init__ function defined on my models, meaning the instance attributes haven't yet been populated at that time, so I can't build my 'lens' on them.
I also wonder if the Attribute event set might be of help. That is my next try, though i still wonder if it is the most appropriate way.
All in all I really wonder if I am missing some really elegant way to approach this problem.

I think the metaclass thing with declarative goes by the old XML saying, "if you have a problem, and use XML, now you have two problems". The metaclass in Python is useful pretty much as a hook to detect the construction of new classes, and that's about it. We now have enough events that there shouldn't be any need to use a metaclass beyond what declarative already does.
In this case I'd go a little further and say that the approach of trying to actively build up these collections is not really worth it - it's much easier to generate them lazily, as below:
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
import collections
from sqlalchemy.orm.properties import RelationshipProperty
class memoized_classproperty(object):
"""A decorator that evaluates once at the class level,
assigns the new value to the class.
"""
def __init__(self, fget, doc=None):
self.fget = fget
self.__doc__ = doc or fget.__doc__
self.__name__ = fget.__name__
def __get__(desc, self, cls):
result = desc.fget(cls)
setattr(cls, desc.__name__, result)
return result
class Lens(object):
#memoized_classproperty
def pk_columns(cls):
return class_mapper(cls).primary_key
#memoized_classproperty
def relations(cls):
props = collections.namedtuple('RelationshipItem', 'one many')(set(), set())
# 0.8 will have "inspect(cls).relationships" here
mapper = class_mapper(cls)
for item in mapper.iterate_properties:
if isinstance(item, RelationshipProperty):
if item.direction.name == ('MANYTOONE' or 'ONETOONE'):
local_column = mapper.get_property_by_column(item.local_side[0]).key
props.one.add((local_column, item.key))
else:
props.many.add(item.key)
return props
Base= declarative_base(cls=Lens)
meetings_topics = Table("meetings_topics", Base.metadata,
Column('topic_id', Integer, ForeignKey('topic.id')),
Column('meetings_id', Integer, ForeignKey('meetings.id')),
)
class Meeting(Base):
__tablename__ = 'meetings'
def __init__(self, memo):
self.memo = memo
id = Column(Integer, primary_key=True)
date = Column(Date)
memo = Column('note', String(60), nullable=True)
category_name = Column('category', String(60), ForeignKey('categories.name'))
category = relationship("Category", backref=backref('meetings'))
topics = relationship("Topic",
secondary=meetings_topics,
backref="meetings")
class Category(Base):
__tablename__ = 'categories'
name = Column(String(50), primary_key=True)
class Topic(Base):
__tablename__ = 'topic'
id = Column(Integer, primary_key=True)
print Meeting.pk_columns
print Meeting.relations.one
# assignment is OK, since prop is memoized
Meeting.relations.one.add("FOO")
print Meeting.relations.one

How does django one-to-one relationships map the name to the child object?

Apart from one example in the docs, I can't find any documentation on how exactly django chooses the name with which one can access the child object from the parent object. In their example, they do the following:
class Place(models.Model):
name = models.CharField(max_length=50)
address = models.CharField(max_length=80)
def __unicode__(self):
return u"%s the place" % self.name
class Restaurant(models.Model):
place = models.OneToOneField(Place, primary_key=True)
serves_hot_dogs = models.BooleanField()
serves_pizza = models.BooleanField()
def __unicode__(self):
return u"%s the restaurant" % self.place.name
# Create a couple of Places.
>>> p1 = Place(name='Demon Dogs', address='944 W. Fullerton')
>>> p1.save()
>>> p2 = Place(name='Ace Hardware', address='1013 N. Ashland')
>>> p2.save()
# Create a Restaurant. Pass the ID of the "parent" object as this object's ID.
>>> r = Restaurant(place=p1, serves_hot_dogs=True, serves_pizza=False)
>>> r.save()
# A Restaurant can access its place.
>>> r.place
<Place: Demon Dogs the place>
# A Place can access its restaurant, if available.
>>> p1.restaurant
So in their example, they simply call p1.restaurant without explicitly defining that name. Django assumes the name starts with lowercase. What happens if the object name has more than one word, like FancyRestaurant?
Side note: I'm trying to extend the User object in this way. Might that be the problem?

If you define a custom related_name then it will use that, otherwise it will lowercase the entire model name (in your example .fancyrestaurant). See the else block in django.db.models.related code:
def get_accessor_name(self):
# This method encapsulates the logic that decides what name to give an
# accessor descriptor that retrieves related many-to-one or
# many-to-many objects. It uses the lower-cased object_name + "_set",
# but this can be overridden with the "related_name" option.
if self.field.rel.multiple:
# If this is a symmetrical m2m relation on self, there is no reverse accessor.
if getattr(self.field.rel, 'symmetrical', False) and self.model == self.parent_model:
return None
return self.field.rel.related_name or (self.opts.object_name.lower() + '_set')
else:
return self.field.rel.related_name or (self.opts.object_name.lower())
And here's how the OneToOneField calls it:
class OneToOneField(ForeignKey):
... snip ...
def contribute_to_related_class(self, cls, related):
setattr(cls, related.get_accessor_name(),
SingleRelatedObjectDescriptor(related))
The opts.object_name (referenced in the django.db.models.related.get_accessor_name) defaults to cls.__name__.
As for
Side note: I'm trying to extend the
User object in this way. Might that be
the problem?
No it won't, the User model is just a regular django model. Just watch out for related_name collisions.

Python dicts in sqlalchemy

I would like to load/save a dict to/from my sqlite DB, but am having some problems figuring out a simple way to do it. I don't really need to be able to filter, etc., based on the contents so a simple conversion to/from string is fine.
The next-best thing would be foreign keys. Please don't post links to huge examples, my head would explode if I ever set eyes on any those.

The SQLAlchemy PickleType is meant exactly for this.
class SomeEntity(Base):
__tablename__ = 'some_entity'
id = Column(Integer, primary_key=True)
attributes = Column(PickleType)
# Just set the attribute to save it
s = SomeEntity(attributes={'baked': 'beans', 'spam': 'ham'})
session.add(s)
session.commit()
# If mutable=True on PickleType (the default) SQLAlchemy automatically
# notices modifications.
s.attributes['parrot'] = 'dead'
session.commit()
You can change the serialization mechanism by changing out the pickler with something else that has dumps() and loads() methods. The underlying storage mechanism by subclassing PickleType and overriding the impl attritbute:
class TextPickleType(PickleType):
impl = Text
import json
class SomeOtherEntity(Base):
__tablename__ = 'some_other_entity'
id = Column(Integer, primary_key=True)
attributes = Column(TextPickleType(pickler=json))

You can create a custom type by subclassing sqlalchemy.types.TypeDecorator to handle serialization and deserialization to Text.
An implementation might look like
import json
import sqlalchemy
from sqlalchemy.types import TypeDecorator
SIZE = 256
class TextPickleType(TypeDecorator):
impl = sqlalchemy.Text(SIZE)
def process_bind_param(self, value, dialect):
if value is not None:
value = json.dumps(value)
return value
def process_result_value(self, value, dialect):
if value is not None:
value = json.loads(value)
return value
Example usage:
class SomeModel(Base):
__tablename__ = 'the_table'
id = Column(Integer, primary_key=True)
json_field = Column(TextPickleType())
s = SomeModel(json_field={'baked': 'beans', 'spam': 'ham'})
session.add(s)
session.commit()
This is outlined in an example in the SQLAlchemy docs, which also shows how to track mutations of that dictionary.
This approach should work for all versions of Python, whereas simply passing json as the value to the pickler argument of PickleType will not work correctly, as AlexGrönholm points out in his comment on another answer.

SQLAlchemy has a built-in JSON type that you can use:
attributes = Column(JSON)

If you need to map a 1-N relation and map it as dict rather then list, then read Custom Dictionary-Based Collections
But if you mean a field, then what you can do it to have a DB field of type string, which is mapped to your Python object. But on the same python object you provide a property which will be kind-of proxy for this mapped string field of type dict().
Code example (not tested):
class MyObject(object):
# fields (mapped automatically by sqlalchemy using mapper(...)
MyFieldAsString = None
def _get_MyFieldAsDict(self):
if self.MyFieldAsString:
return eval(self.MyFieldAsString)
else:
return {} # be careful with None and empty dict
def _set_MyFieldAsDict(self, value):
if value:
self.MyFieldAsString = str(value)
else:
self.MyFieldAsString = None
MyFieldAsDict = property(_get_MyFieldAsDict, _set_MyFieldAsDict)

You can simply save() method to save dicts in sqlalchemy
For example
class SomeModel(Base):
__tablename__ = 'the_table'
id = Column(Integer, primary_key=True)
baked = Column(String, nullable=True)
spam = Column(String, nullable=True)
s = {'baked': 'beans', 'spam': 'ham'})
SomeModel(**s).save()