Develop Reference

How to have one DB URI for read and one for read-write [duplicate]

I have a Flask, SQLAlchemy webapp which uses a single mysql server. I want to expand the database setup to have a read-only slave server such that I can spread the reads between both master and slave while continuing to write to the master db server.
I have looked at few options and I believe I can't do this with plain SQLAlchemy. Instead I'm planning to create 2 database handles in my webapp, one each for master and slave db servers. Then using a simple random value use either the master/slave db handle for "SELECT" operations.
However, I'm not sure if this is the right way to go with using SQLAlchemy. Any suggestion/tips on how to pull this off?

I have an example of how to do this on my blog at http://techspot.zzzeek.org/2012/01/11/django-style-database-routers-in-sqlalchemy/ . Basically you can enhance the Session so that it chooses from master or slave on a query-by-query basis. One potential glitch with that approach is that if you have one transaction that calls six queries, you might end up using both slaves in one request....but there we're just trying to imitate Django's feature :)
A slightly less magic approach that also establishes the scope of usage more explicitly I've used is a decorator on view callables (whatever they're called in Flask), like this:
#with_slave
def my_view(...):
# ...
with_slave would do something like this, assuming you have a Session and some engines set up:
master = create_engine("some DB")
slave = create_engine("some other DB")
Session = scoped_session(sessionmaker(bind=master))
def with_slave(fn):
def go(*arg, **kw):
s = Session(bind=slave)
return fn(*arg, **kw)
return go
The idea is that calling Session(bind=slave) invokes the registry to get at the actual Session object for the current thread, creating it if it doesn't exist - however since we're passing an argument, scoped_session will assert that the Session we're making here is definitely brand new.
You point it at the "slave" for all subsequent SQL. Then, when the request is over, you'd ensure that your Flask app is calling Session.remove() to clear out the registry for that thread. When the registry is next used on the same thread, it will be a new Session bound back to the "master".
Or a variant, you want to use the "slave" just for that call, this is "safer" in that it restores any existing bind back to the Session:
def with_slave(fn):
def go(*arg, **kw):
s = Session()
oldbind = s.bind
s.bind = slave
try:
return fn(*arg, **kw)
finally:
s.bind = oldbind
return go
For each of these decorators you can reverse things, have the Session be bound to a "slave" where the decorator puts it on "master" for write operations. If you wanted a random slave in that case, if Flask had some kind of "request begin" event you could set it up at that point.

Or, we can try another way. Such as we can declare two different class with all the instance attributes the same but the __bind__ class attribute is different. Thus we can use rw class to do read/write and r class to do read only. :)
I think this way is more easy and reliable. :)
We declare two db models because we can have tables in two different db with the same names. This way we can also bypass the 'extend_existing' error when two models with the same __tablename__.
Here is an example:
app = Flask(__name__)
app.config['SQLALCHEMY_BINDS'] = {'rw': 'rw', 'r': 'r'}
db = SQLAlchemy(app)
db.Model_RW = db.make_declarative_base()
class A(db.Model):
__tablename__ = 'common'
__bind_key__ = 'r'
class A(db.Model_RW):
__tablename__ = 'common'
__bind_key__ = 'rw'

Maybe this answer is too late! I use a slave_session to query the slave DB
class RoutingSession(SignallingSession):
def __init__(self, db, bind_name=None, autocommit=False, autoflush=True, **options):
self.app = db.get_app()
if bind_name:
bind = options.pop('bind', None)
else:
bind = options.pop('bind', None) or db.engine
self._bind_name = bind_name
SessionBase.__init__(
self, autocommit=autocommit, autoflush=autoflush,
bind=bind, binds=None, **options
)
def get_bind(self, mapper=None, clause=None):
if self._bind_name is not None:
state = get_state(self.app)
return state.db.get_engine(self.app, bind=self._bind_name)
else:
if mapper is not None:
try:
persist_selectable = mapper.persist_selectable
except AttributeError:
persist_selectable = mapper.mapped_table
info = getattr(persist_selectable, 'info', {})
bind_key = info.get('bind_key')
if bind_key is not None:
state = get_state(self.app)
return state.db.get_engine(self.app, bind=bind_key)
return SessionBase.get_bind(self, mapper, clause)
class RouteSQLAlchemy(SQLAlchemy):
def __init__(self, *args, **kwargs):
SQLAlchemy.__init__(self, *args, **kwargs)
self.slave_session = self.create_scoped_session({'bind_name':
'slave'})
def create_session(self, options):
return orm.sessionmaker(class_=RoutingSession,db=self,**options)
db = RouteSQLAlchemy(metadata=metadata, query_class=orm.Query)

How to update a declarative SQLAlchemy model using a dictionary data while running validators

The question really is how to update a SQLAlchemy declarative model so that it runs the validators. In my case using setters like User.name = name is not really an option.
Below is a runnable example of what I mean
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy import Column, String, Integer
from sqlalchemy.orm import validates
from sqlalchemy.ext.declarative import declarative_base
some_engine = create_engine('sqlite://')
Session = sessionmaker(bind=some_engine)
session = Session()
Base = declarative_base()
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String, nullable=False)
#validates('name')
def validate_name(self, key, value):
if value != 'asd':
raise ValueError('not asd')
return value
Base.metadata.create_all(bind=some_engine)
user = User(id=1, name='qwe')
# >>> ValueError: not asd
user = User(id=1, name='asd')
session.add(user)
session.commit()
session.query(User).filter(User.id=1).update({'name': 'qwe'})
session.query(User).filter(User.id==1)[0].name
# >>> 'qwe'

You could add a mixin to your models that provides a rather simple update method that just uses setattr() to set attributes of an instance.
class UpdateMixin:
"""
Add a simple update() method to instances that accepts
a dictionary of updates.
"""
def update(self, values):
for k, v in values.items():
setattr(self, k, v)
User class would then be defined as
class User(UpdateMixin, Base):
...
And to update a single instance from a given dictionary you could for example run
session.query(User).get(1).update({ 'name': 'qwe' })
# or since you have the user instance from before
user.update({ 'name': 'qwe' })
Note the use of Query.get(). If there is no user with the given id, it will return None and trying to call the method update on it will raise. Another caveat is that if you do not rollback if any exceptions are raised, you cannot predict what, if any, updates took place (were added to the session) because a dictionary has no ordering. So always rollback on any errors.
I'd also recommend actually naming the method updateSelf or some such to prevent risk of confusing it with Query.update().

The short answer is not to use query.update when you want model level constraints. It's exactly for the times when performance is more important than enforcing those sorts of model level constraints. Other answers have provided specifics on solutions, but the fundamental answer is that Query.update is not intended to enforce python-level constraints.
General categories of solutions are:
Use some session-level method and Query.get or a loop on Query.filter.all
Check constraints
Triggers and stored procedures

SQLAlchemy Versioned Object not working

I am using the SQLAlchemy versioned object example as a reference.
Example: http://docs.sqlalchemy.org/en/rel_0_7/orm/examples.html#versioned-objects
When I update a record I am getting no errors. The case_history table is being created, but the version number is staying at '1' and the case_history table is empty.
(Yes I am aware that I am using 'case' as a class name. Is that bad?)
Here are my code snippets:
models.py:
from history_meta import Versioned, versioned_session
# set up the base class
class Base(object):
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
id = Column(Integer, primary_key = True)
header_row = Column(SmallInteger)
def to_dict(self):
serialized = dict((column_name, getattr(self, column_name))
for column_name in self.__table__.c.keys())
return serialized
Base = declarative_base(cls=Base)
class case(Versioned, Base):
title = Column(String(32))
description = Column(Text)
def __repr__(self):
return self.title
app.py:
engine = create_engine(SQLALCHEMY_DATABASE_URI)
Session = sessionmaker(bind=engine)
versioned_session(Session)
db = Session()
...
#app.route('/<name>/:record', method='POST')
def default(name, record):
myClass = getattr(sys.modules[__name__], name)
db.query(myClass).filter(myClass.id == record).update(request.json)
for u in db.query(case).filter(case.id == record):
print u.version # Version is always 1
db.commit() # I added this just to test versioning.
Any clue as to why the versioning isn't happening?

For others who find their way here...
Remember: even if filter() returns a single object, the update() method is a bulk operation, and acts differently. It is possible the version is only incremented on an event like after_update() which does not trigger on bulk operations.
Read more on caveats for the update() operation here.

An update query will not cause the version to increment even though the data changes. There might be ways to 'listen' for that type of change, but I don't know.
You have to change an attribute of a mapped class:
#Get an instance of the class
myItem = db.query(myClass).get(record)
#Change an attribute
myItem.title="foo"
#Commit if necessary
db.commit()

How can I verify Column data types in the SQLAlchemy ORM?

Using the SQLAlchemy ORM, I want to make sure values are the right type for their columns.
For example, say I have an Integer column. I try to insert the value “hello”, which is not a valid integer. SQLAlchemy will allow me to do this. Only later, when I execute session.commit(), does it raise an exception: sqlalchemy.exc.DataError: (DataError) invalid input syntax integer: "hello"….
I am adding batches of records, and I don’t want to commit after every single add(…), for performance reasons.
So how can I:
Raise the exception as soon as I do session.add(…)
Or, make sure the value I am inserting can be converted to the target Column datatype, before adding it to the batch?
Or any other way to prevent one bad record from spoiling an entire commit().

SQLAlchemy doesn't build this in as it defers to the DBAPI/database as the best and most efficient source of validation and coercion of values.
To build your own validation, usually TypeDecorator or ORM-level validation is used. TypeDecorator has the advantage that it operates at the core and can be pretty transparent, though it only occurs when SQL is actually emitted.
To do validation and coercion sooner, this is at the ORM level.
Validation can be ad-hoc, at the ORM layer, via #validates:
http://docs.sqlalchemy.org/en/latest/orm/mapped_attributes.html#simple-validators
The event system that #validates uses is also available directly. You can write a generalized solution that links validators of your choosing to the types being mapped:
from sqlalchemy import Column, Integer, String, DateTime
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import event
import datetime
Base= declarative_base()
def validate_int(value):
if isinstance(value, basestring):
value = int(value)
else:
assert isinstance(value, int)
return value
def validate_string(value):
assert isinstance(value, basestring)
return value
def validate_datetime(value):
assert isinstance(value, datetime.datetime)
return value
validators = {
Integer:validate_int,
String:validate_string,
DateTime:validate_datetime,
}
# this event is called whenever an attribute
# on a class is instrumented
#event.listens_for(Base, 'attribute_instrument')
def configure_listener(class_, key, inst):
if not hasattr(inst.property, 'columns'):
return
# this event is called whenever a "set"
# occurs on that instrumented attribute
#event.listens_for(inst, "set", retval=True)
def set_(instance, value, oldvalue, initiator):
validator = validators.get(inst.property.columns[0].type.__class__)
if validator:
return validator(value)
else:
return value
class MyObject(Base):
__tablename__ = 'mytable'
id = Column(Integer, primary_key=True)
svalue = Column(String)
ivalue = Column(Integer)
dvalue = Column(DateTime)
m = MyObject()
m.svalue = "ASdf"
m.ivalue = "45"
m.dvalue = "not a date"
Validation and coercion can also be built at the type level using TypeDecorator, though this is only when SQL is being emitted, such as this example which coerces utf-8 strings to unicode:
http://docs.sqlalchemy.org/en/latest/core/custom_types.html#coercing-encoded-strings-to-unicode

Improving on the answer of #zzzeek , I suggest the following solution:
from sqlalchemy import String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.event import listen_for
Base = declarative_base()
#listens_for(Base, 'attribute_instrument')
def configure_listener(table_cls, attr, col_inst):
if not hasattr(col_inst.property, 'columns'):
return
validator = getattr(col_inst.property.columns[0].type, 'validator', None)
if validator:
# Only decorate columns, that need to be decorated
#listens_for(col_inst, "set", retval=True)
def set_(instance, value, oldvalue, initiator):
return validator(value)
That lets you do things like:
class Name(String):
def validator(self, name):
if isinstance(name, str):
return name.upper()
raise TypeError("name must be a string")
This has two benefits: Firstly, there is only an event triggered, when there actually is a validator attached to the data field object. It does not waste precious CPU cycles on set events for objects, that have no function for validation defined. Secondly, it allows you to define your own field types and just add a validator method there, so not all things that you want to store as Integer etc run through the same checks, just the ones derived from your new field type.

Controlling the instantiation of python object

My question does not really have much to do with sqlalchemy but rather with pure python.
I'd like to control the instantiation of sqlalchemy Model instances. This is a snippet from my code:
class Tag(db.Model):
__tablename__ = 'tags'
query_class = TagQuery
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(), unique=True, nullable=False)
def __init__(self, name):
self.name = name
I want to achieve that whenever an entry is instantiated (Tag('django')) that a new instance should be created only if there is not yet another tag with the name django inside the database. Otherwise, instead of initializing a new object, a reference to the already existent row inside the database should be returned by (Tag('django')).
As of now I am ensuring the uniqueness of tags inside the Post Model:
class Post(db.Model):
# ...
# code code code
# ...
def _set_tags(self, taglist):
"""Associate tags with this entry. The taglist is expected to be already
normalized without duplicates."""
# Remove all previous tags
self._tags = []
for tag_name in taglist:
exists = Tag.query.filter(Tag.name==tag_name).first()
# Only add tags to the database that don't exist yet
# TODO: Put this in the init method of Tag (if possible)
if not exists:
self._tags.append(Tag(tag_name))
else:
self._tags.append(exists)
It does its job but still I'd like to know how to ensure the uniqueness of tags inside the Tag class itself so that I could write the _set_tags method like this:
def _set_tags(self, taglist):
# Remove all previous tags
self._tags = []
for tag_name in taglist:
self._tags.append(Tag(tag_name))
While writing this question and testing I learned that I need to use the __new__ method. This is what I've come up with (it even passes the unit tests and I didn't forget to change the _set_tags method):
class Tag(db.Model):
__tablename__ = 'tags'
query_class = TagQuery
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(), unique=True, nullable=False)
def __new__(cls, *args, **kwargs):
"""Only add tags to the database that don't exist yet. If tag already
exists return a reference to the tag otherwise a new instance"""
exists = Tag.query.filter(Tag.name==args[0]).first() if args else None
if exists:
return exists
else:
return super(Tag, cls).__new__(cls, *args, **kwargs)
What bothers me are two things:
First: I get a warning:
DeprecationWarning: object.__new__() takes no parameters
Second: When I write it like so I get errors (I also tried to rename the paramater name to n but it did not change anything) :
def __new__(cls, name):
"""Only add tags to the database that don't exist yet. If tag already
exists return a reference to the tag otherwise a new instance"""
exists = Tag.query.filter(Tag.name==name).first()
if exists:
return exists
else:
return super(Tag, cls).__new__(cls, name)
Errors (or similar):
TypeError: __new__() takes exactly 2 arguments (1 given)
I hope you can help me!

I use class method for that.
class Tag(Declarative):
...
#classmethod
def get(cls, tag_name):
tag = cls.query.filter(cls.name == tag_name).first()
if not tag:
tag = cls(tag_name)
return tag
And then
def _set_tags(self, taglist):
self._tags = []
for tag_name in taglist:
self._tags.append(Tag.get(tag_name))
As for __new__, you should not confuse it with __init__. It is expected to be called w/out args, so even if your own constructor asks for some, you should not pass them to super/object unless you know that your super needs them. Typical invocation would be:
def __new__(cls, name=None):
tag = cls.query.filter(cls.name == tag_name).first()
if not tag:
tag = object.__new__(cls)
return tag
However this will not work as expected in your case, since it calls __init__ automatically if __new__ returns instance of cls. You would need to use metaclass or add some checks in __init__.

Don't embed this within the class itself.
Option 1. Create a factory that has the pre-existing pool of objects.
tag_pool = {}
def makeTag( name ):
if name not in tag_pool:
tag_pool[name]= Tag(name)
return tag_pool[name]
Life's much simpler.
tag= makeTag( 'django' )
This will create the item if necessary.
Option 2. Define a "get_or_create" version of the makeTag function. This will query the database. If the item is found, return the object. If no item is found, create it, insert it and return it.

Given the OP's latest error msg:
TypeError: __new__() takes exactly 2 arguments (1 given)
it seems that somewhere the class is getting instantiated without the name parameter, i.e. just Tag(). The traceback for that exception should tell you where that "somewhere" is (but we're not shown it, so that's how far as we can go;-).
That being said, I agree with other answers that a factory function (possibly nicely dressed up as a classmethod -- making factories is one of the best uses of classmethod, after all;-) is the way to go, avoiding the complication that __new__ entails (such as forcing __init__ to find out whether the object's already initialized to avoid re-initializing it!-).