I'm trying to share a simple functionality across all my models (timestamping) and am going with the "augmenting the base" approach as described in SQLA docs. So far I did this:
import sqlalchemy as sa
from datetime import datetime as dt
class EntityBase(object):
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
id = sa.Column(sa.Integer, primary_key=True)
last_update = sa.Column(sa.DateTime, default=dt.utcnow())
def update_entity(mapper, connection, target):
target.last_update = dt.utcnow()
Entity = declarative_base(cls=EntityBase)
sa.event.listen(Entity, 'before_insert', update_entity)
sa.event.listen(Entity, 'before_update', update_entity)
All my models are derived from the Entity class. But at runtime I get sqlalchemy.orm.exc.UnmappedClassError: Class 'sqlalchemy.ext.declarative.Base' is not mapped. What am I doing wrong?
UPDATE
I've circumvented the problem by doing simple preprocessing, like this:
def setupEntities():
...
for cls in Entity.__subclasses__():
listen(cls, 'before_insert', update_entity)
listen(cls, 'before_update', update_entity)
...
...but I'd like to hear about a right way to do it.
interesting here is that the way you did it above probably should work. It would be handy. So I've added a ticket for that http://www.sqlalchemy.org/trac/ticket/2585 .
For now, a way you can do this is to set an event listener for new mappings along with your Base:
Entity = declarative_base(cls=EntityBase):
from sqlalchemy.orm import mapper
#event.listens_for(mapper, 'mapper_configured')
def set_events(mapper, class_):
if issubclass(class_, Entity):
listen(class_, 'before_update', update_entity)
listen(class_, 'before_insert', update_entity)
in fact if I implement the feature for #2585 I'd probably have to do it very similarly to this also.
Related
I came across
several approaches on how to use the vanilla SQLAlchemy models in Flask-SQLAlchemy.
It works like a charm to use models that inherit from Base with Flask-SQLAlchemy.
But I really like that convenience stuff...
Job.query.all() # Does not work
db.session.query(Job).all() # Works
So I started to work on this and put together some code, but I am stuck and need some help to get this nice and clean.
The following block is a general definition that does not inherit from either.
It is imported and supposed to be used from Flask-SQLAlchemy and vanilla SQLAlchemy at some point.
class VanillaMachine():
__tablename__ = 'machine'
id = Column(Integer, primary_key=True)
name = Column(String(100))
status = Column(Integer)
And there is a factory that takes either db.Model or Base and return Machine with the correct parent:
class MachineFactory:
def __init__(self, *args, **kwargs):
pass
def __new__(cls, *args, **kwargs):
return type('Machine',(object, VanillaMachine, args[0]), VanillaMachine.__dict__.copy())
I am quite sure that there's something off with that code, but
I am not sure where.
If I use it like
db = SQLAlchemy()
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()#
Machine1 = MachineFactory(db.Model)
Machine2 = MachineFactory(Base)
there is a error message
sqlalchemy.exc.ArgumentError: Column object 'id' already assigned to Table 'machine'
Can help me to get this straight in a nice, reliable way?
I know that you could just use a function, pass the parent as argument into VanillaMachine and use some if statement, but that would be too straightforward, right? :)
Edit:
Other approaches I came across are
using the Flask context to use Flask-SQLAlchemy models
with app.app_context():
pass
or
app.app_context().push()
But this is too focused on Flask for me and does not allow to clearly separate the models, make them independent and adjust to the context.
supplying an alternative Base class to db = SQLAlchemy(app, model_class=Base), see here. This might work for me, but I did not evaluate this so far.
I found a good solution inspired by a Factory pattern and
Declarative Mixins as mentioned in the SQLAlchemy docs.
For complex multi-level inheritance scenarios a different approach is needed, using #declared_attr.cascading.
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy import Column, Integer, String
from sqlalchemy import MetaData
from sqlalchemy.ext.declarative import declarative_base
from flask_sqlalchemy import SQLAlchemy
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + '/tmp/test_app.db'
engine = create_engine(SQLALCHEMY_DATABASE_URI, echo=True)
# for vanilla
Base = declarative_base()
# for Flask (import from app once initialized)
db = SQLAlchemy()
class MachineMixin:
__tablename__ = 'machine'
id = Column(Integer, primary_key=True)
name = Column(String(100))
status = Column(Integer)
class ModelFactory:
#staticmethod
def create(which_model, which_parent):
if which_parent == 'flask_sqlalchemy':
parent = db.Model
elif which_parent == 'pure_sqlalchemy':
parent = Base
# now use type() to interit, fill __dict__ and assign a name
obj = type(which_model.__name__ + '_' + which_parent,
(which_model, parent),
{})
return obj
test_scenario = 'pure_sqlalchemy' # 'flask_sqlalchemy'
Machine = ModelFactory.create(MachineMixin, test_scenario)
if test_scenario == 'flask_sqlalchemy':
db.metadata.drop_all(bind=engine)
db.metadata.create_all(bind=engine)
elif test_scenario == 'pure_sqlalchemy':
Base.metadata.drop_all(bind=engine)
Base.metadata.create_all(bind=engine)
Session = sessionmaker(bind=engine)
session = Session()
session.add(Machine(name='Bob', status=1))
session.commit()
I want to have my database implementation in a separate module or class. But I am struggling with a few details. A simple example:
from peewee import *
db = SqliteDatabase(':memory:')
class BaseModel(Model):
class Meta:
database = db
class User(BaseModel):
name = CharField()
db.connect()
db.create_tables([User,])
db.commit()
#db.atomic()
def add_user(name):
User.create(name=name).save()
#db.atomic()
def get_user(name):
return User.get(User.name == name)
So far this is working fine. I can implement my interface to the database here and import this as a module.
Now I want to be able to choose the database file at runtime. So I need a way to define the Model classes without defining SqliteDatabase('somefile') before. I tried to encapsulate everything in a new Database class, which I can later import and create an instance from:
from peewee import *
class Database:
def __init__(self, dbfile):
self.db = SqliteDatabase(dbfile)
class BaseModel(Model):
class Meta:
database = self.db
class User(BaseModel):
name = CharField()
self.User = User
self.db.connect()
self.db.create_tables([User,])
self.db.commit()
#self.db.atomic() # Error as self is not known on this level
def add_user(self, name):
self.User.create(name=name).save()
#self.db.atomic() # Error as self is not known on this level
def get_user(self, name):
return self.User.get(self.User.name == name)
Now I can call for example database = Database('database.db') or choose any other file name. I can even use multiple database instance in the same program, each with its own file.
However, there are two problems with this approach:
I still need to specify the database driver (SqliteDatabase) before defining the Model classes. To solve this I define the Model classes within the __init__() method, and then create an alias to with self.User = User. I don't really like this approach (it just doesn't feel like neat code), but at least it works.
I cannot use the #db.atomic() decorator since self is not known at class level, I would an instance here.
So this class approach does not seem to work very well. Is there some better way to define the Model classes without having to choose where you want to store your database first?
If you need to change database driver at the runtime, then Proxy is a way to go
# database.py
import peewee as pw
proxy = pw.Proxy()
class BaseModel(pw.Model):
class Meta:
database = proxy
class User(BaseModel):
name = pw.CharField()
def add_user(name):
with proxy.atomic() as txn:
User.create(name=name).save()
def get_user(name):
with proxy.atomic() as txn:
return User.get(User.name == name)
From now on each time you load the module, it won't need a database to be initialized. Instead, you can initialize it at the runtime and switch between multiple as follows
# main.py
import peewee as pw
import database as db
sqlite_1 = pw.SqliteDatabase('sqlite_1.db')
sqlite_2 = pw.PostgresqlDatabase('sqlite_2.db')
db.proxy.initialize(sqlite_1)
sqlite_1.create_tables([db.User], safe=True)
db.add_user(name="Tom")
db.proxy.initialize(sqlite_2)
sqlite_2.create_tables([db.User], safe=True)
db.add_user(name="Jerry")
But if the connection is the only thing that matters, then init() method will be enough.
Now I want to be able to choose the database file at runtime. So I
need a way to define the Model classes without defining
SqliteDatabase('somefile') before. I tried to encapsulate everything
in a new Database class, which I can later import and create an
instance from
Peewee uses the meta class to define the name of the table (Model.Meta.db_table) and database( Model.Meta.database)
set these attribute before calling a Model specific code ( either to create a table or to DML statements)
'Allow to define database dynamically'
Question: I cannot use the #db.atomic() decorator since self is not known at class level
Do it, as you do it with self.User.
I wonder about atomic() instead of atomic, but you tell is working fine.
class Database:
def __init__(self, dbfile):
self.db = SqliteDatabase(dbfile)
...
#self.db.atomic()
def __add_user(self, name):
self.User.create(name=name).save()
self.add_user = __add_user
#self.db.atomic()
def __get_user(self, name):
return self.User.get(self.User.name == name)
self.get_user = __get_user
Related: Define models separately from Database() initialization
The question really is how to update a SQLAlchemy declarative model so that it runs the validators. In my case using setters like User.name = name is not really an option.
Below is a runnable example of what I mean
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy import Column, String, Integer
from sqlalchemy.orm import validates
from sqlalchemy.ext.declarative import declarative_base
some_engine = create_engine('sqlite://')
Session = sessionmaker(bind=some_engine)
session = Session()
Base = declarative_base()
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String, nullable=False)
#validates('name')
def validate_name(self, key, value):
if value != 'asd':
raise ValueError('not asd')
return value
Base.metadata.create_all(bind=some_engine)
user = User(id=1, name='qwe')
# >>> ValueError: not asd
user = User(id=1, name='asd')
session.add(user)
session.commit()
session.query(User).filter(User.id=1).update({'name': 'qwe'})
session.query(User).filter(User.id==1)[0].name
# >>> 'qwe'
You could add a mixin to your models that provides a rather simple update method that just uses setattr() to set attributes of an instance.
class UpdateMixin:
"""
Add a simple update() method to instances that accepts
a dictionary of updates.
"""
def update(self, values):
for k, v in values.items():
setattr(self, k, v)
User class would then be defined as
class User(UpdateMixin, Base):
...
And to update a single instance from a given dictionary you could for example run
session.query(User).get(1).update({ 'name': 'qwe' })
# or since you have the user instance from before
user.update({ 'name': 'qwe' })
Note the use of Query.get(). If there is no user with the given id, it will return None and trying to call the method update on it will raise. Another caveat is that if you do not rollback if any exceptions are raised, you cannot predict what, if any, updates took place (were added to the session) because a dictionary has no ordering. So always rollback on any errors.
I'd also recommend actually naming the method updateSelf or some such to prevent risk of confusing it with Query.update().
The short answer is not to use query.update when you want model level constraints. It's exactly for the times when performance is more important than enforcing those sorts of model level constraints. Other answers have provided specifics on solutions, but the fundamental answer is that Query.update is not intended to enforce python-level constraints.
General categories of solutions are:
Use some session-level method and Query.get or a loop on Query.filter.all
Check constraints
Triggers and stored procedures
Rather than creating mixin classes that models inherit from, I have a use case that requires me to configure classes the other way around. The classes that would normally be mixin classes need to be the classes that inherit from the models as well as the class that model objects are created from. This is because the models and the mapper configurations are in an external library from the main repository. I need to pass in the host for the engine from the main repository to the models library before any of the models are loaded so they can load with the declarative base already configured. After the engine information is passed in, the session, Base class, and everything is created within a sort of base class that the models inherit from. Here is a simplified example:
class SQLAlchemyBase(object):
metadata = None
Session = None
Base = object
sessionfactory = sessionmaker()
def initialize(self, host):
engine = create_engine(host)
self.metadata = MetaData(bind=engine)
self.Session = scoped_session(self.sessionfactory)
self.Base = declarative_base(metadata=self.metadata)
models = SQLAlchemyBase()
(The models inherit from models.Base)
So the SQLAlchemyBase will be imported into the main repository, the initialize method will be called, passing in the host for the engine, and the models can then be imported. The main repository has its own classes with the same names as the models and have additional methods that a normal mixin class would have to extend functionality. However, I am unable to create model objects using the classes in the main repository because I can't get the mappers to play nice with this unusual inheritance that extends from the external models library. Additionally, in the models library, there are models that have multiple levels of inherited polymorphic relationships. Here is an example that is similar one of the more basic inherited polymorphic relationships:
Models Library
class Foo(models.Base):
__tablename__ = "foo"
id = Column(Integer, primary_key=True)
type = Column(String)
foo_bar_id = Column(Integer, ForeignKey("foo_bar.id"))
foo_bar = relationship(Foo, backref=backref("foos"))
__mapper_args__ = {"polymorphic_on": type}
class Bar(Foo):
__mapper_args__ = {"polymorphic_identity": "bar"}
class FooBar(models.Base):
__tablename__ = "foo_bar"
id = Column(Integer, primary_key=True)
Main Repository
from separate_library.models import models, Foo as BaseFoo, Bar as BaseBar, FooBar as BaseFooBar
class Foo(BaseFoo):
#classmethod
def custom_create_method(cls, **kw):
foo_obj = cls(**kw)
models.session.add(foo_obj)
models.session.flush()
class Bar(BaseBar):
pass
class FooBar(BaseFooBar):
pass
The original error I was getting was something like this:
InvalidRequestError: One or more mappers failed to initialize - can't proceed with initialization of other mappers.
Original exception was: Multiple classes found for path Foo in the registry of this declarative base. Please use a fully module-qualified path.
So I tried putting the full path in the relationships. Then it started giving me an error like this:
FlushError: Attempting to flush an item of type as a member of collection FooBar.foos. Expected an object of type or a polymorphic subclass of this type. If is a subclass of , configure mapper Mapper|Foo|foo to load this subtype polymorphically, or set enable_typechecks=False to allow any subtype to be accepted for flush.
Essentially, the main problem is getting the classes in the main module to point to and act like the model classes. For example, when I try to create relationships, it says it expected an object of type separate_library.models.Foo instead of main_module.models.Foo. Additionally, in the polymorphic relationships, I can't get the polymorphic_identity to populate for the polymorphic_on column. For example, Bar in the main repository will have the type column empty when the object is initially created.
One idea I tried was to add a metaclass to the declarative base in the models library and modify the mappers in the __init__ method during their initialization. I made progress this way, but haven't gotten it to work completely.
Sorry for the complex explanation, but this is a complex problem. I am not able to change anything about the models or the use case, unfortunately. I have to work within these constraints. If anyone can offer ideas on how to configure the mappers for the classes in the main repository to act like the models in the model library, I would be very grateful.
There are three problems here:
When you write foo_bar = relationship(FooBar, backref=backref("foos")) the FooBar needs to refer to the subclass FooBar, not the BaseFooBar.
Bar needs to inherit from Foo for the inheritance mechanism to work; it cannot inherit from BaseFoo.
Your base classes should not have mappers attached to them; otherwise the inheritance mechanism gets out of whack.
The solutions to these problems, in order:
Use a string to refer to the class name. This confines the consumer to name their classes a certain way. Let's accept this restriction for now.
We can use a metaclass to dynamically change the base class. The metaclass needs to derive from the metaclass of Base because SQLAlchemy's declarative extension makes liberal use of metaclasses. We'll see that the metaclass approach can also solve problem 1 in a flexible way.
Use __abstract__ = True.
Simplest possible example:
from sqlalchemy import *
from sqlalchemy.ext.declarative import declarative_base, declared_attr, DeclarativeMeta
class BaseMeta(DeclarativeMeta):
def __new__(cls, name, bases, attrs):
if not attrs.get("__abstract__"):
if len(bases) != 1:
# you'll need to have multiple inheritance if you have that
# as well
raise NotImplementedError()
base, = bases
extra_bases = tuple(b._impl for b in base.__bases__
if hasattr(b, "_impl"))
bases += extra_bases
self = super(BaseMeta, cls).__new__(cls, name, bases, attrs)
if getattr(base, "__abstract__", False):
base._impl = self
return self
else:
return super(BaseMeta, cls).__new__(cls, name, bases, attrs)
Base = declarative_base(metaclass=BaseMeta)
class BaseFoo(Base):
__abstract__ = True
__tablename__ = "foo"
id = Column(Integer, primary_key=True)
type = Column(String)
#declared_attr
def foo_bar_id(cls):
return Column(Integer, ForeignKey("foo_bar.id"))
#declared_attr
def foo_bar(cls):
return relationship(lambda: BaseFooBar._impl, backref=backref("foos"))
__mapper_args__ = {"polymorphic_on": type}
class BaseBar(BaseFoo):
__abstract__ = True
__mapper_args__ = {"polymorphic_identity": "bar"}
class BaseFooBar(Base):
__abstract__ = True
__tablename__ = "foo_bar"
id = Column(Integer, primary_key=True)
class Foo(BaseFoo):
#classmethod
def custom_create_method(cls, **kw):
foo_obj = cls(**kw)
models.session.add(foo_obj)
models.session.flush()
class Bar(BaseBar):
pass
class FooBar(BaseFooBar):
pass
print(Bar.__bases__) # (<class '__main__.BaseBar'>, <class '__main__.Foo'>)
The basic idea of the metaclass is to inject the class Foo into the bases of Bar, based on the fact that BaseBar inherits from BaseFoo, and the fact that Foo implements BaseFoo (by inheriting from it).
You can add more complicated stuff on top, such as multiple inheritance support or graceful error handling (e.g. warning the user that he's missing a subclass for each base class that you have or he's provided multiple subclasses for the same base class).
I'm looking to hook into the model creation cycle for sqlalchemy models. For example on create or on save (like in the Ruby ORM ActiveRecord, in fact I'm moving a model from ActiveRecord to SqlAlchemy).
Events looks like what I need: http://docs.sqlalchemy.org/en/rel_0_7/core/event.html, but I haven't found more detailed examples, yet. I'd like to hear someone's experience with this.
Are there similar facilities in sqlalchemy for doing things with a model/instance based on certain cues, e.g. after_create?
Events are pretty simple once you get the hang of it.
Here is a quick example using events
import uuid
from sqlalchemy.event import listen
from mypackage.models import Base
def generate_license(mapper, connect, target):
target.generate_license()
class User(Base):
__tablename__ = "users"
id = Column(String(36))
license = Column(String(256))
def generate_license(self):
if not self.license:
self.license = str(uuid.uuid4())
return self.license
listen(User, 'before_insert', generate_license)
Alternately, you can use decorators:
from sqlalchemy.event import listens_for
…
class User(Base):
…
#listens_for(User, 'before_insert')
def generate_license(mapper, connect, self):
…
from sqlalchemy.event import listen_for
…
class User(Base):
…
#listen_for(User, 'before_insert')
#staticmethod
def generate_license(mapper, connect, self):
…
This will return
NameError: name 'User' is not defined