In order to handle a growing database table, we are sharding on table name. So we could have database tables that are named like this:
table_md5one
table_md5two
table_md5three
All tables have the exact same schema.
How do we use SQLAlchemy and dynamically specify the tablename for the class that corresponds to this? Looks like the declarative_base() classes need to have tablename pre-specified.
There will eventually be too many tables to manually specify derived classes from a parent/base class. We want to be able to build a class that can have the tablename set up dynamically (maybe passed as a parameter to a function.)
OK, we went with the custom SQLAlchemy declaration rather than the declarative one.
So we create a dynamic table object like this:
from sqlalchemy import MetaData, Table, Column
def get_table_object(self, md5hash):
metadata = MetaData()
table_name = 'table_' + md5hash
table_object = Table(table_name, metadata,
Column('Column1', DATE, nullable=False),
Column('Column2', DATE, nullable=False)
)
clear_mappers()
mapper(ActualTableObject, table_object)
return ActualTableObject
Where ActualTableObject is the class mapping to the table.
In Augmenting the Base you find a way of using a custom Base class that can, for example, calculate the __tablename__ attribure dynamically:
class Base(object):
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
The only problem here is that I don't know where your hash comes from, but this should give a good starting point.
If you require this algorithm not for all your tables but only for one you could just use the declared_attr on the table you are interested in sharding.
Because I insist to use declarative classes with their __tablename__ dynamically specified by given parameter, after days of failing with other solutions and hours of studying SQLAlchemy internals, I come up with the following solution that I believe is simple, elegant and race-condition free.
def get_model(suffix):
DynamicBase = declarative_base(class_registry=dict())
class MyModel(DynamicBase):
__tablename__ = 'table_{suffix}'.format(suffix=suffix)
id = Column(Integer, primary_key=True)
name = Column(String)
...
return MyModel
Since they have their own class_registry, you will not get that warning saying:
This declarative base already contains a class with the same class name and module name as mypackage.models.MyModel, and will be replaced in the string-lookup table.
Hence, you will not be able to reference them from other models with string lookup. However, it works perfectly fine to use these on-the-fly declared models for foreign keys as well:
ParentModel1 = get_model(123)
ParentModel2 = get_model(456)
class MyChildModel(BaseModel):
__tablename__ = 'table_child'
id = Column(Integer, primary_key=True)
name = Column(String)
parent_1_id = Column(Integer, ForeignKey(ParentModel1.id))
parent_2_id = Column(Integer, ForeignKey(ParentModel2.id))
parent_1 = relationship(ParentModel1)
parent_2 = relationship(ParentModel2)
If you only use them to query/insert/update/delete without any reference left such as foreign key reference from another table, they, their base classes and also their class_registry will be garbage collected, so no trace will be left.
you can write a function with tablename parameter and send back the class with setting appropriate attributes.
def get_class(table_name):
class GenericTable(Base):
__tablename__ = table_name
ID= Column(types.Integer, primary_key=True)
def funcation(self):
......
return GenericTable
Then you can create a table using:
get_class("test").__table__.create(bind=engine) # See sqlachemy.engine
Try this
import zlib
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, BigInteger, DateTime, String
from datetime import datetime
BASE = declarative_base()
ENTITY_CLASS_DICT = {}
class AbsShardingClass(BASE):
__abstract__ = True
def get_class_name_and_table_name(hashid):
return 'ShardingClass%s' % hashid, 'sharding_class_%s' % hashid
def get_sharding_entity_class(hashid):
"""
#param hashid: hashid
#type hashid: int
#rtype AbsClientUserAuth
"""
if hashid not in ENTITY_CLASS_DICT:
class_name, table_name = get_class_name_and_table_name(hashid)
cls = type(class_name, (AbsShardingClass,),
{'__tablename__': table_name})
ENTITY_CLASS_DICT[hashid] = cls
return ENTITY_CLASS_DICT[hashid]
cls = get_sharding_entity_class(1)
print session.query(cls).get(100)
Instead of using imperative creating Table object, you can use usual declarative_base and make a closure to set a table name as the following:
def make_class(Base, table_name):
class User(Base):
__tablename__ = table_name
id = Column(Integer, primary_key=True)
name= Column(String)
return User
Base = declarative_base()
engine = make_engine()
custom_named_usertable = make_class(Base, 'custom_name')
Base.metadata.create_all(engine)
session = make_session(engine)
new_user = custom_named_usertable(name='Adam')
session.add(new_user)
session.commit()
session.close()
engine.dispose()
just you need to create class object for Base.
from sqlalchemy.ext.declarative import declarative_base, declared_attr
class Base(object):
#declared_attr
def __tablename__(cls):
return cls.__name.lower()
Base = declarative_base(cls=Base)
Related
We have 1 table with a large amount of data and DBA's partitioned it based on a particular parameter. This means I ended up with Employee_TX, Employee_NY kind of table names. Earlier the models.py was simple as in --
class Employee(Base):
__tablename__ = 'Employee'
name = Column...
state = Column...
Now, I don't want to create 50 new classes for the newly partitioned tables as anyways my columns are the same.
Is there a pattern where I can create a single class and then use it in query dynamically? session.query(<Tablename>).filter().all()
Maybe some kind of Factory pattern or something is what I'm looking for.
So far I've tried by running a loop as
for state in ['CA', 'TX', 'NY']:
class Employee(Base):
__qualname__ = __tablename__ = 'Employee_{}'.format(state)
name = Column...
state = Column...
but this doesn't work and I get a warning as - SAWarning: This declarative base already contains a class with the same class name and module name as app_models.employee, and will be replaced in the string-lookup table.
Also it can't find the generated class when I do from app_models import Employee_TX
This is a flask app with PostgreSQL as a backend and sqlalchemy is used as an ORM
Got it by creating a custom function like -
def get_model(state):
DynamicBase = declarative_base(class_registry=dict())
class MyModel(DynamicBase):
__tablename__ = 'Employee_{}'.format(state)
name = Column...
state = Column...
return MyModel
And then from my services.py, I just call with get_model(TX)
Whenever you think of dynamically constructing classes think of type() with 3 arguments (see this answer for a demonstration, and the docs more generally).
In your case, it's just a matter of constructing the classes and keeping a reference to them so you can access them again later.
Here's an example:
from sqlalchemy import Column, Integer, String
from sqlalchemy.engine import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
Base = declarative_base()
# this produces the set of common attributes that each class should have
def attribute_factory():
return dict(
id=Column(Integer, primary_key=True),
name=Column(String, nullable=False),
state=Column(String, nullable=False),
CLASS_VAR=12345678,
)
states = ["CA", "TX", "NY"]
# here we map the state abbreviation to the generated model, notice the templated
# class and table names
model_map = {
state: type(
f"Employee_{state}",
(Base,),
dict(**attribute_factory(), __tablename__=f"Employee_{state}"),
)
for state in states
}
engine = create_engine("sqlite:///", echo=True)
Session = sessionmaker(bind=engine)
Base.metadata.create_all(engine)
if __name__ == "__main__":
# inserts work
s = Session()
for state, model in model_map.items():
s.add(model(name="something", state=state))
s.commit()
s.close()
# queries work
s = Session()
for state, model in model_map.items():
inst = s.query(model).first()
print(inst.state, inst.CLASS_VAR)
I'm really new to SQLAlchemy, and I love it. Right now I'm doing a lot of things manually and I want to do things more 'pythonically' and dynamic.
So as an example, I have this short script that manually creates/defines a table and then a function that inserts data into that table.
database connection
import os
from sqlalchemy import *
from sqlalchemy import schema, types
from sqlalchemy.ext.declarative import declarative_base
db_url = os.environ.get('DATABASE_URL')
engine = create_engine(db_url)
Base = declarative_base(engine)
meta = Base.metadata
table definition
file_paths = Table('file_paths', meta,
Column('table_id', Integer, primary_key = True),
Column('fullpath', String(255)),
Column('filename', String(255)),
Column('extension', String(255)),
Column('created', String(255)),
Column('modified', String(255)),
Column('size', Integer),
Column('owner', String(255)),
Column('permissions', Integer),
mysql_engine='InnoDB',
)
file_paths.drop(engine, checkfirst = False)
file_paths.create(engine, checkfirst = True)
insert function takes a string and a list as arguments
def push_to_db(fullpath, fileInfo):
i = file_paths.insert()
i.execute( fullpath = str(fullpath),
filename = str(fileInfo[0]),
extension = str(fileInfo[1]),
created = str(fileInfo[2]),
modified = str(fileInfo[3]),
size = str(fileInfo[4]),
owner = str(fileInfo[5]),
permissions = str(fileInfo[6]),
)
This works but it's ugly and taken right out of a tutorial I found somewhere online. My goal is to make these operations dynamic.
example class
class FileMeta(object):
def __init__(self, fullPathFileName, filename):
self.fullPathFileName = fullPathFileName
self.filename = filename
self.extension = os.path.splitext(self.filename)[1].lower()
...
def fileMetaList(self):
return [self.filename, self.extension, self.created, self.modified,\
self.size, self.owner, self.permissions]
So here's the scenario: given a class object
define the table dynamically according to the class member variables
column numbers and names should correspond to variable names
or correspond to the index of that variable in a list of class variables
write a function that can insert data from the class into the corresponding dynamically created table
My intuition tells me this is what SQLAlchemy would be good for. Can someone tell me a good tutorial or reference that can outline this process?
You want to use the declarative extension instead:
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class FilePaths(Base):
__tablename__ = 'file_paths'
__table_args__ = {'mysql_engine':'InnoDB'}
table_id = Column(Integer, primary_key=True)
fullpath = Column(String(255))
filename = Column(String(255))
extension = Column(String(255))
created = Column(String(255))
modified = Column(String(255))
size = Column(Integer)
owner = Column(String(255))
permissions = Column(Integer)
Base.metadata.create_all(engine)
You can define your own __init__() as needed, as well other methods, then create instances of these to insert new rows.
See the SQLAlchemy's own ORM tutorial.
Add Automap extension :
from sqlalchemy.ext.automap import automap_base
# SQLAlchemy
engine = create_engine(DATABASE_URL)
metadata = MetaData()
Base = automap_base()
Base.prepare(engine, reflect=True)
In declarative approach, I want to exclude one property, its working properly when my column name and property name are same. But if I give different name then its not working.
Here is my sample code.
Base = declarative_base()
class tblUser(Base):
__tablename__ = 'tblUser'
User_Id = Column('User_Id', String(100), primary_key=True)
SequenceNo = Column('Sequence_No', Integer)
FullName = Column('FullName', String(50))
__mapper_args__ = {'exclude_properties' :['Sequence_No']}
user = tblUser()
user.User_Id = '1000001'
user.SequenceNo = 101
session.add(user)
session.commit()
In the above sample I don't want the SequenceNo property to be updated in database even if I assign some value to it. So I used exclude_properties but still its updating the value in db. But if I change the property name from SequenceNo to Sequence_No (same as the column name) then its working as per the behaviour. Can anyone help me?
Thanks
Adhi
Unfortunately, __mapper_args__ is probably the wrong approach. It is intended to control the reflection of an existing database table into a mapper, not make a column 'read-only'.
I think a better approach would be to use a hybrid property:
from sqlalchemy.ext.hybrid import hybrid_property
Base = declarative_base()
class tblUser(Base):
__tablename__ = 'tblUser'
User_Id = Column('User_Id', String(100), primary_key=True)
FullName = Column('FullName', String(50))
_Sequence_No = Column('Sequence_No', Integer)
_local_Sequence_No = None
#hybrid_property
#property
def SequenceNo(self):
if self._local_Sequence_No is not None:
return self._local_Sequence_No
return self._SequenceNo
#SequenceNo.setter
def SequenceNo(self, value):
self._local_Sequence_No = value
The original Sequence_No column is available via a private attribute, and the SequenceNo property intercepts writes and stores them on the instance to be re-used later, but not written to the database.
With SQLAlchemy, it is possible to add a default value to every function. As I understand it, this may also be a callable (either without any arguments or with an optional ExecutionContext argument).
Now in a declarative scenario, I wonder if it is somehow possible to have a default function which is called with the object that is being stored. I.e. possibly like so:
Base = sqlalchemy.ext.declarative.declarative_base()
class BaseEntity(Base):
value = Column('value', String(40), default=BaseEntity.gen_default)
def gen_default(self):
# do something with self, for example
# generate a default value using some other data
# attached to the object
return self.default_value
Is something like this possible? Or do I have to somehow set up an before-insertion hook for this (how?)?
before_insert is documented here:
http://docs.sqlalchemy.org/en/rel_0_7/orm/events.html#sqlalchemy.orm.events.MapperEvents.before_insert
examples here:
http://docs.sqlalchemy.org/en/rel_0_7/orm/events.html#mapper-events
i.e.
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import event
Base= declarative_base()
class A(Base):
__tablename__ = "a"
id = Column(Integer, primary_key=True)
data = Column(String)
otherdata = Column(String)
#event.listens_for(A, "before_insert")
def gen_default(mapper, connection, instance):
instance.data = "Some default %s" % instance.otherdata
e = create_engine("sqlite://")
Base.metadata.create_all(e)
a = A(otherdata="some other data")
s = Session(e)
s.add(a)
s.commit()
assert a.data == "Some default some other data"
I am working on a quite large code base that has been implemented using sqlalchemy.ext.declarative, and I need to add a dict-like property to one of the classes. What I need is the same as in this question, but in a declarative fashion. Can anyone with more knowledge in SQLAlchemy give me an example?
Thanks in advance...
Declarative is just another way of defining things. Virtually you end up with the exact same environment than if you used separated mapping.
Since I answered the other question, I'll try this one as well. Hope it gives more upvotes ;)
Well, first we define the classes
from sqlalchemy import Column, Integer, String, Table, create_engine
from sqlalchemy import orm, MetaData, Column, ForeignKey
from sqlalchemy.orm import relation, mapper, sessionmaker
from sqlalchemy.orm.collections import column_mapped_collection
from sqlalchemy.ext.associationproxy import association_proxy
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('sqlite:///:memory:', echo=True)
Base = declarative_base(bind=engine)
class Note(Base):
__tablename__ = 'notes'
id_item = Column(Integer, ForeignKey('items.id'), primary_key=True)
name = Column(String(20), primary_key=True)
value = Column(String(100))
def __init__(self, name, value):
self.name = name
self.value = value
class Item(Base):
__tablename__ = 'items'
id = Column(Integer, primary_key=True)
name = Column(String(20))
description = Column(String(100))
_notesdict = relation(Note,
collection_class=column_mapped_collection(Note.name))
notes = association_proxy('_notesdict', 'value', creator=Note)
def __init__(self, name, description=''):
self.name = name
self.description = description
Base.metadata.create_all()
Now let's make a test:
Session = sessionmaker(bind=engine)
s = Session()
i = Item('ball', 'A round full ball')
i.notes['color'] = 'orange'
i.notes['size'] = 'big'
i.notes['data'] = 'none'
s.add(i)
s.commit()
print i.notes
I get:
{u'color': u'orange', u'data': u'none', u'size': u'big'}
Now let's check the notes table...
for note in s.query(Note):
print note.id_item, note.name, note.value
I get:
1 color orange
1 data none
1 size big
It works!! :D