Develop Reference

How to meaningfully use additional properties of an assosiation prox object?

Consider a many-to-many relationship of Things where the order of the children (or parents) matters. So the association object ("Link") has an additional property "position". When I fill the structure, I obviously can't use the "association-jumping" append() method because I need to explicitly access the Link object to set the "position" attribute. Sad but understandable.
What I'm struggling with now is to retrieve the children of a Thing in the order given by the Link's position property. Is that in any way possible? If not, the association_proxy is not of any use to me in either populating nor retrieving the data which makes me wonder what it is good for at all. I've tried to wrap my head around the use cases in the documentation but they don't seem to apply for my situation.
Here's a self-contained example:
from sqlalchemy import create_engine, CHAR, Column, Integer,\
String, ForeignKey
from sqlalchemy.ext.associationproxy import association_proxy
from sqlalchemy.orm import relationship, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Thing(Base):
__tablename__ = 'thing'
id = Column(Integer, primary_key=True)
name = Column(String(255))
up = association_proxy('coll_up', 'up')
down = association_proxy('coll_down', 'down')
class Link(Base):
__tablename__ = 'link'
id_up = Column(Integer, ForeignKey('thing.id'),
primary_key=True)
id_down = Column(Integer, ForeignKey('thing.id'),
primary_key=True)
position = Column(Integer)
up = relationship('Thing', primaryjoin=id_up == Thing.id,
backref='coll_down')
down = relationship('Thing', primaryjoin=id_down == Thing.id,
backref='coll_up')
if __name__ == '__main__':
# I know that for convenient append()ing stuff I'd need an __init__()
# method, but see comments below
engine = create_engine('sqlite://')
Base.metadata.create_all(engine)
Session = sessionmaker(engine)
if __name__ == '__main__':
db = Session()
root = Thing(name='Root thing')
db.add(root)
# inserting "reversed" to have something to sort
for i in reversed(range(5)):
t = Thing(name='Thing %d' % i)
# If Link had an __init__ method I could use root.append(t), but...
db.add(t)
# ...in order to set position, I need to explicitly use the association
# object anyway
l = Link(up=root, down=t, position=i)
db.add(l)
db.commit()
# OK, so far the association_proxy construct hasn't been of any use. Let's
# see about retrieving the data...
root = db.query(Thing).filter_by(name='Root thing').one()
for thing in root.down:
print(thing.name)
# This, as expected. prints the elements in the order in which they were
# inserted (which is "reversed"). How can I sort them by the "position"
# property of the "Link" element, or is the associaton object useless in
# this scenario?

I'd suggest investigating the order_by parameter of relationships. You can can order the ORM's organization of related object by the child's properties. (Ordering will not work without session.commit()).
Assuming you want to order by Link.position:
class Link(Base):
__tablename__ = 'link'
id_up = Column(Integer, ForeignKey('thing.id'), primary_key=True)
id_down = Column(Integer, ForeignKey('thing.id'), primary_key=True)
position = Column(Integer)
# Note the syntax for order_by -- the ORM model is referred to as a string
up = relationship(
'Thing', primaryjoin=id_up == Thing.id,
order_by='Link.position',
backref='coll_down'
)
down = relationship(
'Thing',
primaryjoin=id_down == Thing.id,
order_by='Link.position',
backref='coll_up'
)
If you need more rigid ordering, meaning, ordering before the child is committed to the session, you can define an orderinglist on the relationship which "intercept" and reorganize relationships in the ORM. (Ordering will work without session.commit()).
from sqlalchemy.ext.orderinglist import ordering_list
class Link(Base):
...
up = relationship(
'Thing',
primaryjoin=id_up == Thing.id,
order_by='Link.position',
collection_class=ordering_list('position')
backref='coll_down'
)
...
As a next step, if you're looking to organize your position based on ids (perhaps position = id_up - id_down), I'd suggest looking into sqlalchemy's event API. An elegant approach would be trigger recalculation of relevant attributes based on a relevant update. (Ex: id_up is increased by 1, increase position by 1).

Creating dynamic classes in SQLAlchemy

We have 1 table with a large amount of data and DBA's partitioned it based on a particular parameter. This means I ended up with Employee_TX, Employee_NY kind of table names. Earlier the models.py was simple as in --
class Employee(Base):
__tablename__ = 'Employee'
name = Column...
state = Column...
Now, I don't want to create 50 new classes for the newly partitioned tables as anyways my columns are the same.
Is there a pattern where I can create a single class and then use it in query dynamically? session.query(<Tablename>).filter().all()
Maybe some kind of Factory pattern or something is what I'm looking for.
So far I've tried by running a loop as
for state in ['CA', 'TX', 'NY']:
class Employee(Base):
__qualname__ = __tablename__ = 'Employee_{}'.format(state)
name = Column...
state = Column...
but this doesn't work and I get a warning as - SAWarning: This declarative base already contains a class with the same class name and module name as app_models.employee, and will be replaced in the string-lookup table.
Also it can't find the generated class when I do from app_models import Employee_TX
This is a flask app with PostgreSQL as a backend and sqlalchemy is used as an ORM

Got it by creating a custom function like -
def get_model(state):
DynamicBase = declarative_base(class_registry=dict())
class MyModel(DynamicBase):
__tablename__ = 'Employee_{}'.format(state)
name = Column...
state = Column...
return MyModel
And then from my services.py, I just call with get_model(TX)

Whenever you think of dynamically constructing classes think of type() with 3 arguments (see this answer for a demonstration, and the docs more generally).
In your case, it's just a matter of constructing the classes and keeping a reference to them so you can access them again later.
Here's an example:
from sqlalchemy import Column, Integer, String
from sqlalchemy.engine import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
Base = declarative_base()
# this produces the set of common attributes that each class should have
def attribute_factory():
return dict(
id=Column(Integer, primary_key=True),
name=Column(String, nullable=False),
state=Column(String, nullable=False),
CLASS_VAR=12345678,
)
states = ["CA", "TX", "NY"]
# here we map the state abbreviation to the generated model, notice the templated
# class and table names
model_map = {
state: type(
f"Employee_{state}",
(Base,),
dict(**attribute_factory(), __tablename__=f"Employee_{state}"),
)
for state in states
}
engine = create_engine("sqlite:///", echo=True)
Session = sessionmaker(bind=engine)
Base.metadata.create_all(engine)
if __name__ == "__main__":
# inserts work
s = Session()
for state, model in model_map.items():
s.add(model(name="something", state=state))
s.commit()
s.close()
# queries work
s = Session()
for state, model in model_map.items():
inst = s.query(model).first()
print(inst.state, inst.CLASS_VAR)

Dynamically setting __tablename__ for sharding in SQLAlchemy?

In order to handle a growing database table, we are sharding on table name. So we could have database tables that are named like this:
table_md5one
table_md5two
table_md5three
All tables have the exact same schema.
How do we use SQLAlchemy and dynamically specify the tablename for the class that corresponds to this? Looks like the declarative_base() classes need to have tablename pre-specified.
There will eventually be too many tables to manually specify derived classes from a parent/base class. We want to be able to build a class that can have the tablename set up dynamically (maybe passed as a parameter to a function.)

OK, we went with the custom SQLAlchemy declaration rather than the declarative one.
So we create a dynamic table object like this:
from sqlalchemy import MetaData, Table, Column
def get_table_object(self, md5hash):
metadata = MetaData()
table_name = 'table_' + md5hash
table_object = Table(table_name, metadata,
Column('Column1', DATE, nullable=False),
Column('Column2', DATE, nullable=False)
)
clear_mappers()
mapper(ActualTableObject, table_object)
return ActualTableObject
Where ActualTableObject is the class mapping to the table.

In Augmenting the Base you find a way of using a custom Base class that can, for example, calculate the __tablename__ attribure dynamically:
class Base(object):
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
The only problem here is that I don't know where your hash comes from, but this should give a good starting point.
If you require this algorithm not for all your tables but only for one you could just use the declared_attr on the table you are interested in sharding.

Because I insist to use declarative classes with their __tablename__ dynamically specified by given parameter, after days of failing with other solutions and hours of studying SQLAlchemy internals, I come up with the following solution that I believe is simple, elegant and race-condition free.
def get_model(suffix):
DynamicBase = declarative_base(class_registry=dict())
class MyModel(DynamicBase):
__tablename__ = 'table_{suffix}'.format(suffix=suffix)
id = Column(Integer, primary_key=True)
name = Column(String)
...
return MyModel
Since they have their own class_registry, you will not get that warning saying:
This declarative base already contains a class with the same class name and module name as mypackage.models.MyModel, and will be replaced in the string-lookup table.
Hence, you will not be able to reference them from other models with string lookup. However, it works perfectly fine to use these on-the-fly declared models for foreign keys as well:
ParentModel1 = get_model(123)
ParentModel2 = get_model(456)
class MyChildModel(BaseModel):
__tablename__ = 'table_child'
id = Column(Integer, primary_key=True)
name = Column(String)
parent_1_id = Column(Integer, ForeignKey(ParentModel1.id))
parent_2_id = Column(Integer, ForeignKey(ParentModel2.id))
parent_1 = relationship(ParentModel1)
parent_2 = relationship(ParentModel2)
If you only use them to query/insert/update/delete without any reference left such as foreign key reference from another table, they, their base classes and also their class_registry will be garbage collected, so no trace will be left.

you can write a function with tablename parameter and send back the class with setting appropriate attributes.
def get_class(table_name):
class GenericTable(Base):
__tablename__ = table_name
ID= Column(types.Integer, primary_key=True)
def funcation(self):
......
return GenericTable
Then you can create a table using:
get_class("test").__table__.create(bind=engine) # See sqlachemy.engine

Try this
import zlib
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, BigInteger, DateTime, String
from datetime import datetime
BASE = declarative_base()
ENTITY_CLASS_DICT = {}
class AbsShardingClass(BASE):
__abstract__ = True
def get_class_name_and_table_name(hashid):
return 'ShardingClass%s' % hashid, 'sharding_class_%s' % hashid
def get_sharding_entity_class(hashid):
"""
#param hashid: hashid
#type hashid: int
#rtype AbsClientUserAuth
"""
if hashid not in ENTITY_CLASS_DICT:
class_name, table_name = get_class_name_and_table_name(hashid)
cls = type(class_name, (AbsShardingClass,),
{'__tablename__': table_name})
ENTITY_CLASS_DICT[hashid] = cls
return ENTITY_CLASS_DICT[hashid]
cls = get_sharding_entity_class(1)
print session.query(cls).get(100)

Instead of using imperative creating Table object, you can use usual declarative_base and make a closure to set a table name as the following:
def make_class(Base, table_name):
class User(Base):
__tablename__ = table_name
id = Column(Integer, primary_key=True)
name= Column(String)
return User
Base = declarative_base()
engine = make_engine()
custom_named_usertable = make_class(Base, 'custom_name')
Base.metadata.create_all(engine)
session = make_session(engine)
new_user = custom_named_usertable(name='Adam')
session.add(new_user)
session.commit()
session.close()
engine.dispose()

just you need to create class object for Base.
from sqlalchemy.ext.declarative import declarative_base, declared_attr
class Base(object):
#declared_attr
def __tablename__(cls):
return cls.__name.lower()
Base = declarative_base(cls=Base)

Automatically Build Database Table based on a Class with SQLAlchemy

I'm really new to SQLAlchemy, and I love it. Right now I'm doing a lot of things manually and I want to do things more 'pythonically' and dynamic.
So as an example, I have this short script that manually creates/defines a table and then a function that inserts data into that table.
database connection
import os
from sqlalchemy import *
from sqlalchemy import schema, types
from sqlalchemy.ext.declarative import declarative_base
db_url = os.environ.get('DATABASE_URL')
engine = create_engine(db_url)
Base = declarative_base(engine)
meta = Base.metadata
table definition
file_paths = Table('file_paths', meta,
Column('table_id', Integer, primary_key = True),
Column('fullpath', String(255)),
Column('filename', String(255)),
Column('extension', String(255)),
Column('created', String(255)),
Column('modified', String(255)),
Column('size', Integer),
Column('owner', String(255)),
Column('permissions', Integer),
mysql_engine='InnoDB',
)
file_paths.drop(engine, checkfirst = False)
file_paths.create(engine, checkfirst = True)
insert function takes a string and a list as arguments
def push_to_db(fullpath, fileInfo):
i = file_paths.insert()
i.execute( fullpath = str(fullpath),
filename = str(fileInfo[0]),
extension = str(fileInfo[1]),
created = str(fileInfo[2]),
modified = str(fileInfo[3]),
size = str(fileInfo[4]),
owner = str(fileInfo[5]),
permissions = str(fileInfo[6]),
)
This works but it's ugly and taken right out of a tutorial I found somewhere online. My goal is to make these operations dynamic.
example class
class FileMeta(object):
def __init__(self, fullPathFileName, filename):
self.fullPathFileName = fullPathFileName
self.filename = filename
self.extension = os.path.splitext(self.filename)[1].lower()
...
def fileMetaList(self):
return [self.filename, self.extension, self.created, self.modified,\
self.size, self.owner, self.permissions]
So here's the scenario: given a class object
define the table dynamically according to the class member variables
column numbers and names should correspond to variable names
or correspond to the index of that variable in a list of class variables
write a function that can insert data from the class into the corresponding dynamically created table
My intuition tells me this is what SQLAlchemy would be good for. Can someone tell me a good tutorial or reference that can outline this process?

You want to use the declarative extension instead:
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class FilePaths(Base):
__tablename__ = 'file_paths'
__table_args__ = {'mysql_engine':'InnoDB'}
table_id = Column(Integer, primary_key=True)
fullpath = Column(String(255))
filename = Column(String(255))
extension = Column(String(255))
created = Column(String(255))
modified = Column(String(255))
size = Column(Integer)
owner = Column(String(255))
permissions = Column(Integer)
Base.metadata.create_all(engine)
You can define your own __init__() as needed, as well other methods, then create instances of these to insert new rows.
See the SQLAlchemy's own ORM tutorial.

Add Automap extension :
from sqlalchemy.ext.automap import automap_base
# SQLAlchemy
engine = create_engine(DATABASE_URL)
metadata = MetaData()
Base = automap_base()
Base.prepare(engine, reflect=True)

How to override a column name in sqlalchemy using reflection and descriptive syntax

Hello I'm trying to port a legacy application to python with sqlalchemy.
The application's existing database has about 300 tables and in every table there is a colum named def such as :
create table accnt (
code varchar(20)
, def varchar(50) --for accnt definition
, ...
)
So when with declarative syntax and reflection I can easily create my class as :
class Accnt(Base):
__table__ = Table('accnt', metadata, autoload = True, autoload_with=engine)
But when I try to reach def column I eventually get an error. For example :
q = session.query(Accnt)
for row in q:
print q.def
Because def is a reserved word for python :(
To overcome this issue I can create my class as :
class Accnt(Base):
__table__ = Table('accnt', metadata, autoload = True, autoload_with=engine)
__mapper_args__ = {'column_prefix':'_'}
But putting a _ in front of every column name is boring and not fancy.
What I'd like to do is access def column with another name / ( key ?).
Any ideas?
--- Edit ---
( Editing original post as requested by TokenMacGuy )
While I've accepted TokenMacGuy's answer I've tried it before as :
engine = create_engine('firebird://sysdba:masterkey#127.0.0.1/d:\\prj\\db2\\makki.fdb?charse‌​t=WIN1254', echo=False)
metadata = MetaData()
DbSession = sessionmaker(bind=engine)
Base = declarative_base()
class Accnt(Base):
__table__ = Table('accnt', metadata, autoload = True, autoload_with=engine)
_def = Column("def", String(50))
And I've got
sqlalchemy.exc.ArgumentError: Can't add additional column 'def' when specifying table
error..
The main difference between me and TokenMacGuy is
mine : _table_ ....
TokenMcGuy : __tablename__ = 'accnt'
__table_args__ = {'autoload': True}
and metadata binding...
So, why my previous attemp generated an error ?

You can have your cake and eat it too. Define the columns you want to rename; sqlalchemy will automatically infer any columns you don't mention.
>>> from sqlalchemy import *
>>> from sqlalchemy.ext.declarative import declarative_base
>>>
>>> engine = create_engine("sqlite:///:memory:")
>>>
>>> engine.execute("""
... create table accnt (
... id integer primary key,
... code varchar(20),
... def varchar(50)
... )
... """)
<sqlalchemy.engine.base.ResultProxy object at 0x2122750>
>>>
>>> Base = declarative_base()
>>>
>>> Base.metadata.bind = engine
>>>
>>> class Accnt(Base):
... __tablename__ = 'accnt'
... __table_args__ = {'autoload': True}
... def_ = Column('def', String)
...
>>> Accnt.def_
<sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x2122e90>
>>> Accnt.code
<sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x2127090>
>>>
EDIT:
By supplying a __table__ argument, you're telling the declarative extension that you already have a properly configured Table that you'd like to use. But that's not true; you want to have the def column referred to by another name in the class. By using __tablename__ and __table_args__, you defer the construction of the table until after you've told declarative how you want to use that table. There's no elegant work-around if you are dead set on using __table__. You can provide a property that aliases the column or you may be able to specify the column as _def = getattr(__table__.c, 'def').
Really, you should just use __tablename__; It's both more convenient and more flexible, and this is a great example of why.
(as an aside, it's most conventional to give alternate identifiers a trailing underscore instead of a leading underscore, use def_ instead of _def; leading underscores usually signify that the name is 'private' or 'an implementation detail', if the name is meant to be public, but looks like a private name, it may cause more confusion than is necessary)

You could define your Table this way:
mymetadata = MetaData()
Base = declarative_base(metadata=mymetadata)
class Accnt(Base):
__tablename__ = 'accnt'
code = Column(String(20))
def_ = Column(String(50))

This may be too much brute force, but another approach is to use the sqlacodegen library to auto-generate all the models for your DB, then alter those manually, or tweak sqlacodegen to build the models using your conventions. It has support for mapping reserved words to other symbols.
See https://pypi.python.org/pypi/sqlacodegen. Nice tool.