I have a mssql database that I am reflecting. Is there a way to add a custom Model class to this kind of database?
db = SQLAlchemy(app)
Base = automap_base()
Base.prepare(db.engine, reflect=True)
Users = Base.classes.Users
Thanks for taking the time to help me with my issue!
automap_base takes declarative_base as an argument, or creates one if not provided.
Hence you can create one of your own and provide.
declarative_base does take an argument cls which you can use to achieve exactly what you need.
class MyBase(...):
...
Base = automap_base(declarative_base(cls=MyBase))
I had asked this question How to create instance of a table entry but not added in ponyorm? where I was asking how I can create an instance of the class defined as a ponyorm table representation without immediately adding it. By using sqlalchemy where an explicit add is needed on a session instance I think I succeeded by using the following code.
I first create a class called AddInstance which has an add method, and then inherit from this in all my table definitions. This seems to work (ie I can create an instance of the class and add it only if I want to relatively easily) but I'm not sure if there are any unintended side effects or this is very far from best practice.
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, relationship
from sqlalchemy import Column, String, Integer
engine = create_engine('sqlite:///:memory:')
Base = declarative_base()
Session = sessionmaker(bind=engine)
class AddInstance:
def add(self):
session = Session()
session.add(self)
session.commit()
class Pizza(Base, AddInstance):
__tablename__ = 'pizzas'
id = Column(Integer, primary_key=True)
name = Column(String(50))
toppings = relationship('Topping', back_populates='name')
class Topping(Base, AddInstance):
__tablename__ = 'fruits'
id = Column(Integer, primary_key=True)
name = Column(String(50))
pizzas = relationship('Pizza', back_populates='name')
Base.metadata.create_all(engine)
There will be no side effects, SQLAlchemy explicitly supports adding methods. It doesn't matter if those methods are defined on a mixin class or directly on the class derived from Base.
Quoting from the SQLAlchemy Object Relational Tutorial documentation section:
Outside of what the mapping process does to our class, the class remains otherwise mostly a normal Python class, to which we can define any number of ordinary attributes and methods needed by our application.
(bold emphasis mine).
There are plenty of examples of code bases that do exactly the same. Flask-SQLAlchemy provides a Model base class to add a query attribute (set when the declarative base is created), which lets you do Topping.query(...) directly from a model class, for example.
Your add() method does have downside: it creates a new Session() instance just to add and commit your object. This keeps it outside of the normal session state management semantics and if you wanted to do anything else with your newly created object you'd have to merge it into an existing session.
The normal, best practice for code involvig SQLAlchemy objects is to create a session to manage a transaction, a set of operations that together must succeed or fail. That includes creating objects; in many real-world applications you'd want to avoid creating extra rows in a database when other operations that rely on those rows fail. Your .add() method unconditionally commits each object in a separate transaction. You may want to revisit this pattern.
As a background: I'm creating an ORM based on a schema of an already existing database. - This due to the fact that the python application won't be the "owner" of said database.
Now in this database there is a table called "task" and a table called "task_notBefore__task_relatedTasks" - this latter is a many-to-many relation between different entries in the "task" table.
now automap_base() has an automated detection of these relationships as described here. However this fails for my case, and no relationship is being build.
I then try to manually create the relationship:
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.ext.automap import generate_relationship
from sqlalchemy.orm import sessionmaker, interfaces, relationship
from sqlalchemy import create_engine
class DBConnection:
def __init__(self, connection_url, **kwargs):
self.engine = create_engine(connection_url, **kwargs)
self._Base = automap_base()
self._Base.prepare(self.engine, reflect=True)
self.Task = self._Base.classes.task
self.Order = self._Base.classes.order
self.Poller = self._Base.classes.poller
rel = generate_relationship(self._Base, interfaces.MANYTOMANY, relationship, 'related', self.Task, self.Task,
secondary=self._Base.classes.task_notBefore__task_relatedTasks, backref='notBefore')
self._Session = sessionmaker()
self._Session.configure(bind=self.engine)
self.session = self._Session()
However this still doesn't "do" anything: it doesn't add anything to the self.Task "class".
How would one do this?
The primary problem in this case is not just the many-to-many relationship, but the fact that it's a self-referential, many-to-many relationship. Because automap is simply translating the mapped class names to relationship names, it constructs the same name, e.g. task_collection, for both directions of the relationship, and the naming collision generates the error. This shortcoming of automap feels significant in that self-referential, many-to-many relationships are not uncommon.
Explicitly adding the relationships you want, using your own names, won't solve the problem because automap will still try to create the task_collection relationships. To deal with this issue, we need to override task_collection.
If you're okay with keeping the name task_collection for the forward direction of the relationship, we can simply pre-define the relationship--specifying whatever name we want for the backref. If automap finds the expected property already in place, it will assume the relationship is being overridden and not try to add it.
Here's a stripped down example, along with the an sqlite database for testing.
Sqlite Database
CREATE TABLE task (
id INTEGER,
name VARCHAR,
PRIMARY KEY (id)
);
CREATE TABLE task_task (
tid1 INTEGER,
tid2 INTEGER,
FOREIGN KEY(tid1) REFERENCES task(id),
FOREIGN KEY(tid2) REFERENCES task(id)
);
-- Some sample data
INSERT INTO task VALUES (0, 'task_0');
INSERT INTO task VALUES (1, 'task_1');
INSERT INTO task VALUES (2, 'task_2');
INSERT INTO task VALUES (3, 'task_3');
INSERT INTO task VALUES (4, 'task_4');
INSERT INTO task_task VALUES (0, 1);
INSERT INTO task_task VALUES (0, 2);
INSERT INTO task_task VALUES (2, 4);
INSERT INTO task_task VALUES (3, 4);
INSERT INTO task_task VALUES (3, 0);
Putting it into a file called setup_self.sql, we can do:
sqlite3 self.db < setup_self.sql
Python Code
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm import Session
from sqlalchemy import create_engine
from sqlalchemy import Table, Column, Integer, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.ext.declarative import declarative_base
DeclBase = declarative_base()
task_task = Table('task_task', DeclBase.metadata,
Column('tid1', Integer, ForeignKey('task.id')),
Column('tid2', Integer, ForeignKey('task.id')))
Base = automap_base(DeclBase)
class Task(Base):
__tablename__ = 'task'
task_collection = relationship('Task',
secondary=task_task,
primaryjoin='Task.id==task_task.c.tid1',
secondaryjoin='Task.id==task_task.c.tid2',
backref='backward')
engine = create_engine("sqlite:///self.db")
Base.prepare(engine, reflect=True)
session = Session(engine)
task_0 = session.query(Task).filter_by(name ='task_0').first()
task_4 = session.query(Task).filter_by(name ='task_4').first()
print("task_0.task_collection = {}".format([x.name for x in task_0.task_collection]))
print("task_4.backward = {}".format([x.name for x in task_4.backward]))
Results
task_0.task_collection = ['task_1', 'task_2']
task_4.backward = ['task_2', 'task_3']
Using a Different Name
If you want to have a name other than task_collection, you need to use automap's function for overriding collection-relationship names:
name_for_collection_relationship(base, local_cls, referred_cls, constraint)
The arguments local_cls and referred_cls are instances of the mapped table classes. For a self-referential, many-to-many relationship, these are both the same class. We can use the arguments to build a key that allows us to identify overrides.
Here is an example implementation of this approach.
from sqlalchemy.ext.automap import automap_base, name_for_collection_relationship
from sqlalchemy.orm import Session
from sqlalchemy import create_engine
from sqlalchemy import Table, Column, Integer, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.ext.declarative import declarative_base
DeclBase = declarative_base()
task_task = Table('task_task', DeclBase.metadata,
Column('tid1', Integer, ForeignKey('task.id')),
Column('tid2', Integer, ForeignKey('task.id')))
Base = automap_base(DeclBase)
class Task(Base):
__tablename__ = 'task'
forward = relationship('Task',
secondary=task_task,
primaryjoin='Task.id==task_task.c.tid1',
secondaryjoin='Task.id==task_task.c.tid2',
backref='backward')
# A dictionary that maps relationship keys to a method name
OVERRIDES = {
'Task_Task' : 'forward'
}
def _name_for_collection_relationship(base, local_cls, referred_cls, constraint):
# Build the key
key = '{}_{}'.format(local_cls.__name__, referred_cls.__name__)
# Did we have an override name?
if key in OVERRIDES:
# Yes, return it
return OVERRIDES[key]
# Default to the standard automap function
return name_for_collection_relationship(base, local_cls, referred_cls, constraint)
engine = create_engine("sqlite:///self.db")
Base.prepare(engine, reflect=True, name_for_collection_relationship=_name_for_collection_relationship)
Note that the overriding of name_for_collection_relationship simply changes the name that automap uses for the relationship. In our case, the relationship is still being pre-defined by Task. But, the override tells automap to look for forward instead of task_collection, which it finds and therefore discontinues defining the relationship.
Other Approaches Considered
Under some circumstances, it would be nice if we could override the relationship names without having to pre-define the actual relationship. On first consideration, this should be possible using name_for_collection_relationship. However, I could not get this approach to work for self-referential, many-to-many relationships, due to a combination of two reasons.
name_for_collection_relationship and the related generate_relationship are called twice, once for each direction of the many-to-many relationship. In both cases, local_cls and referred_cls are the same, because of the self-referentiality. Moreover, the other arguments of name_for_collection_relationship are effectively equivalent. Therefore, we cannot, from the context of the function call, determine which direction we are overriding.
Here is the even-more surprising part of the problem. It appears we cannot even count on one direction happening before the other. In other words, the two calls to name_for_collection_relationship and generate_relationship are very similar. The argument that actually determines the directionality of the relationship is constraint, which is one of the two foreign-key constraints for the relationship; these constraints are loaded, from Base.metadata, into a variable called m2m_const. Herein lies the problem. The order that the constraints end up in m2m_const is nondeterministic, i.e. sometimes it will be one order; other times it will be the opposite (at least when using sqlite3). Because of this, the directionality of the relationship is nondeterministic.
On the other hand, when we pre-define the relationship, the following arguments create the necessary determinism.
primaryjoin='Task.id==task_task.c.tid1',
secondaryjoin='Task.id==task_task.c.tid2',
Of particular note, I actually tried to create a solution that simply overrode the relationship names without pre-defining it. It exhibited the described nondeterminism.
Final Thoughts
If you have a reasonable number of database tables that do not change often, I would suggest just using Declarative Base. It might be a little more work to set up, but it gives you more control.
I'm using python sqlite3 api to create a database.
In all examples I saw on the documentation table names and colum names are hardcoded inside queries..but this could be a potential problem if I re-use the same table multiple times (ie, creating table, inserting records into table, reading data from table, alter table and so on...) because In case of table modification I need to change the hardcoded names in multiple places and this is not a good programming practice..
How can I solve this problem?
I thought creating a class with just constructor method in order to store all this string names..and use it inside the class that will operation on database..but as I'm not an expert python programmer I would like to share my thoughts...
class TableA(object):
def __init__(self):
self.table_name = 'tableA'
self.name_col1 = 'first_column'
self.type_col1='INTEGER'
self.name_col2 = 'second_column'
self.type.col2 = 'TEXT'
self.name_col3 = 'third_column'
self.type_col3 = 'BLOB'
and then inside the DB classe
table_A = TableA()
def insert_table(self):
conn = sqlite3.connect(self._db_name)
query = 'INSERT INTO ' + table_A.table_name + ..... <SNIP>
conn.execute(query)
Is this a proper way to proceed?
I don't know what's proper but I can tell you that it's not conventional.
If you really want to structure tables as classes, you could consider an object relational mapper like SQLAlchemy. Otherwise, the way you're going about it, how do you know how many column variables you have? What about storing a list of 2-item lists? Or a list of dictionaries?
self.column_list = []
self.column_list.append({'name':'first','type':'integer'})
The way you're doing it sounds pretty novel. Check out their code and see how they do it.
If you are going to start using classes to provide an abstraction layer for your database tables, you might as well start using an ORM. Some examples are SQLAlchemy and SQLObject, both of which are extremely popular.
Here's a taste of SQLAlchemy:
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine
Base = declarative_base()
class TableA(Base):
__tablename__ = 'tableA'
id = Column(Integer, primary_key=True)
first_column = Column(Integer)
second_column = Column(String)
# etc...
engine = create_engine('sqlite:///test.db')
Base.metadata.bind = engine
session = sessionmaker(bind=engine)()
ta = TableA(first_column=123, second_column='Hi there')
session.add(ta)
session.commit()
Of course you would choose semantic names for the table and columns, but you can see that declaring a table is something along the lines of what you were proposing in your question, i.e. using a class. Inserting records is simplified by creating instances of that class.
I personally don't like to use libraries and frameworks without proper reason. So, if I'd such reason, so will write a thinking wrapper around sqlite.
class Column(object):
def __init__(self, col_name="FOO", col_type="INTEGER"):
# standard initialization
And then table class that encapsulates operations with database
class Table(object):
def __init__(self, list_of_columns, cursor):
#initialization
#create-update-delete commands
In table class you can encapsulate all operations with the database you want.
How to make dynamic queries in SqlAlchemy ORM (if it is a correct name for them).
I used SqlAlchemy as abstraction for database, with queries in python code, but what if I need to generate these queries dynamically, not only set the parameters of query like "id"?
For example, I need to generate query from list (table names, column names, joined columns) that links three tables like "organisation", "people", "staff". How can I do it properly?
For example, i meant this list:
[{'table':'organisation', 'column':'staff_id'},
{'table':'staff', 'column':'id'}]
And output for example may contain:
organisation.id, organisation.name, organisation.staff_id, staff.id, staff.name
(name column is presented only in output, because I need simple example, recieving all columns of tables, and array must just set joins)
You can use mapper on the result of a call to sqlalchemy.sql.join and/or sqlalchemy.select. This is roughly equivalent to using mapper on a database view; you can query against such classes naturally, but not necessarily create new records. You can also use sqlalchemy.orm.column_property to map computed values to object attributes. As I read your question, a combination of these three techniques should meet your needs.
Haven't tested, but it with the SQLAlchemy ORM, you can link tables together like:
from sqlalchemy import create_engine, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, ForeignKey
from sqlalchemy.orm import relationship
from asgportal.database import Session
Engine = create_engine('mysql+mysqldb://user:password#localhost:3306/mydatabase', pool_recycle=3600)
Base = declarative_base(bind=Engine)
session = Session()
session.configure(bind=Engine)
class DBOrganization(Base):
__tablename__ = 'table_organization'
id = Column(Integer(), primary_key=True)
name = Column(ASGType.sa(ASGType.STRING))
class DBEmployee(Base):
__tablename__ = 'table_employee'
id = Column(Integer(), primary_key=True)
name = Column(String(255))
organization_id = Column(Integer(), ForeignKey('table_organization.id'))
# backref below will be an array[] unless you specify uselist=False
organization = relationship(DBOrganization, backref='employees')
Base.metadata.create_all()
# From here, you can query:
rs = session.query(DBEmployee).join(DBEmployee.organization).filter(DBOrganization.name=='my organization')
for employees in rs:
print '{0} works for {1}'.format(employees.name,employees.organization.name)