I've got a case where most of the time the relationships between objects was such that pre-configuring an eager (joined) load on the relationship made sense. However now I've got a situation where I really don't want the eager load to be done.
Should I be removing the joined load from the relationship and changing all relevant queries to join at the query location (ick), or is there some way to suppress an eager load in a query once it is set up?
Below is an example where eager loading has been set up on the User->Address relationship. Can the query at the end of the program be configured to NOT eager load?
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
import sqlalchemy.orm as orm
##Set up SQLAlchemy for declarative use with Sqlite...
engine = sa.create_engine("sqlite://", echo = True)
DeclarativeBase = declarative_base()
Session = orm.sessionmaker(bind = engine)
class User(DeclarativeBase):
__tablename__ = "users"
id = sa.Column(sa.Integer, primary_key = True, autoincrement = True)
name = sa.Column(sa.String, unique = True)
addresses = orm.relationship("Address",
lazy = "joined", #EAGER LOAD CONFIG IS HERE
)
def __init__(self, Name):
self.name = Name
class Address(DeclarativeBase):
__tablename__ = "addresses"
id = sa.Column(sa.Integer, primary_key = True, autoincrement = True)
address = sa.Column(sa.String, unique = True)
FK_user = sa.Column(sa.Integer, sa.ForeignKey("users.id"))
def __init__(self, Email):
self.address = Email
##Generate data tables...
DeclarativeBase.metadata.create_all(engine)
##Add some data...
joe = User("Joe")
joe.addresses = [Address("joe#example.com"),
Address("joeyjojojs#example.net")]
s1 = Session()
s1.add(joe)
s1.commit()
## Access the data for the demo...
s2 = Session()
#How to suppress the eager load (auto-join) in the query below?
joe = s2.query(User).filter_by(name = "Joe").one() # <-- HERE?
for addr in joe.addresses:
print addr.address
You may override eagerness of properties on query-by-query basis, as far as I remember. Will this work?
from sqlalchemy.orm import lazyload
joe = (s2.query(User)
.options(lazyload('addresses'))
.filter_by(name = "Joe").one())
for addr in joe.addresses:
print addr.address
See the docs.
You can use Query.options(raiseload('*')) or Query.enable_eagerloads(False).
Query.enable_eagerloads(False) will disable all eager loading on the query. That is, even if you put a joinedload() or something, it won't be executed.
Query.options(raiseload('*')) will install a raiseload loader on every column, making sure they're not lazily loaded: an exception is raised instead. Note that this mode is fine for development and testing environments, but may be destructive in production. Make it optional like this:
Query.options(raiseload('*') if development else defaultload([]))
also note that raiseload('*') only works for top-level relationships. It won't spread on joined entities! If you request a relationship, you have to specify it twice:
session.query(User).options(
load_only('id'),
joinedload(User.addresses).options(
load_only('id'),
raiseload('*')
),
raiseload('*')
)
also, raiseload('*') only works for relationships, not columns :)
For columns, use defer(..., raiseload=True)
Related
I am using the following code to retrieve variables from a database that Python uses to run an automated machine. I set the variables through a PHP driven web interface. Python reads the variables and acts according to instructions.
However, during calibration of the machine, we are forced to restart python to accept any variable changes. Python isn't my first language and neither is it the first language of my colleagues. It would obviously save a lot of time if we didn't have to restart python to accept variable changes.
Our variable list class is constructed like the following;
class VariableList():
connectdb = DbConnector(host='localhost', user='a', password='b', database='c')
result = connectdb.selectDb('variablelist','varA,varB')
for row in result:
# INPUTS
varA = row[1]
varB = row[2]
What is the Pythonic way to get around this issue? Getters/Setters? #property? An example to follow would very much appreciated...
easy one. Python implementation goes like this:
class VariableList():
def __init__(self):
self.db_con = DbConnector(host='localhost', user='a', password='b', database='c')
#property
def varA(self):
return self.db_con.selectDb('variablelist','varA')
#property.setter
def varA(self, value):
self.db_con.updateDb('variablelist', value)
and also you can refactor your model with SQLAlchemy framework.
for example
from sqlalchemy (
create_engine,
Column,
Integer,
String,
)
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
Base = declarative_base()
engine = create_engine(DB_DNS,
pool_size=DB_POOL_SIZE,
max_overflow=DB_MAX_OVERFLOW,
pool_recycle=DB_POOL_RECYCLE,
isolation_level="READ UNCOMMITTED", # attention, the last one is important!
)
Session = sessionmaker(bind=engine,
autocommit=False,
expire_on_commit=False)
class MyTable(Base):
__tablename__ = MY_TABLE_NAME
id = Column(Integer, primary=True)
name = Column(String(32), nullable=True, default='', doc='user_name')
# query something
result = Session().query(MyTable).filter(CONDITION).all()
I have to tie database and programming for an assignment and I have an idea for a code but need to make sure that I can use the tables I created in mySQL as my classes or objects in Python.
Example: I use SQL to create a database of houses with specific addresses and zip codes. A client says they live in zipcode x. My program should then parse through the database and return all addresses within zipcode x. Then ideally create a table in SQL with the clients results.
Not the exact assignment but it gets the basic idea across.
You're looking for an ORM. See SQLAlchemy. Example:
from sqlalchemy import Column, String, Integer, Sequence
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
create_session = sessionmaker()
Base = declarative_base()
person_autoincr_seq = Sequence('person_autoincr_seq')
class Person(Base):
__tablename__ = "person"
id = Column(
Integer,
person_autoincr_seq,
server_default=person_autoincr_seq.next_value(),
nullable = False,
primary_key = True
)
name = Column(
String,
nullable = False
)
def __init__(self, name,id=None):
if id is not None:
self.id = id
self.name = name
Using the db:
import logging as log
from contextlib import closing
engine = sqlalchemy.engine.create_engine(
"postgresql://testuser:mypassword#127.0.0.1:5432/testdb"
)
create_session.configure(bind=engine)
try:
with closing(create_session()) as db_session:
name = db_session.query(Person.name).filter_by(id=5).one()[0]
except Exception:
log.exception("Something wrong while querying db")
I wrote a module which is to create an empty database file
def create_database():
engine = create_engine("sqlite:///myexample.db", echo=True)
metadata = MetaData(engine)
metadata.create_all()
But in another function, I want to open myexample.db database, and create tables to it if it doesn't already have that table.
EG of the first, subsequent table I would create would be:
Table(Variable_TableName, metadata,
Column('Id', Integer, primary_key=True, nullable=False),
Column('Date', Date),
Column('Volume', Float))
(Since it is initially an empty database, it will have no tables in it, but subsequently, I can add more tables to it. Thats what i'm trying to say.)
Any suggestions?
I've managed to figure out what I intended to do. I used engine.dialect.has_table(engine, Variable_tableName) to check if the database has the table inside. IF it doesn't, then it will proceed to create a table in the database.
Sample code:
engine = create_engine("sqlite:///myexample.db") # Access the DB Engine
if not engine.dialect.has_table(engine, Variable_tableName): # If table don't exist, Create.
metadata = MetaData(engine)
# Create a table with the appropriate Columns
Table(Variable_tableName, metadata,
Column('Id', Integer, primary_key=True, nullable=False),
Column('Date', Date), Column('Country', String),
Column('Brand', String), Column('Price', Float),
# Implement the creation
metadata.create_all()
This seems to be giving me what i'm looking for.
Note that in 'Base.metadata' documentation it states about create_all:
Conditional by default, will not attempt to recreate tables already
present in the target database.
And if you can see that create_all takes these arguments: create_all(self, bind=None, tables=None, checkfirst=True), and according to documentation:
Defaults to True, don't issue CREATEs for tables already present in
the target database.
So if I understand your question correctly, you can just skip the condition.
The accepted answer prints a warning that engine.dialect.has_table() is only for internal use and not part of the public API. The message suggests this as an alternative, which works for me:
import os
import sqlalchemy
# Set up a connection to a SQLite3 DB
test_db = os.getcwd() + "/test.sqlite"
db_connection_string = "sqlite:///" + test_db
engine = create_engine(db_connection_string)
# The recommended way to check for existence
sqlalchemy.inspect(engine).has_table("BOOKS")
See also the SQL Alchemy docs.
For those who define the table first in some models.table file, among other tables.
This is a code snippet for finding the class that represents the table we want to create ( so later we can use the same code to just query it )
But together with the if written above, I still run the code with checkfirst=True
ORMTable.__table__.create(bind=engine, checkfirst=True)
models.table
class TableA(Base):
class TableB(Base):
class NewTableC(Base):
id = Column('id', Text)
name = Column('name', Text)
form
Then in the form action file:
engine = create_engine("sqlite:///myexample.db")
if not engine.dialect.has_table(engine, table_name):
# Added to models.tables the new table I needed ( format Table as written above )
table_models = importlib.import_module('models.tables')
# Grab the class that represents the new table
# table_name = 'NewTableC'
ORMTable = getattr(table_models, table_name)
# checkfirst=True to make sure it doesn't exists
ORMTable.__table__.create(bind=engine, checkfirst=True)
engine.dialect.has_table does not work for me on cx_oracle.
I am getting AttributeError: 'OracleDialect_cx_oracle' object has no attribute 'default_schema_name'
I wrote a workaround function:
from sqlalchemy.engine.base import Engine
def orcl_tab_or_view_exists(in_engine: Engine, in_object: str, in_object_name: str,)-> bool:
"""Checks if Oracle table exists in current in_engine connection
in_object: 'table' | 'view'
in_object_name: table_name | view_name
"""
obj_query = """SELECT {o}_name FROM all_{o}s WHERE owner = SYS_CONTEXT ('userenv', 'current_schema') AND {o}_name = '{on}'
""".format(o=in_object, on=in_object_name.upper())
with in_engine.connect() as connection:
result = connection.execute(obj_query)
return len(list(result)) > 0
This is the code working for me to create all tables of all model classes defined with Base class
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
class YourTable(Base):
__tablename__ = 'your_table'
id = Column(Integer, primary_key = True)
DB_URL="mysql+mysqldb://<user>:<password>#<host>:<port>/<db_name>"
scoped_engine = create_engine(DB_URL)
Base = declarative_base()
Base.metadata.create_all(scoped_engine)
I have two tables: Eca_users and Eca_user_emails, one user can have many emails. I recive json with users and their emails. And I wont to load them into MS SQL database. Users can update their emails, so in this json I can get the same users with new (or changed) emails.
My code
# some import here
Base = declarative_base()
class Eca_users(Base):
__tablename__ = 'eca_users'
sql_id = sqlalchemy.Column(sqlalchemy.Integer(), primary_key = True)
first_id = sqlalchemy.Column(sqlalchemy.String(15))
name = sqlalchemy.Column(sqlalchemy.String(200))
main_email = sqlalchemy.Column(sqlalchemy.String(200))
user_emails = relationship("Eca_user_emails", backref=backref('eca_users'))
class Eca_user_emails(Base):
__tablename__ = 'user_emails'
sql_id = sqlalchemy.Column(sqlalchemy.Integer(), primary_key = True)
email_address = Column(String(200), nullable=False)
status = Column(String(10), nullable=False)
active = Column(DateTime, nullable=True)
sql_user_id = Column(Integer, ForeignKey('eca_users.sql_id'))
def main()
engine = sqlalchemy.create_engine('mssql+pymssql://user:pass/ECAusers?charset=utf8')
Session = sessionmaker()
Session.configure(bind = engine)
session = Session()
#then I get my json, parse it and...
query = session.query(Eca_users).filter(Eca_users.first_id == str(user_id))
if query.count() == 0:
# not interesting now
else:
for exstUser in query:
exstUser.name = name #update user info
exstUser.user_emails = [:] # empty old emails
# creating new Email obj
newEmail = Eca_user_emails(email_address = email_record['email'],
status = email_record['status'],
active = active_date)
exstUser.user_emails.append(newEmail) # and I get error here because autoflush
session.commit()
if __name__ == '__main__':
main()
Error message:
sqlalchemy.exc.IntegrityError: ...
[SQL: 'UPDATE user_emails SET sql_user_id=%(sql_user_id)s WHERE user_emails.sql_id = %(user_emails_sql_id)s'] [parameters: {'sql_user_id': None, 'user_emails_sql_id': Decimal('1')}]
Can't find any idea why this sql_user_id is None :(
When I chek exstUser and newEmail objects in debugger - it looks like everething fine. I mean all the reference is OK. The session obj and it's dirty attribute looks also OK in the debugger (sql_user_id is set for Eca_user_emails obj).
And what is most strange for me - this code worked absolutely fine when it was without a main function, just all code after the classes declaration. But after I wrote main declaration and put all code here I started to get this error.
I am completely new to Python so maybe this is one of stupid mistakes...
Any ideas how to fix it and what is the reason? Thanks for reading this :)
By the way: Python 3.4, sqlalchemy 1.0, SQL Server 2012
sql_user_id is None because by default SQLAlchemy clears out the foreign key when you delete a child object across a relationship, that is, when you clear exstUser.user_emails SQLAlchemy sets sql_user_id to None for all those instances. If you want SQLAlchemy to issue DELETEs for Eca_user_emails instances when they are detached from Eca_users, you need to add delete-orphan cascade option to the user_emails relationship. If you want SQLAlchemy to issue DELETEs for Eca_user_emails instances when a Eca_users instance is deleted, you need to add the delete cascade option to the user_emails relationship.
user_emails = relationship("Eca_user_emails", backref=backref('eca_users'), cascade="save-update, merge, delete, delete-orphan")
You can find more information about cascades in the SQLAlchemy docs
We are making a game server using SQLAlchemy.
because game servers must be very fast, we have decided to separate databases depending on user ID(integer).
so for example I did it successfully like the following.
from threading import Thread
from sqlalchemy import Column, Integer, String, DateTime, create_engine
from sqlalchemy.ext.declarative import declarative_base, DeferredReflection
from sqlalchemy.orm import sessionmaker
DeferredBase = declarative_base(cls=DeferredReflection)
class BuddyModel(DeferredBase):
__tablename__ = 'test_x'
id = Column(Integer(), primary_key=True, autoincrement=True)
value = Column(String(50), nullable=False)
and the next code will create multiple databases.
There will be test1 ~ test10 databases.
for i in range(10):
url = 'mysql://user#localhost/'
engine = create_engine(url, encoding='UTF-8', pool_recycle=300)
con = engine.connect()
con.execute('create database test%d' % i)
the following code will create 10 separate engines.
the get_engine() function will give you an engine depending on the user ID.
(User ID is integer)
engines = []
for i in range(10):
url = 'mysql://user#localhost/test%d'% i
engine = create_engine(url, encoding='UTF-8', pool_recycle=300)
DeferredBase.metadata.bind = engine
DeferredBase.metadata.create_all()
engines.append(engine)
def get_engine(user_id):
index = user_id%10
return engines[index]
by running prepare function, the BuddyModel class will be prepared, and mapped to the engine.
def prepare(user_id):
engine = get_engine(user_id)
DeferredBase.prepare(engine)
** The next code will do what I want to do exactly **
for user_id in range(100):
prepare(user_id)
engine = get_engine(user_id)
session = sessionmaker(engine)()
buddy = BuddyModel()
buddy.value = 'user_id: %d' % user_id
session.add(buddy)
session.commit()
But the problem is that when I do it in multiple threads, it just raise errors
class MetalMultidatabaseThread(Thread):
def run(self):
for user_id in range(100):
prepare(user_id)
engine = get_engine(user_id)
session = sessionmaker(engine)()
buddy = BuddyModel()
buddy.value = 'user_id: %d' % user_id
session.add(buddy)
session.commit()
threads = []
for i in range(100):
t = MetalMultidatabaseThread()
t.setDaemon(True)
t.start()
threads.append(t)
for t in threads:
t.join()
the error message is ...
ArgumentError: Class '<class '__main__.BuddyModel'>' already has a primary mapper defined. Use non_primary=True to create a non primary Mapper. clear_mappers() will remove *all* current mappers from all classes.
so.. my question is that How CAN I DO MULTIPLE-DATABASE like the above architecture using SQLAlchemy?
this is called horizontal sharding and is a bit of a tricky use case. The version you have, make a session based on getting the engine first, will work fine. There are two variants of this which you may like.
One is to use the horizontal sharding extension. This extension allows you to create a Session to automatically select the correct node.
The other is more or less what you have, but less verbose. Build a Session class that has a routing function, so you at least could share a single session and say, session.using_bind('engine1') for a query instead of making a whole new session.
I have found an answer for my question.
For building up multiple-databases depending on USER ID (integer) just use session.
Before explain this, I want to expound on the database architecture more.
For example if the user ID 114 connects to the server, the server will determine where to retrieve the user's information by using something like this.
user_id%10 # <-- 4th database
Architecture
DATABASES
- DB0 <-- save all user data whose ID ends with 0
- DB1 <-- save all user data whose ID ends with 1
.
.
.
- DB8 <-- save all user data whose ID ends with 9
Here is the answer
First do not use bind parameter.. simply make it empty.
Base = declarative_base()
Declare Model..
class BuddyModel(Base):
__tablename__ = 'test_x'
id = Column(Integer(), primary_key=True, autoincrement=True)
value = Column(String(50), nullable=False)
When you want to do CRUD ,make a session
engine = get_engine_by_user_id(user_id)
session = sessionmaker(bind=engine)()
buddy = BuddyModel()
buddy.value = 'This is Sparta!! %d' % user_id
session.add(buddy)
session.commit()
engine should be the one matched with the user ID.