Hi have have the following tables
nfiletable = Table(
'NFILE', base.metadata,
Column('fileid', Integer, primary_key=True),
Column('path', String(300)),
Column('filename', String(50)),
Column('filesize', Integer),
schema='NATIVEFILES')#,autoload=True,autoload_with=engine)
sheetnames_table=Table(
'SHEETNAMES', base.metadata, schema='NATIVEFILES',
autoload=True, autoload_with=engine)
nfile_sheet_table=Table(
'NFILE_SHEETNAME',base.metadata,
Column('fileid', Integer, ForeignKey(nfiletable.c.fileid)),
Column('sheetid', Integer, ForeignKey(sheetnames_table.c.sheet_id)),
schema='NATIVEFILES')
and mappers:
nfile_mapper=mapper(Nfile,nfiletable)
mapper(Sheet, sheetnames_table, properties={
'files': relation(
Nfile, secondary=nfile_sheet_table,
primaryjoin=(sheetnames_table.c.sheet_id==nfile_sheet_table.c.sheetid),
secondaryjoin=(nfile_sheet_table.c.fileid==nfiletable.c.fileid),
foreign_keys=[nfile_sheet_table.c.sheetid,nfile_sheet_table.c.fileid],
backref='sheets')
})
when i do the following
upl = Session.query(Nfile).filter_by(fileid=k).one()
sheetdb=[]
for sheet in sheetstoadd:
s = sheetcache[sheetname]
sheetdb.append(s)
upl.sheets = sheetdb
Session.save(upl)
Session.flush()
the line upl.sheets = sheetdb takes forever.
It seems that all files for each sheet in sheetdb are loaded from the db.
How can I prevent this?
if NFile.sheets references a huge collection, put "lazy='dynamic'" on the backref:
mapper(Sheet, sheetnames_table, properties={
'files': relation(
Nfile, secondary=nfile_sheet_table,
backref=backref('sheets', lazy='dynamic'))
})
All the primaryjoin/secondaryjoin/foreign_keys stuff is also not needed since your nfile_sheet_table has ForeignKey constructs on it.
Related
I'm trying to do internationalization, and I've encountered one thing I can't seem to figure out. Fair to say I am total novice using SQLAlchemy (coming from Django world).
I am using SQL Alchemy Core (v.1.4.36). (PostgreSQL) (async sessions). Let's assume I have the following tables:
categories = Table(
'category',
catalog_metadata,
Column('id', Integer, primary_key=True, autoincrement=True)
)
category_translation = Table(
'category_translation',
catalog_metadata,
Column('id', Integer, primary_key=True, autoincrement=True),
Column('name', String(50), nullable=False),
Column('language', String(2), nullable=False),
Column('original_id', Integer, ForeignKey('category.id', ondelete="CASCADE"))
)
product = Table(
'product',
catalog_metadata,
Column('id', Integer, primary_key=True, autoincrement=True),
Column('category_id', Integer, ForeignKey('category.id', ondelete="CASCADE")),
nullable=True),
)
product_translation = Table(
'product_translation',
catalog_metadata,
Column('id', Integer, primary_key=True, autoincrement=True),
Column('language', String(2), nullable=False),
Column('original_id', Integer, ForeignKey('product.id', ondelete="CASCADE")),
Column('name', String(50), nullable=False),
Column('description', Text)
)
Explaining in case is not obvious: I have two main tables category and product. Each one of them has "translatable" fields that are being exposed in the secondary tables category_translation and product_translation respectively. The main goal behind this is, based on specific language, retrieve from the DB information based on the requested language and load it on a Category and Product class. Mapper defined next:
mapper.map_imperatively(
model.Category,
categories,
properties={
'products': relationship(model.Product, backref="category"),
'translations': relationship(
model.CategoryTranslation,
backref="category",
collection_class=attribute_mapped_collection('language')
)
},
)
mapper.map_imperatively(model.CategoryTranslation, category_translation)
mapper.map_imperatively(model.ProductTranslation, product_translation)
mapper.map_imperatively(
model.Product,
product,
properties={
'translations': relationship(
model.ProductTranslation,
backref="product",
collection_class=attribute_mapped_collection('language')
)
},
)
The implementation for the model classes is irrelevant, but you can assume it has the needed fields defined. If you must know, I am using FastAPI and pydantic to serialize output. However, that is not the problem.
What I want to know is how can I set the translated fields to the mapped classes when querying the database?
Meaning, that the instantiated objects for model.Category and model.Product have the name and the name, description fields filled, respectively.
As of now I am doing this query:
select(model_cls).join(translation_cls, (model_cls.id == translation_cls.original_id) & (translation_cls.language == requestedLanguage))
Where model_cls is one the main tables, and translation_cls is its respective translation table. For instance:
select(model.Category).join(model.CategoryTranslation, (model.Category.id == model.CategoryTranslation.original_id) & (model.CategoryTranslation.language == requestedLanguage))
Think that when requesting a product, we may need to join and set attributes for product and for its related category. The response may need to look like this:
{
"id": 1,
"name": "TranslatedProductName",
"category": {
"id": 1,
"name": "TranslatedCategoryNme",
"description": "TranslatedCategoryDescription"
}
}
Hope I've explained myself. If anyone needs more info or explaining, please comment
I have already read similar questions in SO and on Google, as well as the official SQLAlchemy docs, but still couldn't figure out how to solve my problem.
Consider the following structure (non-relevant fields removed for simplicity):
header_table = Table(
'header',
metadata,
Column('id', Integer, primary_key=True),
Column('parent_header_id', Integer)
)
item_table = Table(
'item',
dal.metadata,
Column('id', Integer, primary_key=True),
Column('header_id', Integer)
)
class Header:
id: int
parent_header_id: int
# Relationships
items: List[Item]
children: List[Header]
class Item:
id: int
header_id: int
mapper(Header, header_table, properties={
'children': relationship(Header, foreign_keys=[header_table.c.parent_header_id]),
})
Just to summarise: you can nest headers (max of 1 level of nesting), and each header can have items.
I'm trying to load all headers, with their items and children, and the items of the children.
header_alias = aliased(Header)
records = (
session.query(Header)
.outerjoin(Header.items)
.outerjoin(Header.children.of_type(header_alias))
# .outerjoin(Header.children.of_type(header_alias).items) <<< THE PROBLEM IS HERE (READ BELOW)
.options(contains_eager(Header.items))
.options(contains_eager(Header.children.of_type(header_alias)))
.all()
)
How do I load the items of the children?
The code commented out in the example is wrong, I just put it there as an example of what I'm trying to do.
Note: The code above works, but it's lazy loading the items of the children, I'm trying to get rid of this lazy loading.
Big thanks to #zzzeek (Mike Bayer), author of SQLAlchemy, who answered the question in Github.
https://github.com/sqlalchemy/sqlalchemy/discussions/6876
OK you have to alias "items" also, this is SQL so every table has to
be in the FROM clause only once. Here's a full running example
from sqlalchemy import Column
from sqlalchemy import create_engine
from sqlalchemy import ForeignKey
from sqlalchemy import Integer
from sqlalchemy import Table
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import aliased
from sqlalchemy.orm import contains_eager
from sqlalchemy.orm import declarative_base
from sqlalchemy.orm import relationship
from sqlalchemy.orm import Session
Base = declarative_base()
metadata = Base.metadata
header_table = Table(
"header",
metadata,
Column("id", Integer, primary_key=True),
Column("parent_header_id", ForeignKey("header.id")),
)
item_table = Table(
"item",
metadata,
Column("id", Integer, primary_key=True),
Column("header_id", ForeignKey("header.id")),
)
class Header(Base):
__table__ = header_table
children = relationship("Header")
items = relationship("Item")
class Item(Base):
__table__ = item_table
id: int
header_id: int
e = create_engine("sqlite://", echo=True)
Base.metadata.create_all(e)
s = Session(e)
s.add(
Header(
items=[Item(), Item()],
children=[Header(items=[Item()]), Header(items=[Item(), Item()])],
)
)
s.commit()
s.close()
header_alias = aliased(Header)
item_alias = aliased(Item)
records = (
s.query(Header)
.outerjoin(Header.items)
.outerjoin(Header.children.of_type(header_alias))
.outerjoin(header_alias.items.of_type(item_alias))
.options(
contains_eager(Header.items),
contains_eager(Header.children.of_type(header_alias)).options(
contains_eager(header_alias.items.of_type(item_alias))
),
)
.all()
)
s.close()
for r in records:
print(r)
print(r.items)
for c in r.children:
print(c)
print(c.items)
I have three related classes: Parent, Child, SubChild. The relationship is one-many in both cases. I've set it up so child.parent references correctly and of course so that sub_child.child.parent works as well.
The thing is that I never actually need to know sub_child.child, but I do need to know the sub_child's ultimate parent. I'd like to set up a relationship such that sub_child.parent would return a reference to the ultimate Parent object.
Is this possible, or is it just a bad idea? I've read the documentation, but don't see much that looks promising.
I'm using python2 and sqlalchemy orm on mysql as a backend.
Have a look at http://docs.sqlalchemy.org/en/latest/orm/nonstandard_mappings.html
With this method you should be able to create a mapping over the three tables you mentioned and assign the columns of the participating tables as attributes to the mapping class.
metadata = MetaData()
parent = Table('parent', metadata,
Column('id', Integer, primary_key=True),
Column('child', Integer, ForeignKey('child.id')),
)
child = Table('child', metadata,
Column('id', Integer, primary_key=True),
Column('subchild', Integer, ForeignKey('subchild.id')),
)
subchild = Table('subchild', metadata,
Column('id', Integer, primary_key=True),
Column('some_column', String),
)
joined = join(parent, child, subchild)
Base = declarative_base()
class Parent(Base):
__table__ = joined
id = column_property(parent.c.id, child.c.id, subchild.c.id)
subchild_attr = subchild.c.some_column
I have an issue, where I use the following:
class Docs(Base):
__tablename__ = "docs"
__table__ = Table(__tablename__, Base.metadata,
Column("dID", Integer, primary_key=True),
Column("d_type", Integer, primary_key=True),
Column("d_category", Integer, primary_key=True),
autoload_with=get_global_db_engine())
__table_args__ = (UniqueConstraint("dID", "d_type", "d_category"),)
# Class Globals
COLUMNS = __table__.columns.keys()
Problem is, when I loop throughCOLUMNS - it doesn't list all the columns of __table__, it holds only the columns I pre-defined inside the Table ( 3 cols ).
How do I get COLUMNS to return all of them ?
You'll have to set extend_existing to extend your defined table with reflected columns.
From the documenation:
Table.extend_existing will also work in conjunction with Table.autoload to run a new reflection operation against the database, even if a Table of the same name is already present in the target MetaData; newly reflected Column objects and other options will be added into the state of the Table, potentially overwriting existing columns and options of the same name.
So,
__table__ = Table(__tablename__, Base.metadata,
Column("dID", Integer, primary_key=True),
Column("d_type", Integer, primary_key=True),
Column("d_category", Integer, primary_key=True),
extend_existing=True,
autoload_with=get_global_db_engine())
should do the trick.
The situation is a little bit simplified. I have two migration files for sqlalchemy-migrate:
In First I create table volume_usage_cache, then autoload it, create copy of its columns and print it:
from sqlalchemy import Column, DateTime
from sqlalchemy import Boolean, BigInteger, MetaData, Integer, String, Table
def upgrade(migrate_engine):
meta = MetaData()
meta.bind = migrate_engine
# Create new table
volume_usage_cache = Table('volume_usage_cache', meta,
Column('deleted', Boolean(create_constraint=True, name=None)),
Column('id', Integer(), primary_key=True, nullable=False),
Column('curr_write_bytes', BigInteger(), default=0),
mysql_engine='InnoDB',
mysql_charset='utf8'
)
volume_usage_cache.create()
volume_usage_cache = Table('volume_usage_cache', meta, autoload=True)
columns = []
[columns.append(column.copy()) for column in volume_usage_cache.columns]
print columns
And I get in log what I expected:
[Column('deleted', Boolean(), table=None), Column('id', Integer(), table=None,
primary_key=True, nullable=False), Column('curr_write_bytes', BigInteger(),
table=None, default=ColumnDefault(0))]
But if I make a copy of columns in Second migration file (that is runed after First):
from sqlalchemy import MetaData, String, Integer, Boolean, Table, Column, Index
def upgrade(migrate_engine):
meta = MetaData()
meta.bind = migrate_engine
table = Table("volume_usage_cache", meta, autoload=True)
columns = []
for column in table.columns:
columns.append(column.copy())
print columns
I get a different result:
[Column('deleted', INTEGER(), table=None, default=ColumnDefault(0)),
Column(u'id', INTEGER(), table=None, primary_key=True, nullable=False),
Column(u'curr_write_bytes', NullType(), table=None)]
Why curr_write_bytes column has NullType?
The are two problems:
First:
In First file we are using old metadata that already contains all columns with need types
So if we create new MetaData instance, SqlAlchemy will load info about table from database and will get the same result as in Second file.
Second:
There is no support in sqlAlchemy for BigInteger column type (in sqlite). And Sqlite doesn't support types of column at all. So we can create table with column BigInteger (and it will work), but after autoload type of such column will be automatically converted to NullType.