I'd like to add a custom attribute "searchable" to the columns in an sqlalchemy model. The purpose is to extract data for just these columns (using ModelFoo.__ table__.columns) and put the data into solr. Is there a way I can mark certain columns using a custom attribute?
class ModelFoo(AppBase):
__tablename__ = 'foo'
id = Column("id", Integer, primary_key=True, autoincrement=True)
os = Column(String, nullable=False, searchable=True)
platform = Column(String, searchable=True)
by default, I get the following error when I try the above:
sqlalchemy.exc.ArgumentError: Unknown arguments passed to Column: ['searchable']
I am looking for a generic way to add only "searchable" columns to solr, something along these lines:
for table in Base.metadata.tables.values():
keys = [str(key) for key in table.columns if key.searchable]
solr.add(session.query(*keys).all())
in the above code, I am looking for some short solution or alternative to get "key.searchable" to work. Hope this clarifies the question.
I solved this using a separate attribute in model:
class ModelFoo(Base):
__tablename__ = 'foo'
id = Column("id", Integer, primary_key=True, autoincrement=True)
os = Column(String, nullable=False)
platform = Column(String)
search_cols = ["os", "value"]
for k, v in list(Base._decl_class_registry.items()):
if (hasattr(v, "search_cols")):
cols = [getattr(v, val) for val in v.search_cols]
query = sess.query(*cols)
solr.add(query.all())
I'm a little unclear on your question but it looks like your are trying to do something like this from the link you provided:
si.add(Book.objects.all())
Where Book.objects.all() is a list of records from an ORM mapped table. Note that the docs say a list of objects is also acceptable. I think to solution here is to use the SQLAlchemy query method to build records with only the fields you want. Using your example with would look like this:
si.add(session.query(ModelFoo.os, ModelFoo.platform).all())
Related
I have a many-to-many relationship, with instances of OsmAdminUnit (polygon geometries) grouped into OsmAdminAgg instances.
The model definitions are essentially:
class OsmAdminUnit(db.Model):
__tablename__ = 'osm_admin'
id = db.Column(db.Integer, primary_key=True)
geometry = db.Column(Geometry(
geometry_type='GEOMETRY',
srid=3857), nullable=False)
agg_units = db.relationship('OsmAdminAgg',
secondary=aggregations,
backref=db.backref('osm_admin', lazy='dynamic'))
class OsmAdminAgg(db.Model):
__tablename__ = 'admin_agg'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(100), unique=True, nullable=False)
Now what I am struggling to do is selecting OsmAdminUnits that belong to a certain OsmAdminAgg AND getting the polgyons merged by applying ST_Union from GeoAlchemy.
Selecting all admin units that belong to admin agg with id=1 works:
units = OsmAdminUnit.query.filter(OsmAdminUnit.agg_units.any(id=1)).all()
But I don't get how I can apply ST_Union on that result.
My approach so far was:
union = db.session.query(
OsmAdminUnit.geometry.ST_Union().ST_AsGeoJSON().label('agg_union')
).filter(OsmAdminUnit.agg_units.any(id=1)).subquery()
So how do I get the union of these geometries, and get it as GeoJSON?
Btw, I am building this on top of Flask, using SQLAlchemy, Flask-SQLAlchemy, Geoalchemy2.
Try this:
from sqlalchemy.sql.functions import func
union = db.session.query(func.ST_AsGeoJSON(func.ST_Union(
OsmAdminUnit.geometry)).label('agg_union')
).filter(OsmAdminUnit.agg_units.any(id=1)).subquery()
You can see a basic template for this in the GeoAlchemy 2 docs. Essentially, you need to pass func to the query, rather than the model, to select the union itself.
In your case, something like:
import sqlalchemy
union = db.session.query(
sqlalchemy.func.ST_AsGeoJSON(
sqlalchemy.func.ST_Union(OsmAdminUnit.geometry)
).label('agg_union')
).filter(
OsmAdminUnit.agg_units.any(id=1)
).all()
This grabs the union of geometry values for OsmAdminUnit records matching the filter, and returns it as stringified GeoJSON.
The accepted answer didn't work for me, I think the import may be different in the version of sqlalchemy I'm using.
I want to search a SQLAlachmey list (via an association table) and match on multiple items within it via a filter.
I already reviewed this question but I am looking to accomplish this via the ORM filter only (and the second answer is not via an association table).
Database table setup:
tag_ast_table = Table('tag_association',
Base.metadata,
Column('file_id', Integer, ForeignKey('files.id')),
Column('tag_id', Integer, ForeignKey('tags.id')),
PrimaryKeyConstraint('file_id', 'tag_id'))
class File(Base):
__tablename__ = 'files'
id = Column(Integer, primary_key=True)
tags = relationship("Tag", secondary=tag_ast_table)
class Tag(Base):
__tablename__ = 'tags'
id = Column(Integer, primary_key=True)
tag = Column(String)
Current filter to match any I would like to modify to match all:
query = db.query(File).filter(File.tags.any(Tag.tag.in_(my_list))).all()
A reasonable approach to this in SQL (alluded to in your link) is to use having count(distinct tags.id) = <your number of tags>.
So the query needs 2 things: it needs an in that looks for your list of tags, and it needs a having that looks for the full count being present.
query = (
session.query(File)
.join(File.tags)
.filter(Tag.tag.in_(search_tags))
.group_by(File)
.having(func.count(distinct(Tag.id)) == len(search_tags))
)
As an edge case, if search_tags is an empty list you won't get any results, so best to check for that first.
I have a table that does not have a primary key. And I really do not want to apply this constraint to this table.
In SQLAlchemy, I defined the table class by:
class SomeTable(Base):
__table__ = Table('SomeTable', meta, autoload=True, autoload_with=engine)
When I try to query this table, I got:
ArgumentError: Mapper Mapper|SomeTable|SomeTable could not assemble any primary key columns for mapped table 'SomeTable'.
How to loss the constraint that every table must have a primary key?
There is only one way that I know of to circumvent the primary key constraint in SQL Alchemy - it's to map specific column or columns to your table as a primary keys, even if they aren't primary key themselves.
http://docs.sqlalchemy.org/en/latest/faq/ormconfiguration.html#how-do-i-map-a-table-that-has-no-primary-key.
There is no proper solution for this but there are workarounds for it:
Workaround 1
Adding parameter primary_key to the existing column that is not having a primary key will work.
class SomeTable(Base):
__table__ = 'some_table'
some_other_already_existing_column = Column(..., primary_key=True) # just add primary key to it whether or not this column is having primary key or not
Workaround 2
Just declare a new dummy column on the ORM layer, not in actual DB. Just define in SQLalchemy model
class SomeTable(Base):
__table__ = 'some_table'
column_not_exist_in_db = Column(Integer, primary_key=True) # just add for sake of this error, dont add in db
Disclaimer: Oracle only
Oracle databases secretly store something called rowid to uniquely define each record in a table, even if the table doesn't have a primary key. I solved my lack of primary key problem (which I did not cause!) by constructing my ORM object like:
class MyTable(Base)
__tablename__ = 'stupid_poorly_designed_table'
rowid = Column(String, primary_key=True)
column_a = Column(String)
column_b = Column(String)
...
You can see what rowid actually looks like (it's a hex value I believe) by running
SELECT rowid FROM stupid_poorly_designed_table
GO
Here is an example using __mapper_args__ and a synthetic primary_key. Because the table is time-series oriented data, there is no need for a primary key. All rows can be unique addresses with a (timestamp, pair) tuple.
class Candle(Base):
__tablename__ = "ohlvc_candle"
__table_args__ = (
sa.UniqueConstraint('pair_id', 'timestamp'),
)
#: Start second of the candle
timestamp = sa.Column(sa.TIMESTAMP(timezone=True), nullable=False)
open = sa.Column(sa.Float, nullable=False)
close = sa.Column(sa.Float, nullable=False)
high = sa.Column(sa.Float, nullable=False)
low = sa.Column(sa.Float, nullable=False)
volume = sa.Column(sa.Float, nullable=False)
pair_id = sa.Column(sa.ForeignKey("pair.id"), nullable=False)
pair = orm.relationship(Pair,
backref=orm.backref("candles",
lazy="dynamic",
cascade="all, delete-orphan",
single_parent=True, ), )
__mapper_args__ = {
"primary_key": [pair_id, timestamp]
}
MSSQL Tested
I know this thread is ancient but I spent way too long getting this to work to not share it :)
from sqlalchemy import Table, event
from sqlalchemy.ext.compiler import compiles
from sqlalchemy import Column
from sqlalchemy import Integer
class RowID(Column):
pass
#compiles(RowID)
def compile_mycolumn(element, compiler, **kw):
return "row_number() OVER (ORDER BY (SELECT NULL))"
#event.listens_for(Table, "after_parent_attach")
def after_parent_attach(target, parent):
if not target.primary_key:
# if no pkey create our own one based on returned rowid
# this is untested for writing stuff - likely wont work
logging.info("No pkey defined for table, using rownumber %s", target)
target.append_column(RowID('row_id', Integer, primary_key=True))
https://docs-sqlalchemy-org.translate.goog/en/14/faq/ormconfiguration.html?_x_tr_sl=auto&_x_tr_tl=ru&_x_tr_hl=ru#how-do-i-map-a-table-that-has-no-primary-key
One way from there:
In SQLAlchemy ORM, to map to a specific table, there must be at least one column designated as the primary key column; multi-column composite primary keys are of course also perfectly possible. These columns do not need to be known to the database as primary key columns. The columns only need to behave like a primary key, such as a non-nullable unique identifier for a row.
my code:
from ..meta import Base, Column, Integer, Date
class ActiveMinutesByDate(Base):
__tablename__ = "user_computer_active_minutes_by_date"
user_computer_id = Column(Integer(), nullable=False, primary_key=True)
user_computer_date_check = Column(Date(), default=None, primary_key=True)
user_computer_active_minutes = Column(Integer(), nullable=True)
The solution I found is to add an auto-incrementing primary key column to the table, then use that as your primary key. The database should deal with everything else beyond that.
I am creating a website using Flask and SQLAlchemy. This website keeps track of classes that a student has taken. I would like to find a way to search my database using SQLAlchemy to find all unique classes that have been entered. Here is code from my models.py for Class:
class Class(db.Model):
__tablename__ = 'classes'
id = db.Column(db.Integer, primary_key=True)
title = db.Column(db.String(100))
body = db.Column(db.Text)
created = db.Column(db.DateTime, default=datetime.datetime.now)
user_email = db.Column(db.String(100), db.ForeignKey(User.email))
user = db.relationship(User)
In other words, I would like to get all unique values from the title column and pass that to my views.py.
Using the model query structure you could do this
Class.query.with_entities(Class.title).distinct()
query = session.query(Class.title.distinct().label("title"))
titles = [row.title for row in query.all()]
titles = [r.title for r in session.query(Class.title).distinct()]
As #van has pointed out, what you are looking for is:
session.query(your_table.column1.distinct()).all(); #SELECT DISTINCT(column1) FROM your_table
but I will add that in most cases, you are also looking to add another filter on the results. In which case you can do
session.query(your_table.column1.distinct()).filter_by(column2 = 'some_column2_value').all();
which translates to sql
SELECT DISTINCT(column1) FROM your_table WHERE column2 = 'some_column2_value';
I have items, warehouses, and items are in warehouses.
So I have table that has information about items (sku, description, cost ...) and a table that describes warehouses(location, code, name, ...). Now I need a way to store inventory so that I know I have X items in warehouse Y. An item can be in any warehouse.
How would I go about setting up the relationship between them and storing the qty?
class Item(DeclarativeBase):
__tablename__ = 'items'
item_id = Column(Integer, primary_key=True,autoincrement=True)
item_code = Column(Unicode(35),unique=True)
item_description = Column(Unicode(100))
item_long_description = Column(Unicode())
item_cost = Column(Numeric(precision=13,scale=4))
item_list = Column(Numeric(precision=13,scale=2))
def __init__(self,code,description,cost,list):
self.item_code = code
self.item_description = description
self.item_cost = cost
self.item_list = list
class Warehouse(DeclarativeBase):
__tablename__ = 'warehouses'
warehouse_id = Column(Integer, primary_key=True, autoincrement=True)
warehouse_code = Column(Unicode(15),unique=True)
warehouse_description = Column(Unicode(55))
If I am correct I would setup the many to many using an intermediate table something like ...
item_warehouse = Table(
'item_warehouse', Base.metadata,
Column('item_id', Integer, ForeignKey('items.item_id')),
Column('warehouse_id', Integar, ForeignKey('warehouses.warehouse_id'))
)
But i would need to start the qty available on this table but since its not its own class I am not sure how that would work.
What would be the "best" practice for modeling this and having it usable in my app?
Model:
As mentioned by #Lafada, you need an Association Object. As such I would create a SA-persistent object and not only a table:
class ItemWarehouse(Base):
# version-1:
__tablename__ = 'item_warehouse'
__table_args__ = (PrimaryKeyConstraint('item_id', 'warehouse_id', name='ItemWarehouse_PK'),)
# version-2:
#__table_args__ = (UniqueConstraint('item_id', 'warehouse_id', name='ItemWarehouse_PK'),)
#id = Column(Integer, primary_key=True, autoincrement=True)
# other columns
item_id = Column(Integer, ForeignKey('items.id'), nullable=False)
warehouse_id = Column(Integer, ForeignKey('warehouses.id'), nullable=False)
quantity = Column(Integer, default=0)
This covers the model requirement with the following:
added a PrimaryKey
added a UniqueConstraint covering the (item_id, warehouse_id) pairs.
In the code above this is solved in two ways:
version-1: uses composite primary key (which must be unique)
version-2: uses simple primary key, but also adds an explicit unique constraint [I personally prefer this option]
Relationship: Association Object
Now. You can use the Association Object as is, which will look similar to this:
w = Warehouse(...)
i = Item(name="kindle", price=...)
iw = ItemWarehouse(quantity=50)
iw.item = i
w.items.append(i)
Relationship: Association Proxy extension
or, you could go one step further and use the Composite Association Proxies example, and you may configure dictionary-like access to the association object similar to this:
w = Warehouse(...)
i = Item(name="kindle", price=...)
w[i] = 50 # sets the quantity to 50 of item _i_ in warehouse _w_
i[w] = 50 # same as above, if you configure it symmetrically
Beware: the code for the relationships definition might look really not easily readable, but the usage pattern is really nice. So if this option is too much to digest, I would start with Association Object with maybe some helper functions to add/get/update the item stocks, and eventually move to the Association Proxy extesion.
You have to use "Association Object".
I try to give you hint for your problem you have to create table like you mention in your question
item_warehouse = Table( 'item_warehouse',
Base.metadata,
Column('item_id',
Integer,
ForeignKey('items.item_id')
),
Column('warehouse_id',
Integar,
ForeignKey('warehouses.warehouse_id')
),
Column('qty',
Integer,
default=0,
),
)
Now you can add warehouse, item and qty in single object and you have to write method which will take warehouse_id and item_id and get the sum of qty for those itmes.
Hope this will help you to solve your problem.