I am new to ORM's and trying to query a table with a timestamp column. My results however are empty and understandably so since I am querying a timestamp field with a date. I read and found out I can use 'sqlalchemy.sql import func' but my filter is dynamically created based on query params so I was wondering how to go about it.
Code for query model:
def merch_trans_sum_daily_summaries(db_engine, query_params):
query_filters = get_filters(query_params)
page, limit, filters = query_filters.page, query_filters.limit, query_filters.filters
strict_limit = query_filters.strict_limit
with Session(db_engine) as sess:
results = paginate(sess.query(MerchTransSumDaily)
.filter_by(**filters).yield_per(1000),
page, limit, strict_limit)
metadata = results.metadata
query_data = results.data
if not query_data:
raise exc.NoResultFound
data = [record._asdict() for record in query_data]
return data, metadata
Here is my get_filters function
def get_filters(query_parameters, strict_limit=100, default_page=1):
if query_parameters and "batch_type" in query_parameters:
query_parameters.pop('batch_type')
limit = int(query_parameters["limit"]) if query_parameters and "limit" in query_parameters else strict_limit
page = int(query_parameters["page"]) if query_parameters and "page" in query_parameters else default_page
filters = ""
if query_parameters:
filters = {key_: value_ for key_, value_ in query_parameters.items() if key_ not in ["page", "limit", "paginate", "filter"]}
return QueryFilters(limit, page, filters, strict_limit)
Related
I have a pretty reasonable use case: Multiple possible filter_by matches for a single column. Basically, a multiselect JS dropdown on front end posts multiple company industries to the backend. I need to know how to write the SQLAlchemy query and am surprised at how I couldn't find it.
{ filters: { type: "Industry", minmax: false, value: ["Financial Services", "Biotechnology"] } }
#app.route("/dev/api/saved/symbols", methods=["POST"])
#cross_origin(origin="*")
def get_saved_symbols():
req = request.get_json()
# res = None
# if "minmax" in req["filters"]:
# idx = req["filters"].index("minmax")
# if req["filters"][idx] == "min":
# res = db.session.query.filter(Company[req["filter"]["type"]] >= req["filters"]["value"])
# else:
# res = db.session.query.filter(Company[req["filter"]["type"]] <= req["filters"]["value"])
# else:
res = db.session.query.filter_by(Company[req["filters"]["type"]] == req["filters"]["value"])
return jsonify(res)
As you can see I am also working on a minmax which is like an above or below filter for other columns like price or market cap. However, the multiselect OR dynamic statement is really what I am stuck on...
I ended up creating a separate filter function for this that I can than loop over results with.
I will just show the first case for brevity. I am sending a list of strings in which I create a list of filters and then use the or_ operator imported from sqlalchemy package.
def company_filter(db, filter_type, filter_value, minmax):
match filter_type:
case "industry":
filter_list = []
for filter in filter_value:
filter_list.append(Company.industry == filter)
return db.query(Company).with_entities(Company.id, Company.symbol, Company.name, Company.monthly_exp).filter(or_(*filter_list))
...
If I have a sqlalchemy query object
query = db.session(Category)
and Im trying to filter the query further as below
query = query.filter(Category.tree_id.asc())
but instead of Category, I want it to dynamically retrieve the model from the query to use in the filter.
query = query.filter(query.model.tree_id.asc()) #the goal im trying to achieve.
My main goal is to extend wtforms_sqlalchemy.fields.QuerySelectField where I am able to filter the sqlalchemy_mptt model further.
class TreeQuerySelectField(QuerySelectField):
...
def _get_object_list(self):
if self._object_list is None:
query = self.query if self.query is not None else self.query_factory()
if hasattr(self.query.model, 'tree_id'):
query = query.order_by(self.query.model.tree_id, self.query.model.left)
# self.query.model should be the current query model
get_pk = self.get_pk
self._object_list = list((str(get_pk(obj)), obj) for obj in query)
return self._object_list
def iter_choices(self):
if self.allow_blank:
yield ("__None", self.blank_text, self.data is None)
for pk, obj in self._get_object_list():
yield (pk, self.get_label(obj), obj == self.data)
EDIT: This is very similiar to SqlAlchemy - Filtering by Relationship Attribute in that we are both trying to filter on relationship attributes. However, they are filtering on matching an exact value, whereas I am filtering by using like/contains. Because of this, as pointed out in the comments, my solution requires an extra step that was not apparent in the other post.
Let me preface this with: I'm still quite new to SQLAlchemy, so it's entirely possible I'm going about this in the exact wrong way.
I have a Flask API that has defined "alerts" and "events". An alert can only belong to a single event, and an event can have multiple alerts. The schema for the alerts is as follows:
class Alert(PaginatedAPIMixin, db.Model):
__tablename__ = 'alert'
id = db.Column(db.Integer, primary_key=True, nullable=False)
type = db.relationship('AlertType')
type_id = db.Column(db.Integer, db.ForeignKey('alert_type.id'), nullable=False)
url = db.Column(db.String(512), unique=True, nullable=False)
event_id = db.Column(db.Integer, db.ForeignKey('event.id'), nullable=False)
event = db.relationship('Event')
def __str__(self):
return str(self.url)
def to_dict(self):
return {'id': self.id,
'event': self.event.name,
'type': self.type.value,
'url': self.url}
One of the API endpoints lets me get a list of alerts based on various filter criteria. For example, get all alerts of alert_type X, or get all alerts with X inside their alert_url. However, the one that is stumping me is that I want to be able to get all alerts with X inside their associated event name.
Here is the API endpoint function (the commented out events bit is my initial "naive" approach that does not work due to event being a relationship), but you can get the idea what I'm trying to do with the filtering.
def read_alerts():
""" Gets a list of all the alerts. """
filters = set()
# Event filter
#if 'event' in request.args:
# filters.add(Alert.event.name.like('%{}%'.format(request.args.get('event'))))
# URL filter
if 'url' in request.args:
filters.add(Alert.url.like('%{}%'.format(request.args.get('url'))))
# Type filter
if 'type' in request.args:
type_ = AlertType.query.filter_by(value=request.args.get('type')).first()
if type_:
type_id = type_.id
else:
type_id = -1
filters.add(Alert.type_id == type_id)
data = Alert.to_collection_dict(Alert.query.filter(*filters), 'api.read_alerts', **request.args)
return jsonify(data)
The filters set that is built gets fed to the to_collection_dict() function, which essentially returns a paginated list of the query with all of the filters.
def to_collection_dict(query, endpoint, **kwargs):
""" Returns a paginated dictionary of a query. """
# Create a copy of the request arguments so that we can modify them.
args = kwargs.copy()
# Read the page and per_page values or use the defaults.
page = int(args.get('page', 1))
per_page = min(int(args.get('per_page', 10)), 100)
# Now that we have the page and per_page values, remove them
# from the arguments so that the url_for function does not
# receive duplicates of them.
try:
del args['page']
except KeyError:
pass
try:
del args['per_page']
except KeyError:
pass
# Paginate the query.
resources = query.paginate(page, per_page, False)
# Generate the response dictionary.
data = {
'items': [item.to_dict() for item in resources.items],
'_meta': {
'page': page,
'per_page': per_page,
'total_pages': resources.pages,
'total_items': resources.total
},
'_links': {
'self': url_for(endpoint, page=page, per_page=per_page, **args),
'next': url_for(endpoint, page=page + 1, per_page=per_page, **args) if resources.has_next else None,
'prev': url_for(endpoint, page=page - 1, per_page=per_page, **args) if resources.has_prev else None
}
}
return data
I understand that I can get the filtered list of alerts by their associated event name doing something along these lines with options and contains_eager:
alerts = db.session.query(Alert).join(Alert.event).options(contains_eager(Alert.event)).filter(Event.name.like('%{}%'.format(request.args.get('event')))).all()
But I have not gotten something similar to that to work when added to the filters set.
I'm trying to make a query to DynamoDB, and if a LastEvaluatedKey is returned (meaning the query exceeds 1 MB) I want to make other queries in order to fetch all the required data from the table, using LastEvaluatedKey as ExclusiveStartKey for the next query.
This is the code I have for now:
query_response = table.query(
KeyConditionExpression=Key('brand').eq(brand)
)
pagination_key = None
if 'LastEvaluatedKey' in query_response:
pagination_key = query_response['LastEvaluatedKey']
while pagination_key:
next_query_response = table.query(
KeyConditionExpression=Key('brand').eq(brand),
ExclusiveStartKey=pagination_key
)
However, I'd like to refacto this code by extracting the query into a method, passing it pagination_key as an argument. To do this, I'd have to be able to either set ExclusiveStartKey to False, None or some other default value for the first call but I didn't find anything on this, or I'd have to be able to exclude the ExclusiveStartKey alltogether, but I don't know how to do this either.
Using keyword Arguments **kwargs it might look like this. Also,
I am setting up the query before and only updating the ExclusiveStartKeyevery time.
query = { "KeyConditionExpression": Key('brand').eq(brand) }
ExclusiveStartKey = None
while True:
if ExclusiveStartKey is not None:
query['ExclusiveStartKey'] = ExclusiveStartKey
query_response = table.query(**query)
if 'LastEvaluatedKey' in query_response:
ExclusiveStartKey = query_response['LastEvaluatedKey']
else:
break
I found an easy way of building the parameters:
query_params = { 'KeyConditionExpression': Key('brand').eq(brand) }
if pagination_key:
query_params['ExclusiveStartKey'] = pagination_key
query_response = table.query(query_params)
With flask_sqlalchemy, I have two tables (A & B) joined together by an association proxy table (C), in which there are some extras fields.
For now with flask_admin I have a ModelView on C, and in this list view I have appended (with the column_formatters option) a link on each item in the related A table column to its details_view page.
It works fine, but I don't know, in this details-view page, how to append a filtered table, based on C, with just the rows relevant to the selected A item.
Ok, I answer to myself here if someone needs it.
I do the opposite by creating another list_view, but for the selected record I want by modifying the index_view method and passing extras params (details) to the template:
class TableCView(admin.ModelView):
list_template = 'modified_listview.html'
#expose('/')
def index_view(self):
self.a_id = request.args.get('id', None)
self._template_args['a_id'] = self.a_id # add more params here
return super(TableCView, self).index_view()
def get_query(self):
# Filter records by a_id
query = super(TableCView, self).get_query()
if self.a_id:
query = query.filter(self.model.a_id == self.a_id)
return query
def get_count_query(self):
query = super(TableCView, self).get_count_query()
if self.a_id:
query = query.filter(self.model.a_id == self.a_id)
return query