I am pretty new to Django and I am trying to get a query set from a filter function. This filter function is supposed to be able to take 1 to 5 arguments and I am not sure how to handle that.
I have not found anything here that might help me, so if you do know of some other question that might help please let me know.
views.py:
#api_view(('Get',))
def update(request, REQUEST):
if request.method == "Get":
requestlist = REQUEST.split('&')
for keys in requestlist:
if 'module' in keys:
module = keys[8:]
if 'value' in keys:
value = keys[6:]
if 'user' in keys:
user= keys[5:]
if 'time1' in keys:
time1 = keys[6:]
if 'time2' in keys:
time2 = keys[6:]
item = Post.objects.filter(name=Name, user=USER, ...)
The full request string will look like name=NAME&value=VALUE&user=USER&time1=FIRSTTIME&time2=SECONDTIME but it could also be any combination of the individual variables like name&time1.
Now I want to be able to do that with one filter method instead of creating like 2^5 for each different szenario.
The full request string will look like name=NAME&value=VALUE&user=USER&time1=FIRSTTIME&time2=SECONDTIME.
This is a query string [wiki], and Django automatically parses this to a dictionary-like QueryDict, you thus should not specify this yourself. You can work with:
if request.method == 'GET':
Post.objects.filter(**request.GET.dict())
I would however advise to only allow specific keys, and thus not all keys, since then the database is less secure: one can use the filtering mechanism to retrieve data.
It thus might be better to work with:
datas = {}
accept_keys = {'module', 'value', 'user', 'time1', 'time2'}
for key, value in request.GET.dict().items():
if key in accept_keys:
datas[key] = value
if request.method == 'GET':
Post.objects.filter(**datas)
In that case the item after the path is the query string, and the separator between the path and the query string is a question mark (?).
The path thus looks like:
urlpatterns = [
# …,
path('some/path/', views.update, name='update'),
# …
]
and you thus query the path with some.host.com/some/path?name=NAME&value=VALUE&user=USER&time1=FIRSTTIME&time2=SECONDTIME.
I have to make 2 search bars. I don't know how to add multiple fiels...
match= Staff.objects.filter(id=srch1...) here how can I add name=srch1
over here after trying many ways I found it but the problem is all input here is string how to change it to int
def search(request):
# Catch the data and search in Staff model.
if request.method=='POST':
srch1 = request.POST['srch']
print(type(srch1))
if type(srch1)== int:
match= Staff.objects.filter(id=srch1)
if match :
return render(request,'search.html',{'sr': match})
else:
messages.error(request,'no results,found')
elif type(srch1)== str:
catch= Staff.objects.filter(name=srch1)
if catch:
return render(request,'search.html',{'sr': catch})
else:
messages.error(request,'no results,found')
else:
return HttpResponseRedirect("/search")
return render(request,"search.html")
You should be using a GET request with querysting parameters or (url parameters, which is a bit more complicated) for this, not a POST. Here's how I would do this:
def search(request, *args, **kwargs):
# Initial empty query dictionary for use with query later.
query = {}
# Getting 'name' and 'id' from querystring parameters (i.e. ?id=1&name=foo)
name = request.GET.get('name', None)
id = request.GET.get('id', None)
# Add 'id' to the query dictionary if it exists
if id is not None:
query['id'] = id
# Add name to the query dictionary if it exists
if name is not None:
query['name'] = name
# If the query dictionary has name or id, get the Staff entry from the database
if query.get('name', None) or query.get('id', None):
# Note that .filter() returns a QuerySet. You should probably use .get()
# since my guess is that you only want one Staff object (judging by your
# search parameters). Also note that since we are using **query we will be
# using BOTH 'name' AND 'id' to search for the Staff, as long as both exist in
# the query dictionary.
match = Staff.objects.get(**query)
# If there is a match, send it in the rendered response context dict
if match:
return render(request, 'search.html', {'sr': match})
# no match, send message notifying that a Staff entry was not found matching
# the desired criteria
return render(request, 'search.html', {message: 'Not Found'}
# There were no query parameters, so we are not searching for anything.
else:
return render(request, 'search.html')
You can see that the above code is much simpler, and more concise. This will help you or anyone else checking out your code in the future to better understand what you're trying to acheive.
P.S. I typically only use POST requests when I am creating an entry in the database. Maybe this is preference, but to my knowledge it is best practice to use a GET request for search.
I am trying to build an accounting database using flask as the front end. The main page is the ledger, with nine columns "date" "description" "debit" "credit" "amount" "account" "reference" "journal" and "year", I need to be able to query each and some times two at once, there are over 8000 entries, and growing. My code so far displays all the rows, 200 at a time with pagination, I have read "pep 8" which talks about readable code, I have read this multiple parameters and this multiple parameters and like the idea of using
request.args.get
But I need to display all the rows until I query, I have also looked at this nested ifs and I thought perhaps I could use a function for each query and "If" out side of the view function and then call each in the view function, but I am not sure how to. Or I could have a view function for each query. But I am not sure how that would work, here is my code so far,
#bp.route('/books', methods=['GET', 'POST'])
#bp.route('/books/<int:page_num>', methods=['GET', 'POST'])
#bp.route('/books/<int:page_num>/<int:id>', methods=['GET', 'POST'])
#bp.route('/books/<int:page_num>/<int:id>/<ref>', methods=['GET', 'POST'])
#login_required
def books(page_num, id=None, ref=None):
if ref is not None:
books = Book.query.order_by(Book.date.desc()).filter(Book.REF==ref).paginate(per_page=100, page=page_num, error_out=True)
else:
books = Book.query.order_by(Book.date.desc()).paginate(per_page=100, page=page_num, error_out=True)
if id is not None:
obj = Book.query.get(id) or Book()
form = AddBookForm(request.form, obj=obj)
if form.validate_on_submit():
form.populate_obj(obj)
db.session.add(obj)
db.session.commit()
return redirect(url_for('books.books'))
else:
form = AddBookForm()
if form.validate_on_submit():
obj = Book(id=form.id.data, date=form.date.data, description=form.description.data, debit=form.debit.data,\
credit=form.credit.data, montant=form.montant.data, AUX=form.AUX.data, TP=form.TP.data,\
REF=form.REF.data, JN=form.JN.data, PID=form.PID.data, CT=form.CT.data)
db.session.add(obj)
db.session.commit()
return redirect(url_for('books.books', page_num=1))
return render_template('books/books.html', title='Books', books=books, form=form)
With this code there are no error messages, this is a question asking for advice on how to keep my code as readable and as simple as possible and be able to query nine columns of the database whilst displaying all the rows queried and all the rows when no query is activated
All help is greatly appreciated. Paul
I am running this on debian 10 with python 3.7
Edit: I am used to working with Libre Office Base
My question is How do I search one or two columns at a time in My database where I have nine columns out of twelve that I want to be able to search, I want to be able to search one or more at a time, example: column "reference" labels a document reference like "A32", and "account" by a the name of the supplier "FILCUI", possibly both at the same time. I have carried out more research and found that most people advocate a "fulltext" search engine such as "Elastic or Whoosh", But in my case I feel if I search "A32" ( a document number) I will get anything in the model of 12 columns with A 1 2. I have looked at Flask Tutorial 101 search Whoosh all very good tutorials, by excellent people, I thought about trying to use SQLAlchemy as a way, but in the first "Flask Tutorial" he says
but given the fact that SQLAlchemy does not support this functionality,
I thought that this SQLAlchemy-Intergrations will not work either.
So therefor is there a way to "search" "query" "filter" multiple different columns of a model with possibly a form for each search without ending up with a "sack of knots" like code impossible to read or test? I would like to stick to SQLAlchemy if possible
I need just a little pointer in the right direction or a simple personal opinion that I can test.
Warm regards.
EDIT:
I have not answered my question but I have advanced, I can query one row at a time and display all the results on the one page, with out a single "if" statement, i think my code is clear and readable (?) I divided each query into its own view function returning to the same main page, each function has its own submitt button. This has enabled me to render the same page. here is my routes code.
#bp.route('/search_aux', methods=['GET', 'POST'])
#login_required
def search_aux():
page_num = request.args.get('page_num', default = 1, type = int)
books = Book.query.order_by(Book.date.desc()).paginate(per_page=100, page=page_num, error_out=True)
add_form = AddBookForm()
aux_form = SearchAuxForm()
date_form = SearchDateForm()
debit_form = SearchDebitForm()
credit_form = SearchCreditForm()
montant_form = SearchMontantForm()
jn_form = SearchJNForm()
pid_form = SearchPIDForm()
ref_form = SearchREForm()
tp_form = SearchTPForm()
ct_form = SearchCTForm()
des_form = SearchDescriptionForm()
if request.method == 'POST':
aux = aux_form.selectaux.data
books = Book.query.order_by(Book.date.desc()).filter(Book.AUX == str(aux)).paginate(per_page=100, page=page_num, error_out=True)
return render_template('books/books.html', books=books, add_form=add_form, aux_form=aux_form, date_form=date_form, debit_form=debit_form,
credit_form=credit_form, montant_form=montant_form, jn_form=jn_form, pid_form=pid_form, ref_form=ref_form,
tp_form=tp_form, ct_form=ct_form, des_form=des_form)
There is a simple form for each query, it works a treat for each single query. Here is the form and html code:
class SearchAuxForm(FlaskForm):
selectaux = QuerySelectField('Aux', query_factory=AUX, get_label='id')
submitaux = SubmitField('submit')
def AUX():
return Auxilliere.query
html:
<div class="AUX">
<form action="{{ url_for('books.search_aux') }}" method="post">
{{ aux_form.selectaux(class="input") }}{{ aux_form.submitaux(class="submit") }}
</form>
</div>
I tried to do this as a single function with one submit button, but it ended in disaster. I have not submitted this as an answer, Because it does not do all I asked but it is a start.
FINAL EDIT:
I would like to thank the person(s) who reopened this question, allowing mr Lucas Scott to provide a fascinating and informative answer to help me and others.
There are many ways to achieve your desired result of being able to query/filter multiple columns in a table. I will give you an example of how I would approach creating an endpoint that will allow you to filter on one column, or multiple columns.
Here is our basic Books model and the /books endpoint as a stub
import flask
from flask_sqlalchemy import SQLAlchemy
app = flask.Flask(__name__)
db = SQLAlchemy(app) # uses in memory sqlite3 db by default
class Books(db.Model):
__tablename__ = "book"
id = db.Column(db.Integer, primary_key=True, autoincrement=True)
title = db.Column(db.String(255), nullable=False)
author = db.Column(db.String(255), nullable=False)
supplier = db.Column(db.String(255))
published = db.Column(db.Date, nullable=False)
db.create_all()
#app.routes("/books", methods=["GET"])
def all_books():
pass
The first step is to decide on a method of querying a collection by using url parameters. I will use fact that multiple instances of the same key in a query parameter are given as lists to allow us to filter on multiple columns.
For example /books?filter=id&filter=author will turn into {"filter": ["id", "author"]}.
For our querying syntax we will use comma separated values for the filter value.
example:
books?filter=author,eq,jim&suplier,eq,self published
Which turns into {"filter": ["author,eq,jim", "supplier,eq,self published"]}. Notice the space in self published. flask will handle the url-encoding for us and give back a string with a space instead of %20.
Let's clean this up a bit by adding a Filter class to represent our filter query parameter.
class QueryValidationError(Exception):
""" We can handle specific exceptions and
return a http response with flask """
pass
class Filter:
supported_operators = ("eq", "ne", "lt", "gt", "le", "ge")
def __init__(self, column, operator, value):
self.column = column
self.operator = operator
self.value = value
self.validate()
def validate(self):
if operator not in self.supported_operators:
# We will deal with catching this later
raise QueryValidationError(
f"operator `{operator}` is not one of supported "
f"operators `{self.supported_operators}`"
)
Now we will create a function for processing our list of filters into a list of Filter objects.
def create_filters(filters):
filters_processed = []
if filters is None:
# No filters given
return filters_processed
elif isinstance(filters, str):
# if only one filter given
filter_split = filters.split(",")
filters_processed.append(
Filter(*filter_split)
)
elif isinstance(filters, list):
# if more than one filter given
try:
filters_processed = [Filter(*_filter.split(",")) for _filter in filters]
except Exception:
raise QueryValidationError("Filter query invalid")
else:
# Programer error
raise TypeError(
f"filters expected to be `str` or list "
f"but was of type `{type(filters)}`"
)
return filters_processed
and now we can add our helper functions to our endpoint.
#app.route("/books", methods=["GET"])
def all_books():
args = flask.request.args
filters = create_filters(args.get("filter"))
SQLAlchemy allows us to do filtering by using operator overloading. That is using filter(Book.author == "some value"). The == here does not trigger the default == behaviour. Instead the creator of SQLAlchemy has overloaded this operator and instead it creates the SQL query that checks for equality and adds it to the
query. We can leverage this behaviour by using the Pythons operator module. For example:
import operator
from models import Book
authors = Book.query.filter(operator.eq(Book.author, "some author")).all()
This does not seem helpful by it's self, but gets us a step closer to creating a generic and dynamic filtering mechanism. The next important step to making this more dynamic is with the built-in getattr which allows us to look up attributes on a given object using strings. Example:
class Anything:
def say_hi(self):
print("hello")
# use getattr to say hello
getattr(Anything, "say_hi") # returns the function `say_hi`
getattr(Anything, "say_hi")() # calls the function `say_hi`
We can now tie this all together by creating a generic filtering function:
def filter_query(filters, query, model):
for _filter in filters:
# get our operator
op = getattr(operator, _filter.operator)
# get the column to filter on
column = getattr(model, _filter.column)
# value to filter for
value = _filter.value
# build up a query by adding multiple filters
query = query.filter(op(column, value))
return query
We can filter any model with our implementation, and not just by one column.
#app.route("/books", methods=["GET"])
def all_books():
args = flask.request.args
filters = create_filters(args.get("filter"))
query = Books.query
query = filter_query(filters, query, Books)
result = []
for book in query.all():
result.append(dict(
id=book.id,
title=book.title,
author=book.author,
supplier=book.supplier,
published=str(book.published)
))
return flask.jsonify(result), 200
Here is everything all together, and including the error handling of validation errors
import flask
import json
import operator
from flask_sqlalchemy import SQLAlchemy
app = flask.Flask(__name__)
db = SQLAlchemy(app) # uses in memory sqlite3 db by default
class Books(db.Model):
__tablename__ = "book"
id = db.Column(db.Integer, primary_key=True, autoincrement=True)
title = db.Column(db.String(255), nullable=False)
author = db.Column(db.String(255), nullable=False)
supplier = db.Column(db.String(255))
published = db.Column(db.Date, nullable=False)
db.create_all()
class QueryValidationError(Exception):
pass
class Filter:
supported_operators = ("eq", "ne", "lt", "gt", "le", "ge")
def __init__(self, column, operator, value):
self.column = column
self.operator = operator
self.value = value
self.validate()
def validate(self):
if self.operator not in self.supported_operators:
raise QueryValidationError(
f"operator `{self.operator}` is not one of supported "
f"operators `{self.supported_operators}`"
)
def create_filters(filters):
filters_processed = []
if filters is None:
# No filters given
return filters_processed
elif isinstance(filters, str):
# if only one filter given
filter_split = filters.split(",")
filters_processed.append(
Filter(*filter_split)
)
elif isinstance(filters, list):
# if more than one filter given
try:
filters_processed = [Filter(*_filter.split(",")) for _filter in filters]
except Exception:
raise QueryValidationError("Filter query invalid")
else:
# Programer error
raise TypeError(
f"filters expected to be `str` or list "
f"but was of type `{type(filters)}`"
)
return filters_processed
def filter_query(filters, query, model):
for _filter in filters:
# get our operator
op = getattr(operator, _filter.operator)
# get the column to filter on
column = getattr(model, _filter.column)
# value to filter for
value = _filter.value
# build up a query by adding multiple filters
query = query.filter(op(column, value))
return query
#app.errorhandler(QueryValidationError)
def handle_query_validation_error(err):
return flask.jsonify(dict(
errors=[dict(
title="Invalid filer",
details=err.msg,
status="400")
]
)), 400
#app.route("/books", methods=["GET"])
def all_books():
args = flask.request.args
filters = create_filters(args.get("filter"))
query = Books.query
query = filter_query(filters, query, Books)
result = []
for book in query.all():
result.append(dict(
id=book.id,
title=book.title,
author=book.author,
supplier=book.supplier,
published=str(book.published)
))
return flask.jsonify(result), 200
I hope this answers your question, or gives you some ideas on how to tackle your problem.
I would also recommend looking at serialising and marshalling tools like marshmallow-sqlalchemy which will help you simplify turning models into json and back again. It is also helpful for nested object serialisation which can be a pain if you are returning relationships.
We have 2 approach ideas that we are trying to consider here for the Django REST api.
At the moment, we have implemented 1 filter option:
class APINewsViewSet(viewsets.ModelViewSet):
serializer_class = APINewsSerialiser
def get_queryset(self):
queryset = NewsBackup.objects.all()
source = self.request.query_params.get('source', None)
if source is not None:
queryset = queryset.filter(news_source = source)
return queryset
This achieves: 127.0.0.1/news?source=xxx
Something additional that we would like to do is have other filter options that the user can type in such as 127.0.0.1/news?source=xxx&sourcename=xxx which would then return them the JSON object with only data that has a source_id of xx and source_name of xx.
Is this possible with the Django REST framework? We have tried to add other options within the method:
query_params.get()
You can place the filters in a dictionary and then use the dictionary as the filter. Even better you can create a valid filters that safeguards invalid filters. Something like below:
# {'queryparam': 'db_field'}
valid_filters = {
'source': 'news_source',
'source_name': 'source_name',
}
filters = {valid_filters[key]: value for key, value in self.request.query_params.items() if key in valid_filters.keys()}
# So if the query_params is like this: source=xxx&source_name=xxx
# filters value will look something like below:
{
'news_source': 'xxx',
'source_name': 'xxx',
}
# **filters transforms the filters to: (news_source='xxx', source_name='xxx')
queryset = queryset.filter(**filters)
Or if you don't want to customize it heavily, you can use third party packages like Django Filter
UPDATE
If you want to get the multiple values for example: id=1&id=2, you can use getlist
ids = self.request.query_params.getlist('id')
# ids will have the list of id
# ids = [1, 2]
# you can then query the id
queryset = queryset.filter(id__in=ids)
UPDATE 2
If you want to support multiple filters, you can create a dictionary again:
keys = [
'id',
'source_name',
]
filters = {}
for key in keys:
filter = query_params.getlist(key)
if filter:
filters['{}__in'.format(key)] = filter
queryset = queryset.filter(**filters)
You can add the validation if you want.
I want to write universal View with Django, in this function i want to handle several situations: first when i have url like vkusers3/11122233/1/2/ and also i want it working when 2 or third arguments is missing in url, like: vkusers3/11122233/ or vkusers3/11122233/1/
I cannot find it tutorials how to do that (https://docs.djangoproject.com/en/1.6/topics/http/urls/ etc).
The problem that this became a nightmare when you have more than 5 combinations in url parameters, then you should write 5 different url configurations, 5 times in html template hardcode this pattern.
BUT wait, even more!, what about combinatorics: i want /user/group/sex/smoking/ but also i want /user/group/smoking/ i.e. all users from group who is smoking of both man and woman. So the number is huge.
def list_groupmembers_sex(request, group_id, sex=None, smoking=None):
success = False
if group_id and sex and smoking==None:
vkusers = Vkuser._get_collection().find({"member_of_group": int(group_id), 'sex': int(sex)})# 62740364 81099158
success = True
elif group_id and sex and smoking!=None:
vkusers = Vkuser._get_collection().find({"member_of_group": int(group_id), 'sex': int(sex), 'personal.smoking': int(smoking)})
success = True
else:
vkusers = Vkuser._get_collection().find({'personal.smoking': 1})
ctx = {'vkuser_list': vkusers, 'group_id': group_id, 'sex': sex, 'smoking':smoking, 'success': success}
return render_to_response('blog/vkuser_list.html', ctx, context_instance = RequestContext(request))
In my urls.py:
url(r'^vkusers3/(\d{1,8})/(\d{1})/(\d{1})/$', 'blog.views.list_groupmembers_sex', name='vkuser-list3'),
In my base.html:
<li class="menu-level-1">users</li>
Django 1.6.10, MongoDB 2.7.1, mongoengine
At this point, you should probably bite the bullet and just go for query parameters - vkusers3/?sex=1&smoking=2&group= 11122233. You can drop the parameters completely from the URL and the view definition, and just use request.GET['sex'] etc in the view body.
You don't need so hairy logic. Just populate the search critera with arguments passed to the view like this:
criteria = {}
if group_id:
criteria['member_of_group'] = int(group_id)
if sex:
criteria['sex'] = int(sex)
if smoking:
criteria['personal.smoking'] = int(smoking)
vkusers = Vkuser._get_collection().find(criteria)
And yes, consider to switch to the regular GET parameters like #daniel-roseman suggested. With urls like in your question you can't determine the /user/group/sex/ url from the /user/group/smoking/.
UPDATE: request.GET is a dict-like object so you can use the in expression:
if 'sex' in request.GET:
criteria['sex'] = int(request.GET['sex'])