Develop Reference

Change f-string local variable from a while loop?

I want to query a sqlite database using a f-string for the query, it is supposed to return a small description given a specific id that i stored in a local variable var. However some fields will be empty for some values of the variable. I'm specially interested in the value 0 has it will always return an empty description and is an isolated case that needs to be addressed differently from other values.
I want to iterate through a range until I find a not empty description field.
Here is an example:
var = str(self.some_global_variable) # In this case consider var = '0'
query = f'my sql query goes here WHERE this_id={var}'
description = self.function_to_fetch_db(query)
while not description and var == '0':
for i in range (1, 31):
var = str(i)
description = self.function_to_fetch_db(query)
print(description, var)
print(query)
The output of this will be something like:
[] 1
my sql query goes here WHERE this_id=0
[] 2
my sql query goes here WHERE this_id=0
[] 3
my sql query goes here WHERE this_id=0
.
.
.
The local variable is updated but the query always keeps the original value from outside the while loop.
I also tried an if...else instead of the while loop but the result is the same.
I don't think the SQLite part of the problem is relevant, it's just to illustrate my specific case, the query works for other values. I'm just having trouble to figure out this local variable and f-string relationship.

There are two answers to your questions: the first one is formal, the second one is correct.
The formal answer: the string is computed once before the loop in your case. If you want it to alter it for every value, move it inside:
for i in range(31):
query = f'my sql query goes here WHERE this_id={i}'
description = self.function_to_fetch_db(query)
The correct answer: use parameterized queries instead, that will look like:
conn.execute('my sql query goes here WHERE this_id=?', (i,))
(The substitution syntax may vary depending on your database / driver.)

you could use a named variable in a string:
var = str(self.some_global_variable) # In this case consider var = '0'
fquery = 'my sql query goes here WHERE this_id={var}'
description = self.function_to_fetch_db(fquery.format(var=var))
while not description and var == '0':
for i in range(1, 31):
query = fquery.format(var=i)
description = self.function_to_fetch_db(query)
print(description, i)
print(query)

Django append <queryset> for searching each words in a sentece using filter

I want to search each word in a sentence and put that in result['post'] dictionary
and of course this code only looks for last query
queries = querystring.split()
for query in queries:
results['posts'] = Post.objects.filter(text__icontains=query)
i tried append,extend and a lot of things but it didn't work as expected.
Edit Update:
I also tried
count = {}
results = {}
post_results = []
queries = querystring.split()
for query in queries:
post_results.append(Post.objects.filter(text__icontains=query))
results['posts'] = post_results
count['posts'] = results['posts'].count()
But this leads to another error count() takes exactly one argument (0 given)
Error Image
Also Post.objects.filter(text__icontains=query) this line returns a queryset value and Queryset type of django doesn't seem to have append. Is there it's own append like feature ? If so then that wouldn't change it's queryset type to list. Using temp variable like post_results = [] changes to list.
Can we change that list back to queryset? If we can then that would work.

I guess the reason is results['posts'] value is overwritten per every for loop..
How about changing code like
queries = querystring.split()
tmp = []
for query in queries:
tmp.append(Post.objects.filter(text__icontains=query))
results['posts'] = tmp

How about this:
results = {'posts': []}
count = {'posts': []}
for query in querystring.split():
qs = Post.objects.filter(text__icontains=query)
results['posts'].append(qs)
counts['posts'].append(qs.count())
Update after you answered your own question:
Oh, I was under the assumption you wanted a list of queries of each word with their respective result counts. If you only wanted to combine each string of the search term into an OR query you could use Q objects.
results = {}
count = {'posts': []}
from django import models
qobjects = models.Q()
for query in querystring.split():
qobjects |= models.Q( ('text__icontains',query) )
results['posts'] = Post.objects.filter(qobjects)
counts['posts'] = results['posts'].count()

instead of making multiple queries we can use Q object, here is the simple example
q = 'query string'
keywords = []
prevword = ""
for word in q.split(' '):
prevword = prevword + word + " "
keywords.append(prevword.lower())
query = Q(title__icontains=q)
for keyword in keywords:
query.add(Q(title__icontains=keyword), Q.OR)
services = Service.objects.distinct().filter(query).all()

Thanks all of you I tried everything mentioned at first and finally got a solution
I did a crazy thing at first.
I wanted to define an empty queryset so for results['posts'] so I did a bad but a working way results['posts']=Post.objects.filter(text_icontains='sth_that_will_never_b_true')
and I used
Django Merge QuerySets
and solved the problem and then later studied genuine way to define empty QuerySet as
django.db.models.query.QuerySet.none
And gladly final code is
count = {}
results = {}
results['posts']=Post.objects.none() # empty QuerySet
queries = querystring.split()
for query in queries:
results['posts'] = results['posts'] | Post.objects.filter(
text__icontains=query)
count['posts'] = results['posts'].count()
Now everything works as expected.

saving search results as text instead of list

I am using Django 1.8 and currently am working on a Blog application. When i search for tweets( just a name instead of posts) , i want to save the search results obtained after querying the database, as text instead of list. My view function is as below:
def search(request):
query = request.GET.get('q','')
if query:
qset = (
Q(text__icontains=query)
#Q(hashes__icontains=query)
#Q(artist__icontains=query)
)
results = Tweet.objects.filter(qset).distinct()
else:
results = []
number_of_results = len(results)
search_item = query
returned_items = []
for res in results:
text = res.text
returned_items.append(text)
returns = returned_items[:]
search = Search(search_item=search_item,returns=returns)
search.save()
context = {'query':query,'results':results,'number_of_results':number_of_results,'title':'Search results for '+request.GET.get('q','')}
return render_to_response("tweets/search.html",context,context_instance=RequestContext(request))
also, the snapshot of my search table in the database is as shown below:
Please help me out friends.

you should join the returned list using the comma separted values. This will return the string.
returns = ', '.join(returned_items)

This piece of code is setting returns to a list:
returns = returned_items[:]
If you want to access the first string, set it to returned_items[0]. If you want to join all strings in the list, use join()
returns = "".join(returned_items)

How to write multi column in clause with sqlalchemy

Please suggest is there way to write query multi-column in clause using SQLAlchemy?
Here is example of the actual query:
SELECT url FROM pages WHERE (url_crc, url) IN ((2752937066, 'http://members.aye.net/~gharris/blog/'), (3799762538, 'http://www.coxandforkum.com/'));
I have a table that has two columns primary key and I'm hoping to avoid adding one more key just to be used as an index.
PS I'm using mysql DB.
Update: This query will be used for batch processing - so I would need to put few hundreds pairs into the in clause. With IN clause approach I hope to know fixed limit of how many pairs I can stick into one query. Like Oracle has 1000 enum limit by default.
Using AND/OR combination might be limited by the length of the query in chars. Which would be variable and less predictable.

Assuming that you have your model defined in Page, here's an example using tuple_:
keys = [
(2752937066, 'http://members.aye.net/~gharris/blog/'),
(3799762538, 'http://www.coxandforkum.com/')
]
select([
Page.url
]).select_from(
Page
).where(
tuple_(Page.url_crc, Page.url).in_(keys)
)
Or, using the query API:
session.query(Page.url).filter(tuple_(Page.url_crc, Page.url).in_(keys))

I do not think this is currently possible in sqlalchemy, and not all RDMBS support this.
You can always transform this to a OR(AND...) condition though:
filter_rows = [
(2752937066, 'http://members.aye.net/~gharris/blog/'),
(3799762538, 'http://www.coxandforkum.com/'),
]
qry = session.query(Page)
qry = qry.filter(or_(*(and_(Page.url_crc == crc, Page.url == url) for crc, url in filter_rows)))
print qry
should produce something like (for SQLite):
SELECT pages.id AS pages_id, pages.url_crc AS pages_url_crc, pages.url AS pages_url
FROM pages
WHERE pages.url_crc = ? AND pages.url = ? OR pages.url_crc = ? AND pages.url = ?
-- (2752937066L, 'http://members.aye.net/~gharris/blog/', 3799762538L, 'http://www.coxandforkum.com/')
Alternatively, you can combine two columns into just one:
filter_rows = [
(2752937066, 'http://members.aye.net/~gharris/blog/'),
(3799762538, 'http://www.coxandforkum.com/'),
]
qry = session.query(Page)
qry = qry.filter((func.cast(Page.url_crc, String) + '|' + Page.url).in_(["{}|{}".format(*_frow) for _frow in filter_rows]))
print qry
which produces the below (for SQLite), so you can use IN:
SELECT pages.id AS pages_id, pages.url_crc AS pages_url_crc, pages.url AS pages_url
FROM pages
WHERE (CAST(pages.url_crc AS VARCHAR) || ? || pages.url) IN (?, ?)
-- ('|', '2752937066|http://members.aye.net/~gharris/blog/', '3799762538|http://www.coxandforkum.com/')

I ended up using the test() based solution: generated "(a,b) in ((:a1, :b1), (:a2,:b2), ...)" with named bind vars and generating dictionary with bind vars' values.
params = {}
for counter, r in enumerate(records):
a_param = "a%s" % counter
params[a_param] = r['a']
b_param = "b%s" % counter
params[b_param] = r['b']
pair_text = "(:%s,:%s)" % (a_param, b_param)
enum_pairs.append(pair_text)
multicol_in_enumeration = ','.join(enum_pairs)
multicol_in_clause = text(
" (a,b) in (" + multicol_in_enumeration + ")")
q = session.query(Table.id, Table.a,
Table.b).filter(multicol_in_clause).params(params)
Another option I thought about using mysql upserts but this would make whole included even less portable for the other db engine then using multicolumn in clause.
Update SQLAlchemy has sqlalchemy.sql.expression.tuple_(*clauses, **kw) construct that can be used for the same purpose. (I haven't tried it yet)

How to use ResultSet in PyES

I'm using PyES to use ElasticSearch in Python.
Typically, I build my queries in the following format:
# Create connection to server.
conn = ES('127.0.0.1:9200')
# Create a filter to select documents with 'stuff' in the title.
myFilter = TermFilter("title", "stuff")
# Create query.
q = FilteredQuery(MatchAllQuery(), myFilter).search()
# Execute the query.
results = conn.search(query=q, indices=['my-index'])
print type(results)
# > <class 'pyes.es.ResultSet'>
And this works perfectly. My problem begins when the query returns a large list of documents.
Converting the results to a list of dictionaries is computationally demanding, so I'm trying to return the query results already in a dictionary. I came across with this documentation:
http://pyes.readthedocs.org/en/latest/faq.html#id3
http://pyes.readthedocs.org/en/latest/references/pyes.es.html#pyes.es.ResultSet
https://github.com/aparo/pyes/blob/master/pyes/es.py (line 1304)
But I can't figure out what exactly I'm supposed to do.
Based on the previous links, I've tried this:
from pyes import *
from pyes.query import *
from pyes.es import ResultSet
from pyes.connection import connect
# Create connection to server.
c = connect(servers=['127.0.0.1:9200'])
# Create a filter to select documents with 'stuff' in the title.
myFilter = TermFilter("title", "stuff")
# Create query / Search object.
q = FilteredQuery(MatchAllQuery(), myFilter).search()
# (How to) create the model ?
mymodel = lambda x, y: y
# Execute the query.
# class pyes.es.ResultSet(connection, search, indices=None, doc_types=None,
# query_params=None, auto_fix_keys=False, auto_clean_highlight=False, model=None)
resSet = ResultSet(connection=c, search=q, indices=['my-index'], model=mymodel)
# > resSet = ResultSet(connection=c, search=q, indices=['my-index'], model=mymodel)
# > TypeError: __init__() got an unexpected keyword argument 'search'
Anyone was able to get a dict from the ResultSet?
Any good sugestion to efficiently convert the ResultSet to a (list of) dictionary will be appreciated too.

I tried too many ways directly to cast ResultSet into dict but got nothing. The best way I recently use is appending ResultSet items into another list or dict. ResultSet covers every single item in itself as a dict.
Here is how I use:
#create a response dictionary
response = {"status_code": 200, "message": "Successful", "content": []}
#set restul set to content of response
response["content"] = [result for result in resultset]
#return a json object
return json.dumps(response)

Its not that complicated: just iterate over the result set. For example with a for loop:
for item in results:
print item