I have a web application using python on the server (bottle) and VueJS in the client.
one frontend component display's a paginated result of a query which ends in more then 10k records. the records are saved as python dict and sent to the frontend. the size of the result is 5.5MB approximately which with my internet connection ends with more then 2 seconds of waiting.
the api path to handle the request looks like this:
#cmCampaigns.get('/api/campaignManager/campaigns')
#authorize()
def get():
resp = {}
usr = authlayer.current_user
user_id = dao.App().getUserID(usr.username)
resp["campaigns"] = CampaignsData().get_revcontent_campaigns(user_id)
return resp
and the query itself looks like this:
def get_campaigns(self, user_id):
query = "SELECT rc.account_id, rc.campaign_id, " \
"rc.campaign_name, rc.start_date, rc.end_date, " \
"rc.enabled, rc.default_bid, " \
"rc.budget, rc.cost, ctr,rc.country_codes, " \
"'revcontent' AS provider, " \
"replace(JSON_EXTRACT(ac.account_json,'$.client_id'),'\"','') AS account_name, " \
"CASE enabled " \
"WHEN enabled = 'active' THEN 'on' " \
"WHEN enabled = 'inactive' THEN 'off' " \
"END AS enabled_val " \
"FROM prv_campaigns AS rc " \
"INNER JOIN websites AS ws " \
"ON rc.website_id = ws.website_id " \
"INNER JOIN website_users AS wu " \
"ON wu.website_id = ws.website_id " \
"INNER JOIN prv_accounts ac " \
"ON rc.account_id = ac.account_id " \
"WHERE wu.user_id = %s " \
"ORDER BY id DESC"
try:
data = self.db.query(query, user_id)
return data
except Exception as e:
logging.exception(e.message)
return -1, e.message
i tried to optimize the query as possible, but it is still not enogh.
What "best practice" solution's are there to optimize this common logic? compressing the dict before sending it? i wa thinking about pagination but then my filtering and sorting logic is in the client and so lots of data is obviously gone.....any recommendations?? thx
For pagination you can use LIMIT & OFFSET
Optimization ? run "EXPLAIN" before the query and bring the output
Lets take a quick look
SELECT rc.account_id,
rc.campaign_id,
rc.campaign_name,
rc.start_date,
rc.end_date,
rc.enabled,
rc.default_bid,
rc.budget,
rc.cost,
ctr,
rc.country_codes,
'revcontent' AS
provider,
Replace(Json_extract(ac.account_json, '$.client_id'), '"', '') AS
account_name,
CASE enabled
WHEN enabled = 'active' THEN 'on'
WHEN enabled = 'inactive' THEN 'off'
END AS
enabled_val
FROM prv_campaigns AS rc
inner join websites AS ws
ON rc.website_id = ws.website_id
inner join website_users AS wu
ON wu.website_id = ws.website_id
inner join prv_accounts ac
ON rc.account_id = ac.account_id
WHERE wu.user_id = %s
ORDER BY id DESC
enabled column should be numeric or enum, it will save strcmp time
Replace(Json_extract(ac.account_json, '$.client_id'), '"', '') ? extract all account name do your thing in client side
2 inner joins ? might be a design issue
Related
I got the following error occurs when executing sql in Python.
TypeError: not enough arguments for format string
I used executemany function to execute multiple sql statements at once without running a for statement.
And I want to execute the sql statement by receiving the list in "idx".
fruit_idx_list=['123','456','789']
fruit_list = [('123','apple',70,7),('456','strawberry',60,6),('789','banana',100,10)]
sql = "MERGE INTO fruit_test " \
+ "USING DUAL " \
+ "ON idx = {idx}".format(idx = fruit_idx_list) \ #the suspicious part1
+ "WHEN NOT MATCHED THEN " \
+ "INSERT (idx, name, price, vat) VALUES (%s, %s, %s, %s) " \
+ "WHEN MATCHED THEN " \
+ "UPDATE SET idx = %s name = %s, price = %s vat = %s; " #the suspicious part2
cur.executemany(sql, fruit_list)
conn.commit()
I think I have 2 suspicious part to occur sql error.
One is the part which pass fruit_idx_list then execute sql and another is the update statement part.
Please help me to figure out. Thank you in advance
Solution: I had an out of date version of sqlite installed which didn't support RANK(). I fixed this by installing Python3.7.4
I am trying to find the position of a user when they are sorted in descending order based on the sum of two columns. I tested this query in SQLite DB Browser and it returned the correct result, but when I copied it into my Python script it throws a syntax error at one of the "("
I have tried running the code and stopping it just before it executes the query so that I can copy the query verbatim into the db browser. When I do that it works perfectly, but when I execute the query in my script it fails
Here is the plain text query that the string concatenation returns:
SELECT top_rank FROM (
SELECT username, RANK() OVER(ORDER BY summed DESC) AS 'top_rank'
FROM (
SELECT SUM(positive_qc) - SUM(negative_qc) AS 'summed', username
FROM sub_activity
WHERE sub_name IN('cryptocurrency', 'cryptomarkets', 'cryptotechnology', 'blockchain')
GROUP BY username)
) WHERE username = 'someuser'
Here is the code I use to get that query:
select_str = "SELECT top_rank FROM(" + \
"SELECT " + self.KEY2_USERNAME + ", RANK() OVER(ORDER BY summed DESC) AS 'top_rank' " \
"FROM(" \
"SELECT SUM(" + self.KEY2_POSITIVE_QC + ") - SUM(" + self.KEY2_NEGATIVE_QC + ") " \
"AS 'summed', " + self.KEY2_USERNAME + " " \
"FROM " + self.TABLE_SUB_ACTIVITY + " " \
"WHERE " + self.KEY2_SUB_NAME + " IN('" + "', '".join(sub_list) + "') " \
"GROUP BY " + self.KEY2_USERNAME + ")" \
") WHERE " + self.KEY2_USERNAME + " = ?"
The exact error I get when executing it in the script is
sqlite3.OperationalError: near "(": syntax error
Update: I have narrowed down the error to the RANK () OVER(ORDER BY summed DESC) portion of the query
Give this a try... I aliased each subquery. ui and uo.
SELECT top_rank FROM (
SELECT username, RANK() OVER(ORDER BY summed DESC) AS 'top_rank'
FROM (
SELECT SUM(positive_qc) - SUM(negative_qc) AS 'summed', username
FROM sub_activity
WHERE sub_name IN ('cryptocurrency', 'cryptomarkets', 'cryptotechnology', 'blockchain')
GROUP BY username) ui
) uo WHERE username = 'someuser'
I am wondering how to securely parameterize a dynamic mysql query in python. By dynamic, I mean it changes depending on how the if statements evaluate.
I understand how to parameterize a mysql query in python, by using a comma, rather than a percent sign, like as follows.
c.execute("SELECT * FROM foo WHERE bar = %s AND baz = %s", (param1, param2))
Here is an example of a 'dynamic query'. I am looking to find a more secure way than using the percent sign.
def queryPhotos(self, added_from, added, added_to):
sql = "select * from photos where 1=1 "
if added_from is not None:
sql = sql + "and added >= '%s' " % added_from
if added is not None:
sql = sql + "and added = '%s' " % added
if added_to is not None:
sql = sql + "and added <= '%s' " % added_to
Thank you for your insight.
thanks to #Nullman I came to an answer.
def queryPhotos(self, added_from, added, added_to):
vars = []
sql = "select * from photos where 1=1 "
if added_from is not None:
sql = sql + "and added >= %s "
vars.append(added_from)
if added is not None:
sql = sql + "and added = %s "
vars.append(added)
if added_to is not None:
sql = sql + "and added <= %s "
vars.append(added_to)
vars = tuple(vars)
results = c.execute(sql, vars)
I'm trying to build a query in python, I'm looking for an elegant way to append a WHERE condition in the middle of a string:
def get_raw_queryset(frequency=None, where_condition=None):
qs = "SELECT id, COUNT(*) AS count FROM user_transaction_log " \
# I WANT TO APPEND A WHERE ONLY IF not None
if where_condition:
"WHERE .... = 1" \
"GROUP BY type , strftime('{0}', datetime) ORDER BY id" \
.format(frequency)
return qs
This will work as long as you can safely evaluate the WHERE string even if where_condition is not a string:
"SELECT ..." + bool(where_condition) * ("WHERE ...") + "GROUP ..."
I hope you're being extremely careful to avoid SQL injection.
I have a script that pulls data from a weather API & save this info into a MySQL database on localhost. I want to have the UPDATE script prevent any SQL injection but the following doesn't seem to run the UPDATE at all. There isnt an error just the query doesn't seem to have been executed when I check the database.
Can anyone suggest the problem? I am using the mysql.connector import/plugin
def save_to_database(self, uid):
sql = "UPDATE weather_data " \
"SET temperature=%s, temperature_feels=%s, humidity=%s, precipitation=%s, weather_status=%s " \
"WHERE UID =%s"
temperature = self.weather_data['temperature']
temperature_feels = self.weather_data['temperature_feels']
humidity = self.weather_data['humidity']
precipitation = self.weather_data['precipitation']
weather_status = self.weather_data['type']
print(sql)
c = self._db.cursor()
c.execute(sql, (temperature, temperature_feels, humidity, precipitation, weather_status, uid))
UPDATE
The following works fine - but isn't 'safe'
def save_weather_forecast(self, uid):
print(self.weather_data);
sql = "UPDATE weather_data SET temperature = "+ str(self.weather_data['temperature']) + ", " \
+"temperature_feels = "+ str(self.weather_data['temperature_feels']) +", " \
+"humidity = "+ str(self.weather_data['humidity']) +", " \
+"weather_status = '"+ str(self.weather_data['type']) +"', " \
+"precipitation = "+ str(self.weather_data['precipitation']) +"" \
+" WHERE UID = '"+ str(uid) +"'"
print(sql)
c = self._db.cursor()
c.execute(sql)
c.close()
The Python DB API standard explicitly turns off auto commit which means you have to commit any transactions manually otherwise they are not effected at the database.
Committing is done at connection, so you need to add:
self._db.commit()
After the c.execute() line.