I use MySql. When I call the same procedure using sql and python I get different responses:
In MySql it works well but when I call it in python I get another response
(1690, "BIGINT UNSIGNED value is out of range in '(`shop`.`t`.`lvl` - `shop`.`tree`.`lvl`)'")
Procedure:
delimiter $$
CREATE procedure daily_users (us_id bigint)
BEGIN
set #us_id = us_id;
set #u_lvl = (select lvl from users where user_id = #us_id);
set #children_first_lvl = (select count(1)
from tree t
join users u
on t.user_id = u.user_id
join (select * from tree where user_id = #us_id) sub
on (sub.lft < t.lft and sub.rgt > t.rgt)
where (t.lvl - sub.lvl) = 1 and u.lvl > 0);
set #all_children = (select count(1)
from tree t
join users u
on t.user_id = u.user_id
join (select * from tree where user_id = #us_id ) sub
on (sub.lft < t.lft and sub.rgt > t.rgt)
where (t.lvl - sub.lvl) <= (select lvl from users where user_id = #us_id)
and u.lvl > 0);
set #new_first_lvl = (select count(1)
from tree t
join users u
on t.user_id = u.user_id
join (select * from tree where user_id = #us_id ) sub
on (sub.lft < t.lft and sub.rgt > t.rgt) where u.when_joined = date_format(date_sub(now(), interval 1 day), '%y-%m-%d' ));
set #earned_yesterday = (select ifnull(sum(value_of_payment),0) from outcomming_payments
where date_format(time_of_act, '%y-%m-%d') = date_format(date_sub(now(), interval 1 day), '%y-%m-%d')
and user_id = #us_id
and (is_it_end = 1 and is_it_success = 1));
set #erned_all = (select ifnull(sum(value_of_payment),0) from outcomming_payments
where user_id = #us_id
and (is_it_end = 1 and is_it_success = 1));
set #balance = (select balance from users where user_id = #us_id );
set #children_nex_lvl = (select count(1)
from tree t
join users u
on t.user_id = u.user_id
join (select * from tree where user_id = #us_id ) sub
on (sub.lft < t.lft and sub.rgt > t.rgt)
where (t.lvl - sub.lvl) = (select lvl from users where user_id = #us_id ) + 1 );
insert into daily_response(user_id, user_lvl, children_first_lvl, all_children, new_first_lvl, earned_yesterday, erned_all, balance, children_nex_lvl)
value (#us_id , #u_lvl, #children_first_lvl, #all_children, #new_first_lvl, #earned_yesterday, #erned_all, #balance, #children_nex_lvl);
end;$$
delimiter
How I call that in python:
def users_response_db(self,data):
self.cur.execute(f"call daily_users('{data}')")
self.conn.commit()
my_execute = f""";
select * from daily_response
where user_id = '{data}'
order by id_daily desc
limit 1;"""
self.cur.execute(my_execute)
return self.cur.fetchone()
I already tried to use SET sql_mode = 'NO_UNSIGNED_SUBTRACTION';
The problem was in this part of code
(select * from tree where user_id = #us_id )
I don't need to use all from this subtable.
I solved that when I replaced * with required columns. In my solution it looks like that:
(select lft,rgt,lvl from tree where user_id = #us_id )
I have one query which is like:
q = f"""
select distinct
s.user_id
,d.prime_tagging_by_issuer_and_product
from sms_callback_events s
inner join {table} d
on s.user_id = d.user_id
where event_name = 'event_1'
and d.user_id is not null and d.user_id != ''
"""
with redshift_direct() as conn:
sms_df = pd.read_sql(q, conn)
Now I want to discard only those user_ids which were selected in this query and pass rest of the user Id's to my another query which is also similar
q = f"""
select distinct
s.user_id
,d.prime_tagging_by_issuer_and_product
from flash_events s
inner join {table} d
on s.user_id = d.user_id
where event_name = 'event_1'
and d.user_id is not null and d.user_id != ''
"""
with redshift_direct() as conn:
flash_df = pd.read_sql(q, conn)
here {table} is nothing but a dataframe of user_ids and plus some other columns.
How do I achieve this? How do I put a condition where I dont pass the user ids from sms_df to my next query?
Any help is appreciated.
I have Python code that is connected to the SQL Server for this I am using PYODBC with wildcard in order to accept user input, I am trying to perform a Select query
I want to return the the count of records based on the user input where he can input on the displayed fields or leave it empty and then apply the select query based on the requested search.
The problem is that when I try the query on the SQL Server Management Studio it returns the correct result with records that includes Null
But when I tried in the python script it returns just the records that doesn't include Null values
NOTE: the ANSI_NULLS for this database is ON;
In the below query I am trying to return the count of the records where the gender ='F' and status_type = 'Expired'
In SSMS:
select count(*)
from card
where gender = 'F'
and status_type = 3
group by status_type
returns result = 7 ==> correct answer.
In Python Script:
sql = ('''select count(*)
, c3.status
from card C
LEFT JOIN status_type c3 ON c3.ID = c.status_type
where
(c.nickname LIKE CONCAT('%', ?, '%') OR ? IS NULL)
and
c.gender = ?
and
c3.status = ?
Group by c3.status
''')
param1 = nickname_search
param2 = radio_gender
param3 = radio_status
rows = cursor.execute(sql,[
param1,param1,
param2,
param3
]).fetchall()
the return result = 2 ==> the expected result = 7
where 2 is the records that has nickname not Null
My question how to return the values that includes also NULL values in the nickname column?
A rather simple question but for which we surprisingly didn't found a solution.
Here is my current code, for executing a simple SQL query on a PostgreSQL database from Python 3.6.9 using psycopg2 ('2.9.1 (dt dec pq3 ext lo64)'):
import psycopg2
myid = 100
fields = ('p.id', 'p.name', 'p.type', 'p.price', 'p.warehouse', 'p.location', )
sql_query = ("SELECT " + ', '.join(fields) + " FROM product p "
"INNER JOIN owner o ON p.id = o.product_id "
"WHERE p.id = {} AND (o.dateof_purchase IS NOT NULL "
"OR o.state = 'checked_out' );"
).format(myid)
try:
with psycopg2.connect(**DB_PARAMS) as conn:
with conn.cursor(cursor_factory=DictCursor) as curs:
curs.execute(sql_query, )
row = curs.fetchone()
except psycopg2.Error as error:
raise ValueError(f"ERR: something went wrong with the query :\n{sql_query}") from None
We're more and more thinking that this is... not good. (awfully bad to be honest).
Therefore, we're trying to use a modern f-string notation:
sql_query = (f"""SELECT {fields} FROM product p
INNER JOIN owner o ON p.id = o.product_id
WHERE p.id = {myid} AND (o.dateof_purchase IS NOT NULL
OR o.state = 'checked_out' );""")
But then, the query looks like:
SELECT ('p.id', 'p.name', 'p.type', 'p.price', 'p.warehouse', 'p.location', ) FROM ...;
which is not valid in PSQL because 1. of the brackets, and 2. of the single quoted column names.
We'd like to figure out a way to get rid of these.
In between, we went back to the doc and remembered this:
https://www.psycopg.org/docs/usage.html
Ooops! So we refactored it this way:
sql_query = (f"""SELECT %s FROM product p
INNER JOIN owner o ON p.id = o.product_id
WHERE p.id = %s AND (o.dateof_purchase IS NOT NULL
OR o.state = 'checked_out' );""")
try:
with psycopg2.connect(**DB_PARAMS) as conn:
with conn.cursor(cursor_factory=DictCursor) as curs:
# passing a tuple as it only accept one more argument after the query!
curs.execute(sql_query, (fields, myid))
row = curs.fetchone()
and mogrify() says:
"SELECT ('p.id', 'p.name', 'p.type', 'p.price', 'p.warehouse', 'p.location', ) FROM ...;"
here again, the brackets and the single quotes are causing troubles, but no error is actually raised.
The only thing is that row evaluates to this strange result:
['('p.id', 'p.name', 'p.type', 'p.price', 'p.warehouse', 'p.location', )']
So, how could we cleverly and dynamically build a psycopg2 query using a list of parameters for column names without neglecting the security?
(A trick could be to fetch all columns and filter them out after... but there are too many columns, some with quiet large amount of data that we don't need, that's why we want to run a query using a precisely defined selection of columns, which may get dynamically extended by some function, otherwise we would have hard-coded these column names of course).
OS: Ubuntu 18.04
PostgreSQL: 13.3 (Debian 13.3-1.pgdg100+1)
The '%s' insertion will try to turn every argument into an SQL string, as #AdamKG pointed out. Instead, you can use the psycopg2.sql module will allow you to insert identifiers into queries, not just strings:
from psycopg2 import sql
fields = ('id', 'name', 'type', 'price', 'warehouse', 'location', )
sql_query = sql.SQL(
"""SELECT {} FROM product p
INNER JOIN owner o ON p.id = o.product_id
WHERE p.id = %s AND (o.dateof_purchase IS NOT NULL
OR o.state = 'checked_out' );""")
try:
with psycopg2.connect(**DB_PARAMS) as conn:
with conn.cursor(cursor_factory=DictCursor) as curs:
# passing a tuple as it only accept one more argument after the query!
curs.execute(sql_query.format(*[sql.Identifier(field) for field in fields]), (*fields, myid))
row = curs.fetchone()
I finally found a solution. It makes use of map to use a list or a tuple of column names and sql.Literal to use a given id, this is maybe cleaner:
conn = psycopg2.connect(**DB_PARAMS)
myid = 100
# using the simple column identifiers
fields_1 = ('id', 'name', 'type', 'price', 'warehouse', 'location',)
# using the dot notation with the table alias 'p' as the prefix:
fields_2 = ('p.id', 'p.name', 'p.type', 'p.price', 'p.warehouse', 'p.location',)
sql_query_1 = sql.SQL("""
SELECT {f} FROM product p
INNER JOIN owner o ON p.id = o.product_id
WHERE p.id = {j} AND (o.dateof_purchase IS NOT NULL
OR o.state = 'checked_out' );"""
).format(
f = sql.SQL(',').join(map(sql.Identifier, fields_1)),
j = sql.Literal(myid)
)
sql_query_2 = sql.SQL("""
SELECT {f} FROM product p
INNER JOIN owner o ON p.id = o.product_id
WHERE p.id = {j} AND (o.dateof_purchase IS NOT NULL
OR o.state = 'checked_out' );"""
).format(
f = sql.SQL(',').join(map(sql.SQL, fields_2)), # use sql.SQL!
j = sql.Literal(myid)
)
sql_query_2b = sql.SQL("""
SELECT {f} FROM product p
INNER JOIN owner o ON p.id = o.product_id
WHERE p.id = {j} AND (o.dateof_purchase IS NOT NULL
OR o.state = 'checked_out' );"""
).format(
f = sql.SQL(',').join(map(sql.Identifier, fields_2)), # DON'T use sql.Identifier!
j = sql.Literal(myid)
)
# VALID SQL QUERY:
print(sql_query_1.as_string(conn))
# will print:
# SELECT "id","name","type","price","warehouse","location" FROM product p
# INNER JOIN owner o ON p.id = o.product_id
# WHERE p.id = 100 AND (o.dateof_purchase IS NOT NULL
# OR o.state = 'checked_out' );
# VALID SQL QUERY:
print(sql_query_2.as_string(conn))
# will print:
# SELECT p.id,p.name,p.type,p.price,p.warehouse,p.location FROM product p
# INNER JOIN owner o ON p.id = o.product_id
# WHERE p.id = 100 AND (o.dateof_purchase IS NOT NULL
# OR o.state = 'checked_out' );
# /!\ INVALID SQL QUERY /!\:
print(sql_query_2b.as_string(conn))
# will print:
# SELECT "p.id","p.name","p.type","p.price","p.warehouse","p.location" FROM product p
# INNER JOIN owner o ON p.id = o.product_id
# WHERE p.id = 100 AND (o.dateof_purchase IS NOT NULL
# OR o.state = 'checked_out' );
But because of that:
simple columns names are evaluated correctly when in double quotes, eg. id is equivalent to "id", name is equivalent to "name" for PostgreSQL,
column name, when prefixed with the dot notation using the table alias or identifier, e.g. p.id or product.id instead of just id or "id" will miserably fail with the following error:
UndefinedColumn: column "p.id" does not exist
LINE 1: SELECT "p.id","p.type","p.price","p.warehouse","p.location",...
^
HINT: Perhaps you meant to reference the column "p.id".
Suppose I have this multiple-process Python-MySql query:
self.calculateLeadTime = ("""SET #lastDate = (SELECT sessionDate FROM stock
WHERE product = (%s)
ORDER BY stocksessionID DESC LIMIT 1);
SET #secondLastDate = (SELECT sessionDate FROM stock WHERE product = (%s)
ORDER BY stocksessionID DESC LIMIT 1, 1);
SET #leadTime = (SELECT DATEDIFF(#lastDate, #secondLastDate));
SET #lastStockSessionID = (SELECT stocksessionID
FROM stock WHERE product = (%s) ORDER BY stocksessionID DESC LIMIT 1);
UPDATE stock SET leadTime = (#leadTime)
WHERE stocksessionID = #lastStockSessionID;""", (self.productID.get(), self.productID.get(), self.productID.get()))
self.query = self.cur.execute(self.calculateLeadTime, multi=True)
for self.cur in self.results:
print('cursor:', self.cur)
if self.cur.with_rows:
print('result:', self.cur.fetchall())
self.cn.commit()
I am subject to the error:
stmt = operation.encode(self._connection.python_charset)
AttributeError: 'tuple' object has no attribute 'encode'
I have read the MySql Python documentation regarding multi=True when executing multiple SQL statements via Python. However, my implementation does not work. Any ideas?
Currently, you are passing a tuple as first argument in the cursor.execute call when it expects a single scalar query string in first argument and tuple/list of parameters in second argument:
self.calculateLeadTime = """SET #lastDate = (SELECT sessionDate
FROM stock
WHERE product = (%s)
ORDER BY stocksessionID DESC LIMIT 1);
SET #secondLastDate = (SELECT sessionDate
FROM stock WHERE product = (%s)
ORDER BY stocksessionID DESC LIMIT 1, 1);
SET #leadTime = (SELECT DATEDIFF(#lastDate, #secondLastDate));
SET #lastStockSessionID = (SELECT stocksessionID
FROM stock WHERE product = (%s)
ORDER BY stocksessionID DESC LIMIT 1);
UPDATE stock
SET leadTime = (#leadTime)
WHERE stocksessionID = #lastStockSessionID;
"""
self.query = self.cur.execute(self.calculateLeadTime,
params=(self.productID.get(), self.productID.get(), self.productID.get()),
multi=True)
By the way, MySQL supports JOIN in UPDATE statements for a single statement. Run below with two parameters:
UPDATE stock s
INNER JOIN
(SELECT product, MAX(stocksessionID) AS MaxID
FROM stock
WHERE product = %s
GROUP BY product
) agg_id
ON s.stocksessionID = agg_id.MaxID
INNER JOIN
(SELECT product, MAX(sessionDate) As MaxDate
FROM stock
GROUP BY product) max_dt
ON max_dt.product = s.product
INNER JOIN
(SELECT product, MAX(sessionDate) As SecMaxDate
FROM stock
WHERE sessionDate < ( SELECT MAX(sessionDate)
FROM stock
WHERE product = %s )
GROUP BY product
) sec_max_dt
ON max_dt.product = max_dt.product
SET leadTime = DATEDIFF(MaxDate, SecMaxDate);
I am unsure, if this will work or counts even as multiple queries, because Multi = true is for multiple Resultsets from SELECTS.
You must add teh data for the %s
self.query = self.cur.execute(self.calculateLeadTime,('product','product','product') , multi=True)