i found this code in stackoverflow. suits my requirements. i'm using an mysql db that consists of id, date, gender, name and surname. i have made few changes to the code so that i can obtain the date and other info between the start date and the end date.
import mysql
import mysql.connector
import command
import datetime
def findarbs(startdate, enddate):
con = mysql.connector.connect(user='root', password='root',
host='localhost',port='3306',database='testdb')
cur = con.cursor()
command = ("SELECT DISTINCT id, date FROM babyrop", WHERE 'date1'
> ('%s') AND 'date2' < ('%s'), ORDER BY date asc % (startdate,
enddate))
print ('command')
cursor.execute(command)
result = cur.fetchone()
print (result)
while result is not None:
print (result[1])
result = cursor.fetchone()
cur = con.cursor()
cur.execute('select id, date weight from babyrop')
data = cursor.fetchall()
cur.close()
con.close()
when i tried executing it i'm getting an error "invalid syntax" near date. let me know what corrections to are required.
You are terminating the string way too early. You also have invalid commas in the query.
command = ("SELECT DISTINCT id, date FROM babyrop", WHERE 'date1'
> ('%s') AND 'date2' < ('%s'), ORDER BY date asc % (startdate,
enddate))
"should" be
command = ("SELECT DISTINCT id, date FROM babyrop WHERE 'date1'
> ('%s') AND 'date2' < ('%s') ORDER BY date asc" % (startdate,
enddate))
I wrote "should" because you should not use string concatenation in queries since it exposes your code to SQL injection. You should use parametrized queries instead. It also recommended to use triple quotes for query strings so they can be broken apart nicely:
command = """SELECT DISTINCT id, date
FROM babyrop
WHERE 'date1' > ('%s') AND 'date2' < ('%s')
ORDER BY date asc"""
cursor.execute(command, (startdate, enddate))
Also, I'm pretty sure that you don't need the single quotes (') around the columns names and the placeholders in the where clause but that might be a MySQL thing.
Related
I have a function in my main Python file which gets called by main() and executes a SQL Merge (Upsert) statement using pyodbc from a different file & function. Concretely, the SQL statement traverses a source table with transaction details by distinct transaction datetimes and merges customers into a separate target table. The function that executes the statement and the function that returns the completed SQL statement are attached below.
When I run my Python script, it doesn't work as expected and inserts only around 70 rows (sometimes 69, 71, or 72) into the target customer table. However, when I use an identical SQL statement and execute it in the Microsoft SQL Server Management Studio console (attached below), it works fine and inserts 4302 rows (as expected).
I'm not sure what's wrong.. Would really appreciate any help!
SQL Statement Executor in Python main file:
def stage_to_dim(connection, cursor, now):
log(f"Filling {cfg.dim_customer} and {cfg.dim_product}")
try:
cursor.execute(sql_statements.stage_to_dim_statement(now))
connection.commit()
except Exception as e:
log(f"Error in stage_to_dim: {e}" )
sys.exit(1)
log("Stage2Dimensions complete.")
SQL Statement formulator in Python:
def stage_to_dim_statement(now):
return f"""
DECLARE #dates table(id INT IDENTITY(1,1), date DATETIME)
INSERT INTO #dates (date)
SELECT DISTINCT TransactionDateTime FROM {cfg.stage_table} ORDER BY TransactionDateTime;
DECLARE #i INT;
DECLARE #cnt INT;
DECLARE #date DATETIME;
SELECT #i = MIN(id) - 1, #cnt = MAX(id) FROM #dates;
WHILE #i < #cnt
BEGIN
SET #i = #i + 1
SET #date = (SELECT date FROM #dates WHERE id = #i)
MERGE {cfg.dim_customer} AS Target
USING (SELECT * FROM {cfg.stage_table} WHERE TransactionDateTime = #date) AS Source
ON Target.CustomerCodeNK = Source.CustomerID
WHEN MATCHED THEN
UPDATE SET Target.AquiredDate = Source.AcquisitionDate, Target.AquiredSource = Source.AcquisitionSource,
Target.ZipCode = Source.Zipcode, Target.LoadDate = CONVERT(DATETIME, '{now}'), Target.LoadSource = '{cfg.ingest_file_path}'
WHEN NOT MATCHED THEN
INSERT (CustomerCodeNK, AquiredDate, AquiredSource, ZipCode, LoadDate, LoadSource) VALUES (Source.CustomerID,
Source.AcquisitionDate, Source.AcquisitionSource, Source.Zipcode, CONVERT(DATETIME,'{now}'), '{cfg.ingest_file_path}');
END
"""
SQL Statement from MS SQL Server Console:
DECLARE #dates table(id INT IDENTITY(1,1), date DATETIME)
INSERT INTO #dates (date)
SELECT DISTINCT TransactionDateTime FROM dbo.STG_CustomerTransactions ORDER BY TransactionDateTime;
DECLARE #i INT;
DECLARE #cnt INT;
DECLARE #date DATETIME;
SELECT #i = MIN(id) - 1, #cnt = MAX(id) FROM #dates;
WHILE #i < #cnt
BEGIN
SET #i = #i + 1
SET #date = (SELECT date FROM #dates WHERE id = #i)
MERGE dbo.DIM_CustomerDup AS Target
USING (SELECT * FROM dbo.STG_CustomerTransactions WHERE TransactionDateTime = #date) AS Source
ON Target.CustomerCodeNK = Source.CustomerID
WHEN MATCHED THEN
UPDATE SET Target.AquiredDate = Source.AcquisitionDate, Target.AquiredSource = Source.AcquisitionSource,
Target.ZipCode = Source.Zipcode, Target.LoadDate = CONVERT(DATETIME,'6/30/2022 11:53:05'), Target.LoadSource = '../csv/cleaned_original_data.csv'
WHEN NOT MATCHED THEN
INSERT (CustomerCodeNK, AquiredDate, AquiredSource, ZipCode, LoadDate, LoadSource) VALUES (Source.CustomerID, Source.AcquisitionDate,
Source.AcquisitionSource, Source.Zipcode, CONVERT(DATETIME,'6/30/2022 11:53:05'), '../csv/cleaned_original_data.csv');
END
If you think carefully about what your final result ends up, you are actually just taking the latest row (by date) for each customer. So you can just filter the source using a standard row-number approach.
Exactly why the Python code didn't work properly is unclear, but the below query might work better. You are also doing SQL injection, which is dangerous and can also cause correctness problems.
Also you should always use a non-ambiguous date format.
MERGE dbo.DIM_CustomerDup AS t
USING (
SELECT *
FROM (
SELECT *,
rn = ROW_NUMBER() OVER (PARTITION BY s.CustomerID ORDER BY s.TransactionDateTime DESC)
FROM dbo.STG_CustomerTransactions s
) AS s
WHERE s.rn = 1
) AS s
ON t.CustomerCodeNK = s.CustomerID
WHEN MATCHED THEN
UPDATE SET
AquiredDate = s.AcquisitionDate,
AquiredSource = s.AcquisitionSource,
ZipCode = s.Zipcode,
LoadDate = SYSDATETIME(),
LoadSource = '../csv/cleaned_original_data.csv'
WHEN NOT MATCHED THEN
INSERT (CustomerCodeNK, AquiredDate, AquiredSource, ZipCode, LoadDate, LoadSource)
VALUES (s.CustomerID, s.AcquisitionDate, s.AcquisitionSource, s.Zipcode, SYSDATETIME(), '../csv/cleaned_original_data.csv')
;
I have the following query in which i'm trying to pass start dates and end dates in a sql query.
def get_data(start_date,end_date):
ic = Connector()
q = f"""
select * from example_table a
where a.date between {start_date} and {end_date}
"""
result = ic.query(q)
return result
df = pd.DataFrame(get_data('2021-01-01','2021-01-31'))
print(df)
which leads to the following error:
AnalysisException: Incompatible return types 'STRING' and 'BIGINT' of exprs 'a.date' and '2021 - 1 - 1'.\n (110) (SQLExecDirectW)")
I have also tried to parse the dates as follows:
import datetime
start_date = datetime.date(2021,1,1)
end_date = datetime.date(2021,5,13)
df = pd.DataFrame(get_data(start_date,end_date))
but i still get the same error.
Any help will be much appreciated.
It seems to me, that it is because how you inject your values into sql query they don't get recognized as date values. Database will likely 2021-01-01 interpret as mathematical expression with 2019 being the result.
You should try put parentheses around your values.
q = f"""
select * from example_table a
where a.date between '{start_date}' and '{end_date}'
"""
Or preferably if your db library allows it don't inject your values directly
q = """
select * from example_table a
where a.date between %s and %s
"""
result = ic.query(q, (start_date, end_date))
EDIT: Some database libraries may use place-holder with different format than %s. You should probably consult documentation of db library you are using.
I'm having some trouble modifying a sql table that contains a date.
This is the code I'm running:
import pymysql
cnx=pymysql.connect(dbstuff)
cursor=cnx.cursor()
cursor.execute("SELECT * FROM devices ORDER BY ip_addr,Port ASC")
results=cursor.fetchall()
lst = [(i,)+line[1:] for i, line in enumerate(results, 1)]
values = ', '.join(map(str,lst))
query="DELETE FROM devices; INSERT INTO devices VALUES {}".format(values)
query = query.replace("None","NULL")
cursor.execute(query)
cnx.commit()
cnx.close()
If the date column in sql is NULL, this runs without a problem. If there is a date inserted into the field, I get the following error:
"FUNCTION datetime.datetime does not exist."
When I look at the results of the sql select query, the date column value is converted to the following:
datetime.datetime(2016, 8, 16, 11, 24, 4)
I assume this is a python thing and not a sql thing. I haven't been to find anyway to convert this to a format sql would understand.
For datetime values you may use this:
your_datetime = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
This variable will hold a string value like this:
2018-05-06 21:14:37
And then use the following SQL command:
"INSERT INTO your_table (your_datetime_column) VALUES (str_to_date(\'%s\', '%%Y-%%m-%%d %%H:%%i:%%s'))" % (your_datetime)
DO NOT USE FOLLOWING - it won't work with the .format() method:
"INSERT INTO your_table (your_datetime_column) VALUES ({0})".format (your_datetime)
Calling the str function of the datetime.datetime object will convert it to the string format required by SQL. It is currently trying to insert a python object.
Simply wrap your object like this:
dt = str(my_datetime_object)
This error is raised when you pass invalid format date string in SQL query. Even a space in '2018-05-01 11:02:02 ' can cause this error. The best way to get rid of it is to avoid string formatting while making sql queries.
c=db.cursor()
c.execute("INSERT INTO devices VALUES(%s,%s,%s)",(value1,value2,value3))
c.close()
If you are unknown that how many values will have then, you have to generate (%s,%s...) dynamically.
import pymysql
cnx=py
mysql.connect(dbstuff)
cursor=cnx.cursor()
cursor.execute("SELECT * FROM devices ORDER BY ip_addr,Port ASC")
results=cursor.fetchall()
lst = [(i,)+line[1:] for i, line in enumerate(results, 1)]
formatt=','.join(['%s']*len(lst))
query="INSERT INTO devices VALUES (%s)"%formatt
cursor.execute(query,lst)
cnx.commit()
cnx.close()
I have a table populated with values, one being date in the YYYY-MM-DD format. I'm trying to print all "cost" values by month. So,
def SumByMonth():
usrDate = raw_input('Enter month: ')
sql = '''SELECT cost FROM Finance WHERE strftime('%m', date) = ?''', (usrDate)
month_cost = [t[0] for t in cur.execute(sql)]
print month_cost
This code gives me this error:
ValueError: operation parameter must be str or unicode
So I figure the SQL command isn't actually extracting anything? What am I missing here?
There is a problem with line
sql = '''SELECT cost FROM Finance WHERE strftime('%m', date) = ?''', (usrDate)
You need to write SQL query in string or unicode format like the error states
Here is a correct code :
def SumByMonth():
usrDate = raw_input('Enter month: ')
month_cost = [t[0] for t in cur.execute("SELECT cost FROM Finance WHERE strftime('%m', date) = ?", (usrDate,))]
print month_cost
Why do i get this error?
sqlite3.OperationalError: near "?": syntax error
when i run this:
c.execute('UPDATE ? SET Quantity = Quantity + ? WHERE Date = ?', (table, amount, date))
But not when i run this?
c.execute('UPDATE table1 SET Quantity = Quantity + ? WHERE Date = ?', (amount, date))
Variable value is:
table = 'table1'
amount = 20
Date = '12/5/2014'
I'm trying to dynamically create tables, but just doesn't work out.
You can't use placeholders for table names. You have to use normal Python string formatting or concatenation.