I am trying to rewrite some old Access VBA codes to Python but I stuck with the following error:
NameError: name 'ERTZ6635' is not defined
Old VBA Code
Set ConsTable = DB.OpenRecordset("SELECT * FROM table1")
ConsCount = 87404
If ConsCount > 0 Then
ConsTable.MoveFirst
For I = 1 To ConsCount
Set ConsBlendTable = DB.OpenRecordset("SELECT * FROM table2 WHERE CONS_BATCH = " & Char34(ConsTable!Batch))
Python code:
import win32com.client
dbe = win32com.client.Dispatch("DAO.DBEngine.120")
db = dbe.OpenDatabase(r"C:\Users\xyz\Desktop\acess.accdb")
ConsTable = db.OpenRecordset("select * from table1")
ConsCount = 87404
if ConsCount>0:
ConsTable.MoveFirst()
for i in range(1, ConsCount):
ConsBlendTable = db.OpenRecordset("SELECT * FROM table2 WHERE CONS_BATCH = " & eval(ConsTable.Fields["Batch"].Value))
And the ERTZ6635 value is the value in ConsTable.Fields["Batch"].Value
In the VBA code, Char34() likely is a user-defined function since it is not a VBA built-in method. There is however a constant Chr34 for the 34th character in the ASCII table for double quotes. So by its name, this method may wrap double quotes around the input parameter value. This is important since you attempt to translate in Python with eval.
So simple answer is to include double quotes which you can interpolate with F-strings in Python.
sql = f"""SELECT * FROM table2
WHERE CONS_BATCH = "{ConsTable.Fields["Batch"].Value}"
"""
ConsBlendTable = db.OpenRecordset(sql)
However, this string interpolation of parameters into an SQL query is not advisable in any language including VBA and Python. Aside from security and efficiency issues, this code can still break if value itself contains double quotes.
Instead, consider parameterization which is supported in MS Access SQL engine via QueryDefs.
VBA (adjusted from earlier SQL concatenation)
Dim qdef As QueryDef
...
Set ConsTable = DB.OpenRecordset("SELECT * FROM table1")
' PREPARED STATEMENT (NO DATA)
sql = "PARAMETERS batch_prm TEXT(255);" _
& "SELECT * FROM table2 WHERE CONS_BATCH = batch_prm"
ConsTable.MoveFirst
Do While Not ConsTable.EOF
Set qdef = DB.CreateQueryDef("", sql)
qdef!batchprm = ConsTable!Batch ' BIND PARAMETER
Set ConsBlendTable = qdef.OpenRecordset() ' OPEN RECORDSET VIA QUERYDEF
...
ConsBlendTable.Close
ConsTable.MoveNext
Loop
ConsTable.Close
'RELEASE RESOURCES
Set ConsBlendTable = Nothing: Set ConsTable = Nothing
Set qdef = Nothing: Set DB = Nothing
Python (employing try/except for proper COM handling)
Therefore in Python, we similarly interface with QueryDef object. Below loops through every record in recordset with traditional DAO loop (i.e., translation of Do While Not rst.EOF).
import win32com.client
try:
dbe = win32com.client.Dispatch("DAO.DBEngine.120")
db = dbe.OpenDatabase(r"C:\Users\xyz\Desktop\acess.accdb")
ConsTable = db.OpenRecordset("SELECT * FROM table1")
# PREPARED STATEMENT
sql = """PARAMETERS batch_prm TEXT(255);
SELECT * FROM table2 WHERE CONS_BATCH = batch_prm
"""
ConsTable.MoveFirst()
while ConsTable.EOF == False:
qdef = db.CreateQueryDef("", sql)
# BIND PARAMETER
qdef.Parameters["batch_prm"].Value = ConsTable.Fields["Batch"].Value
# OPEN RECORDSET VIA QUERYDEF
ConsBlendTable = qdef.OpenRecordset()
...
ConsBlendTable.Close()
ConsTable.MoveNext()
ConsTable.Close()
except Exception as e:
print(e)
finally:
# RELEASE RESOURCES
ConsBlendTable = None; ConsTable = None
qdef = None; db = None; dbe = None
del ConsBlendTable; del ConsTable; del qdef; del db; del dbe
Finally, I must note. Instead of direct translation of VBA, use Python's DB-API since it can directly query MS Access databases without DAO or COM objects, specifically with the well-maintained: pyodbc. And run with a JOIN instead of iterative loops via WHERE. And yes, pyodbc supports parameters with ? qmarks.
import pyodbc
dbname = r"C:\Users\xyz\Desktop\acess.accdb"
constr = f"DRIVER={{Microsoft Access Driver (*.mdb, *.accdb)}};DBQ={dbname};"
conn = pyodbc.connect(constr)
cur = conn.cursor()
sql = """SELECT t1.Batch, t2.CONS_BATCH, ...
FROM table1 t1
INNER JOIN tbale2 t2
ON t1.Batch = t2.CONS_BATCH
"""
cur.execute(sql)
for row in cur.fetchall():
...
cur.close()
conn.close()
Related
I've a script that makes a query to my database on MySQL. And my doubt is if I can pass any parameter to that query through Python.
For example, on the following script I want to calculate the date_filter using Python and then apply that filter on the query.
now = dt.datetime.now()
date_filter = now - timedelta(days=3)
dq_connection = mysql.connector.connect(user='user', password='pass', host='localhost', database='db')
engine = create_engine('localhost/db')
cursor = connection.cursor(buffered=True)
query = ('''
SELECT *
FROM myTable
WHERE date >= ''' + date_filter + '''
''')
I try it on that way but I got the following error:
builtins.TypeError: can only concatenate str (not "datetime.datetime") to str
It is possible to apply the filter like that?
Thanks!
Yes, you can do it. To avoid sql injections, the best way is not using the python formatting facilities, but the sql parameters & placeholders (see that you don´t need the single quotes ' as the placeholder does the job and converts the type of the variable):
now = dt.datetime.now()
date_filter = now - timedelta(days=3)
dq_connection = mysql.connector.connect(user='user', password='pass', host='localhost', database='db')
engine = create_engine('localhost/db')
cursor = db_connection.cursor(buffered=True)
query = "SELECT * FROM myTable WHERE date >=%s"
cursor.execute(query,(date_filter,))
Also, you had a mistake in your cursor, it should be db_connection.cursor. The last comma after date_filter is ok because you need to send a tuple.
In case you need more than one paremeter, you can place more than one placeholder:
query = "SELECT * FROM myTable WHERE date >=%s and date<=%s"
cursor.execute(query,(date_filter,other_date))
You can just do something like:
WHERE date >= ''' + str(date_filter) + '''
to represent the date as string as not a datetime object.
You can try with this:
date_f = str(date_filter)
query = ('''
SELECT *
FROM myTable
WHERE date >= "{}"
'''.format(date_f))
I'm learning programming with python and trying to implement the safest possible MySQL queries starting with the simple SELECT ones. The problem is whenever I use coma in a query I got the following error:
cursor.execute(query)
File "C:\Users\username\AppData\Local\Programs\Python\Python37-32\lib\site-packages\mysql\connector\cursor.py", line 536, in execute
stmt = operation.encode(self._connection.python_charset)
AttributeError: 'tuple' object has no attribute 'encode'
I am aware of the fact that coma itself isn't a source of a problem but I tried many different MySQL syntax and everytime I use a come I got this "AttributeError: 'tuple' object has no attribute 'encode'" error.
I also tried to change MySQL database encoding - nothing changes. The code is below.
import mysql.connector
conn = mysql.connector.connect(
charset='utf8',
# init_command='SET NAMES UTF8',
host="10.0.0.234",
user="x",
passwd="x>",
database="x",
)
print(conn.is_connected())
param = "test"
cursor = conn.cursor()
# =========== query below does work ========
# query = ("SELECT * from list WHERE username LIKE '%test%'")
# ============ query below does work =======
# query = ("SELECT * from list HAVING username = '%s'" % param)
# ============ query below doesn't work =====
# query = ("SELECT * from list HAVING username = %s", (param,))
# ============= query below doesn't work =====
query = "SELECT * from list WHERE username = :name", {'name': param}
cursor.execute(query)
result = cursor.fetchall()
for x in result:
print(x)
conn.close()
Any ideas what am I doing wrong?
The answer is a little bit tricky, but it is in essence because of what the actual value of the 'query' variable is...
For example:
# 1.
query = ("SELECT * from list WHERE username LIKE '%test%'")
# when you do this, query is a string variable,
# NB: the parentheses are not necessary here
# so when you call
cursor.execute(query)
# the value passed into the execute call is the string "SELECT * from list WHERE username LIKE '%test%'"
# 2.
query = ("SELECT * from list HAVING username = '%s'" % param)
# when you do this, query is the result of a string formatting operation
# This is a Python 2 form of string formatting
# The discussion here probably makes it more clear:
# https://stackoverflow.com/questions/13945749/string-formatting-in-python-3
# it is almost the same as doing this:
query = "SELECT * from list HAVING username = 'test'"
# so when you call
cursor.execute(query)
# the value passed into the execute call is the string "SELECT * from list HAVING username = 'test'"
# 3.
query = ("SELECT * from list HAVING username = %s", (param,))
# This operation is assigning a 2-value tuple into the query variable
# The first value in the tuple is the string "SELECT * from list HAVING username = %s"
# The second value in the tuple is a 1-value, with 'test' as its first value
# 4.
query = "SELECT * from list WHERE username = :name", {'name': param}
# This is similar to #3, but the values in the tuple are instead
# query[0] == "SELECT * from list WHERE username = :name"
# query[1] is a dictionary: {'name': param}
Both 3 and 4 above are not calling the MySQL execute with the parameters you are expecting (see API here). You probably need to do one of:
unpack the query tuple into separate variables, and call the function with them
operation, params = query # unpack the first elem into operation, and second into params
cursor.execute(operation, params)
just index into the query tuple
cursor.execute(query[0], query[1])
# NB: you could also use the 'named parameters' feature in Python
cursor.execute(query[0], params=query[1])
Use the 'unpacking arguments list' (SPLAT operator)
cursor.execute(*query)
I want to read a table stored in HANA directly from python. For that I use the following code:
from hdbcli import dbapi
import pandas as pd
conn = dbapi.connect(
address="address",
port=XYZ,
user="user",
password="password"
)
print (conn.isconnected())
# Fetch table data
stmnt = "select * from '_SYS_NAME'.'part1.part2.part3.part4.part5.part6/table_name'"
cursor = conn.cursor()
cursor.execute(stmnt)
result = cursor.fetchall()
print('Create the dataframe')
The problem is in the line stmnt: I tried different ways of puting the path name so that python can read it as a string but none is working. I know the problem is not relying on the technique, because if the path is simple and not containing the special characters then the code works.
I tried all the following combinations (among others):
stmnt = "select * from '_SYS_NAME'.'part1.part2.part3.part4.part5.part6/table_name'"
stmnt = """select * from '_SYS_NAME'.'part1.part2.part3.part4.part5.part6/table_name'"""
stmnt = "select * from \'_SYS_NAME\'\.\'part1.part2.part3.part4.part5.part6/table_name\'
stmnt = """select * from \'_SYS_NAME\'\.\'part1.part2.part3.part4.part5.part6/table_name\'"""
The error I get is always the following:
hdbcli.dbapi.Error: (257, 'sql syntax error: incorrect syntax near "_SYS_NAME": line 1 col 1 (at pos 1)')
And the original path as I get it from SQL is:
'_SYS_NAME'.'part1.part2.part3.part4.part5.part6/table_name'
Any ideas what I am missing?
You should reverse your quotes:
stmnt = 'select * from "_SYS_BIC"."rwev.dev.bw.project.si.churn/SI_CV_CHU_7_DATA_MODEL"'
I am using Python 3.6, pyodbc, and connect to SQL Server.
I am trying make connection to a database, then creating a query with parameters.
Here is the code:
import sys
import pyodbc
# connection parameters
nHost = 'host'
nBase = 'base'
nUser = 'user'
nPasw = 'pass'
# make connection start
def sqlconnect(nHost,nBase,nUser,nPasw):
try:
return pyodbc.connect('DRIVER={SQL Server};SERVER='+nHost+';DATABASE='+nBase+';UID='+nUser+';PWD='+nPasw)
print("connection successfull")
except:
print ("connection failed check authorization parameters")
con = sqlconnect(nHost,nBase,nUser,nPasw)
cursor = con.cursor()
# make connection stop
# if run WITHOUT parameters THEN everything is OK
ask = input ('Go WITHOUT parameters y/n ?')
if ask == 'y':
# SQL without parameters start
res = cursor.execute('''
SELECT * FROM TABLE
WHERE TABLE.TIMESTAMP BETWEEN '2017-03-01T00:00:00.000' AND '2017-03-01T01:00:00.000'
''')
# SQL without parameters stop
# print result to console start
row = res.fetchone()
while row:
print (row)
row = res.fetchone()
# print result to console stop
# if run WITH parameters THEN ERROR
ask = input ('Go WITH parameters y/n ?')
if ask == 'y':
# parameters start
STARTDATE = "'2017-03-01T00:00:00.000'"
ENDDATE = "'2017-03-01T01:00:00.000'"
# parameters end
# SQL with parameters start
res = cursor.execute('''
SELECT * FROM TABLE
WHERE TABLE.TIMESTAMP BETWEEN :STARTDATE AND :ENDDATE
''', {"STARTDATE": STARTDATE, "ENDDATE": ENDDATE})
# SQL with parameters stop
# print result to console start
row = res.fetchone()
while row:
print (row)
row = res.fetchone()
# print result to console stop
When I run the program without parameters in SQL, it works.
When I try running it with parameters, an error occurred.
Parameters in an SQL statement via ODBC are positional, and marked by a ?. Thus:
# SQL with parameters start
res = cursor.execute('''
SELECT * FROM TABLE
WHERE TABLE.TIMESTAMP BETWEEN ? AND ?
''', STARTDATE, ENDDATE)
# SQL with parameters stop
Plus, it's better to avoid passing dates as strings. Let pyodbc take care of that using Python's datetime:
from datetime import datetime
...
STARTDATE = datetime(year=2017, month=3, day=1)
ENDDATE = datetime(year=2017, month=3, day=1, hour=0, minute=0, second=1)
then just pass the parameters as above. If you prefer string parsing, see this answer.
If you're trying to use pd.to_sql() like me I fixed the problem by passing a parameter called chunksize.
df.to_sql("tableName", engine ,if_exists='append', chunksize=50)
hope this helps
i tryied and have a lot of different errors: 42000, 22007, 07002 and others
The work version is bellow:
import sys
import pyodbc
import datetime
# connection parameters
nHost = 'host'
nBase = 'DBname'
nUser = 'user'
nPasw = 'pass'
# make connection start
def sqlconnect(nHost,nBase,nUser,nPasw):
try:
return pyodbc.connect('DRIVER={SQL Server};SERVER='+nHost+';DATABASE='+nBase+';UID='+nUser+';PWD='+nPasw)
except:
print ("connection failed check authorization parameters")
con = sqlconnect(nHost,nBase,nUser,nPasw)
cursor = con.cursor()
# make connection stop
STARTDATE = '11/2/2017'
ENDDATE = '12/2/2017'
params = (STARTDATE, ENDDATE)
# SQL with parameters start
sql = ('''
SELECT * FROM TABLE
WHERE TABLE.TIMESTAMP BETWEEN CAST(? as datetime) AND CAST(? as datetime)
''')
# SQL with parameters stop
# print result to console start
query = cursor.execute(sql, params)
row = query.fetchone()
while row:
print (row)
row = query.fetchone()
# print result to console stop
say = input ('everething is ok, you can close console')
I fixed this issue with code if you are using values through csv.
for i, row in read_csv_data.iterrows():
cursor.execute('INSERT INTO ' + self.schema + '.' + self.table + '(first_name, last_name, email, ssn, mobile) VALUES (?,?,?,?,?)', tuple(row))
I had a similar issue. Saw that downgrading the version of PyODBC to 4.0.6 and SQLAlchemy to 1.2.9 fixed the error,using Python 3.6
I know that this question has been asked in the past, but thorough searching hasn't seemed to fix my issue. I'm probably just missing something simple, as I'm new to the Python-mysql connector supplied by mysql.
I have a Python script which accesses a mysql database, but I'm having issues with removing quotes from my query. Here is my code:
import mysql.connector
try:
db = mysql.connector.connect(user='root', password='somePassword', host='127.0.0.1', database='dbName')
cursor = db.cursor()
query = "select * from tags where %s = %s"
a = 'tag_id'
b = '0'
cursor.execute(query, (a, b))
print cursor
data = cursor.fetchall()
print data
except mysql.connector.Error as err:
print "Exception tripped..."
print "--------------------------------------"
print err
cursor.close()
db.close()
My database is set up properly (as I'll prove shortly).
My output for this program is:
MySQLCursor: select * from tags where 'tag_id' = '0'
[]
Yet when I change my query to not use variables, for example:
cursor.execute("select * from tags where tag_id = 0")
Then my output becomes:
MySQLCursor: select * from tags where tag_id = 0
[(0, u'192.168.1.110')]
To me, this means that the only difference between my Cursor queries are the quotes.
How do I remove them from the query?
Thanks in advance.
I personally believe this code is correct and safe, but you should be extremely skeptical of using code like this without carefully reviewing it yourself or (better yet) with the help of a security expert. I am not qualified to be such an expert.
Two important things I changed:
I changed b = '0' to b = 0 so it ends up as a number rather than a quoted string. (This part was an easy fix.)
I skipped the built-in parameterization for the column name and replaced it with my own slight modification to the escaping/quoting built in to mysql-connector. This is the scary part that should give you pause.
Full code below, but again, be careful with this if the column name is user input!
import mysql.connector
def escape_column_name(name):
# This is meant to mostly do the same thing as the _process_params method
# of mysql.connector.MySQLCursor, but instead of the final quoting step,
# we escape any previously existing backticks and quote with backticks.
converter = mysql.connector.conversion.MySQLConverter()
return "`" + converter.escape(converter.to_mysql(name)).replace('`', '``') + "`"
try:
db = mysql.connector.connect(user='root', password='somePassword', host='127.0.0.1', database='dbName')
cursor = db.cursor()
a = 'tag_id'
b = 0
cursor.execute(
'select * from tags where {} = %s'.format(escape_column_name(a)),
(b,)
)
print cursor
data = cursor.fetchall()
print data
except mysql.connector.Error as err:
print "Exception tripped..."
print "--------------------------------------"
print err
cursor.close()
db.close()
I encountered a similar problem using pymysql and have shown my working code here, hope this will help.
What I did is overwrite the escape method in class 'pymysql.connections.Connection', which obviously adds "'" arround your string.
better have shown my code:
from pymysql.connections import Connection, converters
class MyConnect(Connection):
def escape(self, obj, mapping=None):
"""Escape whatever value you pass to it.
Non-standard, for internal use; do not use this in your applications.
"""
if isinstance(obj, str):
return self.escape_string(obj) # by default, it is :return "'" + self.escape_string(obj) + "'"
if isinstance(obj, (bytes, bytearray)):
ret = self._quote_bytes(obj)
if self._binary_prefix:
ret = "_binary" + ret
return ret
return converters.escape_item(obj, self.charset, mapping=mapping)
config = {'host':'', 'user':'', ...}
conn = MyConnect(**config)
cur = conn.cursor()