Develop Reference

Python looping to obtain different dataframes from a SQL database

I'm trying to connect to an SQL database and, within a loop, create separate dataframes for each different instance of Id, containing all the data related to that Id. I've tried a number of ways, without any success so far. I'm pretty new to all of this, so I'm probably making some rookie mistakes.
Attempt 1:
import pandas as pd
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=Server_name;'
'Database=Database;'
'UID=Username;'
'PWD=password;'
'Trusted_Connection=yes;')
Name = ['HR','ZA','PR','FW']
for x in Name:
SQL = '''
SELECT *
FROM Database
WHERE Id = {x}'''.format(x = x)
cursor = conn.cursor()
cursor.execute(SQL)
df = pd.read_sql_query(SQL)
On this code, I get an 'invalid column name' programming error on the first Name 'HL'.
Attempt 2:
import pandas as pd
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=Server_name;'
'Database=Database;'
'UID=Username;'
'PWD=password;'
'Trusted_Connection=yes;')
SQL = '''
SELECT *
FROM Database
conn.autocommit = True
cursor.execute(SQL)
for [Id] in cursor:
df = pd.Dataframe(SQL,conn)
On this code, I get a 'ValueError: too many values to unpack (expected 1)' - on the for statement.
I want to put a lot more code in the for loop so I need it to be set up to work through each Id. I hope that makes sense. Any guidance would be greatly appreciated. Thanks
UPDATE:
Thanks for all comments/answers. For some reason I just couldn't get it to work in either of the formats above so I took it back to where I started from now I understand how to include the syntax for the loop variable. The following now works:
import pandas as pd
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=Server_name;'
'Database=Database;'
'UID=Username;'
'PWD=password;'
'Trusted_Connection=yes;')
Name = ['HR','ZA','PR','FW']
for x in Name:
SQL = pd.read_sql_query(
'''
SELECT *
FROM Database_table
WHERE Id = '{x}'
'''.format(x = x), conn)
df = pd.DataFrame(SQL)

I think that if you try a variation on your first attempt like:
for x in Name:
SQL = '''
SELECT *
FROM Database
WHERE Id = ?'''
cursor = conn.cursor()
cursor.execute(SQL)
df = pd.read_sql_query(SQL, params={x})
It should probably work :)

Python / Access NameError: name '' is not defined

I am trying to rewrite some old Access VBA codes to Python but I stuck with the following error:
NameError: name 'ERTZ6635' is not defined
Old VBA Code
Set ConsTable = DB.OpenRecordset("SELECT * FROM table1")
ConsCount = 87404
If ConsCount > 0 Then
ConsTable.MoveFirst
For I = 1 To ConsCount
Set ConsBlendTable = DB.OpenRecordset("SELECT * FROM table2 WHERE CONS_BATCH = " & Char34(ConsTable!Batch))
Python code:
import win32com.client
dbe = win32com.client.Dispatch("DAO.DBEngine.120")
db = dbe.OpenDatabase(r"C:\Users\xyz\Desktop\acess.accdb")
ConsTable = db.OpenRecordset("select * from table1")
ConsCount = 87404
if ConsCount>0:
ConsTable.MoveFirst()
for i in range(1, ConsCount):
ConsBlendTable = db.OpenRecordset("SELECT * FROM table2 WHERE CONS_BATCH = " & eval(ConsTable.Fields["Batch"].Value))
And the ERTZ6635 value is the value in ConsTable.Fields["Batch"].Value

In the VBA code, Char34() likely is a user-defined function since it is not a VBA built-in method. There is however a constant Chr34 for the 34th character in the ASCII table for double quotes. So by its name, this method may wrap double quotes around the input parameter value. This is important since you attempt to translate in Python with eval.
So simple answer is to include double quotes which you can interpolate with F-strings in Python.
sql = f"""SELECT * FROM table2
WHERE CONS_BATCH = "{ConsTable.Fields["Batch"].Value}"
"""
ConsBlendTable = db.OpenRecordset(sql)
However, this string interpolation of parameters into an SQL query is not advisable in any language including VBA and Python. Aside from security and efficiency issues, this code can still break if value itself contains double quotes.
Instead, consider parameterization which is supported in MS Access SQL engine via QueryDefs.
VBA (adjusted from earlier SQL concatenation)
Dim qdef As QueryDef
...
Set ConsTable = DB.OpenRecordset("SELECT * FROM table1")
' PREPARED STATEMENT (NO DATA)
sql = "PARAMETERS batch_prm TEXT(255);" _
& "SELECT * FROM table2 WHERE CONS_BATCH = batch_prm"
ConsTable.MoveFirst
Do While Not ConsTable.EOF
Set qdef = DB.CreateQueryDef("", sql)
qdef!batchprm = ConsTable!Batch ' BIND PARAMETER
Set ConsBlendTable = qdef.OpenRecordset() ' OPEN RECORDSET VIA QUERYDEF
...
ConsBlendTable.Close
ConsTable.MoveNext
Loop
ConsTable.Close
'RELEASE RESOURCES
Set ConsBlendTable = Nothing: Set ConsTable = Nothing
Set qdef = Nothing: Set DB = Nothing
Python (employing try/except for proper COM handling)
Therefore in Python, we similarly interface with QueryDef object. Below loops through every record in recordset with traditional DAO loop (i.e., translation of Do While Not rst.EOF).
import win32com.client
try:
dbe = win32com.client.Dispatch("DAO.DBEngine.120")
db = dbe.OpenDatabase(r"C:\Users\xyz\Desktop\acess.accdb")
ConsTable = db.OpenRecordset("SELECT * FROM table1")
# PREPARED STATEMENT
sql = """PARAMETERS batch_prm TEXT(255);
SELECT * FROM table2 WHERE CONS_BATCH = batch_prm
"""
ConsTable.MoveFirst()
while ConsTable.EOF == False:
qdef = db.CreateQueryDef("", sql)
# BIND PARAMETER
qdef.Parameters["batch_prm"].Value = ConsTable.Fields["Batch"].Value
# OPEN RECORDSET VIA QUERYDEF
ConsBlendTable = qdef.OpenRecordset()
...
ConsBlendTable.Close()
ConsTable.MoveNext()
ConsTable.Close()
except Exception as e:
print(e)
finally:
# RELEASE RESOURCES
ConsBlendTable = None; ConsTable = None
qdef = None; db = None; dbe = None
del ConsBlendTable; del ConsTable; del qdef; del db; del dbe
Finally, I must note. Instead of direct translation of VBA, use Python's DB-API since it can directly query MS Access databases without DAO or COM objects, specifically with the well-maintained: pyodbc. And run with a JOIN instead of iterative loops via WHERE. And yes, pyodbc supports parameters with ? qmarks.
import pyodbc
dbname = r"C:\Users\xyz\Desktop\acess.accdb"
constr = f"DRIVER={{Microsoft Access Driver (*.mdb, *.accdb)}};DBQ={dbname};"
conn = pyodbc.connect(constr)
cur = conn.cursor()
sql = """SELECT t1.Batch, t2.CONS_BATCH, ...
FROM table1 t1
INNER JOIN tbale2 t2
ON t1.Batch = t2.CONS_BATCH
"""
cur.execute(sql)
for row in cur.fetchall():
...
cur.close()
conn.close()

Python & pymysql: Database Table with hyphen in name

I need to query data from a mysqldatabase with the table name containing hyphens.
current_table = "tw3-10_1"
sql2 = "SELECT * FROM " + str(current_table )
cursor.execute(sql2)
Unfortunately I get:
1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '-10_1' at line 1")
Is there any way to work around that issue? Unfortunately I cannot change the names of the tables.....

You can normally use backticks to quote a table name or column name, in case it contains unhelpful characters.
current_table = "`tw3-10_1`"
sql2 = "SELECT * FROM " + current_table
or if you prefer
current_table = "tw3-10_1"
sql2 = "SELECT * FROM `{}`".format(current_table)

try like this, I don't know about MariaDB but quotes should work in SQL
sql2 = """
SELECT
*
FROM "{table_name}"
""".format(
table_name='table-with-hyphens'
)
print(sql2)
# SELECT
# *
# FROM "table-with-hyphens"

How to fetch data from sqlite using python?

I am using this sample sqlite database and my code is
import sqlite3
conn = sqlite3.connect('chinook.db')
conn.execute("SELECT * FROM tracks")
rows = conn.cursor().fetchall()
It should have worked, but rows is empty?
What am I doing wrong here?

The Connection.execute shortcut returns a cursor instance, which you need to use with fetchall. In your code, you're creating a new, independent cursor.
Thus:
import sqlite3
conn = sqlite3.connect('chinook.db')
cursor = conn.execute("SELECT * FROM tracks")
rows = cursor.fetchall()
or even shorter (not recommended, but for those who like obscured one-liners):
rows = sqlite3.connect('chinook.db').execute("SELECT * FROM tracks").fetchall()
Or don't use Connection.execute shortcut, to avoid confusion:
import sqlite3
conn = sqlite3.connect('chinook.db')
cursor = conn.cursor()
cursor.execute("SELECT * FROM tracks")
rows = cursor.fetchall()

OperationalError when using Python sqlite3 module

def StatusUpdate(self, table):
inventoryCurs.execute('SELECT * from Table')
for i in inventoryCurs:
html = urlopen(i[5]).read()
Soup = BeautifulSoup(html)
if table.StockStatus(Soup) == 'Out of Stock':
inventoryCurs.execute('''UPDATE table SET status = 'Out of Stock' WHERE id = %s)''', i[0])
inventoryCurs.execute('''UPDATE table SET status = 'Out of Stock' WHERE id = %s)''', i[0])
OperationalError: near "%": syntax error

Without seeing more of the code, it's difficult to fix the problem completely, but looking at your code, I think the problem might be the %s in this line:
inventoryCurs.execute('''UPDATE table SET status = 'Out of Stock' WHERE id = %s)''', i[0])
According to the documentation for the SQLite module in both Python 2 and Python 3, the sqlite3 module requires a ? as a placeholder, not %s or some other format string.
According to the Python 2 documentation, a %s placeholder could be used like this:
import sqlite3
conn = sqlite3.connect('example.db')
c = conn.cursor()
# Never do this -- insecure!
symbol = 'IBM'
c.execute("select * from stocks where symbol = '%s'" % symbol)
but that's a simple format string, not actually the database's placeholder. Also, as the comment shows, you should never build queries that way because it makes them vulnerable to SQL injection. Rather, you should build them like this, using a ? instead:
import sqlite3
conn = sqlite3.connect('example.db')
c = conn.cursor()
# Do this instead
t = (symbol,)
c.execute('SELECT * FROM stocks WHERE symbol=?', t)
The documentation has more details, but I believe that is the solution to the error you posted.