Develop Reference

Pass variable value to query in for loop

I have a list comprised of several queries which are executed by a for loop. I would like to prompt the user to enter the origin (ilink) that will be utilized by the fourth query in the list.
The script runs fine when the origin is manually defined within the query. I have tried the following syntax which have all failed:
cursor.execute(lines, ilink)
cursor.execute(lines, [ilink])
cursor.execute(lines, (ilink))
I have also run the script with each query defined in its own cursor.execute(query) which accepts the argument, but does not pass any results due to multiple cursors.
import MySQLdb
ilink = raw_input("Choose and ilink to query (include 199N):" )
db = MySQLdb.connect(host="host",user="user",passwd="pass")
queries = [
"""USE monthly_audit;""",
"""DROP TEMPORARY TABLE IF EXISTS monthly_audit.tmp_order_ids;""",
"""DROP TEMPORARY TABLE IF EXISTS monthly_audit.tmp_internalselect;""",
"""CREATE TEMPORARY TABLE monthly_audit.tmp_order_ids AS
(SELECT DISTINCT order_id AS orders
FROM ng_tradeserver_db_history.fix_execution_reports_201906
WHERE FROM_UNIXTIME(TIMESTAMP/1000000) >= '2019-06-19 16:59:59'
AND FROM_UNIXTIME(TIMESTAMP/1000000) <= '2019-06-20 23:59:59'
AND TargetCompID = %s);""",]
cursor = db.cursor()
for lines in queries:
lines.split(",")
cursor.execute(lines, [ilink])
results = cursor.fetchall()
**This is only the relevant snippet of sql, total query is over 500 lines*
I expect the script to run the set of queries and return the results of said query to be stored in a csv. I am currently getting the following error when executing:
_mysql_exceptions.ProgrammingError: not all arguments converted during string formatting

I'm not sure if I understand your questions correct, but you can try using fstrings. I believe the quotes cause the problems during the string formatting.
Example:
query = f'''select ID, lat, lon from tbl order by st_distance(tbl.geom,st_setsrid(st_makepoint({lon},{lat}), 4326)) asc limit 1;'''
cursor.execute(query)
In this query the {lon}, {lat} are variables. Have a look at the docs for f strings https://docs.python.org/3/whatsnew/3.6.html

inconsistent results from LIKE query: pyodbc vs. Access

I got a bunch of queries that should be executed in an Access database as a part of my Python script. Unfortunately, queries that used directly in MS Access are giving some records of output, in Python script return nothing (no error either). Connection with database and general syntax should be fine as simple queries (like select one column from table where something) are working just fine. Here is a code with one of these given queries:
import pyodbc
baza = r"C:\base.mdb"
driver = "{Microsoft Access Driver (*.mdb, *.accdb)}"
access_con_string = r"Driver={};Dbq={};".format(driver, baza)
cnn = pyodbc.connect(access_con_string)
db_cursor = cnn.cursor()
expression = """SELECT F_PARCEL.PARCEL_NR, F_PARCEL_LAND_USE.AREA_USE_CD, F_PARCEL_LAND_USE.SOIL_QUALITY_CD, F_ARODES.TEMP_ADRESS_FOREST, F_SUBAREA.AREA_TYPE_CD, F_AROD_LAND_USE.AROD_LAND_USE_AREA, F_PARCEL.COUNTY_CD, F_PARCEL.DISTRICT_CD, F_PARCEL.MUNICIPALITY_CD, F_PARCEL.COMMUNITY_CD, F_SUBAREA.SUB_AREA
FROM F_PARCEL INNER JOIN (F_PARCEL_LAND_USE INNER JOIN ((F_ARODES INNER JOIN F_AROD_LAND_USE ON F_ARODES.ARODES_INT_NUM = F_AROD_LAND_USE.ARODES_INT_NUM) INNER JOIN F_SUBAREA ON F_ARODES.ARODES_INT_NUM = F_SUBAREA.ARODES_INT_NUM) ON (F_PARCEL_LAND_USE.SHAPE_NR = F_AROD_LAND_USE.SHAPE_NR) AND (F_PARCEL_LAND_USE.PARCEL_INT_NUM = F_AROD_LAND_USE.PARCEL_INT_NUM)) ON F_PARCEL.PARCEL_INT_NUM = F_PARCEL_LAND_USE.PARCEL_INT_NUM
WHERE (((F_ARODES.TEMP_ADRESS_FOREST) Like ?) AND ((F_AROD_LAND_USE.AROD_LAND_USE_AREA)<?) AND ((F_ARODES.TEMP_ACT_ADRESS)= ?))
ORDER BY F_PARCEL.PARCEL_NR, F_PARCEL_LAND_USE.SHAPE_NR;"""
rows = db_cursor.execute(expression, ("14-17-2-03*", 0.0049, True)).fetchall()
for row in rows:
print row
cnn.close()
I know that those queries were generated from query builder in MS Access, so I was wondering that maybe this results in differences, but on the other hand this is still access database.
Anyway it seems, that the problem is in SQL, so I would like to know what elements could possibly result in different output between queries executed directly in MS Access and by pyodbc connection?

You are getting tripped up by the difference in LIKE wildcard characters between queries run in Access itself and queries run from an external application.
When running a query from within Access itself you need to use the asterisk as the wildcard character: "14-17-2-03*".
When running a query from an external application (like your Python app) you need to use the percent sign as the wildcard character: "14-17-2-03%".

Too few parameters error, while no parameters placeholders used

I am trying to execute SQL query within Access database using PYODBC and I get following error:
pyodbc.Error: ('07002', '[07002] [Microsoft][ODBC Microsoft Access Driver]
Too few parameters. Expected 1. (-3010) (SQLExecDirectW)')
The problem is that I am not using any additional parameters. Here is the code:
access_con_string = r"Driver={};Dbq={};".format(driver, base)
cnn = pyodbc.connect(access_con_string)
db_cursor = cnn.cursor()
expression = """SELECT F_ARODES.ARODES_INT_NUM, F_ARODES.TEMP_ADRESS_FOREST,F_AROD_LAND_USE.ARODES_INT_NUM, F_ARODES.ARODES_TYP_CD
FROM F_ARODES LEFT JOIN F_AROD_LAND_USE ON F_ARODES.ARODES_INT_NUM = F_AROD_LAND_USE.ARODES_INT_NUM
WHERE (((F_AROD_LAND_USE.ARODES_INT_NUM) Is Null) AND ((F_ARODES.ARODES_TYP_CD)="wydziel") AND ((F_ARODES.TEMP_ACT_ADRESS)=True));"""
db_cursor.execute(expression)
Query itself, if used inside MS-Access works fine. Also, connection is OK, as other queries are executed properly.
What am I doing wrong?

Constants in such queries are problematic - you never know the exact underlying syntax for booleans, strings etc. - even if it works in MS-Access, it can be different inside the intermediary library you're using.
The safest way is to extract them as parameters anyway:
expression = """SELECT F_ARODES.ARODES_INT_NUM, F_ARODES.TEMP_ADRESS_FOREST,F_AROD_LAND_USE.ARODES_INT_NUM, F_ARODES.ARODES_TYP_CD FROM F_ARODES LEFT JOIN F_AROD_LAND_USE ON F_ARODES.ARODES_INT_NUM = F_AROD_LAND_USE.ARODES_INT_NUM WHERE (((F_AROD_LAND_USE.ARODES_INT_NUM) Is Null)
AND ((F_ARODES.ARODES_TYP_CD)=?) AND ((F_ARODES.TEMP_ACT_ADRESS)=?));"""
db_cursor.execute(expression, "wydziel", True)

I had a similar problem, with an update I was trying to perform with pyodbc. When executed in Access, the query worked fine, same for when using the application (it allows some queries from within the app). But when ran in python with pyodbc the same text would throw errors. I determined the problem is the double quote (OP's query has a set of them as well). The query began to work when I replaced them with single quotes.
This does not work:
Update ApplicationStandards Set ShortCutKey = "I" Where ShortName = "ISO"
This does:
Update ApplicationStandards Set ShortCutKey = 'I' Where ShortName = 'ISO'

Pandas read_sql query with multiple selects

Can read_sql query handle a sql script with multiple select statements?
I have a MSSQL query that is performing different tasks, but I don't want to have to write an individual query for each case. I would like to write just the one query and pull in the multiple tables.
I want the multiple queries in the same script because the queries are related, and it making updating the script easier.
For example:
SELECT ColumnX_1, ColumnX_2, ColumnX_3
FROM Table_X
INNER JOIN (Etc etc...)
----------------------
SELECT ColumnY_1, ColumnY_2, ColumnY_3
FROM Table_Y
INNER JOIN (Etc etc...)
Which leads to two separate query results.
The subsequent python code is:
scriptFile = open('.../SQL Queries/SQLScript.sql','r')
script = scriptFile.read()
engine = sqlalchemy.create_engine("mssql+pyodbc://UserName:PW!#Table")
connection = engine.connect()
df = pd.read_sql_query(script,connection)
connection.close()
Only the first table from the query is brought in.
Is there anyway I can pull in both query results (maybe with a dictionary) that will prevent me from having to separate the query into multiple scripts.

You could do the following:
queries = """
SELECT ColumnX_1, ColumnX_2, ColumnX_3
FROM Table_X
INNER JOIN (Etc etc...)
---
SELECT ColumnY_1, ColumnY_2, ColumnY_3
FROM Table_Y
INNER JOIN (Etc etc...)
""".split("---")
Now you can query each table and concat the result:
df = pd.concat([pd.read_sql_query(q, connection) for q in queries])
Another option is to use UNION on the two results i.e. do the concat in SQL.

Translating a complicated SQL query to python 2 for SQLite3

Below I have some code that connects to database and runs a query. This query runs as expected if I punch it into DB Browser for SQLite, but when I run it in python, I receive the error :
sqlite3.OperationalError: no such column: CHARACTER_INNATES.PC_ID
I have read some documentation and multiple StackOverflow comments about parameters, and their purpose to prevent problems like injection attacks. I tried that approach, which littered '?' everywhere, making it unreadable and unmaintainable, and it didn't work. It returned an empty tuple. This leads me to believe there should be an easier way of performing these kinds of SQL queries. So is there? Or am I missing something obvious?
db = sqlite3.connect('TheGame.db')
db.text_factory = str
conn = db.cursor()
query = 'SELECT CHARACTERS.Full_Name, CHARACTER_INNATES.PC_ID, SKILLS_INNATES.Skill, Sum([CHARACTER_INNATES].[Current]*[Weight]) AS [Current Innate], Sum([CHARACTER_INNATES].[Maximum]*[Weight]) AS [Max Innate]\
FROM CHARACTERS INNER JOIN (SKILLS_INNATES INNER JOIN CHARACTER_INNATES ON SKILLS_INNATES.Innate = CHARACTER_INNATES.Innate) ON CHARACTERS.ID = CHARACTER_INNATES.PC_ID\
GROUP BY CHARACTERS.Full_Name, CHARACTER_INNATES.PC_ID, SKILLS_INNATES.Skill\
ORDER BY CHARACTER_INNATES.PC_ID, Sum([CHARACTER_INNATES].[Current]*[Weight]) DESC;'
conn.execute(query)
print conn.fetchall()
My attempt at a fix looked something like
params = ('CHARACTERS.Full_Name', 'CHARACTER_INNATES.PC_ID', ...) #this continued for awhile
query = 'SELECT ?, ?, ?, ...'
conn.execute(query, params)
print conn.fetchall() # prints empty tuple

This query already is unreadable.
Anyway, some SQLite versions have problem with table names hidden inside parentheses. This typically happens when you're using the Access query builder; just write the joins properly instead:
SELECT CHARACTERS.Full_Name,
CHARACTER_INNATES.PC_ID,
SKILLS_INNATES.Skill,
Sum(CHARACTER_INNATES.Current * Weight) AS "Current Innate",
Sum(CHARACTER_INNATES.Maximum * Weight) AS "Max Innate"
FROM CHARACTERS
INNER JOIN CHARACTER_INNATES ON CHARACTERS.ID = CHARACTER_INNATES.PC_ID
INNER JOIN SKILLS_INNATES USING (Innate)
GROUP BY CHARACTERS.Full_Name,
CHARACTER_INNATES.PC_ID,
SKILLS_INNATES.Skill
ORDER BY CHARACTER_INNATES.PC_ID,
Sum(CHARACTER_INNATES.Current * Weight) DESC;