Develop Reference

Organizing SQL Queries in Python project

I have a python script I'm creating that will replace an set of SQL Server stored procedures to make the process more efficient. However, I have a 20-30 queries I need to execute at different points. To make the main query more simple I organized them into a separate file in a dictionary and created a function to pull the query to be executed.
My question here is there a better way to organize them? An idea I had was to put them into a table on the SQL Server or is this method best or is there another better method? Below is an example of what I'm doing now:
queryDict = {}
queryDict.update({"dbQuery1": "TRUNCATE TABLE MyTable;\
INSERT MyTable (Column1, Column2)\
SELECT Col1, Col2 FROM myTable2;"})
queryDict.update({"dbQuery1": 'SELECT MAX(val) FROM MyTable3;'})
def queryRequest(query):
return queryDict[query]

Is it possible to run multible statements in sqlite3 with python?

I have more than 2 tables in an sqlite3 database. Im trying to INSERT data into one table and update a few columns in another table. Is that possible?
I have tried to user executemany() and executescript() in many shapes and forms. I have recieved a bunch of error messages, and its mostly because execute will not accept my parameters.
with sqlite3.connect('database.db') as conn:
cur = conn.cursor()
cur.executescript("""INSERT INTO prev_users (
imei_prev,
user_id_prev,
start_date,
end_date,
type_prev)
VALUES (?,?,?,?,?);
UPDATE phones
SET owner = (?), date = (?), state = (?)
WHERE imei = (?);
""",(user['imei'], user['user_id'], user['date'], date, user['type'], "None", date, "In Progress", user['imei']))

executescript() will not accept parameters because, well, it doesn't take parameters as an argument, it only takes a "script". From the doc:
executescript(sql_script)
This is a nonstandard convenience method for executing multiple SQL statements at once. It issues a COMMIT statement first, then
executes the SQL script it gets as a parameter.
sql_script can be an instance of str.
For example, it might be used for creating several tables in one script.
executemany() runs one sql against a sequence of parameters.
Neither method is the right tool for the job. You'll likely have to split it into two calls to execute().

how to check/print psycopg2 dynamic query Compose without creating conn.cursor()

I am writing unit test for a query builder, in which I assemble query from a couple of user input fields.
e.g.
query = sql.SQL("SELECT {fields} FROM {table}).format(
fields='*'
table=sql.Identifier(topic)))
I just wanna check if the query is what I desired, no need to execute.
I was trying to print query, and got a composed object which looks like
Composed([SQL('SELECT '), Composed([Identifier('*')]), SQL(' FROM '), Identifier('topic'), SQL(' '), SQL(''), SQL(' ')...)
Is there a try to transform the dynamic sql as a Composed object to a sql query String?
SELECT * FROM topic
I don't have the postgres set up for the unit test, so I cannot use
query.as_string(conn)
Any hints? Many thanks

You need cursor.mogrify(query, params), but to call it you need a cursor and to create a cursor you need to open a connection. I don't believe you can get the final query without a connection; this is because the query depends on server and database (server version, database encoding, quoting style).

inconsistent results from LIKE query: pyodbc vs. Access

I got a bunch of queries that should be executed in an Access database as a part of my Python script. Unfortunately, queries that used directly in MS Access are giving some records of output, in Python script return nothing (no error either). Connection with database and general syntax should be fine as simple queries (like select one column from table where something) are working just fine. Here is a code with one of these given queries:
import pyodbc
baza = r"C:\base.mdb"
driver = "{Microsoft Access Driver (*.mdb, *.accdb)}"
access_con_string = r"Driver={};Dbq={};".format(driver, baza)
cnn = pyodbc.connect(access_con_string)
db_cursor = cnn.cursor()
expression = """SELECT F_PARCEL.PARCEL_NR, F_PARCEL_LAND_USE.AREA_USE_CD, F_PARCEL_LAND_USE.SOIL_QUALITY_CD, F_ARODES.TEMP_ADRESS_FOREST, F_SUBAREA.AREA_TYPE_CD, F_AROD_LAND_USE.AROD_LAND_USE_AREA, F_PARCEL.COUNTY_CD, F_PARCEL.DISTRICT_CD, F_PARCEL.MUNICIPALITY_CD, F_PARCEL.COMMUNITY_CD, F_SUBAREA.SUB_AREA
FROM F_PARCEL INNER JOIN (F_PARCEL_LAND_USE INNER JOIN ((F_ARODES INNER JOIN F_AROD_LAND_USE ON F_ARODES.ARODES_INT_NUM = F_AROD_LAND_USE.ARODES_INT_NUM) INNER JOIN F_SUBAREA ON F_ARODES.ARODES_INT_NUM = F_SUBAREA.ARODES_INT_NUM) ON (F_PARCEL_LAND_USE.SHAPE_NR = F_AROD_LAND_USE.SHAPE_NR) AND (F_PARCEL_LAND_USE.PARCEL_INT_NUM = F_AROD_LAND_USE.PARCEL_INT_NUM)) ON F_PARCEL.PARCEL_INT_NUM = F_PARCEL_LAND_USE.PARCEL_INT_NUM
WHERE (((F_ARODES.TEMP_ADRESS_FOREST) Like ?) AND ((F_AROD_LAND_USE.AROD_LAND_USE_AREA)<?) AND ((F_ARODES.TEMP_ACT_ADRESS)= ?))
ORDER BY F_PARCEL.PARCEL_NR, F_PARCEL_LAND_USE.SHAPE_NR;"""
rows = db_cursor.execute(expression, ("14-17-2-03*", 0.0049, True)).fetchall()
for row in rows:
print row
cnn.close()
I know that those queries were generated from query builder in MS Access, so I was wondering that maybe this results in differences, but on the other hand this is still access database.
Anyway it seems, that the problem is in SQL, so I would like to know what elements could possibly result in different output between queries executed directly in MS Access and by pyodbc connection?

You are getting tripped up by the difference in LIKE wildcard characters between queries run in Access itself and queries run from an external application.
When running a query from within Access itself you need to use the asterisk as the wildcard character: "14-17-2-03*".
When running a query from an external application (like your Python app) you need to use the percent sign as the wildcard character: "14-17-2-03%".

How to access a SQL Server 2008 stored procedure with a table valued parameter in Python

I’m looking for a way to take a result set and use it to find records in a table that resides in SQL Server 2008 – without spinning through the records one at a time. The result sets that will be used to find the records could number in the hundreds of thousands. So far I am pursuing creating a table in memory using sqlite3 and then trying to feed that table to a stored procedure that takes a table valued parameter. The work on the SQL Server side is done, the user defined type is created, the test procedure accepting a table valued parameter exists and I’ve tested it through TSQL and it appears to work just fine. In Python a simple in memory table was created through sqlite3. Now the catch, the only documentation I have found for accessing a stored procedure with a table valued parameter uses ADO.Net and VB, nothing in Python. Unfortunately, I’m not enough of a programmer to translate. Has anyone used a SQL Server stored procedure with a table valued parameter? Is there another approach I should look into?
Here are some links:
Decent explanation of table valued parameters and how to set them up in SQL and using in .Net
http://www.sqlteam.com/article/sql-server-2008-table-valued-parameters
http://msdn.microsoft.com/en-us/library/bb675163.aspx#Y2142
Explanation of using ADO in Python – almost what I need, just need the structured parameter type.
http://www.mayukhbose.com/python/ado/ado-command-3.php
My simple code
--TSQL to create type on SQL database
create Type PropIDList as Table
(Prop_Id BigInt primary key)
--TSQL to create stored procedure on SQL database. Note reference to
create procedure PropIDListTest #PIDList PropIDList READONLY
as
SET NOCOUNT ON
select * from
#PIDList p
SET NOCOUNT OFF
--TSQL to test objects.
--Declare variable as user defined type (table that has prop_id)
declare #pidlist as propidlist
--Populate variable
insert into #pidlist(prop_id)
values(1000)
insert into #pidlist(prop_id)
values(2000)
--Pass table variable to stored procedure
exec PropIDListTest #pidlist
Now the tough part – Python.
Here is the code creating the in memory table
import getopt, sys, string, os, tempfile, shutil
import _winreg,win32api, win32con
from win32com.client import Dispatch
from adoconstants import *
import sqlite3
conn1 = sqlite3.connect(':memory:')
c = conn1.cursor()
# Create table
c.execute('''create table PropList
(PropID bigint)''')
# Insert a row of data
c.execute("""insert into PropList
values (37921019)""")
# Save (commit) the changes
conn1.commit()
c.execute('select * from PropList order by propID')
# lets print out what we have to make sure it works
for row in c:
print row
Ok, my attempt at connecting through Python
conn = Dispatch('ADODB.Connection')
conn.ConnectionString = "Provider=sqloledb.1; Data Source=nt38; Integrated Security = SSPI;database=pubs"
conn.Open()
cmd = Dispatch('ADODB.Command')
cmd.ActiveConnection = conn
cmd.CommandType = adCmdStoredProc
cmd.CommandText = "PropIDListTest #pidlist = ?"
param1 = cmd.CreateParameter('#PIDList', adUserDefined) # I “think” the parameter type is the key and yes it is most likely wrong here.
cmd.Parameters.Append(param1)
cmd.Parameters.Value = conn1 # Yeah, this is probably wrong as well
(rs, status) = cmd.Execute()
while not rs.EOF:
OutputName = rs.Fields("Prop_ID").Value.strip().upper()
print OutputName
rs.MoveNext()
rs.Close()
rs = None
conn.Close()
conn = None
# We can also close the cursor if we are done with it
c.close()
conn1.close()

I have coded TVPs from ADO.NET before.
Here is a question on TVPs in classic ADO that I am interested in, sql server - Classic ADO and Table-Valued Parameters in Stored Procedure - Stack Overflow. It does not give a direct answer but alternatives.
The option of XML is easier, you have probably already considered it; it would require more server side processing.
Here is the msdn link for low level ODBC programming of TVPs.
Table-Valued Parameters (ODBC). This one is the closest answer if you can switch to ODBC.
You could pass a csv string to nvarchar(max) and then pass it to a CLR SplitString function, that one is fast but has default behaviour I disagree with.
Please post back what works or does not here.