I have a MySQL query in Python using MySQLdb that I want to give me a different number of rows depending on if there is a LIMIT provided or not.
So, something like:
sql = "SELECT URL, Name AS Category
FROM sitelist sl
INNER JOIN sitecategory sc ON sc.PlacementCategory_id = sl.Category_id "
if limit > 0 :
sql += "LIMIT %s", (limit)
However, this fails:
TypeError: cannot concatenate 'str' and 'tuple' objects
How would I be able to concatenate the string containing the limit number while keeping the query 'safe'? In the case of the above being an individual string, the above works correctly.
In PHP, I would simply use a prepared PDO statement with bindparam(), but I cannot find anything similar in Python.
I think the neatest way to do this is:
sql = ("SELECT URL, Name AS Category "
"FROM sitelist sl "
"INNER JOIN sitecategory sc ON sc.PlacementCategory_id = sl.Category_id")
if limit > 0:
whatever.execute(sql+" LIMIT %s", (limit,))
else:
whatever.execute(sql)
Note that: sql should have opening and closing quotes on every line; and limit needs to be in a tuple as an argument to execute.
use string formatting.
sql = "SELECT URL, Name AS Category\n\
FROM sitelist sl\n\
INNER JOIN sitecategory sc ON sc.PlacementCategory_id = sl.Category_id "
if limit > 0: # you don't need parenthesis in python
sql += "LIMIT {}" .format(limit)
or use format specifier.
sql = "SELECT URL, Name AS Category\n\
FROM sitelist sl\n\
INNER JOIN sitecategory sc ON sc.PlacementCategory_id = sl.Category_id "
if(limit > 0):
sql += "LIMIT %d" %limit
Related
So I'm reading a list of part numbers from excel using Pandas, which can be just about anything, like:
287380274-87
or
ME982394-01
or
HOU8929
that changes randomly based on what the user is looking for and can contain some bad numbers as well. Such as blanks, invalid characters (<, >, or !), as well as phrases, like '12390-01 to 04'. I don't care about filtering the part numbers for all of the random conditions that throw synxtax errors in SQL. But I am attempting to query a SAP database WHERE part number IN (list):
import pandas as pd
from hdbcli import dbapi
userFile = r'T:\H01 Cell\Projects\Part Breakdown Update Spreadsheet Improvements\2022.03.21 Part Breakdown update - VK.xlsm'
# read input from excel for part numbers to WHERE in queries
partNums = pd.read_excel\
(io=userFile, sheet_name='Inputs', usecols=lambda x: 'Unnamed' not in x,\
skiprows=1, dtype={'Part List' : str})
# Open SAP database connection
conn = dbapi.connect(address="server", port=####, user="XXXXX", password="XXXXXXX")
# Function to convert
def listToString(s):
# use list comprehension
listToStr = ', '.join([str(elem) for elem in s])
# return string
return listToStr
partNumStr = listToString(partNums['Part List'].drop_duplicates().tolist())
# GetInvOnHand()
# DISTINCT List
queryIOHlist = [
"PART_NO",
"PLANT",
"LOCATION_DESCRIPTION",
"VALUATION_TYPE"
]
queryIOHstr = listToString(queryIOHlist)
# our SQL query, select all from ' '
queryInvOnHand = (
"SELECT DISTINCT " +
queryIOHstr +
" FROM ZWILLIAMS.ZV_WI_GU_INVENTORY_ON_HAND_IM_THIN INV" +
" WHERE PART_NO IN " +
"(" +
partNumStr +
")"
)
# pandas read SQL to store SQL table in dataframe
inventoryOnHand = pd.read_sql(queryInvOnHand, conn)
conn.close()
I'm running into synxtax errors for my SQL query because of these bad part numbers, such as:
(257, 'sql syntax error: incorrect syntax near "to": line 1 col 8120 (at pos 8120))
where the part number it doesn't like is: 62219-01 to -04
In SQL, is there a way to just skip that number if not found in the Part Numbers column in the table? Ideally, it would just be something like:
if syntaxError:
continue
and then just not record anything in my dataframe for that part number.
I have to connect to an Oracle database and see if a table exists. While I can get a list of the tables, I'm having trouble seeing if the table I'm looking for is in the list. Some tables have associated table which I'll have to join on, some do not, thus I have to check.
What is in my list: ('NYSDOH_CI_EI_HOSPITAL',)
sql = "SELECT table_name FROM all_tables"
cur.execute(sql)
searchstr = 'NYSDOH_CI_EI_HOSPITAL'
p = re.compile(searchstr)
#create data array to load in SQL results in.
ciDataSet = []
cxRows = cur.fetchall()
for i in cxRows:
#print i # list of tables
if p.match(str(i)):
print i
It doesn't find it, even if I use a wildcard.
fetchall() returns a list of tuples.
So when you do
for i in cxRows:
'i' is of type tuple. In your case, this tuple will have only single value. You can access it using i[0] and match it with p.
Currently you are converting a tuple to string so regular expression will not match.
Corrected code:
sql = "SELECT table_name FROM all_tables"
cur.execute(sql)
searchstr = 'NYSDOH_CI_EI_HOSPITAL'
p = re.compile(searchstr)
#create data array to load in SQL results in.
ciDataSet = []
cxRows = cur.fetchall()
for i in cxRows:
#print i # list of tables
if p.match(str(i[0])):
print i
To improve on the syntax of #vaichidrewar, you could simplify the fetch loop to:
for tabname, in cur:
if p.match(str(tabname)):
print(tabname)
But it's going to be more efficient to do the reg exp matching in the query:
sql = "select table_name from all_tables where regexp_like(table_name, :tn, 'i')"
searchstr = 'EMP'
cur.execute(sql, (searchstr,))
for tabname, in cur:
print(tabname)
The 'i' option does a case-insensitive match. You can adjust the regexp as you like.
I'm newbie to python and trying to update one column value (true/false) of all rows (ids provide in list) using IN on list.
My query is like this :
qry = "Update table Set "\
"col1 = %s "\
"where col2 in ( " + ",".join( "%s" for _ in lstIds)
qry += ")"
cursor.execute(qry,(col1Value, lstIds))
But I'm getting error
not enough arguments for format string
I've tried different solutions but unable to resolve my issue.
When I change my query this way, it works fine
qry = "Update table Set "\
"col1 = 1 "\
"where col2 in ( " + ",".join( "%s" for _ in lstIds)
qry += ")"
cursor.execute(qry,( lstIds))
So I guess issue is in first parameter value but I dont know how to resolve this.
Any kind of help will be appreciated.
You need to include each of the values in lstIds as a separate parameter:
cursor.execute(qry, (col1Value,) + lstIds)
assuming lstIds is a tuple, or
cursor.execute(qry, [col1Value] + lstIds)
if it is a list.
Or you can combine the sequence into one list with the * notation in Python 3.5 and newer:
cursor.execute(qry, [col1Value, *lstIds])
I'm looking to take an array list and attach it to a string.
Python 2.7.10, Windows 10
The list is loaded from a mySQL table and the output is this:
skuArray = [('000381001238',) ('000381001238',) ('000381001238',) ('FA200513652',) ('000614400967',)]
I'm wanting to take this list and attach it to a separate query
the problem:
query = "SELECT ItemLookupCode,Description, Quantity, Price, LastReceived "
query = query+"FROM Item "
query = query+"WHERE ItemLookupCode IN ("+skuArray+") "
query = query+"ORDER BY LastReceived ASC;"
I get the error:
TypeError: cannot concatenate 'str' and 'tuple' objects
My guess here is that I need to format the string as:
'000381001238', '000381001238', '000381001238', 'FA200513652','000614400967'
Ultimately the string needs to read:
query = query+"WHERE ItemLookupCode IN ('000381001238', '000381001238', '000381001238', 'FA200513652','000614400967') "
I have tried the following:
skuArray = ''.join(skuArray.split('(', 1))
skuArray = ''.join(skuArray.split(')', 1))
Second Try:
skus = [sku[0] for sku in skuArray]
stubs = ','.join(["'?'"]*len(skuArray))
msconn = pymssql.connect(host=r'*', user=r'*', password=r'*', database=r'*')
cur = msconn.cursor()
query ='''
SELECT ItemLookupCode,Description, Quantity, Price, LastReceived
FROM Item
WHERE ItemLookupCode IN { sku_params }
ORDER BY LastReceived ASC;'''.format(sku_params = stubs)
cur.execute(query, params=skus)
row = cur.fetchone()
print row[3]
cur.close()
msconn.close()
Thanks in advance for your help!
If you want to do the straight inline SQL you could use a list comprehension:
', '.join(["'{}'}.format(sku[0]) for sku in skuArray])
Note: You need to add commas between tuples (based on example)
That said, if you want to do some sql, I would encourage you to parameterize your request with ?
Here is an example of how you would do something like that:
skuArray = [('000381001238',), ('000381001238',), ('000381001238',), ('FA200513652',), ('000614400967',)]
skus = [sku[0] for sku in skuArray]
stubs = ','.join(["'?'"]*len(skuArray))
qry = '''
SELECT ItemLookupCode,Description, Quantity, Price, LastReceived
FROM Item
WHERE ItemLookupCode IN ({ sku_params })
ORDER BY LastReceived ASC;'''.format(sku_params = stubs)
#assuming pyodbc connection syntax may be off
conn.execute(qry, params=skus)
Why?
Non-parameterized queries are a bad idea because it leaves you vulnerable to sql injection and is easy to avoid.
Assuming that skuArray is a list, like this:
>>> skuArray = [('000381001238',), ('000381001238',), ('000381001238',), ('FA200513652',), ('000614400967',)]
You can format your string like this:
>>> ', '.join(["'{}'".format(x[0]) for x in skuArray])
"'000381001238', '000381001238', '000381001238', 'FA200513652', '000614400967'"
I am trying to python to generate a script that generates unload command in redshift. I not an expert Python programmer. I need to where I can generate all columns for the unload list. If the column is of specific name, I need to replace with a function. The challenge I am facing is it appending "," to last item in the dictionary. Is there a way I can avoid the last comma? Any help would be appreciated.
import psycopg2 from psycopg2.extras
import RealDictCursor
try:
conn = psycopg2.connect("dbname='test' port='5439' user='scott' host='something.redshift.amazonaws.com' password='tiger'");
except:
print "Unable to connect to the database"
conn.cursor_factory = RealDictCursor
cur = conn.cursor()
conn.set_isolation_level( psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT )
try:
cur.execute("SELECT column from pg_table_def where schema_name='myschema' and table_name ='tab1' " );
except:
print "Unable to execute select statement from the database!"
result = cur.fetchall()
print "unload mychema.tab1 (select "
for row in result:
for key,value in row.items():
print "%s,"%(value)
print ") AWS Credentials here on..."
conn.close()
Use the join function on the list of values in each row:
print ",".join(row.values())
Briefly, the join function is called on a string which we can think of as the "glue", and takes a list of "pieces" as its argument. The result is a string of "pieces" "held together" by the "glue". Example:
>>> glue = "+"
>>> pieces = ["a", "b", "c"]
>>> glue.join(pieces)
"a+b+c"
(since row.values() returns a list, you don't actually need the comprehension, so it's even simpler than I wrote it at first)
Infact, this worked better.
columns = []
for row in result:
if (row['column_name'] == 'mycol1') or (row['column_name'] == 'mycol2') :
columns.append("func_sha1(" + row['column_name'] + "||" + salt +")")
else:
columns.append(row['column_name'])
print selstr + ",".join(columns)+" ) TO s3://"
Thanks for your help, Jon