So I'm reading a list of part numbers from excel using Pandas, which can be just about anything, like:
287380274-87
or
ME982394-01
or
HOU8929
that changes randomly based on what the user is looking for and can contain some bad numbers as well. Such as blanks, invalid characters (<, >, or !), as well as phrases, like '12390-01 to 04'. I don't care about filtering the part numbers for all of the random conditions that throw synxtax errors in SQL. But I am attempting to query a SAP database WHERE part number IN (list):
import pandas as pd
from hdbcli import dbapi
userFile = r'T:\H01 Cell\Projects\Part Breakdown Update Spreadsheet Improvements\2022.03.21 Part Breakdown update - VK.xlsm'
# read input from excel for part numbers to WHERE in queries
partNums = pd.read_excel\
(io=userFile, sheet_name='Inputs', usecols=lambda x: 'Unnamed' not in x,\
skiprows=1, dtype={'Part List' : str})
# Open SAP database connection
conn = dbapi.connect(address="server", port=####, user="XXXXX", password="XXXXXXX")
# Function to convert
def listToString(s):
# use list comprehension
listToStr = ', '.join([str(elem) for elem in s])
# return string
return listToStr
partNumStr = listToString(partNums['Part List'].drop_duplicates().tolist())
# GetInvOnHand()
# DISTINCT List
queryIOHlist = [
"PART_NO",
"PLANT",
"LOCATION_DESCRIPTION",
"VALUATION_TYPE"
]
queryIOHstr = listToString(queryIOHlist)
# our SQL query, select all from ' '
queryInvOnHand = (
"SELECT DISTINCT " +
queryIOHstr +
" FROM ZWILLIAMS.ZV_WI_GU_INVENTORY_ON_HAND_IM_THIN INV" +
" WHERE PART_NO IN " +
"(" +
partNumStr +
")"
)
# pandas read SQL to store SQL table in dataframe
inventoryOnHand = pd.read_sql(queryInvOnHand, conn)
conn.close()
I'm running into synxtax errors for my SQL query because of these bad part numbers, such as:
(257, 'sql syntax error: incorrect syntax near "to": line 1 col 8120 (at pos 8120))
where the part number it doesn't like is: 62219-01 to -04
In SQL, is there a way to just skip that number if not found in the Part Numbers column in the table? Ideally, it would just be something like:
if syntaxError:
continue
and then just not record anything in my dataframe for that part number.
I have tried to query all elements from a list into an insertion query, I tried by making the list into a list of tuples and directly by adding the elements from the list. But it did not work, and I don't know the best practice for this as I am no SQL shark. I have below added the two different outputs I have before I do the query. I don't know which is easier to work with for this, but the code example is for the list of elements.
Output
['testuser', 'AskeMeyer']
and
[('testuser',), ('AskeMeyer',)]
Code to query
try:
conn = psycopg2.connect(host=ENDPOINT, port=PORT, database=DBNAME, user=USER, password=PASS, sslmode='require', sslrootcert="SSLCERTIFICATE")
cur = conn.cursor()
var_string = ', '.join(map(str, res))
sql = 'INSERT INTO users_from_group(name) VALUES %s;' % var_string)
cur.execute(sql)
error from above
Database connection failed due to syntax error at or near ")"
You need to use brackets and quotes in your insert statement, for example:
var_string = ', '.join([f"(\"{name}\")" for name in ['testuser', 'AskeMeyer']])
Also, there is a bracket at the end of your sql statement which causes the syntax error:
sql = 'INSERT INTO users_from_group(name) VALUES %s;' % var_string)
Should be
sql = 'INSERT INTO users_from_group(name) VALUES %s;' % var_string
it's not the proper way to store lists.
you should save it as json :
json.dumps(['testuser', 'AskeMeyer'] )
then save it
Try this:
var_string = '(' + ','.join(map(str, res)) + ')'
sql = 'INSERT INTO users_from_group(name) VALUES %s;' % var_string
I have a MySQL query in Python using MySQLdb that I want to give me a different number of rows depending on if there is a LIMIT provided or not.
So, something like:
sql = "SELECT URL, Name AS Category
FROM sitelist sl
INNER JOIN sitecategory sc ON sc.PlacementCategory_id = sl.Category_id "
if limit > 0 :
sql += "LIMIT %s", (limit)
However, this fails:
TypeError: cannot concatenate 'str' and 'tuple' objects
How would I be able to concatenate the string containing the limit number while keeping the query 'safe'? In the case of the above being an individual string, the above works correctly.
In PHP, I would simply use a prepared PDO statement with bindparam(), but I cannot find anything similar in Python.
I think the neatest way to do this is:
sql = ("SELECT URL, Name AS Category "
"FROM sitelist sl "
"INNER JOIN sitecategory sc ON sc.PlacementCategory_id = sl.Category_id")
if limit > 0:
whatever.execute(sql+" LIMIT %s", (limit,))
else:
whatever.execute(sql)
Note that: sql should have opening and closing quotes on every line; and limit needs to be in a tuple as an argument to execute.
use string formatting.
sql = "SELECT URL, Name AS Category\n\
FROM sitelist sl\n\
INNER JOIN sitecategory sc ON sc.PlacementCategory_id = sl.Category_id "
if limit > 0: # you don't need parenthesis in python
sql += "LIMIT {}" .format(limit)
or use format specifier.
sql = "SELECT URL, Name AS Category\n\
FROM sitelist sl\n\
INNER JOIN sitecategory sc ON sc.PlacementCategory_id = sl.Category_id "
if(limit > 0):
sql += "LIMIT %d" %limit
So I'm currently linking Python with SQL to pull out customer information. Unfortunately, I'm getting some errors with regards to SQL. I am trying to use the LIKE operator, and the % wildcard, but I keep getting errors because Python does not like %. As a result, it pretends that variable between the %s do not exist. Here's what I mean:
SELECT custnbr,
firstname,
middleint,
lastname
FROM lqppcusmst
WHERE custnbr = ? AND firstname LIKE ?
Right now, I'm just testing it out, so I'm just using the customer number, and the first name. I give it a value:
remote_system_account_number = request.DATA['remote_system_account_number']
remote_system_first_name = request.DATA['remote_system_first_name']
Since what I'm writing is for searching customers within the database, there's a chance there could be blank entries, so I have it like such:
if remote_system_account_number != '':
SQL_where += ' custnbr = ? '
parameters += "remote_system_account_number"
if remote_system_first_name != '':
SQL_where += ' AND firstname LIKE ? '
parameters += ", %remote_system_first_name%"
So I thought this would work, but it didn't. When I execute it like such:
database_cursor.execute(customer_information_SQLString + SQL_where, parameters)
I get this:
ProgrammingError: ('The SQL contains 2 parameter markers, but 1 parameters were supplied', 'HY000')
Anyone know how to deal with this?
parameters should not be a comma separated string, it should be an enumerable (a list or similar) with a number of values matching the number of placeholders in your SQL. For instance:
parameters = []
if remote_system_account_number != '':
SQL_where += ' custnbr = ? '
parameters.append("remote_system_account_number")
if remote_system_first_name != '':
SQL_where += ' AND firstname LIKE ? '
parameters.append("%remote_system_first_name%")
I am trying to use a dict to do a SQL INSERT. The logic would basically be:
INSERT INTO table (dict.keys()) VALUES dict.values()
However, I am having a tough time figuring out the correct syntax / flow to do this. This is what I currently have:
# data = {...}
sorted_column_headers_list = []
sorted_column_values_list = []
for k, v in data.items():
sorted_column_headers_list.append(k)
sorted_column_values_list.append(v)
sorted_column_headers_string = ', '.join(sorted_column_headers_list)
sorted_column_values_string = ', '.join(sorted_column_values_list)
cursor.execute("""INSERT INTO title (%s)
VALUES (%s)""",
(sorted_column_headers_string, sorted_column_values_string))
From this I get a SQL exception (I think related to the fact that commas are also included in some of the values that I have). What would be the correct way to do the above?
I think the comment on using this with MySQL is not quite complete. MySQLdb doesn't do parameter substitution in the columns, just the values (IIUC) - so maybe more like
placeholders = ', '.join(['%s'] * len(myDict))
columns = ', '.join(myDict.keys())
sql = "INSERT INTO %s ( %s ) VALUES ( %s )" % (table, columns, placeholders)
# valid in Python 2
cursor.execute(sql, myDict.values())
# valid in Python 3
cursor.execute(sql, list(myDict.values()))
You're not getting escaping on the columns though, so you might want to check them first....
See http://mail.python.org/pipermail/tutor/2010-December/080701.html for a more complete solution
You want to add parameter placeholders to the query. This might get you what you need:
qmarks = ', '.join('?' * len(myDict))
qry = "Insert Into Table (%s) Values (%s)" % (qmarks, qmarks)
cursor.execute(qry, myDict.keys() + myDict.values())
Always good answers here, but in Python 3, you should write the following:
placeholder = ", ".join(["%s"] * len(dict))
stmt = "insert into `{table}` ({columns}) values ({values});".format(table=table_name, columns=",".join(dict.keys()), values=placeholder)
cur.execute(stmt, list(dict.values()))
Don't forget to convert dict.values() to a list because in Python 3, dict.values() returns a view, not a list.
Also, do NOT pour the dict.values() in stmt because it tears a quote out of a string by joining it, which caused MySQL error in inserting it. So you should always put it in cur.execute() dynamically.
I'm a little late to the party but there is another way that I tend to prefer since my data is usually in the form of a dict already. If you list the bind variables in the form of %(columnName)s you can use a dictionary to bind them at execute. This partially solves the problem of column ordering since the variables are bound in by name. I say partially because you still have to make sure that the columns & values portion of the insert are mapped correctly; but the dictionary itself can be in any order (since dicts are sort of unordered anyway)
There is probably a more pythonic way to achieve all this, but pulling the column names into a list and working off it ensures we have a static ordering to build the columns & values clauses.
data_dict = {'col1': 'value 1', 'col2': 'value 2', 'col3': 'value 3'}
columns = data_dict.keys()
cols_comma_separated = ', '.join(columns)
binds_comma_separated = ', '.join(['%(' + item + ')s' for item in columns])
sql = f'INSERT INTO yourtable ({cols_comma_separated}) VALUES ({binds_comma_separated})'
cur.execute(sql, data_dict)
Now whether or not it is a good idea to dynamically build your columns & values clause like this is a topic for a SQL injection thread.
table='mytable'
columns_string= '('+','.join(myDict.keys())+')'
values_string = '('+','.join(map(str,myDict.values()))+')'
sql = """INSERT INTO %s %s
VALUES %s"""%(table, columns_string,values_string)
I tried #furicle's solution but it still inputs everything as a string - if your dict is a mixed one then this may not work as you would want it to. I had a similar issue and this is what I came up with - this is only a query builder and you could use it (with changes) to work with any database of your choice. Have a look!
def ins_query_maker(tablename, rowdict):
keys = tuple(rowdict)
dictsize = len(rowdict)
sql = ''
for i in range(dictsize) :
if(type(rowdict[keys[i]]).__name__ == 'str'):
sql += '\'' + str(rowdict[keys[i]]) + '\''
else:
sql += str(rowdict[keys[i]])
if(i< dictsize-1):
sql += ', '
query = "insert into " + str(tablename) + " " + str(keys) + " values (" + sql + ")"
print(query) # for demo purposes we do this
return(query) #in real code we do this
This is crude and still needs sanity checks, etc, but it works as intended.
for a dict:
tab = {'idnumber': 1, 'fname': 'some', 'lname': 'dude', 'dob': '15/08/1947', 'mobile': 5550000914, 'age' : 70.4}
running the query I get the following output
results of query generated by the suite
This code worked for me (Python 3):
fields = (str(list(dictionary.keys()))[1:-1])
values = (str(list(dictionary.values()))[1:-1])
sql = 'INSERT INTO Table (' + fields + ') VALUES (' + values + ')'
cursor.execute(sql)
It does rely on the dictionary outputting its keys and values in the same order. I'm unclear if this is always true :)
When constructing queries dynamically it's important to ensure that both identifiers and values are correctly quoted. Otherwise you risk
SQL injection if untrusted data is processed
Errors if the column names require quoting (for example embedded spaces)
Data corruption or errors if values are incorrectly quoted (for example 2021-07-11 unquoted may be evaluated as 2003)
Quoting values is best delegated to the DB-API connector. However connector packages don't always provide a way to quote identifiers, so you may need to do this manually. MySQL uses backticks (`) to quote identifiers.
This code quotes identifiers and values. It works for MySQLdb, mysql.connector and pymysql and works for Python 3.5+.
data = {'col1': val1, 'col2': val2, ...}
# Compose a string of quoted column names
cols = ','.join([f'`{k}`' for k in data.keys()])
# Compose a string of placeholders for values
vals = ','.join(['%s'] * len(data))
# Create the SQL statement
stmt = f'INSERT INTO `tbl` ({cols}) VALUES ({vals})'
# Execute the statement, delegating the quoting of values to the connector
cur.execute(stmt, tuple(data.values()))
This is based on other answers here, but it uses back ticks around column names for cases in which you are using reserved words as column names and it it ensures that column names only contain letters, numbers, and underscores to thwart SQL injection attacks.
I've also written a similar upsert that works the same way as the insert but which overwrites data that duplicates the primary key.
import mysql.connector
import re
cnx = mysql.connector.connect(...)
def checkColumnNames(data):
for name in data.keys():
assert re.match(r'^[a-zA-Z0-9_]+$',name), "Bad column name: " + name
def insert(table, data):
checkColumnNames(data)
assert table, "No table specified"
placeholders = ', '.join(['%s'] * len(data))
columns = '`,`'.join(data.keys())
sql = "INSERT INTO `%s` (`%s`) VALUES (%s);" % (table, columns, placeholders)
cnx.cursor().execute(sql, list(data.values()))
def upsert(table, data):
checkColumnNames(data)
assert table, "No table specified"
placeholders = ', '.join(['%s'] * len(data))
columns = '`,`'.join(data.keys())
updates = '`' + '`=%s,`'.join(data.keys()) + '`=%s'
sql = "INSERT INTO `%s` (`%s`) VALUES (%s) ON DUPLICATE KEY UPDATE %s" % (table, columns, placeholders, updates)
cnx.cursor().execute(sql, list(data.values()) + list(data.values()))
Example usage
insert("animals", {
"id": 1,
"name": "Bob",
"type": "Alligator"
})
cnx.commit()
I used this thread for my usage and tried to keep it much simpler
ins_qry = "INSERT INTO {tablename} ({columns}) VALUES {values};" .format(
tablename=my_tablename,
columns=', '.join(myDict.keys()),
values=tuple(myDict.values())
)
cursor.execute(ins_qry)
Make sure to commit the data inserted, either using db_connection.commit() and use cursor.lastrowid, if you need the primary key of the inserted row
This works for me
cursor.execute("INSERT INTO table (col) VALUES ( %(col_value) )",
{'col_value': 123})
if you have list in which there are number of dictionaries
for example: lst=[d1,d2,d3,d4]
then below one will worked for me:
for i in lst:
placeholders = ', '.join(['%s'] * len(i))
columns = ', '.join(i.keys())
sql = "INSERT INTO %s ( %s ) VALUES ( %s )" % (table, columns, placeholders)
cursor.execute(sql,list(i.values()))
conn.commit()
Note:Dont ever forget to commit otherwise you wont be able to see columns and values inserted in table
columns = ', '.join(str(x).replace('/', '_') for x in row_dict.keys())
values = ', '.join("'" + str(x).replace('/', '_') + "'" for x in row_dict.values())
sql = "INSERT INTO %s ( %s ) VALUES ( %s );" % ("tablename", columns, values)
applicable for python3
Let's say our data is:
data = {
"name" : "fani",
"surname": "dogru",
"number" : 271990
}
This is my shorter version:
tablo = "table_name"
cols = ','.join([f" {k}" for k in data.keys()])
vals = ','.join([f"'{k}'" for k in data.values()])
stmt = f'INSERT INTO {tablo} ({cols}) VALUES ({vals})'
What about:
keys = str(dict.keys())
keys.replace('[', '(')
keys.replace(']', ')')
keys.replace("'",'')
vals = str(dict.values())
vals.replace('[', '(')
vals.replace(']', ')')
cur.execute('INSERT INTO table %s VALUES %s' % (keys, vals))
For python 3:
keys = str(dict.keys())[9:].replace('[', '').replace(']', '')
vals = str(dict.values())[11:].replace('[', '').replace(']', '')
...