I am using pymyql/mysql-connector to write the messages to mysql database. The messages are processed on callback (paho.mqtt callback) from mqtt broker.I have 4 different tables and based on the message type, I am inserting messages into database. I have written the insert queries as below. this way of writing leads to sql injections it seems.Any suggestions how can I improve the insert query statements?
# callback attached to paho.mqtt.client
def on_message(self, client, userdata, msg):
if msg.topic.startswith("topic1/"):
self.bulkpayload += "(" + msg.payload.decode("utf-8") + "," + datetime + "),"
elif msg.topic.startswith("topic2/"):
self.insertStatement += "INSERT INTO mydatabase.table1 VALUES (" + msg.payload.decode("utf-8") + "," + datetime + ");"
elif msg.topic.startswith("topic3/")
self.insertStatement += "INSERT INTO mydatabase.table2 VALUES (" +msg.payload.decode("utf-8") + "," + datetime + ");"
elif msg.topic.startswith("messages"):
self.insertStatement += "INSERT INTO mydatabase.table3 VALUES ('" + msg.topic + "'," + msg.payload.decode("utf-8") + "," + datetime + ");"
else:
return # do not store in DB
cursor.execute(self.insertStatement)
cursor.commit()
Make your query use parameters. Much less chance of injection:
cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))
credit (and more info) here: How to use variables in SQL statement in Python?
Also, Dan Bracuk is correct - make sure you validate your params before executing the SQL if you aren't already
Related
I have a large csv file of 30 million rows(1.6 gb) and I am using pymysql to load the data from csv to mysql tables.
I have removed all constraints in table schema to make load faster and have also set timeout values to large values.
def setTimeOutLimit(connection):
try:
with connection.cursor() as cursor:
query = "SET GLOBAL innodb_lock_wait_timeout = 28800"
cursor.execute(query)
query2 = "SET innodb_lock_wait_timeout = 28800"
cursor.execute(query2)
query3 = "SET GLOBAL connect_timeout = 28800"
cursor.execute(query3)
query4 = "SET GLOBAL wait_timeout = 28800"
cursor.execute(query4)
query5 = "SET GLOBAL interactive_timeout = 28800"
cursor.execute(query5)
query6 = "SET GLOBAL max_allowed_packet = 1073741824"
cursor.execute(query6)
except:
conn.close()
sys.exit(" Could not set timeout limit ")
The data gets inserted into the table but I need to make one of the column as Primary Key and so I am creating another table that makes that column primary index by ignoring duplicate values. (tableName_1 is old table tableName is new table)
def createNewTableFromOld(connection, tableName):
try:
pprint( " Creating new table from old table with constraints" )
with connection.cursor() as cursor:
query = (" CREATE TABLE " + tableName +
" Like " + tableName + "_1")
cursor.execute(query)
query2 = (" ALTER TABLE " + tableName +
" ADD PRIMARY KEY(TimeStamp) ")
cursor.execute(query2)
query3 = (" INSERT IGNORE INTO " + tableName +
" SELECT * FROM " + tableName + "_1")
cursor.execute(query3)
query4 = ("DROP TABLE " + tableName + "_1")
cursor.execute(query4)
connection.commit()
except:
conn.close()
sys.exit(" Could not create table with Primary Key ")
During this method execution, somewhere after 5-6 minutes I get this error,
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query ([WinError 10054] An existing connection was forcibly closed by the remote host)')
And when I check services, MYSQL80 automatically crashed and stopped. I have also set max_allowed_packet_size to 1 gb in my.ini file and all timeouts are manually set to 8 hours. What could be the issue?
The original table schema is:
query = ("CREATE TABLE IF NOT EXISTS " + table + " ("
" TimeStamp DECIMAL(15, 3), " +
" Value DECIMAL(30, 11), " +
" Quality INT, " +
" TagName varchar(30) )"
)
I finally solved the issue by setting the innodb_buffer_pool_size in my.ini file to 2GB which was earlier only 4M.
I need to insert a record into the table MemberDetails but I end up with the error stating " _mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'I' at line 1") "
sql5="INSERT INTO MemberDetails VALUES"+"("+"'2016-2020','"+PhoneNum+"','VEC','"+Address+"','"+StuID+"','"+DOB+"','1000','"+Name+"','"+Pass+"','"+dept+"','"+year+"','"+sec+"')"
cursor.execute(sql5)
You need to add spacing as well as the columns in the table each variable will be inserted into, your query should be like this:
"INSERT INTO MemberDetails (datetime, phone, code, address, stuid, birthdate, num, name, pass, dept, year, sec) VALUES (" + "'2016-2020','" + PhoneNum + "','VEC','" + Address + "','" + StuID + "','" +DOB +"','1000','" + Name + "','" + Pass + "','" + dept + "','" + year + "','" +sec + "')"
Just replace the variables in the first set of brackets with the column name you want each variable to be put into.
I have the following values In Python values:
_sfwBIPMKEeCTfuMZpaDEBQ Atomic Warehouse Model None
These values are sent through a function and is being put into a db3 table like this:
INSERT INTO [PackageRoster] (PkgID, Package, PkgParentID) VALUES ('_sfwBIPMKEeCTfuMZpaDEBQ', 'Atomic Warehouse Model', 'None');
I do not want the string 'None' put into the table. I would like NULL.
How do I tell Python to inject NULL instead of None?
Here's the function that works the statement (includes debug feature):
def StorePackage(PkgID, Package, PkgParentID):
dbutils.ResetTable()
conn = sqlite3.connect(dbutils.db)
try:
conn.execute("INSERT INTO " + dbutils.table + " (PkgID, Package, PkgParentID) VALUES ('" + PkgID + "', '" + Package + "', '" + PkgParentID + "');")
except:
print("INSERT INTO " + dbutils.table + " (PkgID, Package, PkgParentID) VALUES ('" + PkgID + "', '" + Package + "', '" + str(PkgParentID) + "');")
conn.commit()
Thanks
I initially commented a recommendation to use parameter substitution, assuming it wasn't actually related. On further inspection, though, I think it's exactly what you're looking for.
Your code currently inserts all strings because you're explicitly converting everything to a string in Python. Parameter substitution enables Python to handle how to get its values into the database, and also protects you from SQL injection attacks.
conn.execute("INSERT INTO " + dbutils.table + " (PkgID, Package, PkgParentID) VALUES (?,?,?)", (PkgID, Package, PkgParentID))
A Python None object will be inserted as an SQLite NULL using this method. You can read more about parameter substitution in the docs, as well as how Python values are converted to SQLite values and vice versa.
You are converting all of your values to strings, because you add quotes arround them. I.e, because you do '" + str(PkgParentID) + "' it will end up as 'None' instead of None.
You should for example convert your values to a string before the query, and replace them with NULL when they are None:
def StorePackage(PkgID, Package, PkgParentID):
PkgParentID = "'{}'".format(PkgParentID) if PkgParentID is not None else "NULL"
# Same for other parameters if desired
dbutils.ResetTable()
conn = sqlite3.connect(dbutils.db)
query = "INSERT INTO " + dbutils.table + " (PkgID, Package, PkgParentID) VALUES ('" + PkgID + "', '" + Package + "', " + PkgParentID + ");"
try:
conn.execute(query)
except:
print(query)
conn.commit()
Also you should use parameter substitution, although thats not what is causing your issue in the first place
I have a web form taking user data and putting it into a mysql database. I need to trim leading and trailing blanks/spaces. I currently am utilizing the strip() method, but it only trims them for the fields, NOT for the mysql database.
My code is:
first_name = first_name.strip()
last_name = last_name.strip()
so on and so forth. It strips it perfectly fine for the webpage, but not when it is entered into the SQL database. The spaces still exist. How do I remove them?
EDIT:
db = MySQLdb.connect("localhost","user","pass","db_name")
cursor = db.cursor()
cursor.execute("Select * FROM registrants")
cursor.execute("INSERT INTO registrants VALUES( " + "'" + first_name + "'" + ", " + "'" + last_name + "'" + ");")
db.commit()
db.close()
It could be a scope issue.
If the stripping occurs in a different scope (ex: first_name is a global variable and the strip() occurs in a function) then you will not benefit from it in another scope (if the insert is happening in another function for example).
Have you tried this for a test:
cursor.execute("INSERT INTO registrants VALUES( " + "'" + first_name.strip() + "'" + ", " + "'" + last_name.strip() + "'" + ");")
btw, how do you know there's a space in the db? There could be an issue with the way the data is retrieved or displayed..
I think you should be passing the values into the INSERT differently
cursor.execute("INSERT INTO registrants (fname, lname) VALUES (%s, %s)", (first_name, last_name)
I'm not sure if that's where you're getting the whitespace, but it opens you up to sql injection so it's bad form to put the variables straight into the query.
Im running into this error that I can't work out
Im writing some code in Python using tkinter interface to transfer data from a text file to sqlite.
first, here is the relevant code:
def submit_data(self):
self.new_filename = self.file_entry.get()
self.new_tablename = self.table_entry.get()
self.new_fieldname = self.field_entry.get().split(',')
#print(self.new_fieldname)
self.create_new.destroy()
from sqlite3 import dbapi2 as sqlite
con = sqlite.connect(self.new_filename)
cur = con.cursor()
cur.execute('CREATE TABLE ' + self.new_tablename + '(id INTEGER PRIMARY KEY)')
for field in self.new_fieldname:
cur.execute('ALTER TABLE ' + self.new_tablename + ' ADD ' + field)
with open(self.filename, 'r', encoding='latin-1') as self.the_file:
status = True
#_keynumber=1
while status:
_row = self._next_line()
if _row:
_entry_list = _row.split(',')
# add space after text line comma for formatting
_entry_list = ', '.join(_entry_list)
#print(_entry_list)
#entries = {'row': _keynumber, 'entry': _entry_list}
#row_entry = "INSERT INTO " + self.new_tablename + " VALUES(" + _entry_list + ")"
cur.execute("INSERT INTO " + self.new_tablename + " VALUES(" + _entry_list + ")")
#_colrange = range(_colamount)
#_keynumber+=1
else:
status = False
con.commit()
At the cur.execute("INSERT INTO " ... line (about 6 lines up) I get this error:
** cur.execute("INSERT INTO " + self.new_tablename + " VALUES(" + _entry_list + ")")
sqlite3.OperationalError: near ".": syntax error**
I have changed this around in many different ways. At one time I had the whole "INSERT INTO ... VALUES ...." string as a variable and used
cur.execute(*variable*)
when I did it this way the error was the same except "OperationalError: near "." was "OperationalError: near "of" ... and there was no 'of' anywhere.
Im really confused and frustrated. Someone break this down for my please??
Thanks
F
the text file lines its reading are set up like this:
A Big Star In Hollywood,Sandra Dickinson
so I had figured that if I use .join() to put a space after the comma then the string would be the equivalent of two VALUES for the INSERT INTO statement.
Remove
_entry_list = ', '.join(_entry_list)
and use
cur.execute("INSERT INTO " + self.new_tablename + "(" + ",".join(self.new_fieldname) +") VALUES(" + ",".join(("?" for i in xrange(len(_entry_list)))) + ")", _entry_list)
This will parameterize your query and automatically quote all value in _entry_list.
You still have to manually quote self.new_tablename and self.new_fieldname. This should be before you use them in any sql statements.
You need to quote your strings.
As written, your SQL statement is:
INSERT INTO foo VALUES(Hello there, world, I am, unquoted, string, not good)
You should use:
INSERT INTO foo VALUES("Hello there","world","I am","quoted","string","hooray")
I suggest you do the following:
a) Instead of executing the statement, print it to the console. Change the line:
cur.execute("INSERT INTO " + self.new_tablename + ...)
to:
print "INSERT INTO " + self.new_tablename + ...
b) Run your program after making this change. Take a look at the SQL statements that you print to the console. Are they valid SQL statements? Start a SQLite command-line, and copy/paste the statements produced by your program. Does SQLite give any errors when you try to execute the pasted statements?