During an insert into MySQL 5.x via python 3.x I do receive an error which I can not find the reason for. It is a simple insert:
self.curr.execute("""
INSERT IGNORE INTO test (
`name`
)
VALUES (%s)
""", (
item['test']
)
)
Leading to this error:
Failed to save datasets. INSERT IGNORE INTO test (
`name`
)
VALUES (%s) Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%s)' at line 4
Everything looks good to me, what is the problem with this insert?
Python's DB-API, the standard to which your database module was developed, specifies that there must be a "sequence or mapping" supplied as the parameters argument to .execute().
Your argument was a string, so what was placed into the query was not understood by the database.
What is needed is to make the second (2nd) argument to .execute() into a sequence, and the usual sequence to use is a tuple. It's sometimes a surprise to people that a tuple is not formed by enclosing a value in parentheses/brackets, and that the trailing comma is necessary. Note that if you were inserting multiple values you wouldn't have run into this issue at all.
So what is needed is:
self.curr.execute("""
INSERT IGNORE INTO test (
`name`
)
VALUES (%s)
""", (
item['test'],
# ^ trailing comma!
)
)
There is at least one database module which allows just one value to be used (i.e. not a sequence) -- in a departure from the DB-API specifications; which has thrown me off in the past.
Related
Python mysql default escape function, corrupts the query.
Original Query string is following. It works fine and does add records to database as desired
INSERT IGNORE INTO state (`name`, `search_query`, `business_status`, `business_type`, `name_type`, `link`) VALUES ("test_name1", "test", "test_status", "test_b_typ", "test_n_typ", "test_link"), ("test_name2", "test", "test_status", "test_b_typ", "test_n_typ", "test_link")
But After escaping it to make sql Injection secure using the fuction
safe_sql = self.conn.escape_string(original_sql)
safe_sql being generated is following
b'INSERT IGNORE INTO state (`name`, `search_query`, `business_status`, `business_type`, `name_type`, `link`) VALUES (\\"test_name1\\", \\"test\\", \\"test_status\\", \\"test_b_typ\\", \\"test_n_typ\\", \\"test_link\\"), (\\"test_name2\\", \\"test\\", \\"test_status\\", \\"test_b_typ\\", \\"test_n_typ\\", \\"test_link\\")'
Now if I try to execute the safe_sql I get the syntax error below
MySQLdb._exceptions.ProgrammingError: (1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near \'\\"test_name1\\", \\"test\\", \\"test_status\\", \\"test_b_typ\\", \\"test_n_typ\\", \\"tes\' at line 1')
Which makes me wonder that if escape function I am using is either broken / uncompatibl or I am not using it the right way ? Also i am entering hundreds of records at one time, and due to the fast processing (which i purely assume) of single query as compared to prepared statements running hundreds of time, I am creating a large query.
You can't escape the entire query! You can't construct a query by randomly concatenating strings and then wave a magic wand over it and make it "injection secure". You need to escape every individual value before you put it into the query. E.g.:
"INSERT ... VALUES ('%s', ...)" % self.conn.escape_string(foo)
But really, your MySQL API probably offers prepared statements, which are much easier to use and less error prone. Something like:
self.conn.execute('INSERT ... VALUES (%s, %s, %s, ...)',
(foo, bar, baz))
I keep getting an OperationalError: Unrecognized Token. The error hapens when I'm attempting to insert data into my SQLite database using an SQLite Insert command. What do I need to do to correct this error or is there a better way I should go about inserting data into my database? The data is water level data measured in meters above chart datum and is gathered from water level gauge data loggers throughout the Great Lakes region of Canada and the US. The script uses the Pandas library and is hardcoded to merge data from water level gauging stations that are located in close proximity to each other. I'd like to use the insert command so I can deal with overlapping data when adding future data to the database. I won't even begin to pretend I know what I'm talking about with databases and programming so any help would be appreciated in how I can solve this error!
I've tried altering my script in the parameterized query to try and solve the problem without any luck as my research has said this is the likely culprit
# Tecumseh. Merges station in steps due to inability of operation to merge all stations at once. Starts by merging PCWL station to hydromet station followed by remaining PCWL station and 3 minute time series
final11975 = pd.merge(hydrometDF["Station11975"], pcwlDF["station11995"], how='outer', left_index=True,right_index=True)
final11975 = pd.merge(final11975, pcwlDF["station11965"], how='outer', left_index=True,right_index=True)
final11975 = pd.merge(final11975, cts, how='outer', left_index=True,right_index=True)
final11975.to_excel("C:/Users/Andrew/Documents/CHS/SeasonalGaugeAnalysis_v2/SeasonalGaugeAnalysis/Output/11975_Tecumseh.xlsx")
print "-------------------------------"
print "11975 - Tecumseh"
print(final11975.info())
final11975.index = final11975.index.astype(str)
#final11975.to_sql('11975_Tecumseh', conn, if_exists='replace', index=True)
#Insert and Ignore data into database to eliminate overlaps
testvalues = (final11975.index, final11975.iloc[:,0], final11975.iloc[:,1], final11975.iloc[:,2])
c.execute("INSERT OR IGNORE INTO 11975_Tecumseh(index,11975_VegaRadar(m),11995.11965), testvalues")
conn.commit()
I'd like the data to insert into the database using the Insert And Ignore command as data is often overlapping when its downloaded. I'm new to databases but I'm under the impression that the Insert and Ignore command will illiminate overlapping data. The message I receive when running my script is:
</> <Exception has occurred: OperationalError
unrecognized token: "11975_Tecumseh"
File "C:\Users\Documents\CHS\SeasonalGaugeAnalysis_v2\SeasonalGaugeAnalysis\Script\CombineStations.py", line 43, in <module>>
c.execute("INSERT OR IGNORE INTO 11975_Tecumseh(index,11975_VegaRadar(m),11995.11965), testvalues") </>
As per SQL Standards, You can create table or column name such as "11975_Tecumseh" and also Tecumseh_11975, but cannot create table or column name begin with numeric without use of double quotes.
c.execute("INSERT OR IGNORE INTO '11975_Tecumseh'(index,'11975_VegaRadar(m)',11995.11965), testvalues")
The error you are getting is because the table name 11975_Tecumseh is invalid as it stands as it is not suitably enclosed.
If you want to use a keyword as a name, you need to quote it. There
are four ways of quoting keywords in SQLite:
'keyword' A keyword in single quotes is a string literal.
"keyword" A keyword in double-quotes is an identifier. [keyword] A
keyword enclosed in square brackets is an identifier.
This is not
standard SQL. This quoting mechanism is used by MS Access and SQL
Server and is included in SQLite for compatibility. keyword A
keyword enclosed in grave accents (ASCII code 96) is an identifier.
This is not standard SQL. This quoting mechanism is used by MySQL and
is included in SQLite for compatibility. For resilience when
confronted with historical SQL statements, SQLite will sometimes bend
the quoting rules above:
If a keyword in single quotes (ex: 'key' or 'glob') is used in a
context where an identifier is allowed but where a string literal is
not allowed, then the token is understood to be an identifier instead
of a string literal.
If a keyword in double quotes (ex: "key" or "glob") is used in a
context where it cannot be resolved to an identifier but where a
string literal is allowed, then the token is understood to be a string
literal instead of an identifier.
Programmers are cautioned not to use the two exceptions described in
the previous bullets. We emphasize that they exist only so that old
and ill-formed SQL statements will run correctly. Future versions of
SQLite might raise errors instead of accepting the malformed
statements covered by the exceptions above.
SQL As Understood By SQLite - SQLite Keywords
The above is applied to invalid names, which includes names that start with numbers and names that include a non numeric inside parenthesises.
If 11975_Tecumseh is the actual table name then it must be enclosed e.g. [11975_Tecumseh]
Likewise the columns
index
11975_VegaRadar(m)
and 11995.11965
Also have to be suitably enclosed.
Doing so you'd end up with
"INSERT OR IGNORE INTO [11975_Tecumseh]([index],[11975_VegaRadar(m)],[11995.11965]), testvalues"
The the issues is that ,testvalues is syntactically incorrect. after the columns to insert into i.e. ([index],[11975_VegaRadar(m)],[11995.11965]) the keyword VALUES with the three values should be used.
An example of a valid statement is :
"INSERT INTO [11975_Tecumseh] ([index],[11975_VegaRadar(m)],[11995.11965]) VALUES('value1','value2','value3')"
As such
c.execute("INSERT INTO [11975_Tecumseh] ([index],[11975_VegaRadar(m)],[11995.11965]) VALUES('value1','value2','value3')")
would insert a new row (unless a constrain conflict occurred)
However, I suspect that you want to insert values according to variables in which case you could use:
"INSERT INTO [11975_Tecumseh] ([index],[11975_VegaRadar(m)],[11995.11965]) VALUES(?,?,?)"
the question marks being place-holders/bind values
SQL As Understood By SQLite- INSERT
The above would then be invoked using :
c.execute("INSERT INTO [11975_Tecumseh] ([index],[11975_VegaRadar(m)],[11995.11965]) VALUES(?,?,?)",testvalues);
#Working Example :
import sqlite3
drop_sql = "DROP TABLE IF EXISTS [11975_Tecumseh]"
crt_sql = "CREATE TABLE IF NOT EXISTS [11975_Tecumseh] ([index],[11975_VegaRadar(m)],[11995.11965])"
testvalues = ("X","Y","Z")
c = sqlite3.connect("test.db")
c.execute(drop_sql)
c.execute(crt_sql)
insert_sql1 = "INSERT INTO [11975_Tecumseh] " \
"([index],[11975_VegaRadar(m)],[11995.11965]) " \
"VALUES('value1','value2','value3')"
c.execute(insert_sql1)
insert_sql2 = "INSERT OR IGNORE INTO '11975_Tecumseh'" \
"('index','11975_VegaRadar(m)',[11995.11965])" \
" VALUES(?,?,?)"
c.execute(insert_sql2,(testvalues))
cursor = c.cursor()
cursor.execute("SELECT * FROM [11975_Tecumseh]")
for row in cursor:
print(row[0], "\n" + row[1], "\n" + row[2])
c.commit()
cursor.close()
c.close()
#Result
##Row 1
value1
value2
value3
##Row 2
X
Y
Z
My objective is to store a JSON object into a MySQL database field of type json, using the mysql.connector library.
import mysql.connector
import json
jsonData = json.dumps(origin_of_jsonData)
cnx = mysql.connector.connect(**config_defined_elsewhere)
cursor = cnx.cursor()
cursor.execute('CREATE DATABASE dataBase')
cnx.database = 'dataBase'
cursor = cnx.cursor()
cursor.execute('CREATE TABLE table (id_field INT NOT NULL, json_data_field JSON NOT NULL, PRIMARY KEY (id_field))')
Now, the code below WORKS just fine, the focus of my question is the use of '%s':
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES (%s, %s)"
values_to_insert = (1, jsonData)
cursor.execute(insert_statement, values_to_insert)
My problem with that: I am very strictly adhering to the use of '...{}'.format(aValue) (or f'...{aValue}') when combining variable aValue(s) into a string, thus avoiding the use of %s (whatever my reasons for that, let's not debate them here - but it is how I would like to keep it wherever possible, hence my question).
In any case, I am simply unable, whichever way I try, to create something that stores the jsonData into the mySql dataBase using something that resembles the above structure and uses '...{}'.format() (in whatever shape or form) instead of %s. For example, I have (among many iterations) tried
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES ({}, {})".format(1, jsonData)
cursor.execute(insert_statement)
but no matter how I turn and twist it, I keep getting the following error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[some_content_from_jsonData})]' at line 1
Now my question(s):
1) Is there a way to avoid the use of %s here that I am missing?
2) If not, why? What is it that makes this impossible? Is it the cursor.execute() function, or is it the fact that it is a JSON object, or is it something completely different? Shouldn't {}.format() be able to do everything that %s could do, and more?
First of all: NEVER DIRECTLY INSERT YOUR DATA INTO YOUR QUERY STRING!
Using %s in a MySQL query string is not the same as using it in a python string.
In python, you just format the string and 'hello %s!' % 'world' becomes 'hello world!'. In SQL, the %s signals parameter insertion. This sends your query and data to the server separately. You are also not bound to this syntax. The python DB-API specification specifies more styles for this: DB-API parameter styles (PEP 249). This has several advantages over inserting your data directly into the query string:
Prevents SQL injection
Say you have a query to authenticate users by password. You would do that with the following query (of course you would normally salt and hash the password, but that is not the topic of this question):
SELECT 1 FROM users WHERE username='foo' AND password='bar'
The naive way to construct this query would be:
"SELECT 1 FROM users WHERE username='{}' AND password='{}'".format(username, password)
However, what would happen if someone inputs ' OR 1=1 as password. The formatted query would then become
SELECT 1 FROM users WHERE username='foo' AND password='' OR 1=1
which will allways return 1. When using parameter insertion:
execute('SELECT 1 FROM users WHERE username=%s AND password=%s', username, password)
this will never happen, as the query will be interpreted by the server separately.
Performance
If you run the same query many times with different data, the performance difference between using a formatted query and parameter insertion can be significant. With parameter insertion, the server only has to compile the query once (as it is the same every time) and execute it with different data, but with string formatting, it will have to compile it over and over again.
In addition to what was said above, I would like to add some details that I did not immediately understand, and that other (newbies like me ;)) may also find helpful:
1) "parameter insertion" is meant for only for values, it will not work for table names, column names, etc. - for those, the Python string substitution works fine in the sql syntax defintion
2) the cursor.execute function requires a tuple to work (as specified here, albeit not immediately clear, at least to me: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html)
EXAMPLE for both in one function:
def checkIfRecordExists(column, table, condition_name, condition_value):
...
sqlSyntax = 'SELECT {} FROM {} WHERE {} = %s'.format(column, table, condition_name)
cursor.execute(sqlSyntax, (condition_value,))
Note both the use of .format in the initial sql syntax definition and the use of (condition_value,) in the execute function.
A Python API is giving back u"'HOPPE'S No. 9'" as a value for a particular product attribute. I'm then looking to insert it into the DB, also using Python (python-mysqldb), with the following query:
INSERT INTO mytable (rating, Name) VALUES('5.0 (7)', 'HOPPE'S No. 9';
MySQL rejects this, and the suggested approach to handling a single quote in MySQL is to escape it first. This I need to do in Python, so I try:
In [5]: u"'HOPPE'S No. 9'".replace("'", "\'")
Out[5]: u"'HOPPE'S No. 9'"
When I incorporate this in my program, MySQL still rejects it. So I double-escape the apostrophe, and then an insert happens successfully. Thing is, it contains the escape character (so what gets written is 'HOPPE\'S No. 9').
If I need the second escape character, but when I add it gets left in, then how can I handle the escaping without having the escape character included in the string that gets inserted?
Edit: Based on theBjorn's suggestion, tried:
actualSQL = "INSERT INTO %s (%s) VALUES(%s);"
#cur.execute(queryString)
cur.execute(actualSQL,
(configData["table"], sqlFieldMappingString, sqlFieldValuesString))
but it looks like I'm back to where I was when I was trying to escape using the single escape with .replace():
Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''mytable' ('rating, Name, Image, mfg, price, URL') VALUES('\'5.0 (3)\', \'AR-1' at line 1
You should never construct sql that way. Use parameterized code instead:
cursor.execute(
"insert into mytable (rating, name) values (%s, %s);",
("5.0 (7)", "HOPPE'S No. 9")
)
your latest problem is due to the misconception that this is string interpolation, which it isn't (the use of %s is confusing), thus:
actualSQL = "INSERT INTO %s (%s) VALUES(%s);"
will be wrong. It is possible to construct your sql string, but probably easier to do so in two steps so we don't trip over sql parameter markers looking like string interpolation markers. Assuming you have the values in a tuple named field_values:
params = ["%s"] * len(field_values) # create a list with the correct number of parameter markers
sql = "insert into %s (%s) values (%s)" % ( # here we're using string interpolation, but not with the values
configData["table"],
sqlFieldMappingString,
', '.join(params)
)
if you print sql it should look like my example above. Now you can execute it with:
cursor.execute(sql, field_values)
I'm trying to select a record from my MySQL db, the problem is that it returns with this error which i don't understand, i searched on the web for a solution, and there are many similar cases, but none apply to this specific one.
The expression i'm trying to execute is the following:
result = cursor.execute("""select * from urls where domain = '%s';"""%(found_url,))
it is in a try clause and it always goes right to the except giving me this error:
OperationalError(1054, "Unknown column 'http://..................' in 'field list'")
(i omitted the url, the dots act as placeholders)
after some cycles, being in a loop, it stops giving me that error and changes it to this:
ProgrammingError(2014, "Commands out of sync; you can't run this command now")
any idea? i'm going crazy on this one.
Use query parameters instead of how you are doing it with string interpolation. The protects you against SQL Injection attacks.
cursor.execute("""select * from urls where domain = %s""", (found_url,))
Notice the differences here:
The %s is NOT quoted, as the MySQLdb library will handle this automatically
You are not using string interpolation (no % after the string to replace your placeholders). Instead, you are passing a tuple as the second argument to the execute function.
Since you are only using one place holder and you still need to pass a tuple, there is a comma to indicate it is a tuple (found_url,)