The following python MySQL query to retrieve the feed_id from the MySQL table feed_master.
cur.execute('SELECT feed_id FROM feed_master WHERE url_link = %s',(source,))
rows = cur.fetchall()
for row in rows:
print "Feed id : " + str(row)
Python version : 2.7.10
MySQL version : 5.6.27-0ubuntu1
On executing the above Python code, I get no results. However if I substitute %s with the actual string, I get the result I am expecting. The source variable is also passed to the function, as I tested the "source" variable with a print statement. Why doesn't the above code return the result?
Your query is malformed. The %s needs single quotes around it. So, this might do what you want:
cur.execute("""SELECT feed_id FROM feed_master WHERE url_link = '%s'""",(source,))
I tweaked the code in all possible ways and the following executes perfectly
cur.execute("""SELECT feed_id FROM feed_master WHERE url_link = '%s' """ %(source))
I have no idea why it works, but hope it helps somebody facing the same errors.
The proper way to run the query is your original
cur.execute('SELECT feed_id FROM feed_master WHERE url_link = %s',(source,))
It looks like you can then iterate through the cursor:
for feed_id in cur:
print feed_id
This might wind up printing a tuple with a single element, but that's pretty easy to fix by changing feed_id to feed_id, I can't tell you why the fetchall method isn't working though.
The reason your change works is that you're using Python's (old) string formatting syntax to replace %s with whatever the value of source is rather than using the database client to replace the variable properly. It's incredibly insecure to use Python string replacement in a SQL statement where the value you're replacing %s with is a user input. See https://en.wikipedia.org/wiki/SQL_injection
Related
I am automating a task through Python that will run an SQL statement to insert into an existing table in a DB.
My CSV headers look as such:
ID,ACCOUNTID,CATEGORY,SUBCATEGORY,CREATION_DATE,CREATED_BY,REMARK,ISIMPORTANT,TYPE,ENTITY_TYPE
My values:
seq_addnoteid.nextval,123456,TEST,ADMN_TEST,sysdate,ME,This is a test,Y,1,A
NOTE: Currently, seq_addnote works from DB but in my code i added a small snippet to get the max ID and the rows will increase this by one for each iteration.
Sysdate could also be passed as format '19-MAY-22'
If i was to run from DB, this would work:
insert into notes values(seq_addnoteid.nextval,'123456','TEST','ADMN_TEST',sysdate,'ME','This is a test','Y',1,'A');
# Snippet to get function
cursor.execute("SELECT MAX(ID) from NOTES")
max = cursor.fetchone()
max = int(max[0])
with open ('sample.csv', 'r') as f:
reader = csv.reader(f)
columns = next(reader)
query = 'INSERT INTO NOTES({0}) values ({1})'
query = query.format(','.join(columns), ','.join('?' * len(columns)))
cursor = conn.cursor()
for data in reader:
cursor.execute(query, data)
conn.commit()
print("Records inserted successfully")
cursor.close()
conn.close()
Currently, i'm getting Oracle-Error-Message: ORA-01036: illegal variable name/number and i think its because of my query.format line. However, I'm looking for help to get this code to handle the data types properly.
Thanks!
Try printing your query before you execute it. I think you'll find that it's printing this:
INSERT INTO NOTES(ID,ACCOUNTID,CATEGORY,SUBCATEGORY,CREATION_DATE,CREATED_BY,REMARK,ISIMPORTANT,TYPE,ENTITY_TYPE)
values(seq_addnoteid.nextval,123456,TEST,ADMN_TEST,sysdate,ME,This is a test,Y,1,A);
Which will also give you a ORA-01036 if you try to run it manually.
The problem is that you want some of your column values to be literal values, and some of them to be strings escaped in single-quotes, and your code doesn't do that. I don't think there's an easy to way to do it with ','.join(), so you'll either need to modify your CSVs to quote the strings, like:
seq_addnoteid.nextval,"'123456'","'TEST'","'ADMN_TEST'",sysdate,"'ME'","'This is a test'","'Y'",1,"'A'"
Or modify your query.format to add the quotes around the parameters that you want to treat as strings:
query.format(','.join(columns), "?,'?','?','?',?,'?','?','?',?,'?'")
As the commenters mentioned, pandas does handle this all very nicely.
EDIT: I see what you're saying. I'm not sure pandas will help with the literal functions you want to pass to the insert. But yes, you should be able to change your CSV and then do:
query.format(','.join(columns) + ',ID,CREATION_DATE', "'?','?','?','?','?','?',?,'?',seq_addnoteid.nextval,sysdate")
As a side note, a lot of people do this sort of thing on the database side in a BEFORE INSERT trigger, e.g.:
create or replace trigger NOTES_INS_TRG
before insert on NOTES
for each row
begin
:NEW.ID := seq_addnoteid.nextval;
:NEW.CREATION_DATE := sysdate;
end;
/
Then you could leave those columns out of your insert entirely.
Edit again:
I'm not sure you can use ? for bind/substitution variables in cx_oracle (see documentation ). So where your raw query is currently:
INSERT INTO NOTES(ACCOUNTID,CATEGORY,SUBCATEGORY,CREATED_BY,REMARK,ISIMPORTANT,TYPE,ENTITY_TYPE,ID,CREATION_DATE)
values (seq_addnoteid.nextval,sysdate,'?','?','?','?','?','?',?,'?')
You'd need something like:
INSERT INTO NOTES(ACCOUNTID,CATEGORY,SUBCATEGORY,CREATED_BY,REMARK,ISIMPORTANT,TYPE,ENTITY_TYPE,ID,CREATION_DATE)
values (seq_addnoteid.nextval,sysdate,:1,:2,:3,:4,:5,:6,:7,:8)
We can probably do that by modifying the format string again to generate some bind variables:
query.format('ID,CREATION_DATE,' + ','.join(columns),
"seq_addnoteid.nextval,sysdate," + ','.join([':'+c for c in columns])
Again, try printing the query before executing it to make sure the column names and values are lining up correctly.
My objective is to store a JSON object into a MySQL database field of type json, using the mysql.connector library.
import mysql.connector
import json
jsonData = json.dumps(origin_of_jsonData)
cnx = mysql.connector.connect(**config_defined_elsewhere)
cursor = cnx.cursor()
cursor.execute('CREATE DATABASE dataBase')
cnx.database = 'dataBase'
cursor = cnx.cursor()
cursor.execute('CREATE TABLE table (id_field INT NOT NULL, json_data_field JSON NOT NULL, PRIMARY KEY (id_field))')
Now, the code below WORKS just fine, the focus of my question is the use of '%s':
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES (%s, %s)"
values_to_insert = (1, jsonData)
cursor.execute(insert_statement, values_to_insert)
My problem with that: I am very strictly adhering to the use of '...{}'.format(aValue) (or f'...{aValue}') when combining variable aValue(s) into a string, thus avoiding the use of %s (whatever my reasons for that, let's not debate them here - but it is how I would like to keep it wherever possible, hence my question).
In any case, I am simply unable, whichever way I try, to create something that stores the jsonData into the mySql dataBase using something that resembles the above structure and uses '...{}'.format() (in whatever shape or form) instead of %s. For example, I have (among many iterations) tried
insert_statement = "INSERT INTO table (id_field, json_data_field) VALUES ({}, {})".format(1, jsonData)
cursor.execute(insert_statement)
but no matter how I turn and twist it, I keep getting the following error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '[some_content_from_jsonData})]' at line 1
Now my question(s):
1) Is there a way to avoid the use of %s here that I am missing?
2) If not, why? What is it that makes this impossible? Is it the cursor.execute() function, or is it the fact that it is a JSON object, or is it something completely different? Shouldn't {}.format() be able to do everything that %s could do, and more?
First of all: NEVER DIRECTLY INSERT YOUR DATA INTO YOUR QUERY STRING!
Using %s in a MySQL query string is not the same as using it in a python string.
In python, you just format the string and 'hello %s!' % 'world' becomes 'hello world!'. In SQL, the %s signals parameter insertion. This sends your query and data to the server separately. You are also not bound to this syntax. The python DB-API specification specifies more styles for this: DB-API parameter styles (PEP 249). This has several advantages over inserting your data directly into the query string:
Prevents SQL injection
Say you have a query to authenticate users by password. You would do that with the following query (of course you would normally salt and hash the password, but that is not the topic of this question):
SELECT 1 FROM users WHERE username='foo' AND password='bar'
The naive way to construct this query would be:
"SELECT 1 FROM users WHERE username='{}' AND password='{}'".format(username, password)
However, what would happen if someone inputs ' OR 1=1 as password. The formatted query would then become
SELECT 1 FROM users WHERE username='foo' AND password='' OR 1=1
which will allways return 1. When using parameter insertion:
execute('SELECT 1 FROM users WHERE username=%s AND password=%s', username, password)
this will never happen, as the query will be interpreted by the server separately.
Performance
If you run the same query many times with different data, the performance difference between using a formatted query and parameter insertion can be significant. With parameter insertion, the server only has to compile the query once (as it is the same every time) and execute it with different data, but with string formatting, it will have to compile it over and over again.
In addition to what was said above, I would like to add some details that I did not immediately understand, and that other (newbies like me ;)) may also find helpful:
1) "parameter insertion" is meant for only for values, it will not work for table names, column names, etc. - for those, the Python string substitution works fine in the sql syntax defintion
2) the cursor.execute function requires a tuple to work (as specified here, albeit not immediately clear, at least to me: https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-execute.html)
EXAMPLE for both in one function:
def checkIfRecordExists(column, table, condition_name, condition_value):
...
sqlSyntax = 'SELECT {} FROM {} WHERE {} = %s'.format(column, table, condition_name)
cursor.execute(sqlSyntax, (condition_value,))
Note both the use of .format in the initial sql syntax definition and the use of (condition_value,) in the execute function.
I am using mysqldb to try to update a lot of records in a database.
cur.executemany("""UPDATE {} set {} =%s Where id = %s """.format(table, ' = %s, '.join(col)),updates.values.tolist())
I get the error message...
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near...
So I tried outputting the actual sql update statement as that error message wasn't helpful using the following code:
cur.execute('set profiling = 1')
try:
cur.executemany("""UPDATE {} set {} =%s Where id = %s """.format(table, ' = %s, '.join(col)),updates.values.tolist())
except Exception:
cur.execute('show profiles')
for row in cur:
print(row)
That print statement seems to cut off the update statement at 300 characters. I can't find anything in the documentation about limits so I am wondering is this the print statement limiting or is it mysqldb?
Is there a way I can generate the update statement with just python rather than mysqldb to see the full statement?
To see exactly what the cursor was executing, you can use the cursor.statement command as shown here in the API. That may help with the debugging.
I don't have experience with the mySQL adapter, but I work with the PostgreSQL adapter on a daily basis. At least in that context, it is recommended not to format your query string directly, but let the second argument in the cursor.execute statement do the substitution. This avoids problems with quoted strings and such. Here is an example, the second one is correct (at least for Postgres):
cur.execute("""UPDATE mytbl SET mycol = %s WHERE mycol2 = %s""".format(val, cond))
cur.execute("""UPDATE mytbl SET mycol = %(myval)s WHERE mycol2 = %(mycond)s""", {'myval': val, 'mycond': cond})
This can result in the query
UPDATE mytbl SET mycol = abc WHERE mycol2 = xyz
instead of
UPDATE mytbl SET mycol = 'abc' WHERE mycol2 = 'xyz'.
You would have needed to explicitly add those quotes if you do the value substitution in the query yourself, which becomes annoying and circumvents the type handling of the database adapter (keep in mind this was only a text example). See the API for a bit more information on this notation and the cursor.executemany command.
What I want is execute the sql
select * from articles where author like "%steven%".
For the sake of safety, i used like this way :
cursor.execute('select * from articles where %s like %s', ('author', '%steven%')
Then the result is just empty, not get a syntax error, but just empty set.
But I am pretty sure there is some thing inside, I can get result use the first sql. Is there anything run with my code ?
You can't set a column name like a parameter where you're doing where %s like %s. To dynamically set the column name you need to do actual string manipulation like:
sql = 'select * from articles where '+ sql_identifier('author') +' like %s'
cursor.execute(sql, ('%steven%',))
Where sql_identifier is your lib's function for making an identifier safe for SQL injection. Something like:
# don't actually use this!
def sql_identifier(s):
return '"%s"' % s.replace('"','')
But with actual testing and knowledge of the DB engine you're using.
The problem here is fact a minor mistake. Thanks to #Asad Saeeduddin, when I try to use print cursor._last_executed to check what has happened. I found that what is in fact executed is
SELECT * FROM articles WHERE 'title' LIKE '%steven%', look the quotation mark around the title, that's the reason why I got empty set.
So always remember the string after formatting will have a quotation around
I'm trying to return a hard coded value in my SQL query, but when running the query using pyodbc, random records return '\x0e' instead of the hard coded value (in this case '16'). If I run the query on the server (MS SQL Server 2008), the query returns all the correct results and values.
The beginning of the query looks like this:
My SQL Code:
Select '"16","' + S.ShipNum + '","'
My python code:
cursor.execute("""Select '\"16\",\"' + SS.ShipNum + '\",\"'
Is there another way to guarantee a value is returned from a query?
\016 is the oct representation of \x0e
So I would think that it has more to do with the way in which you are escaping your double quotes. In your python you are actually geting \16 and not "16" as you desire.
You should try a prepared statment maybe.
ps = db.prepare("SELECT 16")
ps()
returns:
[(16,)]
Addtional examples can be seen here:
[http://python.projects.pgfoundry.org/docs/0.8/driver.html#parameterized-statements]
You can see all of the ascii and other character sets here
[http://donsnotes.com/tech/charsets/ascii.html]
It looks like you're trying to create a comma-delimited, quoted, string representation of the row. Don't try to do this in the database query, string formatting isn't one of T-SQL's strengths.
Pass the static value using a parameter, then join the row values. Using sys.databases for the example:
params = ("Some value",)
sql = "SELECT ?, name, user_access_desc FROM sys.databases"
for row in cursor.execute(sql):
print(','.join('"{0}"'.format(column) for column in row))