In making some slow code run more efficiently, I ran into a snag with escape characters. The original code (which works) was something like this:
sql = f"""INSERT INTO {schema}.{table}
(seg_id, road_type, road_name)
VALUES ({s_id}, '{road_type}', %s);"""
# road_name may have an apostrophe. Let the execute() handle it.
cur.execute(sql, (road_name,))
This query ran many thousands of times, and the execute statements make the code sluggish due to communicating with a remote server. I would like to get a series of sql statements and execute them all at once, something like this:
sql += f"""INSERT INTO {schema}.{table}
(seg_id, road_type, road_name)
VALUES ({s_id}, '{road_type}', '{road_name}');\n"""
The sql statement is not executed immediately, but is "stacked" so that I can execute a collection of sql statements in a single cur.execute(sql) which happens later. However, since it does not have the road_name as an argument in the execute statement, it fails when there's a road name with an apostrophe (like Eugene O'Neill Drive).
What is a viable alternative?
NOTE: The query is in a function that is called from multiple processes. The {variables} are being passed as arguments.
Related
I am currently working with cx_oracle
Here with SELECT statements I am able to use the fetchall() function to get rows.
But how to get the outputs for queries that fall under Data Definition Language (DDL) category.
For example, after executing a GRANT statement with cursor.execute(), the expected output assuming the query is valid would be,
"GRANT executed successfully"
But how do I get this with cx_oracle, Python.
The answer is that you have print it yourself, which is what SQL*Plus does.
DDL statements are statements not queries because they do not return data. They return a success or error condition to the tool that executed them, which can then print any message. In your case the tool is cx_Oracle. There isn't a way to get the type (GRANT, CREATE etc) of the statement automatically in cx_Oracle. Your application can either print a generic message like 'statement executed successfully', or you can extract the first keyword(s) from the SQL statement so you can print a message like SQL*Plus does.
I am trying to update a table from the Snowflake Connector in python to a Snowflake table;
update TABLE set X = 100 where Y = 'Applebee's'
To escape the single quote in the Snowflake UI, I would format the "where" clause to be
where Y = 'Applebee\'s'
Also tried:
where Y = 'Applebee''s'
However, no fix that I have tried is succeeding to workaround this error in python. Is there a way to do this from python, in a one step process, without any steps in Snowflake? I only care about workarounds from python.
In all SQL, the only true way to avoid weird characters and not have to account for each possibility is to parametrized your sql calls either as some sort of stored procedure call or a parametrized call from your client - in this case python.
In general this means, never have a case where you are concatenating a sql string to the value of a where clause like you have above.
Here's an example of how to parametrize a statement in python here
Python mysql default escape function, corrupts the query.
Original Query string is following. It works fine and does add records to database as desired
INSERT IGNORE INTO state (`name`, `search_query`, `business_status`, `business_type`, `name_type`, `link`) VALUES ("test_name1", "test", "test_status", "test_b_typ", "test_n_typ", "test_link"), ("test_name2", "test", "test_status", "test_b_typ", "test_n_typ", "test_link")
But After escaping it to make sql Injection secure using the fuction
safe_sql = self.conn.escape_string(original_sql)
safe_sql being generated is following
b'INSERT IGNORE INTO state (`name`, `search_query`, `business_status`, `business_type`, `name_type`, `link`) VALUES (\\"test_name1\\", \\"test\\", \\"test_status\\", \\"test_b_typ\\", \\"test_n_typ\\", \\"test_link\\"), (\\"test_name2\\", \\"test\\", \\"test_status\\", \\"test_b_typ\\", \\"test_n_typ\\", \\"test_link\\")'
Now if I try to execute the safe_sql I get the syntax error below
MySQLdb._exceptions.ProgrammingError: (1064, 'You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near \'\\"test_name1\\", \\"test\\", \\"test_status\\", \\"test_b_typ\\", \\"test_n_typ\\", \\"tes\' at line 1')
Which makes me wonder that if escape function I am using is either broken / uncompatibl or I am not using it the right way ? Also i am entering hundreds of records at one time, and due to the fast processing (which i purely assume) of single query as compared to prepared statements running hundreds of time, I am creating a large query.
You can't escape the entire query! You can't construct a query by randomly concatenating strings and then wave a magic wand over it and make it "injection secure". You need to escape every individual value before you put it into the query. E.g.:
"INSERT ... VALUES ('%s', ...)" % self.conn.escape_string(foo)
But really, your MySQL API probably offers prepared statements, which are much easier to use and less error prone. Something like:
self.conn.execute('INSERT ... VALUES (%s, %s, %s, ...)',
(foo, bar, baz))
I have a table of three columnsid,word,essay.I want to do a query using (?). The sql sentence is sql1 = "select id,? from training_data". My code is below:
def dbConnect(db_name,sql,flag):
conn = sqlite3.connect(db_name)
cursor = conn.cursor()
if (flag == "danci"):
itm = 'word'
elif flag == "wenzhang":
itm = 'essay'
n = cursor.execute(sql,(itm,))
res1 = cursor.fetchall()
return res1
However, when I print dbConnect("data.db",sql1,"danci")
The result I obtained is [(1,'word'),(2,'word'),(3,'word')...].What I really want to get is [(1,'the content of word column'),(2,'the content of word column')...]. What should I do ? Please give me some ideas.
You can't use placeholders for identifiers -- only for literal values.
I don't know what to suggest in this case, as your function takes a database nasme, an SQL string, and a flag to say how to modify that string. I think it would be better to pass just the first two, and write something like
sql = {
"danci": "SELECT id, word FROM training_data",
"wenzhang": "SELECT id, essay FROM training_data",
}
and then call it with one of
dbConnect("data.db", sql['danci'])
or
dbConnect("data.db", sql['wenzhang'])
But a lot depends on why you are asking dbConnect to decide on the columns to fetch based on a string passed in from outside; it's an unusual design.
Update - SQL Injection
The problems with SQL injection and tainted data is well documented, but here is a summary.
The principle is that, in theory, a programmer can write safe and secure programs as long as all the sources of data are under his control. As soon as they use any information from outside the program without checking its integrity, security is under threat.
Such information ranges from the obvious -- the parameters passed on the command line -- to the obscure -- if the PATH environment variable is modifiable then someone could induce a program to execute a completely different file from the intended one.
Perl provides direct help to avoid such situations with Taint Checking, but SQL Injection is the open door that is relevant here.
Suppose you take the value for a database column from an unverfied external source, and that value appears in your program as $val. Then, if you write
my $sql = "INSERT INTO logs (date) VALUES ('$val')";
$dbh->do($sql);
then it looks like it's going to be okay. For instance, if $val is set to 2014-10-27 then $sql becomes
INSERT INTO logs (date) VALUES ('2014-10-27')
and everything's fine. But now suppose that our data is being provided by someone less than scrupulous or downright malicious, and your $val, having originated elsewhere, contains this
2014-10-27'); DROP TABLE logs; SELECT COUNT(*) FROM security WHERE name != '
Now it doesn't look so good. $sql is set to this (with added newlines)
INSERT INTO logs (date) VALUES ('2014-10-27');
DROP TABLE logs;
SELECT COUNT(*) FROM security WHERE name != '')
which adds an entry to the logs table as before, end then goes ahead and drops the entire logs table and counts the number of records in the security table. That isn't what we had in mind at all, and something we must guard against.
The immediate solution is to use placeholders ? in a prepared statement, and later passing the actual values in a call to execute. This not only speeds things up, because the SQL statement can be prepared (compiled) just once, but protects the database from malicious data by quoting every supplied value appropriately for the data type, and escaping any embedded quotes so that it is impossible to close one statement and another open another.
This whole concept was humourised in Randall Munroe's excellent XKCD comic
I'm running a relatively simple python script that is meant to read a text file that has a series of stored procedures - one per line. The script should run the stored procedure on the first line, move to the second line, run the stored procedure on the second line, etc etc. Running these stored procedures should populate a particular table.
So my problem is that these procedures aren't populating the table with all of the results that they should be. For example, if my file looks like
exec myproc 'data1';
exec myproc 'data2';
Where myproc 'data1' should populate this other table with about ~100 records, and myproc 'data2' should populate this other table with an additional ~50 records. Instead, I end up with about 9 results total - 5 from the first proc, 4 from the second.
I know the procedures work, because if I run the same sql file (with the procs) through OSQL and I get the correct ~150 records in the other table, so obviously it's something to do with my script.
Here's the code I'm running to do this:
import pypyodbc
conn = pypyodbc.connect(CONN_STR.format("{SQL Server}",server,database,user,password))
conn.autoCommit = True
procsFile = openFile('otherfile.txt','r+')
#loop through each proc (line) in the file
for proc in procsFile:
#run the procedure
curs = conn.cursor()
curs.execute(proc)
#commit results
conn.commit()
curs.close();
conn.close();
procsFile.close();
I'm thinking this has something to do with the procedures not committing...or something?? Frankly I don't really understand why only 5/100 records would be committed.
I dont know. any help or advice would be much appreciated.
There a couple things to check. One is that your data1 is actually a string 'data1', or if you want the value of data1? If you want the string 'data1', then you will have to add quotes around it. So your string to execute would look like this:
exec_string = 'exec my_proc \'data1\';'
In your case you turn ON auto-commit and also you manually commit for the entire connection.
I would comment out the auto-commit line:
#conn.autoCommit = True
And then change conn.commit() to the cursor instead
curs.commit()
As a one-liner:
conn.cursor().execute('exec myproc \'data1\';').commit()
Also, your Python semi-colons (;) at the end of the python line are unnecessary, and may be doing something weird in your for-loop. (Keep the SQL ones though.)