I'm using python to read values from REST API and write to a MySQL table. One of the returned values is a JSON, which I want to store in the DB.
The value returned by the API has escaped quotes and looks something like:
{\"dashboard\":\"val\"}
When I use print, I see that the escape characters are replaced with the actual quotes (which is the desired outcome):
{"dashboard":"val"}
However, when I'm using the MySQLdb execute or executemany (with tokenised params) - the value written to the database has all the double quotes replaced with single quotes, making it a non-valid json:
{'dashboard':'val'}
How do I avoid that?
You should change your library into mysql.connector or pymysql or etc , because MySQLdb has some problems you can not predict. Even though your parameters and base_sql are correct. I recommend mysql.connector, because it is MySQL official library. https://dev.mysql.com/downloads/connector/python/
Related
This question already has an answer here:
How to prevent pandas dataframe from adding double quotes around #tmp when using sqlalchemy and sybase?
(1 answer)
Closed 2 years ago.
I using a framework, that creates temporary tables (sqlalchemy and pandas).
However, it creates a table surrounded by quotes, and in my case, I'm using Sybase and it returns an permission error.
When I create manually, without quotes works perfectly.
To workround it, I put \b at the beggining of string:
table_name=f'\b{table_name}'
When I test with \b, it erases the left quote, but I can't find a way to delete the closing quote.
Obs: I already tested table_name=f'\b{table_name}' + \u'\u007f'
For example:
table_name="#test"
df.to_sql(con=engine,name=table_name,index=False)
Generates following create:
CREATE TABLE "#test" (nome TEXT NULL)
I'm getting error, because quotes.
However, with this code, I can remove left quote:
table_name="\b#test"
df.to_sql(con=engine,name=table_name,index=False)
It generates:
CREATE TABLE #test" (nome TEXT NULL)
Thanks
If you have access to the string itself, you can always slice it to remove whatever characters you want. To remove the first and last ones:
>>> table_name = '"some test table"'
>>> table_name[1:-1] # No surrounding " characters.
some test table
Adding delete characters to the string (what you currently have) just affects the console output. Those characters are still present.
Assuming this is Sybase ASE and the error the OP is receiving is related to a syntax issue with the use of quotes, consider:
using double quotes around (table/column) names is referred to (in ASE) as quoted identifiers
ASE's default behavior is for quoted identifiers to be off (ie, generate an error if double quotes are used around (table/column) names)
in order to use quoted identifiers the client application needs to explicitly enable quoted identifiers either via a connection parameter or via the explicit command set quoted_identifier on
ASE also supports the use of square brackets ([]) around (table/column) names with the added benefit that there is no need to explicitly set quoted_identifier on
Again, assuming this is Sybase ASE, I'd want to find out if the client side app a) has the ability to use square brackets ([]) in place of double quotes or b) has the ability to enable quoted_identifier support or c) has the ability to disable quoted identifiers (ie, not wrap names in double quotes).
My problem is with pandas function to_sql, because when I pass the table name as parameter, it automatically adds quotes surround table name
You appear to be using the internal SQLAlchemy dialect for Sybase which is in the process of being replaced by the external SAP ASE (Sybase) dialect that I maintain. The issue with DDL rendering of #temp tables was fixed in version 1.0.1.
I have a program in python and I want to insert data into a table(using insert into statement). I receive data from web(web scraping) and the data contain both single and double quotes. As you know MySQL allows to insert both single and double quotes to a table so the error is not from database. Problem appears when I use that data in python and an error appears.
No matters if i use single or double quotes in the string (insert into statement values) in python, in both ways error appears because of the data(that contain single or double quotes).I use MySQL and Connector/python and in my script I import mysql. I hope you got this, sorry about bad English.
Most likely explanation for the behavior is a SQL Injection vulnerability. (That's just a guess because we are speculating about code we haven't seen; only a description of the behavior.)
The short answer is to use prepared statements with bind placeholders
https://pynative.com/python-mysql-execute-parameterized-query-using-prepared-statement/
If for some reason that is not possible, then at a bare minimum, any potentially unsafe values included in SQL text must be properly escaped to make them safe for inclusion
(The single quote in Little Bobby Tables https://xkcd.com/327/ is not escaped.)
As example, this SQL will throw an error, because the second single quote ends the string literal, and what follows the end of the string literal "s wrong" is gibberish in terms of SQL:
INSERT INTO mytab (mycol) VALUES ( 'It's wrong' )
^
But this will work:
INSERT INTO mytab (mycol) VALUES ( 'It''ll work' )
^^
Because the single quote within the string literal is escaped, by preceding it with another single quote.
The OWASP project provides a good overview of SQL Injection.
https://www.owasp.org/index.php/SQL_Injection
https://www.owasp.org/index.php/SQL_Injection_Prevention_Cheat_Sheet
So I am trying to create a script with a MySQL Query "show databases like 'login';"
I am able to use string substitution and use "login" but i am unable to get the single quotes around it.
Below is how I am trying to do it but i cant get the single quotes even if I escape it using "\".
db = "'"+val+"'"
print "DB...", db
run.cmd("echo 'res = cur.execute(\"SHOW DATABASES like %s;\")\n' >> /run.py" % (db))
Since your string is wrapped in double quotes you shouldn't need to escape the single quotes. Just add the val directly into the format, then add the single quotes to the string being formatted.
run.cmd("echo 'res = cur.execute(\"SHOW DATABASES like '%s';\")\n' >> /run.py" % (val))
Also, is there any reason why you're appending a bash command into a python script?
SHOW DATABASES like '\'%s\'';\" -> this is what worked.
Is there a way to store a mix of quote characters as a value in Redis? My specific need is to store PHP and HTML strings, and keep the mix of quote types.
In Python:
import redis
db = redis.Redis('localhost')
phpfile = /root/test.php
with open(phpfile) as f:
bar = f.read()
db.set('foo', bar)
If I try to pass the string any other direct way, it fails, of course, because of all the competing quotes.
Edit:
I'm going to close this question. I cannot get the behavior to repeat in Python using the code above, so it must be an issue with another layer in my code and not an issue with Redis.
Redis does not care the structure of your data you store. You can throw random bytes at redis or strings with double or single quotes.
Simply store the string and be happy.
I would like to store windows path in MySQL without escaping the backslashes. How can I do this in Python? I am using MySQLdb to insert records into the database. When I use MySQLdb.escape_string(), I notice that the backslashes are removed.
Have a look at os.path.normpath(thePath)
I can't remember if it's that one, but there IS a standard os.path formating function that gives double backslashes, that can be stored in a db "as is" and reused later "as is". I have no more windows machine and cannot test it anymore.
Just use a dictionary to add slashes wherever required to make the query valid :
http://codepad.org/7mjbwKBf
def addslashes(s):
dict = {"\0":"\\\0", "\\":"\\\\"} #add more here
return ''.join(dict.get(x,x) for x in s)
query = "INSERT INTO MY_TABLE id,path values(23,'c:\windows\system\')";
print(addslashes(query));