Hi I'm trying to check if I already have an entry in a mysql table through python. The key I use to check if their is already an entry is called PID and it uses the ascii_bin collation. my problem is when when I try something like...
q = """select * from table_name where PID = '%s'"""%("Hello")
db = MySQLdb.connect("xxxx", "xxxx", "xxxx","temp",cursorclass=MySQLdb.cursors.DictCursor)
cursor = db.cursor()
res = cursor.execute(q)
rowOne = cursor.fetchone() #fetches the row where pid = "hello"
rowOne ends up being the row where pid = hello. However when I use sqlyog and execute the query it properly prints out the row where pid = Hello(Properly functions as a case sensitive query). I'm looking for a way to get the mysqldb module to work properly as a lot of my code already is using this module
By default string comparison operations in MySQL are not case sensitive (or accent sensitive): http://dev.mysql.com/doc/refman/5.0/en/case-sensitivity.html
You have two easy options though, use COLLATE:
For example, if you are comparing a column and a string that both have the latin1 character set, you can use the COLLATE operator to cause either operand to have the latin1_general_cs or latin1_bin collation:
or set the column to be case sensitive:
If you want a column always to be treated in case-sensitive fashion, declare it with a case sensitive or binary collation. See Section 13.1.10, “CREATE TABLE Syntax”.
Related
I am trying to execute mysql query from python. I want the output
query = "UPDATE 'college_general' SET 'fees' = '180000' WHERE ('college_id' = '2')"
Below is the snippet of the code
def update(table, column, value):
return f"UPDATE '{table}' SET '{column}' = '{value}' WHERE ('college_id' = '{id}')"
query = update("college_general", "fees", fee)
cursor.execute(query)
Instead Python is storing it like
query = 'UPDATE \'college_general\' SET \'fees\' = \'180000\' WHERE (\'college_id\' = \'2\')'
which is causing the script to fail. How can I achieve the desired output?
Thanks in advance!
You can replace the identifiers single quotes with backticks. For more detailed answers visit this question.
There are two types of quotes in MySQL:
' for enclosing string literals
` for enclosing identifiers such as table and column names
There are multiple issues here:
First, I suspect that the string handling bit of your program is actually working, but you are being confused by the external representation of strings. For example, if you do
x = "O'Reilly"
Python will, in some circumstances, display the string as
'O\'Reilly'
Second, I think you are using the wrong kind of quotes. Single quotes in SQL are for strings; MySQL uses backticks for names when necessary, while other SQL implementations usually use double quotes for this.
Third, AND THIS IS IMPORTANT! Do not use string manipulation for building SQL queries. The database library almost certainly has a feature for parametrized queries and you should be using that. Your query should look something like this:
query = 'UPDATE college_general SET fees = ? WHERE college_ID = ?'
cursor.execute(query, [180000, '2'])
but the details will depend on the DB library you are using. For example, some use %s instead of ?. This saves you from all kinds of headaches with quoting strings.
raw string is the simplest solution to your problem.
I believe the code below will achieve what you wanted.
def update(table, column, value):
return fr"UPDATE '{table}' SET '{column}' = '{value}' WHERE ('college_id' = '{id}')"
query = update("college_general", "fees", fee)
cursor.execute(query)
I have a 250GB sqlite database file on an SSD drive and need to search through this file and search for a specific value in a table.
I wrote a script to perform the lookup in python and here is a similar sql statement to the one that I wrote:
SELECT table FROM database WHERE table like X'003485FAd480'.
I am looking to compare between hex values stored in a table to a given hex value.I am using Anaconda command prompt and not sure if this is the best route.
My question is about possible recommendations or tools to help speed up the lookup?
Thanks!
LIKE converts both operands into strings, so it might not work correctly if a value contains zero bytes or bytes that are not valid in the UTF-8 encoding.
To compare for equality, use =:
SELECT ... FROM MyTable WHERE MyColumn = x'003485FAD480';
This search can be sped up with an index on the lookup column; if you do not already have a primary key or unique constraint on this column, you can create an index manually:
CREATE INDEX MyLittleIndex ON MyTable(MyColumn);
I don't know if this is what your looking for, you mentioned using Python. If you're searching different values that are in Python, have you thought about writing two functions, one to search the database and one to compare those results and do something with them?
def queryFuntion():
cnxn = pyodbc.connect('DRIVER={SQLite3 ODBC Driver};SERVER=localhost;DATABASE=test.db;Trusted_connection=yes') #for production use only
cursor = cnxn.cursor()
query = cursor.execute("SELECT table FROM database")
for row in cursor.fetchall():
yield str(row.table)
def compareFunction(row):
search = '003485FAd480'
if row == search:
print('Yes')
else:
print('No')
I trying to get an SQL selection with the LIKE parameter, but it gives my only:
false,"error_code":400,"description":"Bad Request: Message text is empty"
Here is my code:
textvar = message.text
c.execute("SELECT * FROM run WHERE act LIKE '+textvar+'")
res = c.fetchone()
bot.send_message(message.chat.id, res)
print textvar gives me an expected sting, taken from message.text. I've tried to get some data without LIKE and that seems everything else works pretty well.
What am I doing wrong?
You are testing if the string '+textvar+' (not the value from the variable textvar!) is present in your act column. There are no such columns.
Don't use string interpolation to insert variables into SQL queries, because that opens you wide to SQL injection attacks. Use query parameters:
textvar = '%{}%'.format(message.text)
c.execute("SELECT * FROM run WHERE act LIKE ?", (textvar,))
Note that I do use string interpolation to add % wildcards to the string to make sure LIKE searches for text anywhere in the column. Without wildcards LIKE is nothing more than an equality test. % matches 0 or more arbitrary characters, _ matches exactly 1 arbitrary character. See the LIKE operator documentation.
So, for your sample text 'sting', the above code produces the string '%sting%', and then the database adapter takes care of proper quoting and executes a SELECT * FROM run WHERE act LIKE '%sting%' SQL query against your database. Any row where the act column contains the substring sting will match and be returned.
I have a table of three columnsid,word,essay.I want to do a query using (?). The sql sentence is sql1 = "select id,? from training_data". My code is below:
def dbConnect(db_name,sql,flag):
conn = sqlite3.connect(db_name)
cursor = conn.cursor()
if (flag == "danci"):
itm = 'word'
elif flag == "wenzhang":
itm = 'essay'
n = cursor.execute(sql,(itm,))
res1 = cursor.fetchall()
return res1
However, when I print dbConnect("data.db",sql1,"danci")
The result I obtained is [(1,'word'),(2,'word'),(3,'word')...].What I really want to get is [(1,'the content of word column'),(2,'the content of word column')...]. What should I do ? Please give me some ideas.
You can't use placeholders for identifiers -- only for literal values.
I don't know what to suggest in this case, as your function takes a database nasme, an SQL string, and a flag to say how to modify that string. I think it would be better to pass just the first two, and write something like
sql = {
"danci": "SELECT id, word FROM training_data",
"wenzhang": "SELECT id, essay FROM training_data",
}
and then call it with one of
dbConnect("data.db", sql['danci'])
or
dbConnect("data.db", sql['wenzhang'])
But a lot depends on why you are asking dbConnect to decide on the columns to fetch based on a string passed in from outside; it's an unusual design.
Update - SQL Injection
The problems with SQL injection and tainted data is well documented, but here is a summary.
The principle is that, in theory, a programmer can write safe and secure programs as long as all the sources of data are under his control. As soon as they use any information from outside the program without checking its integrity, security is under threat.
Such information ranges from the obvious -- the parameters passed on the command line -- to the obscure -- if the PATH environment variable is modifiable then someone could induce a program to execute a completely different file from the intended one.
Perl provides direct help to avoid such situations with Taint Checking, but SQL Injection is the open door that is relevant here.
Suppose you take the value for a database column from an unverfied external source, and that value appears in your program as $val. Then, if you write
my $sql = "INSERT INTO logs (date) VALUES ('$val')";
$dbh->do($sql);
then it looks like it's going to be okay. For instance, if $val is set to 2014-10-27 then $sql becomes
INSERT INTO logs (date) VALUES ('2014-10-27')
and everything's fine. But now suppose that our data is being provided by someone less than scrupulous or downright malicious, and your $val, having originated elsewhere, contains this
2014-10-27'); DROP TABLE logs; SELECT COUNT(*) FROM security WHERE name != '
Now it doesn't look so good. $sql is set to this (with added newlines)
INSERT INTO logs (date) VALUES ('2014-10-27');
DROP TABLE logs;
SELECT COUNT(*) FROM security WHERE name != '')
which adds an entry to the logs table as before, end then goes ahead and drops the entire logs table and counts the number of records in the security table. That isn't what we had in mind at all, and something we must guard against.
The immediate solution is to use placeholders ? in a prepared statement, and later passing the actual values in a call to execute. This not only speeds things up, because the SQL statement can be prepared (compiled) just once, but protects the database from malicious data by quoting every supplied value appropriately for the data type, and escaping any embedded quotes so that it is impossible to close one statement and another open another.
This whole concept was humourised in Randall Munroe's excellent XKCD comic
I'm trying to return a hard coded value in my SQL query, but when running the query using pyodbc, random records return '\x0e' instead of the hard coded value (in this case '16'). If I run the query on the server (MS SQL Server 2008), the query returns all the correct results and values.
The beginning of the query looks like this:
My SQL Code:
Select '"16","' + S.ShipNum + '","'
My python code:
cursor.execute("""Select '\"16\",\"' + SS.ShipNum + '\",\"'
Is there another way to guarantee a value is returned from a query?
\016 is the oct representation of \x0e
So I would think that it has more to do with the way in which you are escaping your double quotes. In your python you are actually geting \16 and not "16" as you desire.
You should try a prepared statment maybe.
ps = db.prepare("SELECT 16")
ps()
returns:
[(16,)]
Addtional examples can be seen here:
[http://python.projects.pgfoundry.org/docs/0.8/driver.html#parameterized-statements]
You can see all of the ascii and other character sets here
[http://donsnotes.com/tech/charsets/ascii.html]
It looks like you're trying to create a comma-delimited, quoted, string representation of the row. Don't try to do this in the database query, string formatting isn't one of T-SQL's strengths.
Pass the static value using a parameter, then join the row values. Using sys.databases for the example:
params = ("Some value",)
sql = "SELECT ?, name, user_access_desc FROM sys.databases"
for row in cursor.execute(sql):
print(','.join('"{0}"'.format(column) for column in row))