I have created a little script that allows me to save data to MySQLdb. At first it was working fine when I was using:
cursor.execute('INSERT INTO people (name, text) VALUES ("dan", "test2")')
The above would save "dan" into the title and "test2" into the text. I wanted to test to see if I was able to define something and fill it in this way. For example if I was to scrape a site and say (dan = soup.title.string) or something like that it would be able to populate this data into the database. I have tried to have a look around but cannot seem to find anything.
import MySQLdb
import sys
try:
db = MySQLdb.connect(
host = 'localhost',
user = 'root',
passwd = '',
db = 'python',
)
except:
print "db not found"
dan = "dandandan"
test2 = "testing101"
cursor = db.cursor()
cursor.execute('INSERT INTO people (name, text) VALUES (dan, test2)')
cursor.execute('SELECT * FROM people')
result = cursor.fetchall()
db.commit()
db.close()
The error I am receiving is:
C:\Users\********\Desktop>python mysqltest.py
Traceback (most recent call last):
File "mysqltest.py", line 18, in <module>
cursor.execute('INSERT INTO people (name) VALUES (dan)')
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "C:\Python27\lib\site-packages\MySQLdb\connections.py", line 36, in defau
lterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (1054, "Champ 'dan' inconnu dans field list"
)
You need to use parameters.
cursor.execute('INSERT INTO people (name, text) VALUES (%s,%s)', (dan, test2))
Use prepared statements:
cursor.execute("INSERT INTO people (name, text) VALUES (%s,%s)", (dan, test2))
From the documentation :
paramstyle
String constant stating the type of parameter marker formatting
expected by the interface. Set to 'format' = ANSI C printf format
codes, e.g. '...WHERE name=%s'. If a mapping object is used for
conn.execute(), then the interface actually uses 'pyformat' = Python
extended format codes, e.g. '...WHERE name=%(name)s'. However, the API
does not presently allow the specification of more than one style in
paramstyle.
Note that any literal percent signs in the query string passed to
execute() must be escaped, i.e. %%.
Parameter placeholders can only be used to insert column values. They
can not be used for other parts of SQL, such as table names,
statements, etc.
Related
Thank you for reading. I have some experience with SQL, very new to python.
In the below code, i am accessing 2 databases in python 2.7
The connections work. I can query a tables that has a serial #s for devices in one statement with no issue. I then want to query a table which name matches that serial number in another database, pulling the latest value of the "Stamp" field. All of this works when i explictly name the table ccnbsc00000001, but when using variable subsitution, it fails.
When the variable currentdevice is substituted, extras characters are included. When i print that variable, those character are not present in that output. here is the code, and the error result at the bottom
#!/usr/bin/python
### Imports
import datetime
import mysql.connector
#Connect to heartbeat results database
hb_db = mysql.connector.connect(
host="localhost",
user="otheruser",
passwd="******",
database="active_devices"
)
#Connect to heartbeat results database
device_Settings_db = mysql.connector.connect(
host="localhost",
user="otheruser",
passwd="******",
database="active_devices"
)
device_settings_cursor = device_settings_db.cursor()
hb_cursor = hb_db.cursor()
## Get deviuce serial#
device_settings_cursor.execute('select device_serial from devices')
active_devices = device_settings_cursor.fetchall()
print ("these are the current devices:")
print (active_devices)
for device in active_devices:
currentdevice = device[0]
print(currentdevice)
print ("SELECT MAX(stamp) FROM (%s)" , (currentdevice,) )
hb_cursor.execute('SELECT MAX(stamp) FROM (%s)' , (currentdevice,) )
laststamp = hb_cursor.fetchone
laststamp = laststamp[0]
print("Last time stamp is:")
print(laststamp)
*
Output of print(active_devices)
[(u'ccnbsc00000001',), (u'ccnbsc00000002',)]
output of print(currentdevice)
ccnbsc00000001
(This is the correct output/value)
but I get this error in the SQL query that implies it has kept the surrounding characters ' and ')
Traceback (most recent call last):
File "./hb_notify.py", line 61, in <module>
hb_cursor.execute('SELECT MAX(stamp) FROM (%s)' , (currentccn,) )
File "/usr/lib/python2.7/site-packages/mysql/connector/cursor.py", line 551, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "/usr/lib/python2.7/site-packages/mysql/connector/connection.py", line 490, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "/usr/lib/python2.7/site-packages/mysql/connector/connection.py", line 395, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your **SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''ccnbsc00000001')' at line 1**
Python MySQL libraries commonly insert quotation marks when you pass string arguments to them as arguments, because usually you do actually want those quotation marks. This is why you're seeing quotation marks.
The fix here is easy: instead of passing those values as arguments to your cursor, you can just insert those values directly into the string like you would if it were any other Python string. Like so:
hb_cursor.execute('SELECT MAX(stamp) FROM {0}'.format(currentdevice))
Python string arguments will remove quotes around a string, MySQL cursor arguments will keep the quotes.
I am working with Python 3.3, pypyodbc 1.2.1, and a Quickbooks Enterprise 12 company file being access over Flexquarters QODBC version 14. I'm new to programming and python, so still learning :) I can run a query using the pypyodbc examples just fine, and produces expected results.
Notice the hardcoded email address in the execute. This works as expected:
def get_customer_id(search_col,search_str):
'''(str,str) --> str
>>>get_customer_id(email, foo#foo.com)
80000001-1385782702
'''
cur.execute("SELECT listid FROM CUSTOMER WHERE email='foo#foo.com'")
for row in cur.fetchall():
for field in row:
return field
If I try to do the same thing using the parameters that I am reading from the pypyodbc documentation, I throw an error. I'm having problems with the quotes, and parameter markers I think.
def get_customer_id(search_col,search_str):
'''(str,str) --> str
>>>get_customer_id(email, foo#foo.com)
80000001-1385782702
'''
cur.execute("SELECT listid FROM CUSTOMER WHERE email=?",(search_str,))
for row in cur.fetchall():
for field in row:
return field
Trying to be more pythonistic? I really want to reuse the function to search different columns. Something like:
cur.execute("SELECT listid FROM CUSTOMER WHERE search_str=search_col")
I have looked at a few other threads, and most of them seem to just be dealing with the parameter, and not the column to search. Can anyone help me learn this?
PS forgot to include the traceback:
Traceback (most recent call last):
File "C:\Users\Mike\Documents\Projects\qb_sync\quickbooks.py", line 32, in <module>
print(get_customer_id('email','foo#foo.com'))
File "C:\Users\Mike\Documents\Projects\qb_sync\quickbooks.py", line 27, in get_customer_id
cur.execute("SELECT listid FROM CUSTOMER WHERE email=?",[search_str,])
File "C:\Python\lib\site-packages\pypyodbc.py", line 1457, in execute
self._BindParams(param_types)
File "C:\Python\lib\site-packages\pypyodbc.py", line 1420, in _BindParams
check_success(self, ret)
File "C:\Python\lib\site-packages\pypyodbc.py", line 982, in check_success
ctrl_err(SQL_HANDLE_STMT, ODBC_obj.stmt_h, ret, ODBC_obj.ansi)
File "C:\Python\lib\site-packages\pypyodbc.py", line 960, in ctrl_err
raise Error(state,err_text)
pypyodbc.Error: ('HY004', '[HY004] [Microsoft][ODBC Driver Manager] SQL data type out of range')
[Finished in 1.7s]
I think the use of
cur.execute("""SELECT listid FROM CUSTOMER WHERE ?=?""",[column, email])
can not be accepted by database engine rather than pypyodbc or any other odbc interface. It's the database engine refuse to accept the query for the use of parameter on column names.
Probably you would have to try this instead to reuse the function:
# First construct your dynamic query for the targeted column
sql = """SELECT listid FROM CUSTOMER WHERE %s=?""" %(column)
# Then provide the dynamic value for the dynamic query string
cur.execute(sql, (value,))
Python 3 also has the str.format() method which will do string replacement on {index} items within your string. This is useful if you have many values to inject into your strings like:
myStr = "I like {0} and {1}, but I don't like {2}.".format("apples","bananas","spinach")
myStr
"I like apples and bananas, but I don't like spinach."
# First construct your dynamic query for the targeted column
sql = """SELECT listid FROM CUSTOMER WHERE {0}=?""".format(column)
# Then provide the dynamic value for the dynamic query string
cur.execute(sql, (value,))
It's worth noting that this method of replacing values in a string query can be subject to sql injection.
The safer way to do this would be with parameterized stored procs.
I got 1/2 the answer so far. This works for one parameter, IF I format the string before calling the function;
print(get_custid_email(b'foo#foo.org'))
cur.execute("""SELECT listid FROM CUSTOMER WHERE email=?""",[email])
I still can't get it to do the same thing with column name though.
print(get_custid_email(b'email',b'foo#foo.org'))
cur.execute("""SELECT listid FROM CUSTOMER WHERE ?=?""",[column, email])
That throws a differnt error:
Traceback (most recent call last):
File "C:\Users\Mike\Documents\Projects\qb_sync\quickbooks.py", line 34, in <module>
print(get_custid_email(b'wendy.lindsay#gmail.com'))
File "C:\Users\Mike\Documents\Projects\qb_sync\quickbooks.py", line 29, in get_custid_email
cur.execute("""SELECT listid FROM CUSTOMER WHERE ?=?""",['email',email])
pyodbc.ProgrammingError: ('42S00', '[42S00] [QODBC] Data type of parameter cannot be determined (11023) (SQLPrepare)')
I have the below result from my python code:
[
{filename:'1,2',Name:'Gorge',registration number: '6657', registration date: '2012-09-10 14:31:13'},
{filename:'5,43',Name:'mazu',registration number:'45', registration date:'2012-10-08 17:28:47'}]
and as soon as I want to put it in a MySQL table, I got this error:
Traceback (most recent call last):
File "C:\Users\jio\Datasets 1\MyTable_info.py", line 63, in <module>
cur.executemany(query,records)
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 243, in executemany
self.errorhandler(self, ProgrammingError, msg.args[0])
File "C:\Python27\lib\site-packages\MySQLdb\connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: not enough arguments for format string
My python code to insert the result in MySQL table is the below code:
con = MySQLdb.connect(host = "******", port=***, user = "***", passwd="*****", db="****")
with con:
cur=con.cursor()
cur.execute('''CREATE TABLE IF NOT EXISTS info(id INT(10) auto_increment primary key,file_name VARCHAR(10),
Name VARCHAR(50),Registration ID INT(50),registration time INT(50))''')
query= "INSERT INTO info (file_name, Name, Registration ID, registration time) VALUES ( %s, %s, %s, %s )"
cur.executemany(query,records)
con.commit()
Does anyone has an idea why I get this error and what does the error mean?
You are trying to insert a String into a field INT(50).
Take a look in the last field of the table, registration time field. It is an Integer and you are trying to insert values like '2012-10-08 17:28:47' or '2012-09-10 14:31:13'.
For a quick fix just change registration time field type as a VARCHAR(50).
But maybe, for perfomance issues, you should think to use some kind of TIMESTAMP field instead of a VARCHAR for this kind of purposes.
Apart from don't use INT type for field where you would like to add some kind of String.
Modify the records variable from
[ {filename:'1,2',Name:'Gorge',registration number: '6657', registration date: '2012-09-10 14:31:13'},
{filename:'5,43',Name:'mazu',registration number:'45', registration date:'2012-10-08 17:28:47'}]
to
[('1,2','Gorge','6657', '2012-09-10 14:31:13'),('5,43','mazu','45','2012-10-08 17:28:47')]
And avoid use whitespaces for the columns names as well
Try to quote field names. What the contents of records?
Don't use spaces in field names.
If you are required to, quote them with the ` character.
I have a database called project1 with the following tables:
_systbl1
_systbl2
_systbl3
dataset1
dataset2
dataset3
MySQL user odbc will need to be granted SELECT permissions on dataset% tables whenever a new one is added.
To accomplish this, I'm using a simple python script, like so:
#!/usr/bin/python
import MySQLdb
db = MySQLdb.connect(
host="localhost",
user="user",
passwd="pass",
db="project1"
)
# Create Cursor Object
cur = db.cursor()
# get list of tables beginning with dataset
cur.execute("SHOW TABLES FROM project1 LIKE 'dataset%';")
# run GRANT statement for each table
for row in cur.fetchall() :
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`localhost`;", row)
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`%`;", row)
Unfortunately, it gives me the following error:
Traceback (most recent call last):
File "mysql_query.py", line 20, in <module>
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`localhost`;", row)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1146, "Table 'project1.'dataset1'' doesn't exist")
As you can see in the last line of the error, the problem is that python is putting a single quote around the table names when generating the query.
What am I missing here?
Do not use SQL parameters for table names. SQL parameters are escaped by the database adapter to not be interpreted as anything but literal values.
You'll have to interpolate those yourself instead, but be absolutely certain that your table name does not hold untrusted data (prevent SQL injection attacks):
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`localhost`;" % row)
cur.execute("GRANT SELECT ON `project1`.`%s` TO `odbc`#`%%`;" % row)
(where the % character in the grant has been escaped by doubling it to %%).
Instead use:
cur.execute("GRANT SELECT ON `project`.`%s` TO `odbc`#`localhost`" % row)
This will not use the normal escaping of the input. Beware of a backtick in any of your table names, though.
sql = """CREATE TABLE IF NOT EXISTS `""" + project + """` ( `id` INT(11) NOT NULL AUTO_INCREMENT, PRIMARY KEY (`id`))"""
This is my first post! I also just started programming, so please bear with me!
I am trying to load a bunch of .csv files into a database, in order to later perform various reports on the data. I started off by creating a few tables in mysql with matching field names and data types to what will be loaded into the tables. I am manipulating the filename (in order to parse out the date to use as a field in my table) and cleaning up the data with python.
So my problem right now (haha...) is that I get this error message when I attempt the 'Insert Into' query to mysql.
Traceback (most recent call last):
File "C:\Program Files\Python\load_domains2.py", line 80, in <module>
cur.execute(sql)
File "C:\Program Files\Python\lib\site-packages\MySQLdb\cursors.py", line 166, in execute
self.errorhandler(self, exc, value)
File "C:\Program Files\Python\lib\site-packages\MySQLdb\connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1054, "Unknown column 'a1200e.com' in 'field list'")
'a1200e.com' refers to a specific domain name I'm inserting into that column. My query is as follows:
sql="""INSERT INTO temporary_load
(domain_name, session_count, search_count, click_count,
revenue, revenue_per_min, cost_per_click, traffic_date)
VALUES (%s, %d, %d, %d, %d, %d, %d, %s)""" %(cell[0],
int(cell[1]),
int(cell[2].replace (",","")),
int(cell[3].replace(",","")),
float(cell[4].replace("$","")),
float(cell[5].replace("$","")),
float(cell[6].replace("$","")),
parsed_date)
cur.execute(sql)
I am very new at all this, so I'm sure my code isn't at all efficient, but I just wanted to lay everything out so it's clear to me. What I don't understand is that I have ensured my table has correctly defined data types (corresponding to those in my query). Is there something I'm missing? I've been trying to work this out for a while, and don't know what could be wrong :/
Thanks so much!!!
Val
Thomas is, as usual, absolutely correct: feel free to let MySQLdb handle the quoting issues.
In addition to that recommendation:
The csv module is your friend.
MySQLdb uses the "format" parameter style as detailed in PEP 249.
What does that mean for you?
All parameters, whatever type, should be passed to MySQLdb as strings (like this %s). MySQLdb will make sure that the values are properly converted to SQL literals.
By the way, MySQLdb has some good documentation.
Feel free to include more detail about your source data. That may make diagnosing the problem easier.
Here's one way to insert values to a MySQL database from a .csv file:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import csv
import MySQLdb
import os
def main():
db = MySQLdb.connect(db="mydb",passwd="mypasswd",) # connection string
filename = 'data.csv'
f = open(filename, "rb") # open your csv file
reader = csv.reader(f)
# assuming the first line of your csv file has column names
col_names = reader.next() # first line of .csv file
reader = csv.DictReader(f, col_names) # apply column names to row values
to_db = [] # this list holds values you will insert to db
for row in reader: # loop over remaining lines in .csv file
to_db.append((row['col1'],row['col2']))
# or if you prefer one-liners
#to_db = [(row['col1'],row['col2']) for row in reader]
f.close() # we're done with the file now
cursor = db.cursor()
cursor.executemany('''INSERT INTO mytable (col1,col2)
VALUES (%s, %s)''', to_db) # note the two arguments
cursor.close()
db.close()
if __name__ == "__main__":
main()
You should be using DB-API quoting instead of including the data in the SQL query directly:
sql = """INSERT INTO temporary_load
(domain_name, session_count, search_count, click_count,
revenue, revenue_per_min, cost_per_click, traffic_date)
VALUES (%s, %d, %d, %d, %d, %d, %d, %s)"""
args = (cell[0],
int(cell[1]),
int(cell[2].replace (",","")),
int(cell[3].replace(",","")),
float(cell[4].replace("$","")),
float(cell[5].replace("$","")),
float(cell[6].replace("$","")),
parsed_date)
cur.execute(sql, args)
This makes the DB-API module quote the values appropriately, and resolves a whole host of issues that you might get when doing it by hand (and usually incorrectly.)