I have 304MB text file which contains comma(,) separated values in each line, I am reading file line by line and inserting values using for loop, but I keep getting this error after 21 rows are inserted.
here is the code
import MySQLdb
myfile = open('/home/vaibhav/Desktop/question1.csv', 'r')
db = MySQLdb.connect(host="localhost",
user="root",
passwd="admin",
db="question1agg")
for line in myfile:
my_line_list = line.split(',')
string = ''
for value in my_line_list:
string = string + "'" + value + "',"
query_string = string[:-1]
final_query = "insert into question1 values"+"("+query_string+");"
cur = db.cursor()
cur.execute(final_query)
db.commit()
db.close()
And this is the error i get
Traceback (most recent call last):
File "da.py", line 19, in <module>
cur.execute(final_query)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 's Solids','Men','NULL','Solid','/Products/Legwear','/Products/Legwear/AmericanEs' at line 1")
Looks like on one of your lines there is a apostrophe that is messing MySQL up. From the error message -
's Solids','Men','NULL','Solid'
Your problem and it's solution is explained nicely here - Python MySQL escape special characters
It looks like one of your values have an apostrophe in it. You need to escape the apostrophe to avoid that error.
For example, you have a value like this:
That's right!
So your insert looks like this:
insert into question1 values('value1','That's right!','value2');
You should escape the apostrophe to make the insert like this:
insert into question1 values('value1','That''s right!','value2');
Related
This question already has answers here:
Python MySQLdb TypeError: not all arguments converted during string formatting
(8 answers)
Closed 4 years ago.
I am trying to export CSV files into MySQL databases, but I can't seem to figure out the error with my code. I believe it should be working.
import csv
import MySQLdb
mysql_conn = MySQLdb.connect(host='localhost', user='root', passwd='******', db='accounting')
mysql_cursor = mysql_conn.cursor()
f = open('C:/Users/Pops/Desktop/List of Payees.csv')
csv_f = csv.reader(f)
for row in csv_f:
mysql_cursor.execute("""INSERT INTO list_of_payees (list_of_payees) VALUES (%s)""", (row[0]))
mysql_conn.commit()
mysql_cursor.close()
Error Code:
Traceback (most recent call last):
File "C:\Users\Pops\Desktop\Martin\Programming\Python\venv\lib\site-packages\MySQLdb\cursors.py", line 238, in execute
query = query % args
TypeError: not all arguments converted during string formatting
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/Pops/Desktop/Martin/Programming/Python/ExcelToMySQL.py", line 11, in <module>
mysql_cursor.execute("""INSERT INTO list_of_payees (list_of_payees) VALUES (%s)""", (row[0]))
File "C:\Users\Pops\Desktop\Martin\Programming\Python\venv\lib\site-packages\MySQLdb\cursors.py", line 240, in execute
self.errorhandler(self, ProgrammingError, str(m))
File "C:\Users\Pops\Desktop\Martin\Programming\Python\venv\lib\site-packages\MySQLdb\connections.py", line 52, in defaulterrorhandler
raise errorclass(errorvalue)
_mysql_exceptions.ProgrammingError: not all arguments converted during string formatting
Process finished with exit code 1
LOAD DATA INFILE is a faster and better approach to upload the csv files to MYSQL table, Reading CSV and Inserting into MYSQL table is a slow and time taking process. Use MYSQLdb to execute this command from python.
Sample Query:
LOAD DATA INFILE '/path_of_csv_file/sampledata.csv'
INTO TABLE expenses
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n' -- or \r\n
(column1, column2, column3, column4, column5)
Execute's second parameter represents a list of the objects to be converted.
mysql_cursor.execute("""INSERT INTO list_of_payees (list_of_payees) VALUES (%s)""", (row[0]))
This line is the problem, as row[0] is expected to be a list
You could try going with mysql_cursor.execute("INSERT INTO list_of_payees (list_of_payees) VALUES '%s'", [row[0]])
Also alternatively you can try adding a comma to the end of your row value mysql_cursor.execute("INSERT INTO list_of_payees (list_of_payees) VALUES '%s'", (row[0],))
I am trying to create a sql table using python script.
Here is the code :
import MySQLdb
db1 = MySQLdb.connect(host="localhost",user="root",passwd="")
cursor = db1.cursor()
sql = 'use test'
cursor.execute(sql)
query='CREATE TABLE IF NOT EXISTS 3112017(service_area_code VARCHAR(100),phone_numbers VARCHAR(100),preferences VARCHAR(100),opstype VARCHAR(100),phone_type VARCHAR(100))'
cursor.execute(query)
db1.commit()
Following is the error I am getting:
cursor.execute(query)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 226, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '3112017(service_area_code VARCHAR(100),phone_numbers VARCHAR(100),preferences VA' at line 1")
As mentioned in the traceback, you have an error in your SQL syntax. Table name is wrong.
From the SQL manual -
Identifiers may begin with a digit but unless quoted may not consist
solely of digits.
I am trying to pass a variable to an SQL statement which I will eventually use in an iterator in order to process a list of key values and store in a CSV, however I am having trouble getting the variable into the statement?
Here is my code:
import MySQLdb as mdb
from MySQLdb import cursors
import csv
con = mdb.connect('172.16.7.50', 'root', 'abcd2014', 'templog')
tablename = 'pitemp'
with con:
cursor = con.cursor()
cursor.execute("SELECT temp, time FROM %s", (tablename,))
fid = open('new.csv','w')
writer = csv.writer(fid, delimiter=',')
writer.writerow([ i[0] for i in cursor.description ]) # heading row
writer.writerows(cursor.fetchall())
print 'finished!'
I have tried a selection of different bracket combinations as found on stack overflow but they all result in the following error:
Traceback (most recent call last):
File "/home/tom/PycharmProjects/TomsSQL2CSV/sql2csv.py", line 11, in <module>
cursor.execute("SELECT temp, time FROM %s", (vari,))
File "/home/tom/.local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "/home/tom/.local/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''pitemp'' at line 1")
You should be using '?' for parameter bindings in your sql string not
python format specifiers (you are after all writing sql here not
python).
cursor.execute("SELECT temp, time FROM ?", (tablename,))
I'm trying to execute an insert query. It works when I directly copy and paste it to the mysql command prompt, but fails when I execute it from Python. I'm getting this error with MySQLdb (also tried using _mysql directly and get the same error).
The error is the same as this question, but the answer does not work for my problem (my query is on a single line): MySQL the right syntax to use near '' at line 1 error
query = """INSERT INTO %s(%s) VALUES (%f) ON DUPLICATE KEY UPDATE %s=%f"""%(table_name,measurement_type,value,measurement_type,value)
print query
cur.execute(query)
Result (it prints the query, which when copied directly into MySQL command prompt executes fine, and then crashes):
INSERT INTO D02CA10B13E5$accelerometer_X(periodic) VALUES (79.000000) ON DUPLICATE KEY UPDATE periodic=79.000000
Traceback (most recent call last):
File "collect_data.py", line 145, in <module>
process_data_array(data_bytes[start:start+sensor_packet_size])
File "collect_data.py", line 105, in process_data_array
record_data(MAC,sensor_name,"X",code_name,X,cur)
File "collect_data.py", line 58, in record_data
cur.execute(query)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1")
I tried turning on profiling because I can't seem be able to access _last_executed from my cursor (something others have suggested). It seems that my cursor does not have this property (was it removed?).
query = """INSERT INTO %s(%s) VALUES (%f) ON DUPLICATE KEY UPDATE %s=%f"""%(table_name,measurement_type,value,measurement_type,value)
print query
#cur.execute(query)
try:
cur.execute(query)
except:
cur.execute('show profiles')
for row in cur:
print row
cur.execute("set profiling = 0")
exit()
This shows me an incomplete query:
INSERT INTO D02CA10B13E5$accelerometer_X(periodic) VALUES (80.000000) ON DUPLICATE KEY UPDATE periodic=80.000000
(1L, 5.925e-05, 'INSERT INTO D02CA10B13E5')
Some suggest splitting the query into multiple lines (which contradicts the suggestion in the link I posted above). Anyway, it narrows the search to one line (apparently the Python module doesn't like either my table name or column name which are on the third line)
query = "INSERT \nINTO \n%s(%s) \nVALUES \n(%f) \nON \nDUPLICATE \nKEY \nUPDATE %s=%f"%(table_name,measurement_type,value,measurement_type,value)
print query
cur.execute(query)
Result:
INSERT
INTO
D02CA10B13E5$accelerometer_X(periodic)
VALUES
(80.000000)
ON
DUPLICATE
KEY
UPDATE periodic=80.000000
Traceback (most recent call last):
File "collect_data.py", line 145, in <module>
process_data_array(data_bytes[start:start+sensor_packet_size])
File "collect_data.py", line 105, in process_data_array
record_data(MAC,sensor_name,"X",code_name,X,cur)
File "collect_data.py", line 58, in record_data
cur.execute(query)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 3")
Again, this query executes fine on MySQL, so I'm not sure what's going on here in Python. Any pointers are appreciated. Thanks!
Ok so '' in the MySQL error apparently referred to a \0 character that got appended to my table name. This was not visible as you can see for yourself:
>>> print chr(0)
>>> print "hello" + chr(0)
hello
The \0 character is completely invisible and doesn't get copied onto the clipboard from the terminal (so it worked when pasted onto the MySQL console). Sneaky!
I found this out by comparing string lengths with expected string lengths and finding a difference of 1 character that was invisible to my eyes. The \0 character came about in my Python code when I read the string (sensor ID) from a socket (the application on the other end was a C program that was appending these \0 characters).
Hope my answer saves someone else a lot of trouble!
I keep getting the same error everytime I try to INSERT data into a MySQL table in my Python script:
tools.cerabot#tools-login:~/wikitool-tasks2$ python didyouknow.py
didyouknow.py:62: Warning: Table 'did_you_know' already exists
self.cursor.execute(self.create_query)
Traceback (most recent call last):
File "didyouknow.py", line 121, in <module>
test._parse_page()
File "didyouknow.py", line 109, in _parse_page
self.cursor.execute(record_exists.format(item["name"]))
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ':Did you know nominations/Cirrus (song)' at line 1")
The code for this is openly viewable on GitHub, though you'd be particularly interested in lines 95 through 116. I've tried escaping and unicoding the string, modifying my query, nothing. (Admittedly, I'm a basic MySQL programmer.) Could anyone experienced in the area help me figure this out please?
The problem is that the "funny characters" in your Wikipedia titles are messing with the SQL syntax. You need to handle that. This can be done by escaping, but the best practice is to use SQL parameterization, like so:
record_exists = u"SELECT COUNT(*) FROM did_you_know WHERE " \
"name = %s"
# and later on
self.cursor.execute(record_exists, item["name"])
You're actually already doing this later on (line 112).