I am trying to import a large text file into a MySQL database. The SQL statement is as follows:
LOAD DATA INFILE '/tmp/epf/full/album_popularity_per_genre'
INTO TABLE album_popularity_per_genre
CHARACTER SET UTF8 FIELDS TERMINATED BY X'01' LINES TERMINATED BY '\n'
IGNORE 45 LINES (export_date, storefront_id, genre_id, album_id, album_rank)
The above works when I run it in phpMyAdmin, however when I write a simple function in Python that uses the above SQL statement I get an error.
Here is the Python code,
def test():
dbConnection = MySQLdb.connect(
charset='utf8',
host='localhost',
user='root',
passwd='root',
db='epf')
cursor = dbConnection.cursor()
exStr = """LOAD DATA INFILE '/tmp/epf/full/album_popularity_per_genre'
INTO TABLE album_popularity_per_genre CHARACTER SET UTF8
FIELDS TERMINATED BY X'01' LINES TERMINATED BY '\n'
IGNORE 45 LINES
(export_date, storefront_id, genre_id, album_id, album_rank)"""
try:
cursor.execute(exStr)
except MySQLdb.Warning, e:
print "Warning %s" % (str(e))
except MySQLdb.IntegrityError, e:
print "Error %d: %s" % (e.args[0], e.args[1])
#Clean up
cursor.close()
dbConnection.close()
The error I get is as follows,
Warning Data truncated for column 'album_rank' at row 1
My question now is, why does the raw SQL statement work but when I try to run the Python code, no data is imported into the database?
The Python DBAPI is implicitly transactional. Try adding dbConnection.commit() after the execute.
Related
I am having trouble trying to copy a few tables from a mysql server to postgres server. Here is the code that I am currently using to connect, fetch the data and write the data.
my_sqlconnection = MySQLdb.connect(host='a.b.c.d',
user='user',
passwd='admin'
)
try:
pl_sqlconnection = psycopg2.connect("host='x.y.z.c' dbname='user'
user='uadmin' password='uadmin'" )
except psycopg2.Error as e:
print('PSQL: Unable to connect!\n{0}')
print (e)
print (e.pgcode)
print (e.pgerror)
print (traceback.format_exc())
sys.exit(1)
cursor1 = my_sqlconnection.cursor(MySQLdb.cursors.DictCursor)
cursor2 = pl_sqlconnection.cursor()
for db in db_list.db_set:
restaurant_name = db[:-3]
my_sqlconnection.select_db(db)
sql = "SELECT * FROM Orders"
cursor1.execute(sql)
for row in cursor1:
try:
cursor2.execute("INSERT INTO Orders (all the values) Values
(%(all the values)s)", row)
except psycopg2.Error as e:
print ("cannot execute that query!!")
print (e)
print (e.pgcode)
print (e.pgerror)
print (traceback.format_exc())
sys.exit("Some problem occured with that query! leaving early")
sql2 = "SELECT * FROM USERS"
cursor1.execute(sql2)
for row in cursor1:
try:
cursor2.execute("INSERT INTO Users (all the values) Values
(%(all the values)s)", row)
except psycopg2.Error as e:
print ("cannot execute that query!!")
print (e)
print (e.pgcode)
print (e.pgerror)
print (traceback.format_exc())
sys.exit("Some problem occured with that query! leaving early")
cursor1.close()
cursor2.close()
pl_sqlconnection.commit()
my_sqlconnection.close()
pl_sqlconnection.close()
Now the error that I am getting is
python backup.py
cannot execute that query!!
invalid byte sequence for encoding "UTF8": 0xeef1e5
HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
22021
ERROR: invalid byte sequence for encoding "UTF8": 0xeef1e5
HINT: This error can also happen if the byte sequence does not match
the encoding expected by the server, which is controlled by "client_encoding".
This error is shown specifically when I am trying to execute the 2nd query. When I run only the 1st query, everything runs as it is supposed to. Both the tables are present in the same database. Why is it that the encoding error is shown while executing the 2nd query.
I've seen some similar questions about this on StackOverflow but haven't found an answer that works; see http://stackoverflow.com/questions/4408714/execute-sql-file-with-python-mysqldb AND http://stackoverflow.com/questions/10593876/execute-sql-file-in-python-with-mysqldb?lq=1
Here is my code:
import pymysql
import sys
import access # holds credentials
import mysql_connector # connects to MySQL, is fully functional
class CreateDB(object):
def __init__(self):
self.cursor = None
self.conn = pymysql.connect(host, user, passwd)
def create_database(self):
try:
with self.conn.cursor() as cursor:
for line in open('file.sql'):
cursor.execute(line)
self.conn.commit()
except Warning as warn:
f = open(access.Credentials().error_log, 'a')
f.write('Warning: %s ' % warn + '\nStop.\n')
sys.exit()
create = CreateDB()
create.create_database()
When I run my script I get the following error:
pymysql.err.InternalError: (1065, 'Query was empty')
My .sql file is successfully loaded when I import directly through MySQL and there is a single query on each line of the file. Does anybody have a solution for this? I have followed the suggestions on other posts but have not had any success.
Take care of empty lines in the end of the file by:
if line.strip(): cursor.execute(line)
You can execute all the SQL in the file at once, by using the official MySQL Connector/Python and the Multi parameter in its cursor.execute method.
Quote from the second link:
If multi is set to True, execute() is able to execute multiple statements specified in the operation string. It returns an iterator that enables processing the result of each statement.
Example code from the link, slightly modified:
import mysql.connector
file = open('script.sql')
sql = file.read()
cnx = mysql.connector.connect(user='u', password='p', host='h', database='d')
cursor = cnx.cursor()
for result in cursor.execute(sql, multi=True):
if result.with_rows:
print("Rows produced by statement '{}':".format(
result.statement))
print(result.fetchall())
else:
print("Number of rows affected by statement '{}': {}".format(
result.statement, result.rowcount))
cnx.close()
I am using following command to load multiple .csv files into Mysql database but i am getting no errors on (the IDLE window) and the data does not load
Here is the erroneous script
#!C:\Python27\python.exe
import MySQLdb
import os
import string
# Open database connection
db = MySQLdb.connect (host="localhost",port=3307,user="root",\
passwd="gamma123",db="test")
cursor=db.cursor()
l = os.listdir(".")
for file_name in l:
print file_name
cursor=db.cursor()
if (file_name.find("DIV.csv")>-1):
#Query under testing
sql = """LOAD DATA LOCAL INFILE file_name \
INTO TABLE system_work \
FIELDS TERMINATED BY ',' \
OPTIONALLY ENCLOSED BY '"' \
LINES TERMINATED BY '\r\n' \
IGNORE 1 LINES;;"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
But when i try to load a single file using the following python script then its works fine.
please help....
#!C:\Python27\python.exe
import MySQLdb
import os
import string
# Open database connection
db = MySQLdb.connect (host="localhost",port=3307,user="root",\
passwd="gamma123",db="test")
cursor=db.cursor()
#Query under testing
sql = """LOAD DATA LOCAL INFILE 'Axle.csv' \
INTO TABLE system_work \
FIELDS TERMINATED BY ',' \
OPTIONALLY ENCLOSED BY '"' \
LINES TERMINATED BY '\r\n' \
IGNORE 1 LINES;;"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
You need to interpolate the filename into the SQL string; you are just sending the literal text file_name to the server. You could use the str.format() method for that, any {} placeholder can then be replaced by a variable of your choosing.
You also must indent the try and except blocks to be within the for loop:
sql = """LOAD DATA LOCAL INFILE '{}'
INTO TABLE system_work
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\\r\\n'
IGNORE 1 LINES;;"""
for file_name in l:
print file_name
if file_name.endswith('DIV.csv'):
try:
cursor = db.cursor()
cursor.execute(sql.format(file_name))
db.commit()
except Exception:
# Rollback in case there is any error
db.rollback()
The cursor.execute() method is passed the sql string with the file_name variable interpolated. The {} part on the first line (LOAD DATA LOCAL INFILE '{}') will be replaced by the value in file_name before passing the SQL statement to MySQL.
I also simplified the filename test; presumably it is enough if the filename ends with DIV.csv.
Note that it might just be easier to use the mysqlimport utility; you can achieve the exact same results with:
mysqlimport --fields-terminated-by=, --fields-optionally-enclosed-by=\" \
--local --lines-terminated-by=\r\n --user=root --password=gamma123 \
test *DIV.csv
if (file_name.find("DIV.csv")>-1): unless all of your files are actually called DIV.csv should that be if (file_name.find(".csv")>-1): (that would probably be more efficient testing the last four letters of the file name by the way)
I am trying to test the following with python but I get the invalid syntax error:
db = None
try:
db = mdb.connect("localhost","user","pass","dbName")
with db:
cur = db.cursor()
cur.execute("SELECT * from product")
rows = cur.fetchall()
for row in rows:
print row
except mdb.Error, e:
print "Error %d: %s" % (e.args[0],e.args[1])
sys.exit(1)
The error is the following:
File "script.py", line 11
with db:
^
SyntaxError: invalid syntax
How do I fix this?
I'd be nice to see all your imports declarations too to see what could be missing from your code. I assume that you at least have the import MySQLdb somewhere. I made a dummy database with a couple of data rows to test this out. Not sure how python let you get away with the indentation blocks being all messed up, but maybe it's just your code indentation posting error on here.
This is the code I tried and seemed to have no issues:
import MySQLdb as mdb
db = None
try:
db = mdb.connect("localhost","user", "password", "test_data")
## with db: ## try taking this out for Python 2.4
cur = db.cursor()
cur.execute("SELECT * FROM PRODUCT")
rows = cur.fetchall()
for row in rows:
print row
except mdb.Error, e:
print "Error %d: %s" % (e.args[0],e.args[1])
sys.exit(1)
I wanted a script that iterates through csv files in a folder and dump them into a MySQL database. I was able to dump one csv file into it.. But have troubles passing the file name in to the SQL script.
This is the code I use
file_path="C:\csv-files"
files=os.listdir(file_path)
files.sort()
for n in files:
cursor.execute(" LOAD DATA LOCAL INFILE '%s' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES ",(n))
And I get the following error
raise errorclass, errorvalue
ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'file1.csv'' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY' at line 1")
If I use the file name directly instead of passing it, it works fine.
If you can see in the error thrown out, there seems to be an error in the SQL Script.
This would be the whole code
import csv
import MySQLdb
import sys
import os
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db')
cursor = connection.cursor()
file_path="C:\csv-files"
files=os.listdir(file_path)
files.sort()
for n in files:
print n
cursor.execute(" LOAD DATA LOCAL INFILE %s INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES " %n)
connection.commit()
cursor.close()
First, replace '%s' with %s in the query. MySQLdb handles any quoting automatically.
Here's the code with some corrections and changes:
import MySQLdb
import os
CSV_DIR = "C:\csv-files"
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db',
local_infile=1)
cursor = connection.cursor()
try:
for filename in sorted(os.listdir(CSV_DIR)):
cursor.execute("""LOAD DATA LOCAL INFILE %s
INTO TABLE new_table
FIELDS
TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES""",
(os.path.join(CSV_DIR, filename),))
connection.commit()
finally:
cursor.close()
NOTE: I set local_infile parameter to 1 in MySQLdb.connect and pass filename in tuple to execute.
Works for me.