I am using following command to load multiple .csv files into Mysql database but i am getting no errors on (the IDLE window) and the data does not load
Here is the erroneous script
#!C:\Python27\python.exe
import MySQLdb
import os
import string
# Open database connection
db = MySQLdb.connect (host="localhost",port=3307,user="root",\
passwd="gamma123",db="test")
cursor=db.cursor()
l = os.listdir(".")
for file_name in l:
print file_name
cursor=db.cursor()
if (file_name.find("DIV.csv")>-1):
#Query under testing
sql = """LOAD DATA LOCAL INFILE file_name \
INTO TABLE system_work \
FIELDS TERMINATED BY ',' \
OPTIONALLY ENCLOSED BY '"' \
LINES TERMINATED BY '\r\n' \
IGNORE 1 LINES;;"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
But when i try to load a single file using the following python script then its works fine.
please help....
#!C:\Python27\python.exe
import MySQLdb
import os
import string
# Open database connection
db = MySQLdb.connect (host="localhost",port=3307,user="root",\
passwd="gamma123",db="test")
cursor=db.cursor()
#Query under testing
sql = """LOAD DATA LOCAL INFILE 'Axle.csv' \
INTO TABLE system_work \
FIELDS TERMINATED BY ',' \
OPTIONALLY ENCLOSED BY '"' \
LINES TERMINATED BY '\r\n' \
IGNORE 1 LINES;;"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
You need to interpolate the filename into the SQL string; you are just sending the literal text file_name to the server. You could use the str.format() method for that, any {} placeholder can then be replaced by a variable of your choosing.
You also must indent the try and except blocks to be within the for loop:
sql = """LOAD DATA LOCAL INFILE '{}'
INTO TABLE system_work
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\\r\\n'
IGNORE 1 LINES;;"""
for file_name in l:
print file_name
if file_name.endswith('DIV.csv'):
try:
cursor = db.cursor()
cursor.execute(sql.format(file_name))
db.commit()
except Exception:
# Rollback in case there is any error
db.rollback()
The cursor.execute() method is passed the sql string with the file_name variable interpolated. The {} part on the first line (LOAD DATA LOCAL INFILE '{}') will be replaced by the value in file_name before passing the SQL statement to MySQL.
I also simplified the filename test; presumably it is enough if the filename ends with DIV.csv.
Note that it might just be easier to use the mysqlimport utility; you can achieve the exact same results with:
mysqlimport --fields-terminated-by=, --fields-optionally-enclosed-by=\" \
--local --lines-terminated-by=\r\n --user=root --password=gamma123 \
test *DIV.csv
if (file_name.find("DIV.csv")>-1): unless all of your files are actually called DIV.csv should that be if (file_name.find(".csv")>-1): (that would probably be more efficient testing the last four letters of the file name by the way)
Related
I've seen some similar questions about this on StackOverflow but haven't found an answer that works; see http://stackoverflow.com/questions/4408714/execute-sql-file-with-python-mysqldb AND http://stackoverflow.com/questions/10593876/execute-sql-file-in-python-with-mysqldb?lq=1
Here is my code:
import pymysql
import sys
import access # holds credentials
import mysql_connector # connects to MySQL, is fully functional
class CreateDB(object):
def __init__(self):
self.cursor = None
self.conn = pymysql.connect(host, user, passwd)
def create_database(self):
try:
with self.conn.cursor() as cursor:
for line in open('file.sql'):
cursor.execute(line)
self.conn.commit()
except Warning as warn:
f = open(access.Credentials().error_log, 'a')
f.write('Warning: %s ' % warn + '\nStop.\n')
sys.exit()
create = CreateDB()
create.create_database()
When I run my script I get the following error:
pymysql.err.InternalError: (1065, 'Query was empty')
My .sql file is successfully loaded when I import directly through MySQL and there is a single query on each line of the file. Does anybody have a solution for this? I have followed the suggestions on other posts but have not had any success.
Take care of empty lines in the end of the file by:
if line.strip(): cursor.execute(line)
You can execute all the SQL in the file at once, by using the official MySQL Connector/Python and the Multi parameter in its cursor.execute method.
Quote from the second link:
If multi is set to True, execute() is able to execute multiple statements specified in the operation string. It returns an iterator that enables processing the result of each statement.
Example code from the link, slightly modified:
import mysql.connector
file = open('script.sql')
sql = file.read()
cnx = mysql.connector.connect(user='u', password='p', host='h', database='d')
cursor = cnx.cursor()
for result in cursor.execute(sql, multi=True):
if result.with_rows:
print("Rows produced by statement '{}':".format(
result.statement))
print(result.fetchall())
else:
print("Number of rows affected by statement '{}': {}".format(
result.statement, result.rowcount))
cnx.close()
Writing a script to convert raw data for MySQL import I worked with a temporary textfile so far which I later imported manually using the LOAD DATA INFILE... command.
Now I included the import command into the python script:
db = mysql.connector.connect(user='root', password='root',
host='localhost',
database='myDB')
cursor = db.cursor()
query = """
LOAD DATA INFILE 'temp.txt' INTO TABLE myDB.values
FIELDS TERMINATED BY ',' LINES TERMINATED BY ';';
"""
cursor.execute(query)
cursor.close()
db.commit()
db.close()
This works but temp.txt has to be in the database directory which isn't suitable for my needs.
Next approch is dumping the file and commiting directly:
db = mysql.connector.connect(user='root', password='root',
host='localhost',
database='myDB')
sql = "INSERT INTO values(`timestamp`,`id`,`value`,`status`) VALUES(%s,%s,%s,%s)"
cursor=db.cursor()
for line in lines:
mode, year, julian, time, *values = line.split(",")
del values[5]
date = datetime.strptime(year+julian, "%Y%j").strftime("%Y-%m-%d")
time = datetime.strptime(time.rjust(4, "0"), "%H%M" ).strftime("%H:%M:%S")
timestamp = "%s %s" % (date, time)
for i, value in enumerate(values[:20], 1):
args = (timestamp,str(i+28),value, mode)
cursor.execute(sql,args)
db.commit()
Works as well but takes around four times as long which is too much. (The same for construct was used in the first version to generate temp.txt)
My conclusion is that I need a file and the LOAD DATA INFILE command to be faster. To be free where the textfile is placed the LOCAL option seems useful. But with MySQL Connector (1.1.7) there is the known error:
mysql.connector.errors.ProgrammingError: 1148 (42000): The used command is not allowed with this MySQL version
So far I've seen that using MySQLdb instead of MySQL Connector can be a workaround. Activity on MySQLdb however seems low and Python 3.3 support will probably never come.
Is LOAD DATA LOCAL INFILE the way to go and if so is there a working connector for python 3.3 available?
EDIT: After development the database will run on a server, script on a client.
I may have missed something important, but can't you just specify the full filename in the first chunk of code?
LOAD DATA INFILE '/full/path/to/temp.txt'
Note the path must be a path on the server.
To use LOAD DATA INFILE with every accessible file you have to set the
LOCAL_FILES client flag while creating the connection
import mysql.connector
from mysql.connector.constants import ClientFlag
db = mysql.connector.connect(client_flags=[ClientFlag.LOCAL_FILES], <other arguments>)
I have a SQL query which opens up a csv file and dumps it in a table of a database. I am trying to dump multiple files at once using a python script to iterate among the files. I tried embedding the same SQL query inside the script, but it throws out an error.
This is the script I have.
import csv
import MySQLdb
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db')
cursor = connection.cursor()
query = """ LOAD DATA INFILE 'c:\\example.csv' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES """
cursor.execute(query)
conenction.commit()
cursor.close()
And for some reason the python script looks up example.csv at a different location
This is the error that is thrown out :
raise errorclass, errorvalue
InternalError: (29, "File 'C:\\Documents and Settings\\All Users\\Application Data\\MySQL\\MySQL Server 5.5\\data\\example.csv' not found (Errcode: 2)")
Any help would be greatly appreciated. I am also searching on stackoverflow for help to dump the scv files into differnt tables of a database. Any ideas on that?
You probably need the load data local syntax to make sure the data is being read relative to the client and not the server. Change
query = """ LOAD DATA INFILE 'c:\\example.csv' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES """
to
query = """ LOAD DATA LOCAL INFILE 'c:\\example.csv' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES """
Watch your spelling! conenction.commit() should be connection.commit()
I wanted a script that iterates through csv files in a folder and dump them into a MySQL database. I was able to dump one csv file into it.. But have troubles passing the file name in to the SQL script.
This is the code I use
file_path="C:\csv-files"
files=os.listdir(file_path)
files.sort()
for n in files:
cursor.execute(" LOAD DATA LOCAL INFILE '%s' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES ",(n))
And I get the following error
raise errorclass, errorvalue
ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'file1.csv'' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY' at line 1")
If I use the file name directly instead of passing it, it works fine.
If you can see in the error thrown out, there seems to be an error in the SQL Script.
This would be the whole code
import csv
import MySQLdb
import sys
import os
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db')
cursor = connection.cursor()
file_path="C:\csv-files"
files=os.listdir(file_path)
files.sort()
for n in files:
print n
cursor.execute(" LOAD DATA LOCAL INFILE %s INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES " %n)
connection.commit()
cursor.close()
First, replace '%s' with %s in the query. MySQLdb handles any quoting automatically.
Here's the code with some corrections and changes:
import MySQLdb
import os
CSV_DIR = "C:\csv-files"
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db',
local_infile=1)
cursor = connection.cursor()
try:
for filename in sorted(os.listdir(CSV_DIR)):
cursor.execute("""LOAD DATA LOCAL INFILE %s
INTO TABLE new_table
FIELDS
TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES""",
(os.path.join(CSV_DIR, filename),))
connection.commit()
finally:
cursor.close()
NOTE: I set local_infile parameter to 1 in MySQLdb.connect and pass filename in tuple to execute.
Works for me.
I am trying to import a large text file into a MySQL database. The SQL statement is as follows:
LOAD DATA INFILE '/tmp/epf/full/album_popularity_per_genre'
INTO TABLE album_popularity_per_genre
CHARACTER SET UTF8 FIELDS TERMINATED BY X'01' LINES TERMINATED BY '\n'
IGNORE 45 LINES (export_date, storefront_id, genre_id, album_id, album_rank)
The above works when I run it in phpMyAdmin, however when I write a simple function in Python that uses the above SQL statement I get an error.
Here is the Python code,
def test():
dbConnection = MySQLdb.connect(
charset='utf8',
host='localhost',
user='root',
passwd='root',
db='epf')
cursor = dbConnection.cursor()
exStr = """LOAD DATA INFILE '/tmp/epf/full/album_popularity_per_genre'
INTO TABLE album_popularity_per_genre CHARACTER SET UTF8
FIELDS TERMINATED BY X'01' LINES TERMINATED BY '\n'
IGNORE 45 LINES
(export_date, storefront_id, genre_id, album_id, album_rank)"""
try:
cursor.execute(exStr)
except MySQLdb.Warning, e:
print "Warning %s" % (str(e))
except MySQLdb.IntegrityError, e:
print "Error %d: %s" % (e.args[0], e.args[1])
#Clean up
cursor.close()
dbConnection.close()
The error I get is as follows,
Warning Data truncated for column 'album_rank' at row 1
My question now is, why does the raw SQL statement work but when I try to run the Python code, no data is imported into the database?
The Python DBAPI is implicitly transactional. Try adding dbConnection.commit() after the execute.