I wanted a script that iterates through csv files in a folder and dump them into a MySQL database. I was able to dump one csv file into it.. But have troubles passing the file name in to the SQL script.
This is the code I use
file_path="C:\csv-files"
files=os.listdir(file_path)
files.sort()
for n in files:
cursor.execute(" LOAD DATA LOCAL INFILE '%s' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES ",(n))
And I get the following error
raise errorclass, errorvalue
ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'file1.csv'' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY' at line 1")
If I use the file name directly instead of passing it, it works fine.
If you can see in the error thrown out, there seems to be an error in the SQL Script.
This would be the whole code
import csv
import MySQLdb
import sys
import os
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db')
cursor = connection.cursor()
file_path="C:\csv-files"
files=os.listdir(file_path)
files.sort()
for n in files:
print n
cursor.execute(" LOAD DATA LOCAL INFILE %s INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES " %n)
connection.commit()
cursor.close()
First, replace '%s' with %s in the query. MySQLdb handles any quoting automatically.
Here's the code with some corrections and changes:
import MySQLdb
import os
CSV_DIR = "C:\csv-files"
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db',
local_infile=1)
cursor = connection.cursor()
try:
for filename in sorted(os.listdir(CSV_DIR)):
cursor.execute("""LOAD DATA LOCAL INFILE %s
INTO TABLE new_table
FIELDS
TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES""",
(os.path.join(CSV_DIR, filename),))
connection.commit()
finally:
cursor.close()
NOTE: I set local_infile parameter to 1 in MySQLdb.connect and pass filename in tuple to execute.
Works for me.
Related
I'm writing code using Python to connect Postgresql.
When I try to copy the CSV file to SQL table, the syntax is '\copy TABLE_NAME from PATH_OF_FILE delimiter ',' csv header;
the '\' make the error when I execute the sql query.
Because it will become '\\copy'
Is there any way I can get rid of this '\\' ??
def copy_file(file_name, table, path):
copy_csv = '\copy ' + table +''' from '''+ "'" + path + "/" + file_name + "'" + ''' delimiter ',' csv header;'''
try:
cursor = connection.cursor()
cursor.execute(copy_csv)
connection.commit()
answer = "Data successfully loaded in PostgreSQL"
except (Exception, psycopg2.DatabaseError) as error :
answer = "Error while Inserting Data into PostgreSQL table: " + str(error)
return answer
I expect the output is \copy table from path....,
but the actual output is \\copy table from path...
Why you need the \ at the beginning? Try this:
import os
def copy_file(file_name, table, path):
try:
cursor = connection.cursor()
cursor.execute(
f"copy {table} from %(filepath}s delimiter ',' csv header",
{"filepath": os.path.join(path, file_name)}
)
connection.commit()
answer = "Data successfully loaded in PostgreSQL"
except (Exception, psycopg2.DatabaseError) as error :
answer = "Error while Inserting Data into PostgreSQL table: " + str(error)
return answer
This uses:
f-Strings to build the string (This is support since python3.6)
the os module to join path name and filename
named parameter in the query
EDIT:
Instead of f-Strings it its more save to use SQL string composition.
import os
from psycopg2 import sql
def copy_file(file_name, table, path):
try:
cursor = connection.cursor()
cursor.execute(
sql.SQL("copy {} from %(filepath}s delimiter ',' csv header").format(sql.Identifier(table)),
{"filepath": os.path.join(path, file_name)}
)
connection.commit()
answer = "Data successfully loaded in PostgreSQL"
except (Exception, psycopg2.DatabaseError) as error :
answer = "Error while Inserting Data into PostgreSQL table: " + str(error)
return answer
I am using following command to load multiple .csv files into Mysql database but i am getting no errors on (the IDLE window) and the data does not load
Here is the erroneous script
#!C:\Python27\python.exe
import MySQLdb
import os
import string
# Open database connection
db = MySQLdb.connect (host="localhost",port=3307,user="root",\
passwd="gamma123",db="test")
cursor=db.cursor()
l = os.listdir(".")
for file_name in l:
print file_name
cursor=db.cursor()
if (file_name.find("DIV.csv")>-1):
#Query under testing
sql = """LOAD DATA LOCAL INFILE file_name \
INTO TABLE system_work \
FIELDS TERMINATED BY ',' \
OPTIONALLY ENCLOSED BY '"' \
LINES TERMINATED BY '\r\n' \
IGNORE 1 LINES;;"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
But when i try to load a single file using the following python script then its works fine.
please help....
#!C:\Python27\python.exe
import MySQLdb
import os
import string
# Open database connection
db = MySQLdb.connect (host="localhost",port=3307,user="root",\
passwd="gamma123",db="test")
cursor=db.cursor()
#Query under testing
sql = """LOAD DATA LOCAL INFILE 'Axle.csv' \
INTO TABLE system_work \
FIELDS TERMINATED BY ',' \
OPTIONALLY ENCLOSED BY '"' \
LINES TERMINATED BY '\r\n' \
IGNORE 1 LINES;;"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
You need to interpolate the filename into the SQL string; you are just sending the literal text file_name to the server. You could use the str.format() method for that, any {} placeholder can then be replaced by a variable of your choosing.
You also must indent the try and except blocks to be within the for loop:
sql = """LOAD DATA LOCAL INFILE '{}'
INTO TABLE system_work
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\\r\\n'
IGNORE 1 LINES;;"""
for file_name in l:
print file_name
if file_name.endswith('DIV.csv'):
try:
cursor = db.cursor()
cursor.execute(sql.format(file_name))
db.commit()
except Exception:
# Rollback in case there is any error
db.rollback()
The cursor.execute() method is passed the sql string with the file_name variable interpolated. The {} part on the first line (LOAD DATA LOCAL INFILE '{}') will be replaced by the value in file_name before passing the SQL statement to MySQL.
I also simplified the filename test; presumably it is enough if the filename ends with DIV.csv.
Note that it might just be easier to use the mysqlimport utility; you can achieve the exact same results with:
mysqlimport --fields-terminated-by=, --fields-optionally-enclosed-by=\" \
--local --lines-terminated-by=\r\n --user=root --password=gamma123 \
test *DIV.csv
if (file_name.find("DIV.csv")>-1): unless all of your files are actually called DIV.csv should that be if (file_name.find(".csv")>-1): (that would probably be more efficient testing the last four letters of the file name by the way)
I am having trouble getting a variable path into a MySQL query in a python script. The path variable is either resolved with double backslashes or none at all.
This works:
cursor.execute ("""load data local infile 'M:/Users/Jonathan/Dropbox/BCHS_3015/Spatial Data/Cartographic Data/USA/acs_data/Sequence_Number_and_Table_Number_Lookup.csv'
into table Sequence_Table_Lookup
fields terminated by ','enclosed by '"'
lines terminated by '\r\n'
ignore 1 lines
(File_ID,Table_ID,Sequence_Number,Line_Number, Subject_Area)""");
This following returns the error:
_mysql_exceptions.InternalError: (22, "File 'M:UsersJonathanDropbox\x08chs_3015spatial datacartographic datausaacs_dataSequence_Number_and_Table_Number_Lookup.txt' not found (Errcode: 22)")
cursor.execute ("""load data local infile '%s'
into table Sequence_Table_Lookup
fields terminated by ','enclosed by '"'
lines terminated by '\r\n'
ignore 1 lines
(File_ID,Table_ID,Sequence_Number,Line_Number, Subject_Area)""" % filepath);
Removing the single quotes around %s yields
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax;
check the manual that corresponds to your MySQL server version for the right
syntax to use near 'M:\\Users\\Jonathan\\Dropbox\\bchs_3015\\spatial data\\cartographic data\\usa\\acs_data\\' at line 1")
I would appreciate any help in understanding how to insert a variable path into a MySQL query.
I am using PyDev in Eclipse on a windows machine. Python 2.7 and MySQLdb connector.
The full block of relevant code
conn = MySQLdb.connect (host = "localhost",
user = "user",
passwd = "pwd",
db = "gis_census_acs")
#finds census directory
dropbox = navigation.get_dropbox_home()
acs_data = os.path.join(dropbox,'bchs_3015','spatial data','cartographic data','usa','acs_data');
for filepath in navigation.get_filepaths(acs_data):
filename = os.path.split(filepath)[1]
if filename == 'Sequence_Number_and_Table_Number_Lookup.txt':
print filepath;
tablename = filename.split('.')[0].replace(' ','_')[0:64]
cursor = conn.cursor()
cursor.execute ('create table if not exists ' + tablename + """(
File_ID varchar(255),
Table_ID varchar(255),
Sequence_Number varchar(255),
Line_Number varchar(255),
Start_Position varchar(255),
Total_cells_in_Table varchar(255),
Total_Cells_in_Sequence varchar(255),
Table_title text,
Subject_Area text
)""");
cursor.execute ("""load data local infile '%s'
into table Sequence_Table_Lookup
fields terminated by ','enclosed by '"'
lines terminated by '\r\n'
ignore 1 lines
(File_ID,Table_ID,Sequence_Number,Line_Number, Start_Position,
Total_cells_in_Table, Total_Cells_in_Sequence, Table_title, Subject_Area)""" % filepath);
print "Number of rows inserted: %d" % cursor.rowcount
cursor.close()
else:
print "not the file"
conn.close ()
This file exists:
M:/Users/Jonathan/Dropbox/BCHS_3015/Spatial Data/Cartographic Data/USA/acs_data/Sequence_Number_and_Table_Number_Lookup.csv
As you expect, this strange one doesnt:
M:UsersJonathanDropbox\x08chs_3015spatial datacartographic datausaacs_dataSequence_Number_and_Table_Number_Lookup.txt
Seems like there's something wrong with your filepath. Try checking it out.
I have a SQL query which opens up a csv file and dumps it in a table of a database. I am trying to dump multiple files at once using a python script to iterate among the files. I tried embedding the same SQL query inside the script, but it throws out an error.
This is the script I have.
import csv
import MySQLdb
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db')
cursor = connection.cursor()
query = """ LOAD DATA INFILE 'c:\\example.csv' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES """
cursor.execute(query)
conenction.commit()
cursor.close()
And for some reason the python script looks up example.csv at a different location
This is the error that is thrown out :
raise errorclass, errorvalue
InternalError: (29, "File 'C:\\Documents and Settings\\All Users\\Application Data\\MySQL\\MySQL Server 5.5\\data\\example.csv' not found (Errcode: 2)")
Any help would be greatly appreciated. I am also searching on stackoverflow for help to dump the scv files into differnt tables of a database. Any ideas on that?
You probably need the load data local syntax to make sure the data is being read relative to the client and not the server. Change
query = """ LOAD DATA INFILE 'c:\\example.csv' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES """
to
query = """ LOAD DATA LOCAL INFILE 'c:\\example.csv' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES """
Watch your spelling! conenction.commit() should be connection.commit()
I am trying to import a large text file into a MySQL database. The SQL statement is as follows:
LOAD DATA INFILE '/tmp/epf/full/album_popularity_per_genre'
INTO TABLE album_popularity_per_genre
CHARACTER SET UTF8 FIELDS TERMINATED BY X'01' LINES TERMINATED BY '\n'
IGNORE 45 LINES (export_date, storefront_id, genre_id, album_id, album_rank)
The above works when I run it in phpMyAdmin, however when I write a simple function in Python that uses the above SQL statement I get an error.
Here is the Python code,
def test():
dbConnection = MySQLdb.connect(
charset='utf8',
host='localhost',
user='root',
passwd='root',
db='epf')
cursor = dbConnection.cursor()
exStr = """LOAD DATA INFILE '/tmp/epf/full/album_popularity_per_genre'
INTO TABLE album_popularity_per_genre CHARACTER SET UTF8
FIELDS TERMINATED BY X'01' LINES TERMINATED BY '\n'
IGNORE 45 LINES
(export_date, storefront_id, genre_id, album_id, album_rank)"""
try:
cursor.execute(exStr)
except MySQLdb.Warning, e:
print "Warning %s" % (str(e))
except MySQLdb.IntegrityError, e:
print "Error %d: %s" % (e.args[0], e.args[1])
#Clean up
cursor.close()
dbConnection.close()
The error I get is as follows,
Warning Data truncated for column 'album_rank' at row 1
My question now is, why does the raw SQL statement work but when I try to run the Python code, no data is imported into the database?
The Python DBAPI is implicitly transactional. Try adding dbConnection.commit() after the execute.