I'm writing code using Python to connect Postgresql.
When I try to copy the CSV file to SQL table, the syntax is '\copy TABLE_NAME from PATH_OF_FILE delimiter ',' csv header;
the '\' make the error when I execute the sql query.
Because it will become '\\copy'
Is there any way I can get rid of this '\\' ??
def copy_file(file_name, table, path):
copy_csv = '\copy ' + table +''' from '''+ "'" + path + "/" + file_name + "'" + ''' delimiter ',' csv header;'''
try:
cursor = connection.cursor()
cursor.execute(copy_csv)
connection.commit()
answer = "Data successfully loaded in PostgreSQL"
except (Exception, psycopg2.DatabaseError) as error :
answer = "Error while Inserting Data into PostgreSQL table: " + str(error)
return answer
I expect the output is \copy table from path....,
but the actual output is \\copy table from path...
Why you need the \ at the beginning? Try this:
import os
def copy_file(file_name, table, path):
try:
cursor = connection.cursor()
cursor.execute(
f"copy {table} from %(filepath}s delimiter ',' csv header",
{"filepath": os.path.join(path, file_name)}
)
connection.commit()
answer = "Data successfully loaded in PostgreSQL"
except (Exception, psycopg2.DatabaseError) as error :
answer = "Error while Inserting Data into PostgreSQL table: " + str(error)
return answer
This uses:
f-Strings to build the string (This is support since python3.6)
the os module to join path name and filename
named parameter in the query
EDIT:
Instead of f-Strings it its more save to use SQL string composition.
import os
from psycopg2 import sql
def copy_file(file_name, table, path):
try:
cursor = connection.cursor()
cursor.execute(
sql.SQL("copy {} from %(filepath}s delimiter ',' csv header").format(sql.Identifier(table)),
{"filepath": os.path.join(path, file_name)}
)
connection.commit()
answer = "Data successfully loaded in PostgreSQL"
except (Exception, psycopg2.DatabaseError) as error :
answer = "Error while Inserting Data into PostgreSQL table: " + str(error)
return answer
Related
I have a Python Script that I am running to take data from a CSV and insert it into my MS SQL Server. The CSV is about 35 MB and contains about 200,000 records with 15 columns. It takes the SQL Server Import Tool less than 5 min to insert all the data into the Server. The python script, using pypyodbc takes 30 minutes or longer.
What am I doing wrong with my code that is causing this to take so long
import pypyodbc
import csv
import datetime
now = datetime.datetime.now()
cnxn = pypyodbc.connect('DRIVER={SQL Server};SERVER=;DATABASE=')
cursor = cnxn.cursor()
cursor.execute("""
DELETE FROM DataMaster
""")
cnxn.commit()
FileName = "Data - " + str('{:02d}'.format(now.month)) + "-" + str('{:02d}'.format(now.day-1)) + "-" + str(now.year) + ".csv"
myCSV = open(FileName)
myCSV = csv.reader(myCSV)
next(myCSV, None) # this skips the headers
listlist = list(myCSV)
cursor.executemany('''
INSERT INTO DataMaster (Column1,Column2,Column3,Column4,Column5,Column6,Column7,Column8,Column9,Column10,Column11,Column12,Column13,Column14,Column15)
VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)''',
(
listlist
)
)
cursor.commit() # commits any changes
cnxn.close() # closes the connection
print "Import Completed."
Below code, should work in your case. This should not much time.
import pymysql
import logging
import numpy as np
try:
conn = pymysql.connect(hostname='', user='user', passwd='password', db='dbname', connect_timeout=5)
except Exception as e:
logger.error("ERROR: Unexpected error: Could not connect to MySql instance.")
logger.error(e)
sys.exit()
logger.info("SUCCESS: Connection to SQL Server instance succeeded")
local_cursor = conn.cursor()
path ='path/*/csv'
for files in glob.glob(path + "/*.csv"):
add_csv_file="""LOAD DATA LOCAL INFILE '%s' INTO TABLE tablename FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' IGNORE 1 LINES;;;""" %(files)
cursor.execute(add_csv_file)
cnx.commit()
cursor.close();
cnx.close();
Please select the answer if this works.
I am using following command to load multiple .csv files into Mysql database but i am getting no errors on (the IDLE window) and the data does not load
Here is the erroneous script
#!C:\Python27\python.exe
import MySQLdb
import os
import string
# Open database connection
db = MySQLdb.connect (host="localhost",port=3307,user="root",\
passwd="gamma123",db="test")
cursor=db.cursor()
l = os.listdir(".")
for file_name in l:
print file_name
cursor=db.cursor()
if (file_name.find("DIV.csv")>-1):
#Query under testing
sql = """LOAD DATA LOCAL INFILE file_name \
INTO TABLE system_work \
FIELDS TERMINATED BY ',' \
OPTIONALLY ENCLOSED BY '"' \
LINES TERMINATED BY '\r\n' \
IGNORE 1 LINES;;"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
But when i try to load a single file using the following python script then its works fine.
please help....
#!C:\Python27\python.exe
import MySQLdb
import os
import string
# Open database connection
db = MySQLdb.connect (host="localhost",port=3307,user="root",\
passwd="gamma123",db="test")
cursor=db.cursor()
#Query under testing
sql = """LOAD DATA LOCAL INFILE 'Axle.csv' \
INTO TABLE system_work \
FIELDS TERMINATED BY ',' \
OPTIONALLY ENCLOSED BY '"' \
LINES TERMINATED BY '\r\n' \
IGNORE 1 LINES;;"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
You need to interpolate the filename into the SQL string; you are just sending the literal text file_name to the server. You could use the str.format() method for that, any {} placeholder can then be replaced by a variable of your choosing.
You also must indent the try and except blocks to be within the for loop:
sql = """LOAD DATA LOCAL INFILE '{}'
INTO TABLE system_work
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\\r\\n'
IGNORE 1 LINES;;"""
for file_name in l:
print file_name
if file_name.endswith('DIV.csv'):
try:
cursor = db.cursor()
cursor.execute(sql.format(file_name))
db.commit()
except Exception:
# Rollback in case there is any error
db.rollback()
The cursor.execute() method is passed the sql string with the file_name variable interpolated. The {} part on the first line (LOAD DATA LOCAL INFILE '{}') will be replaced by the value in file_name before passing the SQL statement to MySQL.
I also simplified the filename test; presumably it is enough if the filename ends with DIV.csv.
Note that it might just be easier to use the mysqlimport utility; you can achieve the exact same results with:
mysqlimport --fields-terminated-by=, --fields-optionally-enclosed-by=\" \
--local --lines-terminated-by=\r\n --user=root --password=gamma123 \
test *DIV.csv
if (file_name.find("DIV.csv")>-1): unless all of your files are actually called DIV.csv should that be if (file_name.find(".csv")>-1): (that would probably be more efficient testing the last four letters of the file name by the way)
I'm using the MySQLdb module in Python to create a table, but I've ran into some issues when trying to name it using the datetime module.
class DbPipeline(object):
def __init__(self):
vendor = "vendorname"
curDate = time.strftime('%Y-%m-%d').replace("-", ".")
tableName = vendor + ":" + curDate
# connect to the database
self.conn = MySQLdb.connect(host='127.0.0.1', user='root', passwd='', db='dbname', charset="utf8", use_unicode=True)
self.cursor = self.conn.cursor()
#create a new table
sql = "CREATE TABLE %s (name CHAR(40))" %tableName
self.cursor.execute(sql)
This results in the following error:
" raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ':2013.12.21 (name CHAR(40))' at line 1")
I'm pretty sure it might be something to do with character escaping or the way my sql query is defined, but plenty of Googling and REPL sessions later I've not managed to fix it.
Some code from REPL:
vendor = "vendorname"
curDate = time.strftime('%Y-%m-%d').replace("-", ".")
tableName = vendor + ":" + curDate
sql = "CREATE TABLE %s (name CHAR(40))" %tableName
sql
'CREATE TABLE vendorname:2013.12.21 (name CHAR(40))'
Final little note, this works perfectly if you were to just assign %tableName a-z characters. Thanks for reading and apologies if this is something blindingly obvious I've missed!
According to the MySQL documentation:
Database and table names cannot contain “/”, “\”, “.”, or characters that are not permitted in file names.
Try:
curDate = time.strftime('%Y%m%d')
tableName = "%s_%s" % (vendor, curDate)
For example, "vendorname_20131221"
I am having trouble getting a variable path into a MySQL query in a python script. The path variable is either resolved with double backslashes or none at all.
This works:
cursor.execute ("""load data local infile 'M:/Users/Jonathan/Dropbox/BCHS_3015/Spatial Data/Cartographic Data/USA/acs_data/Sequence_Number_and_Table_Number_Lookup.csv'
into table Sequence_Table_Lookup
fields terminated by ','enclosed by '"'
lines terminated by '\r\n'
ignore 1 lines
(File_ID,Table_ID,Sequence_Number,Line_Number, Subject_Area)""");
This following returns the error:
_mysql_exceptions.InternalError: (22, "File 'M:UsersJonathanDropbox\x08chs_3015spatial datacartographic datausaacs_dataSequence_Number_and_Table_Number_Lookup.txt' not found (Errcode: 22)")
cursor.execute ("""load data local infile '%s'
into table Sequence_Table_Lookup
fields terminated by ','enclosed by '"'
lines terminated by '\r\n'
ignore 1 lines
(File_ID,Table_ID,Sequence_Number,Line_Number, Subject_Area)""" % filepath);
Removing the single quotes around %s yields
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax;
check the manual that corresponds to your MySQL server version for the right
syntax to use near 'M:\\Users\\Jonathan\\Dropbox\\bchs_3015\\spatial data\\cartographic data\\usa\\acs_data\\' at line 1")
I would appreciate any help in understanding how to insert a variable path into a MySQL query.
I am using PyDev in Eclipse on a windows machine. Python 2.7 and MySQLdb connector.
The full block of relevant code
conn = MySQLdb.connect (host = "localhost",
user = "user",
passwd = "pwd",
db = "gis_census_acs")
#finds census directory
dropbox = navigation.get_dropbox_home()
acs_data = os.path.join(dropbox,'bchs_3015','spatial data','cartographic data','usa','acs_data');
for filepath in navigation.get_filepaths(acs_data):
filename = os.path.split(filepath)[1]
if filename == 'Sequence_Number_and_Table_Number_Lookup.txt':
print filepath;
tablename = filename.split('.')[0].replace(' ','_')[0:64]
cursor = conn.cursor()
cursor.execute ('create table if not exists ' + tablename + """(
File_ID varchar(255),
Table_ID varchar(255),
Sequence_Number varchar(255),
Line_Number varchar(255),
Start_Position varchar(255),
Total_cells_in_Table varchar(255),
Total_Cells_in_Sequence varchar(255),
Table_title text,
Subject_Area text
)""");
cursor.execute ("""load data local infile '%s'
into table Sequence_Table_Lookup
fields terminated by ','enclosed by '"'
lines terminated by '\r\n'
ignore 1 lines
(File_ID,Table_ID,Sequence_Number,Line_Number, Start_Position,
Total_cells_in_Table, Total_Cells_in_Sequence, Table_title, Subject_Area)""" % filepath);
print "Number of rows inserted: %d" % cursor.rowcount
cursor.close()
else:
print "not the file"
conn.close ()
This file exists:
M:/Users/Jonathan/Dropbox/BCHS_3015/Spatial Data/Cartographic Data/USA/acs_data/Sequence_Number_and_Table_Number_Lookup.csv
As you expect, this strange one doesnt:
M:UsersJonathanDropbox\x08chs_3015spatial datacartographic datausaacs_dataSequence_Number_and_Table_Number_Lookup.txt
Seems like there's something wrong with your filepath. Try checking it out.
I wanted a script that iterates through csv files in a folder and dump them into a MySQL database. I was able to dump one csv file into it.. But have troubles passing the file name in to the SQL script.
This is the code I use
file_path="C:\csv-files"
files=os.listdir(file_path)
files.sort()
for n in files:
cursor.execute(" LOAD DATA LOCAL INFILE '%s' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES ",(n))
And I get the following error
raise errorclass, errorvalue
ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'file1.csv'' INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY' at line 1")
If I use the file name directly instead of passing it, it works fine.
If you can see in the error thrown out, there seems to be an error in the SQL Script.
This would be the whole code
import csv
import MySQLdb
import sys
import os
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db')
cursor = connection.cursor()
file_path="C:\csv-files"
files=os.listdir(file_path)
files.sort()
for n in files:
print n
cursor.execute(" LOAD DATA LOCAL INFILE %s INTO TABLE new_table FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' Lines terminated by '\n' IGNORE 1 LINES " %n)
connection.commit()
cursor.close()
First, replace '%s' with %s in the query. MySQLdb handles any quoting automatically.
Here's the code with some corrections and changes:
import MySQLdb
import os
CSV_DIR = "C:\csv-files"
connection = MySQLdb.connect(host='localhost',
user='root',
passwd='password',
db='some_db',
local_infile=1)
cursor = connection.cursor()
try:
for filename in sorted(os.listdir(CSV_DIR)):
cursor.execute("""LOAD DATA LOCAL INFILE %s
INTO TABLE new_table
FIELDS
TERMINATED BY ','
OPTIONALLY ENCLOSED BY '"'
ESCAPED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES""",
(os.path.join(CSV_DIR, filename),))
connection.commit()
finally:
cursor.close()
NOTE: I set local_infile parameter to 1 in MySQLdb.connect and pass filename in tuple to execute.
Works for me.