import MySQLdb
db = MySQLdb.connect(host="?",
user="root",
passwd="?",
db="test")
cursor = db.cursor()
file = open('...../EM.txt', 'r')
file_content = file.read()
file.close()
query = "INSERT INTO EM VALUES (%s,%s,%s,%s,%s,%s)"
cursor.execute(query, (file_content,))
db.commit()
db.close()
I have try this code to read from a text file and insert into EM table ...can any one help me make this work?
I have no idea how your text file is formatted, but file.read() gives you the whole file as a single string and it seems like you have six fields to fill. So maybe the file consists of 6 tab- or space-separated fields?
First, split the file into lines with file.readlines() instead of file.read(). Next, build a list of rows that you can feed into executemany:
values = [line.split() for line in file_content]
cursor.executemany(query, values)
The split method splits the lines in the file on whitespace into a tuple, e.g., the string a b c is turned into the tuple ('a', 'b', 'c'), so that the list comprehension produces a list of tuples that can be fed into cursor.executemany to perform a bulk insert.
As #Evan points out, you also have to specify the columns the values are associated to in your SQL query, e.g.,
INSERT INTO EM (field, spam, ham, eggs, price, ni) VALUES (%s, %s, %s, %s, %s, %s)
Are you trying to insert the entire text file into a string column in the database, or do you want to import tabular data into the database? It seems like you want to import tabular data based on the way you specified the values, so I'll use that as my assumption.
To do this, you need to read each row from your file and insert it into the database one row at a time. You also need to update your query syntax to specify the column names you are inserting into.
import csv
import MySQLdb
db = MySQLdb.connect(host='localhost',
user='root',
passwd='?',
db='test')
cursor = mydb.cursor()
with open('...../EM.txt') as f:
reader = csv.reader(f)
for row in reader:
cursor.execute("""INSERT INTO testcsv(col1, col2, col3, col4, col5, col6)
VALUES(%s, %s, %s, %s, %s, %s)
""", row)
#close the connection to the database.
mydb.commit()
cursor.close()
print "Done"
i think your problem is readlines() and values = [line.split() for line in file_content]
To start off in order for python to communicate with your SQl database you have to use an adapter such as psycopg2. Once you have installed psycopg2 you have to import it. Another segment that would also improve the simplicity and functionality of your code, is by using a csv reader. This can be done by simply importing csv. SQL databases are stored as a csv file so using this will help you read csv files while using python.
You can find more about using csv reader here:
https://docs.python.org/2/library/csv.html
In terms of your code try this:
import MySQLdb
import psycopg2
db = MySQLdb.connect(host="?",
user="root",
passwd="?",
db="test")
cursor = db.cursor()
file = open('...../EM.txt', 'r')
file_content = file.read()
file.close()
query = "INSERT INTO EM VALUES (%s,%s,%s,%s,%s,%s)"
cursor.execute(query, (file_content,))
db.commit()
db.close()
By installing and importing psycopg2 will allow python to understand the SQL quires you are trying to execute.
Related
I have a .csv file to be loaded into snowsql table using python API.
My question is how to load one row at a time, so to check if every row is successfully loaded.
Although it's possible, I do not recommend you to do single inserts to Snowflake:
import snowflake.connector
import csv
ctx = snowflake.connector.connect(
...
)
cursor = ctx.cursor()
with open('test.csv') as f:
reader = csv.reader(f)
for row in reader:
cursor.execute("""INSERT INTO table1 (col1, col2, col3 )
VALUES(%s, %s, %s )
""", row)
cursor.close()
You can validate the files before COPY command:
https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#validating-staged-files
And you can also check the errors after COPY command:
https://docs.snowflake.com/en/sql-reference/functions/validate.html
I'm trying to insert some values from a csv file through Python but I'm getting a no viable alternative at input error. When I specify the values instead of %s the code works but when I try to use %s it fails. This is my code:
import jaydebeapi
import jpype
import pyodbc
import pandas as pd
import csv
conn = pyodbc.connect("myconnection")
cursor = conn.cursor()
with open('/Users/user/Desktop/TEST.csv') as f:
reader = csv.reader(f)
for row in reader:
cursor.execute("INSERT INTO mytable (user_id, email) VALUES(%s,%s)", row)
#close the connection to the database.
mydb.commit()
cursor.close()
This is my cs50w project here i'm trying to import books.csv file into the postgresql database but i'm getting some errors, i think i'm having some problem with my script can someone correct it...
import psycopg2
import csv
#For connecting to the database
conn = psycopg2.connect("host=hostname_here port=5432 dbname=dbname_here user=username_here password=pass_here")
cur = conn.cursor()
#importing csv file
with open('books.csv', 'r') as f:
reader = csv.reader(f)
next(reader)
for row in reader:
cur.execute("INSERT INTO book VALUES (%s, %s, %s, %s)",
row
)
conn.commit()
Traceback (most recent call last):
File "import.py", line 15, in <module>
row
psycopg2.errors.SyntaxError: INSERT has more expressions than target columns
LINE 1: INSERT INTO book VALUES ('0380795272', 'Krondor: The Betraya...
sample of csv file :
sample of csv file :
INSERT has more expressions than target columns.
You are trying to insert a row with 4 values in a table that has less than 4 columns.
However, if the table indeed has 4 columns, you need to review your data source (books.cvs.) The source data may have some single quotes or commas. Either remove the problematic data from the file or modify your program to handle the data correctly.
Problem solved in my postgres table i set isbn to integer but i didnt see the alphabets with it now i changed isbn column to varchar and problem is solved
I want to fetch the SQL query from a text file and run it in Python program. This is my code:
csvfilelist=os.listdir(inputPath)
mycursor = mydb.cursor()
for csvfilename in csvfilelist:
with open(inputPath + csvfilename, 'r') as csvFile:
reader = csv.reader(csvFile)
for row in reader:
'''r = "INSERT INTO Terminate.RAW VALUES('%s','%s','%s','%s','%s')" %(row[0],row[1],row[2],row[3],row[4],row[5])'''
try:
result = mycursor.execute(r)
mydb.commit()
except mysql.connector.Error as err:
print(err)
csvFile.close()
Say you have a INI file containing the query
[main]
query=INSERT INTO Terminate.RAW VALUES('%s','%s','%s','%s','%s')
you may load it
config = configparser.ConfigParser()
config.read('myfile.ini')
query = config['main']['query']
and later you can call it with
r = query % (row[0],row[1],row[2],row[3],row[4],row[5])
As pointed out in comments, using "%" in queries is not a good solution, you should bind your variables when executing the query. I don't remember the exact syntax, it's something like
r = query
mycursor.execute(r, (row[0],row[1],row[2],row[3],row[4],row[5]))
Edit: sorry, I just read that your file is JSON, not INI. You wrote that in the title, not in the post. If so, you should use the json module instead of configparser module.
I have a CSV file without headers and am trying to create a SQL table from certain columns in the file. I tried the solutions given here: Importing a CSV file into a sqlite3 database table using Python,
but keep getting the error that col1 is not defined. I then tried inserting headers in my CSV file and am still getting a KeyError.
Any help is appreciated! (I am not very familiar with SQL at all)
If the .csv file has no headers, you don't want to use DictReader; DictReader assumes line 1 is a set of headers and uses them as keys for every subsequent line. This is probably why you're getting KeyErrors.
A modified version of the example from that link:
import csv, sqlite3
con = sqlite3.connect(":memory:")
cur = con.cursor()
cur.execute("CREATE TABLE t (col1, col2);")
with open('data.csv','rb') as fin:
dr = csv.reader(fin)
dicts = ({'col1': line[0], 'col2': line[1]} for line in dr)
to_db = ((i['col1'], i['col2']) for i in dicts)
cur.executemany("INSERT INTO t (col1, col2) VALUES (?, ?);", to_db)
con.commit()
This below code, will read all the csv files from the path and load all the data into table present in sqllite 3 database.
import sqllite3
import io
import os.path
import glob
cnx = sqlite3.connect(user='user', host='localhost', password='password',
database='dbname')
cursor=cnx.cursor(buffered= True);
path ='path/*/csv'
for files in glob.glob(path + "/*.csv"):
add_csv_file="""LOAD DATA LOCAL INFILE '%s' INTO TABLE tabkename FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' IGNORE 1 LINES;;;""" %(files)
print ("add_csv_file: %s" % files)
cursor.execute(add_csv_file)
cnx.commit()
cursor.close();
cnx.close();
Let me know if this works.