How to Insert csv files to database using python_ columns - python

This is my code
import pymysql
import csv
conn=pymysql.connect("localhost", "root","****","finance")
cursor=conn.cursor()
print "done"
csv_data = csv.reader(file('naver_loan_output.csv'))
for row in csv_data:
cursor.execute('INSERT INTO 'daily_new' (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond )' 'VALUES("%s", "%s", "%s", "%s", "%s", "%s")',row)
conn.commit()
cursor.close()
print "Done"
And this is the error:
File "D:\dropbox\Dropbox\lab\7_0218\insert_daily_new.py", line 13
cursor.execute('INSERT INTO 'daily_new' (date,cust_bal, cust_credit,
fund_stock, fund_hyb, fund_bond )' 'VALUES("%s", "%s", "%s", "%s",
"%s", "%s")',row)
^ SyntaxError: invalid syntax [Finished in 0.104s]
I tried a lot, but I'm not sure about the proper SQL insert query syntax. How do I get columns from csv? There are 6 columns in my csv file.
With this updated example, code works:
import pymysql
import csv
csv_data= csv.reader(file('naver_loan_output.csv'))
conn=pymysql.connect("localhost","finance")
cursor=conn.cursor()
print "Done"
for row in csv_data:
#cursor.execute('INSERT INTO \'daily_new\' (date, cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond ) VALUES({}, {}, {}, {}, {}, {})'.format(row[0], row[1], row[2], row[3], row[4], row[5]))
sql="INSERT INTO daily_n (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond ) VALUES('2017-01-01','2','2','2','2','2')"
cursor.execute(sql)
conn.commit()
cursor.close()
So, I think the for row or %s is the problem.


Mainly, your quotes is the issue.
You need to escape single quotes if larger sql string is wrapped in single quotes; or simply wrap larger string in double quotes like your updated example. And note: the SyntaxError is a Python compile error (not MySQL runtime exception).
For parameterized queries, do not quote the placeholder, %s.
MySQL (and practically for all RDMS's) do not use single quotes to enclose table and column name references as you do with 'daily_new'. Use backticks for names in MySQL. A few databases and the ANSI standard allows the double quote for object identifiers (not string literals).
Consider the following adjustment with a more efficient read process of csv file using with() as opposed to all at once as you have it with file(). And as shown with parameterization, you can prepare the sql statement once and then just bind values iteratively in loop.
strSQL = "INSERT INTO `daily_new` (`date`, cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond )" + \
" VALUES(%s, %s, %s, %s, %s, %s)"
with open('naver_loan_output.csv', 'r') as f:
csv_data = csv.reader(f)
for row in csv_data:
cursor.execute(strSQL, row)
conn.commit()
cursor.close()
conn.close()
print "Done"


I think that you can try something like this:
query = """'INSERT INTO 'daily_new' (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond )' 'VALUES("""+row+""")'"""
cursor.execute(query)
Just see this code fragment from CSV File Reading and Writing doc:
>>> import csv
>>> with open('eggs.csv', 'rb') as csvfile:
... spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
... for row in spamreader:
... print ', '.join(row)
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam
I hope it is useful to you or that puts you in the way.


There are three problems tripping you up:
String escape characters
In your code is that the single quote before daily_new is stopping the string and python is interpreting daily_new as a variable, which is not defined.
To solve this you should use the escape character "\" directly before each single quote you want in the string like this:
cursor.execute('INSERT INTO \'daily_new\' (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond ) VALUES("%s", "%s", "%s", "%s", "%s", "%s")',row)
Column access
The csv module returns rows as a list. To access an element in a list (or row in your case), use bracket notation. For row[0] accesses the first element (column) in a row, and row[5] accesses the 6th column.
String substitution
The third problem you are facing is how to pass the values into the string substitution correctly. While there are many ways to do this, an easy one is the format() method. For example: if I wanted to build a string that says "Count is 1", I could run "Count is {}".format(1).
In your case you want to add 6 values to the string, so you add a pair of {} wherever you want a value substituted into a string and add another parameter to the format() function.
Putting it all together
So, to correct your loop code you would want something like this:
csv_data = csv.reader(file('naver_loan_output.csv'))
for row in csv_data:
cursor.execute('INSERT INTO daily_new (date, cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond ) VALUES ({}, {}, {}, {}, {}, {})'.format(row[0], row[1], row[2], row[3], row[4], row[5]))

Related

How to fix Python putting wrong value in MySQL database

I am trying to add the value to MySQL database using Python, but for some reason it puts different number than it should.
I tried many syntax and look for help online but no luck. I am using mysql.connector, Python 3.7.
I expected the output:
(12, 'TEST_JOB', 'SOME_TESTING', '20/7/2019')
but I got:
(12, 'TEST_JOB', 'SOME_TESTING', '0.001415127715205547')
Code:
TODAY = date.today().strftime("%d/%m/%Y")
def add_row_to_database(row = ''):
if row == '': return "This is not gonna happened, wrong row"
query = f"INSERT INTO my_table (col1, col2, col3) VALUES (%s, %s, {TODAY})"
cursor.execute(query, row)
connection.commit()
add_row_to_database('TEST_JOB', 'SOME_TESTING')

Answer from #furas solved the problem.
you should use ' ' like in '12345' - '{TODAY}'. If you don't use ' ' then date 20/7/2019 is treated by SQL as division which gives result 0.001415127715205547

remove quotes from csv file data in python

I have a csv that is being imported from url and placed into a database, however it imports with quotes around the names and id like to remove them. The originial format of the csv file is
"Apple Inc.",113.08,113.07
"Alphabet Inc.",777.61,777.30
"Microsoft Corporation",57.730,57.720
the code I currently have is as follows.
def csv_new(conn, cursor, filename):
with open(filename, 'rt') as csv_file:
csv_data = csv.reader(csv_file)
for row in csv_data:
if(not row[0][0].isdigit()):
continue
split = [int(x) for x in row[0].split('/')]
row[0] = datetime.datetime(split[2], split[0],
split[1]).date().isoformat()
print(row);
cursor.execute('INSERT INTO `trade_data`.`import_data`'
'(date, name, price) VALUES(%s, "%s", %s)',
row)
conn.commit()
final database looks like this
Name | Price1| Price 2|
'Apple Inc.' 113.08 113.07
'Alphabet Inc.' 777.61 777.30
'Microsoft Corporation' 57.730 57.720
and I would like it to look like
Name | Price1| Price 2|
Apple Inc. 113.08 113.07
Alphabet Inc. 777.61 777.30
Microsoft Corporation 57.730 57.720
I tried using for row in csv.reader(new_data.splitlines(), delimiter=', skipinitialspace=True): but it threw errors

csv.reader removes the quotes properly. You may be viewing a quoted string representation of the text instead of the actual text.
>>> new_data = '''"Apple Inc.",113.08,113.07
... "Alphabet Inc.",777.61,777.30
... "Microsoft Corporation",57.730,57.720'''
>>>
>>> import csv
>>>
>>> for row in csv.reader(new_data.splitlines()):
... print(','.join(row))
...
Apple Inc.,113.08,113.07
Alphabet Inc.,777.61,777.30
Microsoft Corporation,57.730,57.720
>>>

Figured it out, the problem was as tdelaney mentioned was that the quotes were not acually in the string it was python, so my changing value in
cursor.execute('INSERT INTO `trade_data`.`import_data`'
'(date, name, price) VALUES(%s, "%s", %s)',
row)
to %s instead of "%s" it fixed the problem and removed the extra quotes.

psycopg, double and single quotes insert

I ran into problems, while trying to insert to database:
ur_psql.execute("""insert into smth(data, filedate, filedby)"""
""" values('%s', NOW(), %s)""" % (data, userid))
where data=""" "5.09,448,1418112000"; d="scan'208" """(a string containing double and single quotes)
Any ideas how to insert such string to db ?
Thanks

You can read about it at: http://initd.org/psycopg/docs/usage.html#the-problem-with-the-query-parameters
Simply do not use quotes in SQL and instead of % string Python operator use 2nd parameter of execute() which is data you want to pass to SQL query:
sql = "insert into smth (data, filedate, filedby) values (%s, NOW(), %s)"
ur_psql.execute(sql, (data, userid))

Using a Python dict for a SQL INSERT statement

I am trying to use a dict to do a SQL INSERT. The logic would basically be:
INSERT INTO table (dict.keys()) VALUES dict.values()
However, I am having a tough time figuring out the correct syntax / flow to do this. This is what I currently have:
# data = {...}
sorted_column_headers_list = []
sorted_column_values_list = []
for k, v in data.items():
sorted_column_headers_list.append(k)
sorted_column_values_list.append(v)
sorted_column_headers_string = ', '.join(sorted_column_headers_list)
sorted_column_values_string = ', '.join(sorted_column_values_list)
cursor.execute("""INSERT INTO title (%s)
VALUES (%s)""",
(sorted_column_headers_string, sorted_column_values_string))
From this I get a SQL exception (I think related to the fact that commas are also included in some of the values that I have). What would be the correct way to do the above?

I think the comment on using this with MySQL is not quite complete. MySQLdb doesn't do parameter substitution in the columns, just the values (IIUC) - so maybe more like
placeholders = ', '.join(['%s'] * len(myDict))
columns = ', '.join(myDict.keys())
sql = "INSERT INTO %s ( %s ) VALUES ( %s )" % (table, columns, placeholders)
# valid in Python 2
cursor.execute(sql, myDict.values())
# valid in Python 3
cursor.execute(sql, list(myDict.values()))
You're not getting escaping on the columns though, so you might want to check them first....
See http://mail.python.org/pipermail/tutor/2010-December/080701.html for a more complete solution

You want to add parameter placeholders to the query. This might get you what you need:
qmarks = ', '.join('?' * len(myDict))
qry = "Insert Into Table (%s) Values (%s)" % (qmarks, qmarks)
cursor.execute(qry, myDict.keys() + myDict.values())

Always good answers here, but in Python 3, you should write the following:
placeholder = ", ".join(["%s"] * len(dict))
stmt = "insert into `{table}` ({columns}) values ({values});".format(table=table_name, columns=",".join(dict.keys()), values=placeholder)
cur.execute(stmt, list(dict.values()))
Don't forget to convert dict.values() to a list because in Python 3, dict.values() returns a view, not a list.
Also, do NOT pour the dict.values() in stmt because it tears a quote out of a string by joining it, which caused MySQL error in inserting it. So you should always put it in cur.execute() dynamically.

I'm a little late to the party but there is another way that I tend to prefer since my data is usually in the form of a dict already. If you list the bind variables in the form of %(columnName)s you can use a dictionary to bind them at execute. This partially solves the problem of column ordering since the variables are bound in by name. I say partially because you still have to make sure that the columns & values portion of the insert are mapped correctly; but the dictionary itself can be in any order (since dicts are sort of unordered anyway)
There is probably a more pythonic way to achieve all this, but pulling the column names into a list and working off it ensures we have a static ordering to build the columns & values clauses.
data_dict = {'col1': 'value 1', 'col2': 'value 2', 'col3': 'value 3'}
columns = data_dict.keys()
cols_comma_separated = ', '.join(columns)
binds_comma_separated = ', '.join(['%(' + item + ')s' for item in columns])
sql = f'INSERT INTO yourtable ({cols_comma_separated}) VALUES ({binds_comma_separated})'
cur.execute(sql, data_dict)
Now whether or not it is a good idea to dynamically build your columns & values clause like this is a topic for a SQL injection thread.

table='mytable'
columns_string= '('+','.join(myDict.keys())+')'
values_string = '('+','.join(map(str,myDict.values()))+')'
sql = """INSERT INTO %s %s
VALUES %s"""%(table, columns_string,values_string)

I tried #furicle's solution but it still inputs everything as a string - if your dict is a mixed one then this may not work as you would want it to. I had a similar issue and this is what I came up with - this is only a query builder and you could use it (with changes) to work with any database of your choice. Have a look!
def ins_query_maker(tablename, rowdict):
keys = tuple(rowdict)
dictsize = len(rowdict)
sql = ''
for i in range(dictsize) :
if(type(rowdict[keys[i]]).__name__ == 'str'):
sql += '\'' + str(rowdict[keys[i]]) + '\''
else:
sql += str(rowdict[keys[i]])
if(i< dictsize-1):
sql += ', '
query = "insert into " + str(tablename) + " " + str(keys) + " values (" + sql + ")"
print(query) # for demo purposes we do this
return(query) #in real code we do this
This is crude and still needs sanity checks, etc, but it works as intended.
for a dict:
tab = {'idnumber': 1, 'fname': 'some', 'lname': 'dude', 'dob': '15/08/1947', 'mobile': 5550000914, 'age' : 70.4}
running the query I get the following output
results of query generated by the suite

This code worked for me (Python 3):
fields = (str(list(dictionary.keys()))[1:-1])
values = (str(list(dictionary.values()))[1:-1])
sql = 'INSERT INTO Table (' + fields + ') VALUES (' + values + ')'
cursor.execute(sql)
It does rely on the dictionary outputting its keys and values in the same order. I'm unclear if this is always true :)

When constructing queries dynamically it's important to ensure that both identifiers and values are correctly quoted. Otherwise you risk
SQL injection if untrusted data is processed
Errors if the column names require quoting (for example embedded spaces)
Data corruption or errors if values are incorrectly quoted (for example 2021-07-11 unquoted may be evaluated as 2003)
Quoting values is best delegated to the DB-API connector. However connector packages don't always provide a way to quote identifiers, so you may need to do this manually. MySQL uses backticks (`) to quote identifiers.
This code quotes identifiers and values. It works for MySQLdb, mysql.connector and pymysql and works for Python 3.5+.
data = {'col1': val1, 'col2': val2, ...}
# Compose a string of quoted column names
cols = ','.join([f'`{k}`' for k in data.keys()])
# Compose a string of placeholders for values
vals = ','.join(['%s'] * len(data))
# Create the SQL statement
stmt = f'INSERT INTO `tbl` ({cols}) VALUES ({vals})'
# Execute the statement, delegating the quoting of values to the connector
cur.execute(stmt, tuple(data.values()))

This is based on other answers here, but it uses back ticks around column names for cases in which you are using reserved words as column names and it it ensures that column names only contain letters, numbers, and underscores to thwart SQL injection attacks.
I've also written a similar upsert that works the same way as the insert but which overwrites data that duplicates the primary key.
import mysql.connector
import re
cnx = mysql.connector.connect(...)
def checkColumnNames(data):
for name in data.keys():
assert re.match(r'^[a-zA-Z0-9_]+$',name), "Bad column name: " + name
def insert(table, data):
checkColumnNames(data)
assert table, "No table specified"
placeholders = ', '.join(['%s'] * len(data))
columns = '`,`'.join(data.keys())
sql = "INSERT INTO `%s` (`%s`) VALUES (%s);" % (table, columns, placeholders)
cnx.cursor().execute(sql, list(data.values()))
def upsert(table, data):
checkColumnNames(data)
assert table, "No table specified"
placeholders = ', '.join(['%s'] * len(data))
columns = '`,`'.join(data.keys())
updates = '`' + '`=%s,`'.join(data.keys()) + '`=%s'
sql = "INSERT INTO `%s` (`%s`) VALUES (%s) ON DUPLICATE KEY UPDATE %s" % (table, columns, placeholders, updates)
cnx.cursor().execute(sql, list(data.values()) + list(data.values()))
Example usage
insert("animals", {
"id": 1,
"name": "Bob",
"type": "Alligator"
})
cnx.commit()

I used this thread for my usage and tried to keep it much simpler
ins_qry = "INSERT INTO {tablename} ({columns}) VALUES {values};" .format(
tablename=my_tablename,
columns=', '.join(myDict.keys()),
values=tuple(myDict.values())
)
cursor.execute(ins_qry)
Make sure to commit the data inserted, either using db_connection.commit() and use cursor.lastrowid, if you need the primary key of the inserted row

This works for me
cursor.execute("INSERT INTO table (col) VALUES ( %(col_value) )",
{'col_value': 123})

if you have list in which there are number of dictionaries
for example: lst=[d1,d2,d3,d4]
then below one will worked for me:
for i in lst:
placeholders = ', '.join(['%s'] * len(i))
columns = ', '.join(i.keys())
sql = "INSERT INTO %s ( %s ) VALUES ( %s )" % (table, columns, placeholders)
cursor.execute(sql,list(i.values()))
conn.commit()
Note:Dont ever forget to commit otherwise you wont be able to see columns and values inserted in table

columns = ', '.join(str(x).replace('/', '_') for x in row_dict.keys())
values = ', '.join("'" + str(x).replace('/', '_') + "'" for x in row_dict.values())
sql = "INSERT INTO %s ( %s ) VALUES ( %s );" % ("tablename", columns, values)
applicable for python3

Let's say our data is:
data = {
"name" : "fani",
"surname": "dogru",
"number" : 271990
}
This is my shorter version:
tablo = "table_name"
cols = ','.join([f" {k}" for k in data.keys()])
vals = ','.join([f"'{k}'" for k in data.values()])
stmt = f'INSERT INTO {tablo} ({cols}) VALUES ({vals})'

What about:
keys = str(dict.keys())
keys.replace('[', '(')
keys.replace(']', ')')
keys.replace("'",'')
vals = str(dict.values())
vals.replace('[', '(')
vals.replace(']', ')')
cur.execute('INSERT INTO table %s VALUES %s' % (keys, vals))
For python 3:
keys = str(dict.keys())[9:].replace('[', '').replace(']', '')
vals = str(dict.values())[11:].replace('[', '').replace(']', '')
...

get formated output from mysql via python

Hi I want the following output from my query:
OK|Abortedclients=119063 Aborted_connects=67591 Binlog_cache_disk_use=0
But I dont know how to generate it. this is my script:
#!/usr/bin/env python
import MySQLdb
conn = MySQLdb.connect (host = "...", user="...", passwd="...")
cursor = conn.cursor ()
cursor.execute ("SHOW GLOBAL STATUS")
rs = cursor.fetchall ()
#print rs
print "OK|"
for row in rs:
print "%s=%s" % (row[0], row[1])
cursor.close()
this is what I get now:
OK|
Aborted_clients=119063
Aborted_connects=67591
Binlog_cache_disk_use=0

You can print the rows together in one string:
output = []
for row in rs:
output.append('%s=%s' % (row[0], row[1])
print ''.join(output)

Build the string using join:
print('OK|'+' '.join(['{0}={1}'.format(*row) for row in rs]))
' '.join(iterable) creates a string out of the strings in iterable, joined together with a space ' ' in between the strings.
To fix the code you posted with minimal changes, you could add a comma at the end of the print statements:
print "OK|",
for row in rs:
print "%s=%s" % (row[0], row[1]),
This suppresses the automatic addition of a newline after each print statement.
It does, however, add a space (which is not what you said you wanted):
OK| Aborted_clients=0 ...

Join each pair with '=' and then each result with ' ', appended to 'OK|':
'OK|' + (' '.join(['='.join(r) for r in rs]))

Categories

Resources