I obtaines a dictionary 'p' from the following code,but cannot able to insert into the mysql database.please help me to insert the datas into database.
dictionary is :[('Casssandraw', 'Cooking'), ('Archanea', 'Playing'), ('Adarshan', 'Programming'), ('Leelal', 'Baking')]
should be stored to Names and Hobby fields.
Name Hobby
Cassandraw Cooking
Archanea Playing
... ...
Program:
import MySQLdb
import re
db = MySQLdb.connect(host="localhost", # your host, usually localhost
user="root", # your username
passwd="mysql", # your password
db="sakila") # n
with open('qwer2.txt','r') as file, db as cursor:
f = open('qwer2.txt', 'r')
lines = f.readlines()
for x in lines:
p=re.findall(r'(?:name is|me)\s+(\w+).*?(?:interest|hobby)\s+is\s+(\w+)',x, re.I)
print p
cursor.execute(
'''INSERT INTO Details (Names, Hobby)
VALUES (%s, %s)''',
(name, hobby))#<-donot know what to provide
db.commit()
It looks like you have a list of tuples containing name/hobby not a dict:
You can unpack the two and insert:
for name, hobby in p: # I am presuming p is the list you posted in your question
cursor.execute(
'''INSERT INTO Details (Names, Hobby)
VALUES (%s, %s)''',
(name, hobby))#<-donot know what to provide
for name,hobby in p:
print name,hobby
Casssandraw Cooking
Archanea Playing
Adarshan Programming
Leelal Baking
Related
Upsert to MySQL using python and data from excel.
Im working on populating a MySQL DB, using python.
The data is stored on excel sheets.
Because the DB is suppossed to be used for monitoring "projects", there's a posibility for repeated pk, so in that case it need to be updated instead of insert, because a project can have many stages.
Also, there's a value to be inserted in the DB table, that can't be added from the spreadsheet. So i'm wondering if in that case, the insert of this value, most be done using a separated query for it or if theres a way to insert it in the same query. The value is the supplier ID and needs to be inserted between id_ops and cif_store.
And to finish, I need to perform an inner join, to import the store_id using the store_cif, from another table called store. I know how do it, but im wondering if it also must be executed from a sepparated query or can be performed at the sameone.
So far, i have done this.
import xlrd
import MySQLdb
def insert():
book = xlrd.open_workbook(r"C:\Users\DevEnviroment\Desktop\OPERACIONES.xlsx")
sheet = book.sheet_by_name("Sheet1")
database = MySQLdb.connect (host="localhost", user = "pytest", passwd = "password", db = "opstest1")
cursor = database.cursor()
query = """INSERT INTO operation (id_ops, cif_store, date, client,
time_resp, id_area_service) VALUES (%s, %s, %s, %s, %s, %s)"""
for r in range(1, sheet.nrows):
id_ops = sheet.cell(r,0).value
cif_store = sheet.cell(r,1).value
date = sheet.cell(r,2).value
client = sheet.cell(r,3).value
time_resp = sheet.cell(r,4).value
id_area_service = sheet.cell(r,5).value
values = (id_ops, cif_store, date, client, time_resp, id_area_service)
cursor.execute(query, values)
# Close the cursor
cursor.close()
# Commit the transaction
database.commit()
# Close the database connection
database.close()
# Print results
print ("")
print ("")
columns = str(sheet.ncols)
rows = str(sheet.nrows)
print ("Imported", columns,"columns and", rows, "rows. All Done!")
insert()
What you are looking for is INSERT ... ON DUPLICATE KEY UPDATE ...
Take a look here https://dev.mysql.com/doc/refman/8.0/en/insert-on-duplicate.html
Regarding the extraneous data, if its a static value for all rows you can just hard code it right into the INSERT query. If it's dynamic you'll have to write some additional logic.
For example:
query = """INSERT INTO operation (id_ops, hard_coded_value, cif_store, date, client,
time_resp, id_area_service) VALUES (%s, "my hard coded value", %s, %s, %s, %s, %s)"""
I have a Python string (or potentially a Python dictionary) that I'd like to insert to MySql table.
My String is the following:
{'ticker': 'BTC', 'avail_supply': 16479075.0, 'prices': 2750.99, 'name': 'Bitcoin', '24hvol': 678995000.0}
I have the same kind of error if I want to insert the Dict format.
I really don't understand this kind of error (i.e. the '\' in-between the components of the string).
How can I deal with this error? Any why to properly insert a whole string to a particular TEXT cell in SQL?
Many thanks !!
Here is how to connect, make a table, and insert in the table.
import MySQLdb as mdb
import sys
#connect
con = mdb.connect('localhost', 'testuser', 'test623', 'testdb');
with con:
#need the cursor object so you can pass sql commands, also there is a dictionary cursor
cur = con.cursor()
#create example table
cur.execute("CREATE TABLE IF NOT EXISTS \
Writers(Id INT PRIMARY KEY AUTO_INCREMENT, Name VARCHAR(25))")
#insert what you want
cur.execute("INSERT INTO Writers(Name) VALUES('Jack London')")
cur.execute("INSERT INTO Writers(Name) VALUES('Honore de Balzac')")
cur.execute("INSERT INTO Writers(Name) VALUES('Lion Feuchtwanger')")
cur.execute("INSERT INTO Writers(Name) VALUES('Emile Zola')")
cur.execute("INSERT INTO Writers(Name) VALUES('Truman Capote')")
Example above will make a table with 2 cols, one ID and one name
look here on an example on how to insert stuff from dictionary with keys and list as value to sql, basically you need place holders
sql = "INSERT INTO mytable (a,b,c) VALUES (%(qwe)s, %(asd)s, %(zxc)s);"
data = {'qwe':1, 'asd':2, 'zxc':None}
conn = MySQLdb.connect(**params)
cursor = conn.cursor()
cursor.execute(sql, data)
cursor.close()
conn.close()
or you can go with this as an example for a simple straight forward dict
placeholders = ', '.join(['%s'] * len(myDict))
columns = ', '.join(myDict.keys())
sql = "INSERT INTO %s ( %s ) VALUES ( %s )" % (table, columns, placeholders)
cursor.execute(sql, myDict.values())
I've been trying to find out what causes the error. I believe it is in the last query to the database. I've marked it with comments.
This error has been giving me a headache for the past 30 minutes.
import MySQLdb
import time
# Create a database connection
db = MySQLdb.connect(host="******", user="******", passwd="*****", db="*****")
cur = db.cursor()
# Create a query to select all IDs
cur.execute("SELECT id FROM users")
clientArray = []
# Loop over all IDs returned from query,
# save all IDs in the clientArray
for row in cur.fetchall():
clientID = str(row[0])
clientArray.append(clientID)
clientIDInput = ""
while True:
# Check and wait for input
clientIDInput = raw_input("")
if clientIDInput in clientArray:
# Check to see whether user is already signed in to the device
cur.execute("SELECT fitnessStatus FROM users WHERE id=%s", (clientIDInput))
data = cur.fetchone()
if data[0] == False:
cur.execute("UPDATE users SET fitnessStatus='1' WHERE id=%s", (clientIDInput))
checkInTime = time.strftime('%Y-%m-%d %H:%M:%S')
checkOutID = raw_input("")
if checkOutID == clientIDInput:
cur.execute("UPDATE users SET fitnessStatus='0' WHERE id=%s", (clientIDInput))
checkOutTime = time.strftime('%Y-%m-%d %H:%M:%S')
print checkInTime
print checkOutTime
### I BELIEVE THIS IS THE CAUSE OF THE ERROR ###
cur.execute("INSERT INTO activities (id, machinename, checkin, checkout, clientid) VALUES (NULL, Cross Trainer #5, %s, %s, %s)", (checkInTime, checkOutTime, clientIDInput))
# Send checkInTime and checkOutTime to database
There is a syntax error in your INSERT statement. Try to enclose the string 'Cross Trainer #5' in single quotes:
cur.execute("INSERT INTO activities (id, machinename, checkin, checkout, clientid) VALUES (NULL, 'Cross Trainer #5', %s, %s, %s)", (checkInTime, checkOutTime, clientIDInput))`
Luckily, the statement itself is already enclosed in double quotes " so that no further change would be required :)
The error 1064 is a bit misleading. It indicates, amongst others, abuse of a reserved word. And indeed: CROSS is a reserved word.
I need to take data from a csv file and import it into two mysql tables within the same database.
CSV file:
username,password,path
FP_Baby,7tO0Oj/QjRSSs16,FP_Baby
lukebryan,uu00U62SKhO.sgE,lukebryan
saul,r320QdyLJEXKEsQ,saul
jencarlos,LOO07D5ZxpyzMAg,jencarlos
abepark,HUo0/XGUeJ28jaA,abepark
From the CSV file
username and password go into the USERS table
path goes into VFS_PERMISSIONS table
The USERS table looks like
INSERT INTO `USERS` (`userid`, `username`, `password`, `server_group`) VALUES
(23, 'username', 'password', 'MainUsers'),
INSERT INTO `VFS_PERMISSIONS` (`userid`, `path`, `privs`) VALUES
(23, '/path/', '(read)(write)(view)(delete)(resume)(share)(slideshow)(rename)(makedir)(deletedir)'),
if possible I'd like to start the userid in both tables at 24 and increment +1 for each row in the csv.
SO far I can read the csv files but I can't figure out how to insert into two mysql tables.
#!/usr/bin/env python
import csv
import sys
import MySQLdb
conn = MySQLdb.connect(host= "localhost",
user="crushlb",
passwd="password",
db="crushlb")
x = conn.cursor()
f = open(sys.argv[1], 'rt')
try:
reader = csv.reader(f)
for row in reader:
## mysql stuff goes here right?
finally:
f.close()
You can reduce the number of calls to cursor.execute by preparing the arguments in advance (in the loop), and calling cursor.executemany after the loop has completed:
cursor = conn.cursor()
user_args = []
perm_args = []
perms = '(read)(write)(view)(delete)(resume)(share)(slideshow)(rename)(makedir)(deletedir)'
with open(sys.argv[1], 'rt') as f:
for id, row in enumerate(csv.reader(f), start = 24):
username, password, path = row
user_args.append((id, username, password, 'MainUsers'))
perm_args.append((id, path, perms))
insert_users = '''
INSERT IGNORE INTO `USERS`
(`userid`, `username`, `password`, `server_group`)
VALUES (%s, %s, %s, %s)
'''
insert_vfs_permissions = '''
INSERT IGNORE INTO `VFS_PERMISSIONS`
(`userid`, `path`, `privs`)
VALUES (%s, %s, %s)
'''
cursor.executemany(insert_users,user_args)
cursor.executemany(insert_vfs_permissions,perm_args)
INSERT IGNORE tells MySQL to try to insert rows into the MySQL table, but ignore the command if there is a conflict. For example, if userid is the PRIMARY KEY, and there is already a row with the same userid, then the INSERT IGNORE SQL will ignore the command to insert a new row since that would create two rows with the same PRIMARY KEY.
Without the IGNORE, the cursor.executemany command would raise an exception and fail to insert any rows.
I used INSERT IGNORE so you can run the code more than once without cursor.executemany raising an exception.
There is also a INSERT ... ON DUPLICATE KEY UPDATE command which tells MySQL to try to insert a row, but update it if there is a conflict, but I'll leave it at this unless you want to know more about ON DUPLICATE KEY.
Since you already know the sql statements that you wan to execute, it should be more or less straightforward to use the cursor.execute method:
offset = 23
for row_number, row in enumerate(reader):
username, password, path = row
x.execute("INSERT INTO `USERS` (`userid`, `username`, `password`, `server_group`) "
"VALUES (%s, %s, %s, 'MainUsers')", (row_number+offset, username, password))
x.execute("INSERT INTO `VFS_PERMISSIONS` (`userid`, `path`, `privs`) "
"VALUES (%s, %s, '(read)(write)(view)(delete)(resume)(share)(slideshow)(rename)(makedir)(deletedir)'", (row_number+offset, path))
I am trying to get a fast i.e. fast and not a lot of code, way to get csv data into postgres data base. I am reading into python using csvDictreader which works fine. Then I need to generate code somehow that takes the dicts and puts it into a table. I want to do this automaticaly as my tables often have hundreds of variables. (I don't want to read directly to Postgres because in many cases I must transform the data and python is good for that)
This is some of what I have got:
import psycopg2
import sys
import itertools
import sys, csv
import psycopg2.extras
import psycopg2.extensions
csvReader=csv.DictReader(open( '/home/matthew/Downloads/us_gis_data/statesp020.csv', "rb"), delimiter = ',')
#close.cursor()
x = 0
ConnectionString = "host='localhost' dbname='mydb' user='postgres' password='######"
try:
connection = psycopg2.extras.DictConnection(ConnectionString)
print "connecting"
except:
print "did not work"
# Create a test table with some data
dict_cur = connection.cursor()
#dict_cur.execute("CREATE TABLE test (id serial PRIMARY KEY, num integer, data varchar);")
for i in range(1,50):
x = x+1
print x
dict_cur.execute("INSERT INTO test (num, data) VALUES(%s, %s)",(x, 3.6))#"abc'def"))
### how to I create the table and insert value using the dictreader?
dict_cur.execute("SELECT * FROM test")
for k in range(0,x+1):
rec = dict_cur.fetchone()
print rec['num'], rec['data']
Say you have a list of field names (presumably you can get this from the header of your csv file):
fieldnames = ['Name', 'Address', 'City', 'State']
Assuming they're all VARCHARs, you can create the table "TableName":
sql_table = 'CREATE TABLE TableName (%s)' % ','.join('%s VARCHAR(50)' % name for name in fieldnames)
cursor.execute(sql_table)
You can insert the rows from a dictionary "dict":
sql_insert = ('INSERT INTO TableName (%s) VALUES (%s)' %
(','.join('%s' % name for name in fieldnames),
','.join('%%(%s)s' % name for name in fieldnames)))
cursor.execute(sql_insert, dict)
Or do it in one go, given a list dictionaries:
dictlist = [dict1, dict2, ...]
cursor.executemany(sql_insert, dictlist)
You can adapt this as necessary based on the type of your fields and the use of DictReader.
I am a novice but this worked for me. I used PG Admin to create the 'testCSV' table.
import psycopg2 as dbapi
con = dbapi.connect(database="testpg", user="postgres", password="secret")
cur = con.cursor()
import csv
csvObject = csv.reader(open(r'C:\testcsv.csv', 'r'), dialect = 'excel', delimiter = ',')
passData = "INSERT INTO testCSV (param1, param2, param3, param4, param5) VALUES (%s,%s,%s,%s,%s);"
for row in csvObject:
csvLine = row
cur.execute(passData, csvLine)
con.commit()