I am attempting to write a simple python script to import from a text file to a mysql database, and encounter a perplexing error
Windows 10, Mysql 5.7.18, Python 3.6, pymysql
The contents of the text file:
nickname|fullname|cell|email|updatedt
andrew|Andrew Jones|+12395551172|arj#domain.com|2017-05-04 13:26:10
laurelai|Laurelai Smith||lsmith#domain.net|2017-05-04 13:27:47
I read in the data to construct a sql string:
insert into contacts (nickname,fullname,cell,email,updatedt) values(%,%,%,%,%)
The field values to be inserted are read in as follows:
['andrew', 'Andrew Jones', '+12395551172', 'arj#domain.com', '2017-05-04 13:26:10']
This is of course a Python list object. I have tried converting it to a tuple, but with same result
The routine to insert the row into the table is as follows:
def insert(sql, values):
#insert a single row of data from the input file
connection = getconn()
with connection.cursor() as cursor:
try:
cursor.execute(sql, values)
#except ValueError:
#print('Value error from pymysql')
finally:
cursor.close()
The following ValueError is returned:
ValueError: unsupported format character ',' (0x2c) at index 69
if, however, I extract the data values and insert them into the sqlstring by concatenation, I get:
insert into contacts (nickname,fullname,cell,email,updatedt) values('laurelai','Laurelai Smith','','lsmith#domain.net','2017-05-04 13:27:47')
This inserts the rows without error
What causes the ValueError?
Change your insert query to %s instead of %:
insert into contacts (nickname,fullname,cell,email,updatedt) values(%s,%s,%s,%s,%s)
Refer to doc.
Related
I'm new to MySQL and database in general, however I'm having some issue when I try to add a new column of integer inside my table. To add a new column I'm doing so:
import mysql.connector
mydb = mysql.connector.connect(
# host, user, password and database
)
mycursor = mydb.cursor(buffered = True)
# some stuff to get the variable domain
mycursor.execute('ALTER TABLE domainsMoreUsed ADD {} INTEGER(10)'.format(domain)) # domain is a string
but i get this error:
raise errors.get_mysql_exception(exc.errno, msg=exc.msg,
mysql.connector.errors.ProgrammingError: 1064 (42000): You
have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right
syntax to use near 'in INTEGER(10)' at line 1
I get the same error above also trying:
mycursor.execute('ALTER TABLE domainsMoreUsed ADD %s INTEGER(10)' % domain)
Instead when I use:
mycursor.execute('ALTER TABLE domainsMoreUsed ADD %s INTEGER(10)', (domain))
i get:
raise ValueError("Could not process parameters")
ValueError: Could not process parameters
I read some post of other users about the same error, but I couldn't find what I need. I'm pretty sure about the SQL syntax being correct.
I'm using MySQL 8.0 with Python 3.8.3 on Windows 10.
Thank you in advance for your help.
What is the string domain set to? The error message syntax to use near 'in INTEGER(10)' at line 1, implies "in", which is a reserved word. If you want to use that for a table or column name, you need to add backticks: " ` " (left of '1' on the top row of your keyboard) around them.
Change your queries like this:
mycursor.execute('ALTER TABLE domainsMoreUsed ADD `{}` INTEGER(10)'.format(domain))
mycursor.execute('ALTER TABLE domainsMoreUsed ADD `%s` INTEGER(10)', (domain))
Whenever I want to insert data in a pandas dataframe to the into a postgresql database I get this error
error: extra data after last expected column CONTEXT: COPY recommendations, line 1: "0,4070,"[5963, 8257, 9974, 7546, 11251, 5203, 102888, 8098, 101198, 10950]""
The dataframe consist of three column, the first and second column are of type integers and the third column is a list of integers.
I created a table in PostgreSQL using this function below
def create_table(query: str) -> None:
"""
:param query: A string of the query to create table in the database
:return: None
"""
try:
logger.info("Creating the table in the database")
conn = psycopg2.connect(host=HOST, dbname=DATABASE_NAME, user=USER, password=PASSWORD, port=PORT)
cur = conn.cursor()
cur.execute(query)
conn.commit()
logger.info("Successfully created a table in the database using this query {}".format(query))
return
except (Exception, psycopg2.Error) as e:
logger.error("An error occurred while creating a table using the query {} with exception {}".format(query, e))
finally:
if conn is not None:
conn.close()
logger.info("Connection closed!")
The query passed into this function is this:
create_table_query = '''CREATE TABLE Recommendations
(id INT NOT NULL,
applicantId INT NOT NULL,
recommendation INTEGER[],
PRIMARY KEY(id),
CONSTRAINT applicantId
FOREIGN KEY(applicantId)
REFERENCES public."Applicant"(id)
ON DELETE CASCADE
ON UPDATE CASCADE
); '''
I then use the function below to copy the data frame to the created table in postgres.
def copy_from_file(df: pd.DataFrame, table: str = "recommendations") -> None:
"""
Here we are going save the dataframe on disk as
a csv file, load the csv file
and use copy_from() to copy it to the table
"""
conn = psycopg2.connect(host=HOST, dbname=DATABASE_NAME, user=USER, password=PASSWORD, port=PORT)
# Save the dataframe to disk
tmp_df = "./tmp_dataframe.csv"
df.to_csv(tmp_df, index_label='id', header=False)
f = open(tmp_df, 'r')
cursor = conn.cursor()
try:
cursor.copy_from(f, table, sep=",")
conn.commit()
except (Exception, psycopg2.DatabaseError) as error:
os.remove(tmp_df)
logger.error("Error: %s" % error)
conn.rollback()
cursor.close()
logger.info("copy_from_file() done")
cursor.close()
os.remove(tmp_df)
And then I still get this error: extra data after last expected column CONTEXT: COPY recommendations, line 1: "0,4070,"[5963, 8257, 9974, 7546, 11251, 5203, 102888, 8098, 101198, 10950]"" please any recommendations on how to fix this issue? Thanks
copy_from uses the text format, not the csv format. You told it to use , as the separator, but that doesn't change the protecting method it is trying to use. So the commas inside the quotes are not treated as being protected, they are treated as field separators, and so of course there are too many of them.
I think you need to use copy_expert and tell it to use the csv format.
I am trying to update my mysql database field with a concatenation. I have to read my file line by line, and i need to append the existing string with the loaded line. I have to do it like this because my goal is to insert a 3gb long whitespace separated text file into one longtext field, and mysql only capable of handling 1gb text to insert.
The problem with my code is that if i add the field name to the concat function like seq=concat(seq, %s) I get a SQL syntax error, but when I add the field name as a variable, python acts like it's a string.
So short story long with this input file:
aaa
bbb
ccc
I want to have an updated mysql field like this:
aaabbbccc
But I get this: seqccc
Any idea how should i work with the fieldname to get this work?
import mysql.connector
connection = mysql.connector.connect(host='localhost',
database='sys',
user='Pannka',
password='???')
cursor = connection.cursor()
with open('test.txt', 'r') as f:
for line in f:
sql = "update linedna set seq=concat(%s, %s) where id=1"
val=('seq', line.rstrip())
print(line.rstrip())
cursor.execute(sql, val)
connection.commit()
cursor.close()
connection.close()
f.close()
print(0)
I think that you want:
sql = "update linedna set seq = concat(seq, %s) where id=1"
val=(line.rstrip())
cursor.execute(sql, val)
connection.commit()
This will append each new line at the end of the already existing database value in column seq.
I want to use sqlite3 to deal with data in Ubuntu with python. But I always failed and get errors. Codes related to database are as follows:
sqlite = "%s.db" % name
#connnect to the database
conn = sqlite3.connect(sqlite)
print "Opened database successfully"
c = conn.cursor()
#set default separator to "\t" in database
c.execute(".separator "\t"")
print "Set separator of database successfully"
#create table data_node
c.execute('''create table data_node(Time int,Node Text,CurSize int,SizeVar int,VarRate real,Evil int);''')
print "Table data_node created successfully"
node_info = "%s%s.txt" % (name,'-PIT-node')
c.execute(".import %\"s\" data_node") % node_info
print "Import to data_node successfully"
#create table data_face
data_info = "%s%s.txt" % (name,'-PIT-face')
c.execute('''create table data_face(Time int,Node Text,TotalEntry real,FaceId int,FaceEntry real,Evil int);''')
c.execute(".import \"%s\" data_face") % face_info
#get the final table : PIT_node
c.execute('''create table node_temp as select FIRST.Time,FIRST.Node,ROUND(FIRST.PacketsRaw/SECOND.PacketsRaw,4) as SatisRatio from tracer_temp FIRST,tracer_temp SECOND WHERE FIRST.Time=SECOND.Time AND FIRST.Node=SECOND.Node AND FIRST.Type='InData' AND SECOND.Type='OutInterests';''')
c.execute('''create table PIT_node as select A.Time,A.Node,B.SatisRatio,A.CurSize,A.SizeVar,A.VarRate,A.Evil from data_node A,node_temp B WHERE A.Time=B.Time AND A.Node=B.Node;''')
#get the final table : PIT_face
c.execute('''create table face_temp as select FIRST.Time,FIRST.Node,FIRST.FaceId,ROUND(FIRST.PacketsRaw/SECOND.PacketsRaw,4) as SatisRatio,SECOND.Packets from data_tracer FIRST,data_tracer SECOND WHERE FIRST.Time=SECOND.Time AND FIRST.Node=SECOND.Node AND FIRST.FaceId=SECOND.FaceId AND FIRST.Type='OutData' AND SECOND.Type='InInterests';''')
c.execute('''create table PIT_face as select A.Time,A.Node,A.FaceId,B.SatisRatio,B.Packets,ROUND(A.FaceEntry/A.TotalEntry,4),A.Evil from data_face as A,face_temp as B WHERE A.Time=B.Time AND A.Node=B.Node AND A.FaceId = B.FaceId;''')
conn.commit()
conn.close()
These sql-commands are right. When I run the code, it always shows sqlite3.OperationalError: near ".": syntax error. So how to change my code and are there other errors in other commands such as create table?
You have many problems in your code as posted, but the one you're asking about is:
c.execute(".separator "\t"")
This isn't valid Python syntax. But, even if you fix that, it's not valid SQL.
The "dot-commands" are special commands to the sqlite3 command line shell. It intercepts them and uses them to configure itself. They mean nothing to the actual database, and cannot be used from Python.
And most of them don't make any sense outside that shell anyway. For example, you're trying to set the column separator here. But the database doesn't return strings, it returns row objects—similar to lists. There is nowhere for a separator to be used. If you want to print the rows out with tab separators, you have to do that in your own print statements.
So, the simple fix is to remove all of those dot-commands.
However, there is a problem—at least one of those dot-commands actually does something:
c.execute(".import %\"s\" data_node") % node_info
You will have to replace that will valid calls to the library that do the same thing as the .import dot-command. Read what it does, and it should be easy to understand. (You basically want to open the file, parse the columns for each row, and do an executemany on an INSERT with the rows.)
I am trying to read a file, parse the data using python 2.7, and import the data into sqlite3. However, I'm running into a problem when inserting the data. After I parse a line from the file, the é in my string is replaced with \xe9. After I split the line from my file, I want a list that contains [73,'Misérables, Les'] but instead I'm getting [73,'Mis\xe9rables, Les'] which is screwing up the SQL INSERT statement. How can I fix this?
#!/usr/bin/python
# -*- coding: latin-1 -*-
import sqlite3
line = '73::Misérables, Les'.decode('latin-1')
vals = line.split("::")
con = sqlite3.connect('myDb.db')
cur = con.cursor()
cur.execute("DROP TABLE IF EXISTS movie")
cur.execute('CREATE TABLE movie (id INT, title TEXT)')
sql = 'INSERT INTO movie VALUES (?,?)'
cur.execute(sql,tuple(vals))
cur.execute('SELECT * FROM movie')
for record in cur:
print record
Your program inserts data into the db perfectly. It subsequently retrieves the correct data. Your problem is when you display the result.
When you print a tuple, the system displays the repr() of each item, not the str() of each item. Thus you see \xe9 instead of é in the output.
To get what you want, try replacing the loop at the end of your program:
for record in cur:
print record[0], record[1]