This question already has answers here:
ValueError: need more than 2 values to unpack in Python 2.6.6
(4 answers)
Closed 6 years ago.
I'm struggling about this ValueError, it happens when I want to define an init_db function with resetting the database and adding some data from local document (hardwarelist.txt), the code was:
def init_db():
"""Initializes the database."""
db = get_db()
with app.open_resource('schema.sql', mode='r') as f:
db.cursor().executescript(f.read())
with open('hardwarelist.txt') as fl:
for eachline in fl:
(model,sn,user,status)=eachline.split(',')
db.execute('insert into entries (model,sn,user,status) values (?, ?, ?, ?)',
(model,sn,user,status))
fl.close()
db.commit()
And the error was:
File "/home/ziyma/Heroku_pro/flaskr/flaskr/flaskr.py", line 48, in init_db
(model,sn,user,status)=eachline.split(',')
ValueError: need more than 3 values to unpack
What should I do?
One of my mentors told me "If half your code is error handling, you aren't doing enough error handling." But we can leverage python's exception handling to make the job easier. Here, I've reworked your example so that if an error is detected, a message is displayed, and nothing is committed to the database.
When you hit the bad line, its printed and you can figure out what's wrong from there.
import sys
def init_db():
"""Initializes the database."""
db = get_db()
with app.open_resource('schema.sql', mode='r') as f:
db.cursor().executescript(f.read())
with open('hardwarelist.txt') as fl:
try:
for index, eachline in enumerate(fl):
(model,sn,user,status)=eachline.strip().split(',')
db.execute('insert into entries (model,sn,user,status) values (?, ?, ?, ?)',
(model,sn,user,status))
db.commit()
except ValueError as e:
print("Failed parsing {} line {}: {} ({})".format('hardwarelist.txt',
index, eachline.strip(), e), file=sys.stderr)
# TODO: Your code should have its own exception class
# that is raised. Your users would catch that exception
# with a higher-level summary of what went wrong.
raise
You should expand that exception handler to catch exceptions from your database code so that you can catch more errors.
As a side note, you need to strip the line before splitting to remove the \n newline character.
UPDATE
From the comments, here's an example on splitting multiple forms of the comma. In this case its Unicode FULLWIDTH COMMA U+FF0C. Whether you can enter unicode directly into your python scripts depends on your text editor and etc..., but that comma could be represented by "\uff0c" or "，". Anyway could can use a regular expression to split on multiple characters.
I create the text using unicode escapes
>>> text='a,b,c\uff0cd\n'
>>> print(text)
a,b,c，d
and I can write the regex with excapes
>>> re.split('[,\uff0c]', text.strip())
['a', 'b', 'c', 'd']
or by copy/paste of the alternate comma character
>>> re.split('[,，]', text.strip())
['a', 'b', 'c', 'd']
Related
I am attempting to write a simple python script to import from a text file to a mysql database, and encounter a perplexing error
Windows 10, Mysql 5.7.18, Python 3.6, pymysql
The contents of the text file:
nickname|fullname|cell|email|updatedt
andrew|Andrew Jones|+12395551172|arj#domain.com|2017-05-04 13:26:10
laurelai|Laurelai Smith||lsmith#domain.net|2017-05-04 13:27:47
I read in the data to construct a sql string:
insert into contacts (nickname,fullname,cell,email,updatedt) values(%,%,%,%,%)
The field values to be inserted are read in as follows:
['andrew', 'Andrew Jones', '+12395551172', 'arj#domain.com', '2017-05-04 13:26:10']
This is of course a Python list object. I have tried converting it to a tuple, but with same result
The routine to insert the row into the table is as follows:
def insert(sql, values):
#insert a single row of data from the input file
connection = getconn()
with connection.cursor() as cursor:
try:
cursor.execute(sql, values)
#except ValueError:
#print('Value error from pymysql')
finally:
cursor.close()
The following ValueError is returned:
ValueError: unsupported format character ',' (0x2c) at index 69
if, however, I extract the data values and insert them into the sqlstring by concatenation, I get:
insert into contacts (nickname,fullname,cell,email,updatedt) values('laurelai','Laurelai Smith','','lsmith#domain.net','2017-05-04 13:27:47')
This inserts the rows without error
What causes the ValueError?
Change your insert query to %s instead of %:
insert into contacts (nickname,fullname,cell,email,updatedt) values(%s,%s,%s,%s,%s)
Refer to doc.
I have created a little script that allows me to save data to MySQLdb. At first it was working fine when I was using:
cursor.execute('INSERT INTO people (name, text) VALUES ("dan", "test2")')
The above would save "dan" into the title and "test2" into the text. I wanted to test to see if I was able to define something and fill it in this way. For example if I was to scrape a site and say (dan = soup.title.string) or something like that it would be able to populate this data into the database. I have tried to have a look around but cannot seem to find anything.
import MySQLdb
import sys
try:
db = MySQLdb.connect(
host = 'localhost',
user = 'root',
passwd = '',
db = 'python',
)
except:
print "db not found"
dan = "dandandan"
test2 = "testing101"
cursor = db.cursor()
cursor.execute('INSERT INTO people (name, text) VALUES (dan, test2)')
cursor.execute('SELECT * FROM people')
result = cursor.fetchall()
db.commit()
db.close()
The error I am receiving is:
C:\Users\********\Desktop>python mysqltest.py
Traceback (most recent call last):
File "mysqltest.py", line 18, in <module>
cursor.execute('INSERT INTO people (name) VALUES (dan)')
File "C:\Python27\lib\site-packages\MySQLdb\cursors.py", line 174, in execute
self.errorhandler(self, exc, value)
File "C:\Python27\lib\site-packages\MySQLdb\connections.py", line 36, in defau
lterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (1054, "Champ 'dan' inconnu dans field list"
)
You need to use parameters.
cursor.execute('INSERT INTO people (name, text) VALUES (%s,%s)', (dan, test2))
Use prepared statements:
cursor.execute("INSERT INTO people (name, text) VALUES (%s,%s)", (dan, test2))
From the documentation :
paramstyle
String constant stating the type of parameter marker formatting
expected by the interface. Set to 'format' = ANSI C printf format
codes, e.g. '...WHERE name=%s'. If a mapping object is used for
conn.execute(), then the interface actually uses 'pyformat' = Python
extended format codes, e.g. '...WHERE name=%(name)s'. However, the API
does not presently allow the specification of more than one style in
paramstyle.
Note that any literal percent signs in the query string passed to
execute() must be escaped, i.e. %%.
Parameter placeholders can only be used to insert column values. They
can not be used for other parts of SQL, such as table names,
statements, etc.
I am still learning Python and as a little Project I wrote a script that would take the values I have in a text file and insert them into a sqlite3 database. But some of the names have weird letter (I guess you would call them non-ASCII), and generate an error when they come up. Here is my little script (and please tell me if there is anyway it could be more Pythonic):
import sqlite3
f = open('complete', 'r')
fList = f.readlines()
conn = sqlite3.connect('tpb')
cur = conn.cursor()
for i in fList:
exploaded = i.split('|')
eList = (
(exploaded[1], exploaded[5])
)
cur.execute('INSERT INTO magnets VALUES(?, ?)', eList)
conn.commit()
cur.close()
And it generates this error:
Traceback (most recent call last):
File "C:\Users\Admin\Desktop\sortinghat.py", line 13, in <module>
cur.execute('INSERT INTO magnets VALUES(?, ?)', eList)
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a te
xt_factory that can interpret 8-bit bytestrings (like text_factory = str). It is
highly recommended that you instead just switch your application to Unicode str
ings.
To get the file contents into unicode you need to decode from whichever encoding it is in.
It looks like you're on Windows so a good bet is cp1252.
If you got the file from somewhere else all bets are off.
Once you have the encoding sorted, an easy way to decode is to use the codecs module, e.g.:
import codecs
# ...
with codecs.open('complete', encoding='cp1252') as fin: # or utf-8 or whatever
for line in fin:
to_insert = (line.split('|')[1], line.split('|')[5])
cur.execute('INSERT INTO magnets VALUES (?,?)', to_insert)
conn.commit()
# ...
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 9 years ago.
I am porting a tcl script to python, because I can't seem to get the tclSqlite thing going on my Nokia N810. The script prompts for inputs, passes them to a sqlite db with 3 tables: Notes, Tags, and a NotesXTags many-to-many tbl. There are triggers that keep me from storing any tags more than once. Being a noob/hobbyist I went line by line through the tcl script replacing each with a Python line. Not very Pythonic, but I'm a hobbyist with no intention of using the language after I get this one script to work on N810. I did look at every Q&A S.O. suggested and I've been working on this for hours. I've got at least 3 bugs-of-ignorance. A chunk of the script in a module called 'pythonmakenote.py':
the crunch-bang ... and some comments ....
import sys, tty
import sqlite3
def mn():
conn = sqlite3.connect('/home/j...notes.sqlite')
db = conn.cursor()
tagsofar =db.execute('select tag_text from tag')
print tagsofar
print "Enter note text, remember to let console wrap long lines"
notetxt = input("note: ")
print "Enter 1 or more tags separated by spaces"
taglist = input("tags: ")
taglist = taglist.split(" ")
db.execute('INSERT INTO note (note_txt) VALUES (?)', notetxt)
db.commit
fknote = db.execute('select last_insert_rowid()')
#records new tags since db trigger stops dups, updates many-many tbl
for tagtxt in taglist:
db.execute('INSERT INTO tag VALUES (?)',tagtxt)
db.commit
fktag = db.execute('select rowid from tag where tag_text = (?)',tagtxt)
db.execute('INSERT INTO fkeys VALUES (?,?)',fknote,fktag)
db.commit
So I do 'import pythonmakenote'. So far so good. I type 'mn' and get an error:
>>> mn
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'mn' is not defined
Then I try this:
>>> from pythonmakenote import mn
>>> mn
<function mn at 0xb76b2a74>
But 'mn' still doesn't work. So I remove the Def altogether and copy the file and name it 'mn.py' and it sort-of works...
>>> import mn
<sqlite3.Cursor object at 0xb75fb740>
Enter note text, remember to let console wrap long lines
note: 'this is a note'<--------------------------Quotes are a MUST (but not in tcl version)
Enter 1 or more tags separated by spaces
tags: 'dev'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "mn.py", line 19, in <module>
db.execute('INSERT INTO note (note_txt) VALUES (?)', notetxt)
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 14 supplied.<-----------------------------Huh?
Where in the world are the S.O. instructions on code block tags and other markdown?
Why can't I Def mn in a module and use it? Is Python: NameError: global name 'foobar' is not defined pertinent? (to my problem)
I've got several other Defs to do for getting notes by tag, getting the tag list, etc.. and I thought they could all go in one module.
I don't want to put quotes around my inputs (the notes or the space-delimited tag list). Is that doable? I have the import tty thing in there but I'm not using it (don't know how but I'm beginning to suspect I'll have to learn)
If I put 3 tags in when prompted, without quotes, I get the 'unexpected EOF' error. Why?
I see strings are immutable so maybe assigning the list/split to a var that was a string before could be a problem?
Where does sqlite get '14' bindings supplied? I'm splitting on the space char but it's being ignored (because I'm doing it wrong)?
Would it be easier to do my little project in Bash?
Thanks to anyone who takes the time to get this far. I have a bad habit of needing help in areas off-topic in S.U. and too noob-RTFM here. I struggled a little with the tcl version but it now works like a champ. I expected Python to be somewhat straightforward. Still think it might be.
edit: WOW. A bunch of newlines got stripped out. Sorry I have no idea how to fix. I'm off to see if "raw_input" works better.
You'll want raw_input instead of input in your script. input evaluates what you type, which is why you have to enter quotes.
You can markdown code using the {} buttons above the input window. That actual markdown for code is a preceding 4 spaces.
db.commit needs to be db.commit().
If you do this:
>>> import pythonmakenote
To run mn do this:
>>> pythonmakenote.mn()
You can also do:
>>> from pythonmakenote import mn
>>> mn()
For lines like:
db.execute('INSERT INTO note (note_txt) VALUES (?)', notetxt)
You need:
db.execute('INSERT INTO note (note_txt) VALUES (?)', (notetxt,))
execute expects a sequence, so if you pass a string, it acts as a sequence of single characters, hence your 14 bindings error (it was a string of 14 characters). (xxx,) is the syntax for a 1-element tuple. Making it a list [xxx] would work too.
Here's my best guess at something that works. I don't have your database:
import sys
import sqlite3
def mn():
conn = sqlite3.connect('data.db')
db = conn.cursor()
db.execute('select tag_text from tag')
tagssofar = db.fetchall()
print tagssofar
print "Enter note text, remember to let console wrap long lines"
notetxt = raw_input("note: ")
print "Enter 1 or more tags separated by spaces"
taglist = raw_input("tags: ")
taglist = taglist.split()
db.execute('INSERT INTO note (note_txt) VALUES (?)', [notetxt])
conn.commit()
db.execute('select last_insert_rowid()')
fknote = db.fetchone()[0]
print fknote
#records new tags since db trigger stops dups, updates many-many tbl
for tagtxt in taglist:
db.execute('INSERT INTO tag VALUES (?)',[tagtxt])
conn.commit()
db.execute('select rowid from tag where tag_text = (?)',[tagtxt])
fktag = db.fetchone()[0]
print fktag
db.execute('INSERT INTO fkeys VALUES (?,?)',[fknote,fktag])
conn.commit()
There are a couple of things going on here. First, mn does not return anything, you would want something like:
>>> def mn():
... conn = sqlite3.connect('tester.db')
... cur = conn.cursor()
... return cur
...
>>> c = mn()
This leaves an open connection, so when you are done with c you would call:
>>> c.connection.close()
Also, the executing on the cursor does not return anything, you need to call some fetch method, ie fetchone or fetchall. Putting a few of these things together I would start to modify as follows:
import sys, tty
import sqlite3
def mn():
conn = sqlite3.connect('/home/j...notes.sqlite')
cur = conn.cursor()
return cur
db = mn()
tags_so_far = db.execute('select tag_text from tag').fetchall()
print tags_so_far
print "Enter note text, remember to let console wrap long lines \n"
notetxt = raw_input("note: ")
print "Enter 1 or more tags separated by spaces \n"
taglist = raw_input("tags: ").split()
db.execute('INSERT INTO note (note_txt) VALUES (?)', notetxt)
db.commit()
fknote = db.execute('select last_insert_rowid()').fetchone()[0]
#records new tags since db trigger stops dups, updates many-many tbl
for tagtxt in taglist:
db.execute('INSERT INTO tag VALUES (?)', (tagtxt,))
db.commit()
fktag = db.execute('select rowid from tag where tag_text = (?)',tagtxt)
db.execute('INSERT INTO fkeys VALUES (?,?)',fknote,fktag)
This is my first post! I also just started programming, so please bear with me!
I am trying to load a bunch of .csv files into a database, in order to later perform various reports on the data. I started off by creating a few tables in mysql with matching field names and data types to what will be loaded into the tables. I am manipulating the filename (in order to parse out the date to use as a field in my table) and cleaning up the data with python.
So my problem right now (haha...) is that I get this error message when I attempt the 'Insert Into' query to mysql.
Traceback (most recent call last):
File "C:\Program Files\Python\load_domains2.py", line 80, in <module>
cur.execute(sql)
File "C:\Program Files\Python\lib\site-packages\MySQLdb\cursors.py", line 166, in execute
self.errorhandler(self, exc, value)
File "C:\Program Files\Python\lib\site-packages\MySQLdb\connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1054, "Unknown column 'a1200e.com' in 'field list'")
'a1200e.com' refers to a specific domain name I'm inserting into that column. My query is as follows:
sql="""INSERT INTO temporary_load
(domain_name, session_count, search_count, click_count,
revenue, revenue_per_min, cost_per_click, traffic_date)
VALUES (%s, %d, %d, %d, %d, %d, %d, %s)""" %(cell[0],
int(cell[1]),
int(cell[2].replace (",","")),
int(cell[3].replace(",","")),
float(cell[4].replace("$","")),
float(cell[5].replace("$","")),
float(cell[6].replace("$","")),
parsed_date)
cur.execute(sql)
I am very new at all this, so I'm sure my code isn't at all efficient, but I just wanted to lay everything out so it's clear to me. What I don't understand is that I have ensured my table has correctly defined data types (corresponding to those in my query). Is there something I'm missing? I've been trying to work this out for a while, and don't know what could be wrong :/
Thanks so much!!!
Val
Thomas is, as usual, absolutely correct: feel free to let MySQLdb handle the quoting issues.
In addition to that recommendation:
The csv module is your friend.
MySQLdb uses the "format" parameter style as detailed in PEP 249.
What does that mean for you?
All parameters, whatever type, should be passed to MySQLdb as strings (like this %s). MySQLdb will make sure that the values are properly converted to SQL literals.
By the way, MySQLdb has some good documentation.
Feel free to include more detail about your source data. That may make diagnosing the problem easier.
Here's one way to insert values to a MySQL database from a .csv file:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import csv
import MySQLdb
import os
def main():
db = MySQLdb.connect(db="mydb",passwd="mypasswd",) # connection string
filename = 'data.csv'
f = open(filename, "rb") # open your csv file
reader = csv.reader(f)
# assuming the first line of your csv file has column names
col_names = reader.next() # first line of .csv file
reader = csv.DictReader(f, col_names) # apply column names to row values
to_db = [] # this list holds values you will insert to db
for row in reader: # loop over remaining lines in .csv file
to_db.append((row['col1'],row['col2']))
# or if you prefer one-liners
#to_db = [(row['col1'],row['col2']) for row in reader]
f.close() # we're done with the file now
cursor = db.cursor()
cursor.executemany('''INSERT INTO mytable (col1,col2)
VALUES (%s, %s)''', to_db) # note the two arguments
cursor.close()
db.close()
if __name__ == "__main__":
main()
You should be using DB-API quoting instead of including the data in the SQL query directly:
sql = """INSERT INTO temporary_load
(domain_name, session_count, search_count, click_count,
revenue, revenue_per_min, cost_per_click, traffic_date)
VALUES (%s, %d, %d, %d, %d, %d, %d, %s)"""
args = (cell[0],
int(cell[1]),
int(cell[2].replace (",","")),
int(cell[3].replace(",","")),
float(cell[4].replace("$","")),
float(cell[5].replace("$","")),
float(cell[6].replace("$","")),
parsed_date)
cur.execute(sql, args)
This makes the DB-API module quote the values appropriately, and resolves a whole host of issues that you might get when doing it by hand (and usually incorrectly.)