I have a set of data in a list, embedded into a dictionary as:
{'list1': ['Freddy','36','fred','123f','2017/04/25'],
'list2':['Tara','25','mtara','123t','2018/03/22']}
Ref notations:
{ key1: [name, age,nickname, userid, account_created_date],
..key2:[name, age,nickname, userid, account_created_date] }
All the data is inserted in variables in a Python function, one for each, as described above. When I call the function I would get the output right-away as
Output:
Freddy
Tara
But when I try to insert the data into a sqlite database, I get the output in the following manner:
Output:
F
R
E
D
D
Y
T
A
R
A
Code:
conn = sqlite3.connect(dbPath)
cur = conn.cursor()
results = cur.executemany("INSERT INTO Field (Field1) VALUES (?)", name)
conn.commit()
Requesting your assistance on this issue. Your help is much appreciated. Thank You.
New Answer
To suit your dataset more directly. You need a list of dictionaries, not a dictionary of lists because each dictionary has the column name as the key and a value associated with that column (key).
names = [{'name': "Freddy"}, {'name': "Joan"}]
results = c.executemany("INSERT INTO names ('name') VALUES (:name)", names)
The rest of the SQLite3 code working, this outputs:
Freddy
Joan
Option to Use a List of Tuples
You can also use a list of tuples. If you're not using a dictionary, you need to make sure each row is in tuple form. You need each of these tuples in a list. You need to call this list in the executemany function after the SQL query.
Essential code:
names = [("Freddy",), ("Joan",)]
results = c.executemany("INSERT INTO names (name) VALUES (?)", names)
Output:
Freddy
Joan
So make sure each entry is in tuple or dictionary form. Then put these tuples or dictionaries into a list. If it's in dictionary form, you cannot use the '?' placeholder. Use ':key_name' for each placeholder. (There might be a different option here, but I haven't found it yet).
Original Response
I suspect that executemany(...) should be execute(...). I haven't used SQLite3 in a while, so I'll test it now to make sure and then get back to you.
Update 1:
I remember the "?'s" needing to be sent in tuple form. I.e.
cur.executemany("INSERT INTO Field (Field1) VALUES (?)", name)
needs to be at least
cur.executemany("INSERT INTO Field (Field1) VALUES (?)", (name, ...))
But I've only gotten the execute() (and not executemany()) to work, with this, else it's back to the same error. So, here's my working code so far:
cur.execute("INSERT INTO Field (Field1) VALUES (?)", (name,))
It gave F, R, E, D, D, Y because it was reading each letter as values in the tuple. If it was in tuple form, it would read "Freddy" all in one go. But I'm still having issues with the executemany(), which I think works differently than we think it does.
Update 2:
Here's what I got to work. You need to make sure each row is in tuple form. You need each of these tuples in a list. You need to call this list in the executemany function after the SQL query.
Essential code:
names = [("Freddy",), ("Joan",)]
results = c.executemany("INSERT INTO names (name) VALUES (?)", names)
Full code:
import sqlite3
conn = sqlite3.connect('test.db')
c = conn.cursor()
# Create table
c.execute('''CREATE TABLE IF NOT EXISTS names
(name text)''')
names = [("Freddy",), ("Joan",)]
results = c.executemany("INSERT INTO names (name) VALUES (?)", names)
conn.commit()
for row in c.execute('SELECT * FROM names'):
print(row[0])
This prints out:
Freddy
Joan
Update 3:
I've updated it to suit your dataset more directly. You need a list of dictionaries, not a dictionary of lists because each dictionary has the column name as the key and a value associated with that column (key).
Here's some more code, I've gotten to work.
names = [{'name': "Freddy"}, {'name': "Joan"}]
results = c.executemany("INSERT INTO names ('name') VALUES (:name)", names)
With the rest of the code as it was, it outputs:
Freddy
Joan
(Inspired from this answer.)
you might consider to change
results = cur.executemany("INSERT INTO Field (Field1) VALUES (?)", name)
to
results = cur.executemany("INSERT INTO Field (Field1) VALUES (?)", (name))
since the expected parameter is a tuple, the string is not considered as a whole but splited
Related
I am trying to write a row of observations into my database, but I have some unique variable called list_variable which is a list of strings that can be of length 1-3. So sometimes ['string1'] but sometimes also ['string1','string2'] or ['string1','string2','string3'].
When I try to add this to my database by:
def add_to_cockroach_db():
cur.execute(f"""
INSERT INTO database (a, b, c)
VALUES ({time.time()}, {event},{list_variable}; <--- this one
""")
conn.commit()
I would get the following error (values have been changed for readability):
SyntaxError: at or near "[": syntax error
DETAIL: source SQL:
INSERT INTO database (a, b, c)
VALUES (a_value, b_value, ['c_value_1', 'c_value_2'])
^
HINT: try \h VALUES
It seems that having a variable that is a list is not allowed, how could I make this work out?
Thanks in advance!
**edit
list_variable looks e.g., like this = ['value1','value2']
You can either cast it to string using
str(['c_value_1', 'c_value_2'])
which looks like this:
"['c_value_1', 'c_value_2']"
or join the elements of your list with a delimiter you choose. This for example generates a comma separated string.
",".join(['c_value_1', 'c_value_2'])
which looks like this:
'c_value_1,c_value_2'
Like Maurice Meyer has already pointed out in the comments, it is better to pass your values as a list or as a tuple instead of formatting the query yourself.
Your command could look like this depending on the solution you choose:
cur.execute("INSERT INTO database (a, b, c) VALUES (%s, %s, %s)", (time.time(), event, ",".join(list_variable)))
There are a few ways you could accomplish this.
The simplest way is to call str on the list and insert the result into a string (VARCHAR) column. While this works, it's not easy to work with the values in database queries, and when it's retrieved from the database it's a string, not a list.
Using a VARCHAR[] column type - an array of string values - reflects the actual data type, and enables use of PostgreSQL's array functions in queries.
Finally, you could use a JSONB column type. This allows storage of lists or dicts, or nested combinations of both, so it's very flexible, and PostgreSQL provides functions for working with JSON objects too. However it might be overkill if you don't need the flexibility, or if you want to be strict about the data.
This script shows all three methods in action:
import psycopg2
from psycopg2.extras import Json
DROP = """DROP TABLE IF EXISTS t73917632"""
CREATE = """\
CREATE TABLE t73917632 (
s VARCHAR NOT NULL,
a VARCHAR[] NOT NULL,
j JSONB NOT NULL
)
"""
INSERT = """INSERT INTO t73917632 (s, a, j) VALUES (%s, %s, %s)"""
SELECT = """SELECT s, a, j FROM t73917632"""
v = ['a', 'b', 'c']
with psycopg2.connect(dbname='test') as conn:
with conn.cursor() as cur:
cur.execute(DROP)
cur.execute(CREATE)
conn.commit()
cur.execute(INSERT, (str(v), v, Json(v)))
conn.commit()
cur.execute(SELECT)
for row in cur:
print(row)
Output:
("['a', 'b', 'c']", ['a', 'b', 'c'], ['a', 'b', 'c'])
It's worth observing that if the array of strings represents some kind of child relationship to the table - for example the table records teams, and the string array contains the names of team members - it is usually a better design to insert each element in the array into a separate row in a child table, and associate them with the parent row using a foreign key.
I'm trying to use this code to update a table on mySQL, but I'm getting error with the update part
table_name = 'my_table'
sql_select_Query = """
INSERT {0} (age, city, gender,UniqueId)
VALUES ({1},{2},{3},{4})
ON DUPLICATE KEY UPDATE
age=VALUES(age,city=VALUES(city),gender=VALUES(gender),height=VALUES(height)
""".format(table_name, '877','2','1','2898989')
cursor = mySQLconnection .cursor()
cursor.execute(sql_select_Query)
mySQLconnection.commit()
For example, to update the city I get:
Unknow columns '877'
Hence it seems it is taking the value as a column name and search for then in my_table.
The correct way to use parameters with Python is to pass the values in the execute() call, not to interpolate the values into the SQL query.
Except for identifiers like the table name in your case.
sql_select_Query = """
INSERT `{0}` (age, city, gender, UniqueId)
VALUES (%s, %s, %s, %s)
ON DUPLICATE KEY UPDATE
age=VALUES(age), city=VALUES(city), gender=VALUES(gender)
""".format(table_name)
cursor.execute(sql_select_Query, ('877', '2', '1', '2898989'))
See also MySQL parameterized queries
You forgot the ) after VALUES(age. Maybe that was just a typo you made transcribing the question into Stack Overflow. I've fixed it in the example above.
Your INSERT statement sets the height column, but it's not part of the tuple you insert. I removed it in the example above. If you want height in the UPDATE clause, then you need to include it in the tuple and pass a value in the parameters.
Also I put back-quotes around the table name, just in case the table name is a reserved keyword or contains a space or something.
I select 1 column from a table in a database. I want to iterate through each of the results. Why is it when I do this it’s a tuple instead of a single value?
con = psycopg2.connect(…)
cur = con.cursor()
stmt = "SELECT DISTINCT inventory_pkg FROM {}.{} WHERE inventory_pkg IS NOT NULL;".format(schema, tableName)
cur.execute(stmt)
con.commit()
referenced = cur.fetchall()
for destTbl in referenced:#why is destTbl a single element tuple?
print('destTbl: '+str(referenced))
stmt = "SELECT attr_name, attr_rule FROM {}.{} WHERE ppm_table_name = {};".format(schema, tableName, destTbl)#this fails because the where clause gets messed up because ‘destTbl’ has a comma after it
cur.execute(stmt)
Because that's what the db api does: always returns a tuple for each row in the result.
It's pretty simple to refer to destTbl[0] wherever you need to.
Because you are getting rows from your database, and the API is being consistent.
If your query asked for * columns, or a specific number of columns that is greater than 1, you'd also need a tuple or list to hold those columns for each row.
In other words, just because you only have one column in this query doesn't mean the API suddenly will change what kind of object it returns to model a row.
Simply always treat a row as a sequence and use indexing or tuple assignment to get a specific value out. Use:
inventory_pkg = destTbl[0]
or
inventory_pkg, = destTbl
for example.
i am new to sqlite and i think this question should have been answered before but i havent been able to find an answer.
i have a list of around 50 elements that i need to write to an sqlite database with 50 columns.
went over the documentation # https://docs.python.org/2/library/sqlite3.html but in the examples the values are specified by ? (so for writing 3 values, 3 ? are specified
sample code:
row_to_write = range(50)
conn = sqlite3.connect('C:\sample_database\sample_database')
c = conn.cursor()
tried these:
approach 1
c.execute("INSERT INTO PMU VALUES (?)", row_to_write)
ERROR: OperationalError: table PMU has 50 columns but 1 values were supplied
approach 2...tried writing a generator for iterating over list
def write_row_value_generator(row_to_write):
for val in row_to_write:
yield (val,)
c.executemany("INSERT INTO PMU VALUES (?)", write_row_value_generator(row_to_write))
ERROR: OperationalError: table PMU has 50 columns but 1 values were supplied
What is the correct way of doing this?
Assuming that your row_to_write has exactly the same number of items as PMU has columns, you can create a string of ? marks easily using str.join : ','.join(['?']*len(row_to_write))
import sqlite3
conn = sqlite3.connect(':memory:')
c = conn.cursor()
c.execute("create table PMU (%s)" % ','.join("col%d"%i for i in range(50)))
row_to_write = list(range(100,150,1))
row_value_markers = ','.join(['?']*len(row_to_write))
c.execute("INSERT INTO PMU VALUES (%s)"%row_value_markers, row_to_write)
conn.commit()
You need to specify the names of the columns. Sqlite will not guess those for you.
columns = ['A', 'B', 'C', ...]
n = len(row_to_write)
sql = "INSERT INTO PMU {} VALUES ({})".format(
', '.join(columns[:n]) , ', '.join(['?']*n))
c.execute(sql, row_to_write)
Note also that if your rows have a variable number of columns, then you might want to rethink your database schema. Usually each row should have a fixed number of columns, and the variability expresses itself in the number of rows inserted, not the number of columns used.
For example, instead of having 50 columns, perhaps you need just one extra column, whose value is one of 50 names (what used to be a column name). Each value in row_to_write would have its own row, and for each row you would have two columns: the value and the name of the column.
How would I add something in sqlite to an already existing table this is what I have so far
>>> rid
'26539249'
>>> for t in [(rid,("billy","jim"))]:
c.execute("insert into whois values (?,?)",t)
How would I add onto jim and create a list? or is there some way to add onto it so It can have multiple values?
I'll take a guess here, but I suspect I'm wrong.
You can't insert ("billy", "jim") as a column in the database. This is intentional. The whole point of RDBMSs like sqlite is that each field holds exactly one value, not a list of values. You can't search for 'jim' in the middle of a column shared with other people, you can't join tables based on 'jim', etc.
If you really, really want to do this, you have to pick some way to convert the multiple values into a single string, and to convert them back on reading. You can use json.dumps/json.loads, repr/ast.literal_eval, or anything else that seems appropriate. But you have to write the extra code yourself. And you won't be getting any real benefit out of the database if you do so; you'd be better off just using shelve.
So, I'm guessing you don't want to do this, and you want to know what you want to do instead.
Assuming your schema looks something like this:
CREATE TABLE whois (Rid, Names);
What you want is:
CREATE TABLE whois (Rid);
CREATE TABLE whois_names (Rid, Name, FOREIGN KEY(Rid) REFERENCES whois(Rid);
And then, to do the insert:
tt = [(rid,("billy","jim"))]
for rid, names in tt:
c.execute('INSERT INTO whois VALUES (?)', (rid,))
for name in names:
c.execute('INSERT INTO whois_names VALUES (?, ?)', (rid, name))
Or (probably faster, but not as interleaved):
c.executemany('INSERT INTO whois VALUES (?)', (rid for rid, names in tt))
c.executemany('INSERT INTO whois_names VALUES (?, ?),
(rid, name for rid, names in tt for name in names))
Not tested but should do the trick
conn = sqlite3.connect(db)
cur = conn.cursor()
cur.execute('''CREATE TABLE if not exists Data
(id integer primary key autoincrement, List)''')
cur.execute("INSERT INTO Data (id,List) values (?,?)",
(lid, str(map(lambda v : v, My_list) ) ))