How would I add something in sqlite to an already existing table this is what I have so far
>>> rid
'26539249'
>>> for t in [(rid,("billy","jim"))]:
c.execute("insert into whois values (?,?)",t)
How would I add onto jim and create a list? or is there some way to add onto it so It can have multiple values?
I'll take a guess here, but I suspect I'm wrong.
You can't insert ("billy", "jim") as a column in the database. This is intentional. The whole point of RDBMSs like sqlite is that each field holds exactly one value, not a list of values. You can't search for 'jim' in the middle of a column shared with other people, you can't join tables based on 'jim', etc.
If you really, really want to do this, you have to pick some way to convert the multiple values into a single string, and to convert them back on reading. You can use json.dumps/json.loads, repr/ast.literal_eval, or anything else that seems appropriate. But you have to write the extra code yourself. And you won't be getting any real benefit out of the database if you do so; you'd be better off just using shelve.
So, I'm guessing you don't want to do this, and you want to know what you want to do instead.
Assuming your schema looks something like this:
CREATE TABLE whois (Rid, Names);
What you want is:
CREATE TABLE whois (Rid);
CREATE TABLE whois_names (Rid, Name, FOREIGN KEY(Rid) REFERENCES whois(Rid);
And then, to do the insert:
tt = [(rid,("billy","jim"))]
for rid, names in tt:
c.execute('INSERT INTO whois VALUES (?)', (rid,))
for name in names:
c.execute('INSERT INTO whois_names VALUES (?, ?)', (rid, name))
Or (probably faster, but not as interleaved):
c.executemany('INSERT INTO whois VALUES (?)', (rid for rid, names in tt))
c.executemany('INSERT INTO whois_names VALUES (?, ?),
(rid, name for rid, names in tt for name in names))
Not tested but should do the trick
conn = sqlite3.connect(db)
cur = conn.cursor()
cur.execute('''CREATE TABLE if not exists Data
(id integer primary key autoincrement, List)''')
cur.execute("INSERT INTO Data (id,List) values (?,?)",
(lid, str(map(lambda v : v, My_list) ) ))
Related
rows_order = "SELECT COUNT (*) FROM 'Order'"
cursor.execute(rows_order)
ordernum = cursor.fetchall()
connection.commit()
cursor.execute("INSERT INTO 'Order' (OrderNo, CustomerID, Date, TotalCost) VALUES (?,?,?,?)", (
[ordernum], custid_Sorder, now, total_item_price))
This is what I am trying but this error popped up;
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
How do I fix this? I want to make it so the OrderNo is = to the amount of orders before it, hence why I want to assign the orderno to it. (I am using sqlite3)
as you have only one value you need only fetchone
import sqlite3
con = sqlite3.connect("tutorial.db")
cursor = con.cursor()
rows_order = "SELECT COUNT (*) FROM 'Order'"
cursor.execute(rows_order)
ordernum = cursor.fetchone()[0]
cursor.execute("INSERT INTO 'Order' (OrderNo, CustomerID, Date, TotalCost) VALUES (?,?,?,?)", (
ordernum, custid_Sorder, now, total_item_price))
tl;dr Don't do this. Use an auto-incremented primary key.
fetchall returns all rows as a list, even if there is only one row.
Instead, use fetchone. This will return a single tuple which you can then select the first item. ordernum = cursor.fetchone()[0]
However, you appear to be writing a query to get the next ID. Using count(*) is wrong. If there are any gaps in OrderNo, for example if something gets deleted, it can return a duplicate. Consider [1, 3, 4]; count(*) will return 3. Use max(OrderNo) instead.
Furthermore, if you try to insert two orders at the same time you might get a race condition and one will try to duplicate the other.
process 1 process 2
select max(orderNo)
fetchone # 4
select max(orderNo)
fetchone # 4
insert into orders...
insert into orders... # duplicate OrderNo
To avoid this, you have to do both the select and insert in a transaction.
process 1 process 2
begin
select max(orderNo)...
fetchone # 4 begin
select max(orderNo)
fetchone
insert into orders... # wait
commit # wait
# 5
insert into orders...
commit
Better yet, do them as a single query.
insert into "Order" (OrderNo, CustomerID, Date, TotalCost)
select max(orderNo), ?, ?, ?
from "order"
Even better don't do it at all. There is a built-in mechanism to do this use an auto-incremented primary keys.
-- order is a keyword, pluralizing table names helps to avoid them
create table orders (
-- It is a special feature of SQLite that this will automatically be unique.
orderNo integer primary key
customerID int,
-- date is also a keyword, and vague. Use xAt.
orderedAt timestamp,
totalCost int
)
-- orderNo will automatically be set to a unique number
insert into orders (customerID, orderedAt, totalCost) values (...)
I am trying to write a row of observations into my database, but I have some unique variable called list_variable which is a list of strings that can be of length 1-3. So sometimes ['string1'] but sometimes also ['string1','string2'] or ['string1','string2','string3'].
When I try to add this to my database by:
def add_to_cockroach_db():
cur.execute(f"""
INSERT INTO database (a, b, c)
VALUES ({time.time()}, {event},{list_variable}; <--- this one
""")
conn.commit()
I would get the following error (values have been changed for readability):
SyntaxError: at or near "[": syntax error
DETAIL: source SQL:
INSERT INTO database (a, b, c)
VALUES (a_value, b_value, ['c_value_1', 'c_value_2'])
^
HINT: try \h VALUES
It seems that having a variable that is a list is not allowed, how could I make this work out?
Thanks in advance!
**edit
list_variable looks e.g., like this = ['value1','value2']
You can either cast it to string using
str(['c_value_1', 'c_value_2'])
which looks like this:
"['c_value_1', 'c_value_2']"
or join the elements of your list with a delimiter you choose. This for example generates a comma separated string.
",".join(['c_value_1', 'c_value_2'])
which looks like this:
'c_value_1,c_value_2'
Like Maurice Meyer has already pointed out in the comments, it is better to pass your values as a list or as a tuple instead of formatting the query yourself.
Your command could look like this depending on the solution you choose:
cur.execute("INSERT INTO database (a, b, c) VALUES (%s, %s, %s)", (time.time(), event, ",".join(list_variable)))
There are a few ways you could accomplish this.
The simplest way is to call str on the list and insert the result into a string (VARCHAR) column. While this works, it's not easy to work with the values in database queries, and when it's retrieved from the database it's a string, not a list.
Using a VARCHAR[] column type - an array of string values - reflects the actual data type, and enables use of PostgreSQL's array functions in queries.
Finally, you could use a JSONB column type. This allows storage of lists or dicts, or nested combinations of both, so it's very flexible, and PostgreSQL provides functions for working with JSON objects too. However it might be overkill if you don't need the flexibility, or if you want to be strict about the data.
This script shows all three methods in action:
import psycopg2
from psycopg2.extras import Json
DROP = """DROP TABLE IF EXISTS t73917632"""
CREATE = """\
CREATE TABLE t73917632 (
s VARCHAR NOT NULL,
a VARCHAR[] NOT NULL,
j JSONB NOT NULL
)
"""
INSERT = """INSERT INTO t73917632 (s, a, j) VALUES (%s, %s, %s)"""
SELECT = """SELECT s, a, j FROM t73917632"""
v = ['a', 'b', 'c']
with psycopg2.connect(dbname='test') as conn:
with conn.cursor() as cur:
cur.execute(DROP)
cur.execute(CREATE)
conn.commit()
cur.execute(INSERT, (str(v), v, Json(v)))
conn.commit()
cur.execute(SELECT)
for row in cur:
print(row)
Output:
("['a', 'b', 'c']", ['a', 'b', 'c'], ['a', 'b', 'c'])
It's worth observing that if the array of strings represents some kind of child relationship to the table - for example the table records teams, and the string array contains the names of team members - it is usually a better design to insert each element in the array into a separate row in a child table, and associate them with the parent row using a foreign key.
I have the following code in python to update db where the first column is "id" INTEGER PRIMARY KEY AUTOINCREMENT UNIQUE:
con = lite.connect('test_score.db')
with con:
cur = con.cursor()
cur.execute("INSERT INTO scores VALUES (NULL,?,?,?)", (first,last,score))
item = cur.fetchone()
on.commit()
cur.close()
con.close()
I get table "scores" with following data:
1,Adam,Smith,68
2,John,Snow,76
3,Jim,Green,88
Two different users (userA and userB) copy test_score.db and code to their computer and use it separately.
I get back two db test_score.db but now with different content:
user A test_score.db :
1,Adam,Smith,68
2,John,Snow,76
3,Jim,Green,88
4,Jim,Green,91
5,Tom,Hanks,15
user A test_score.db :
1,Adam,Smith,68
2,John,Snow,76
3,Jim,Green,88
4,Chris,Prat,99
5,Tom,Hanks,09
6,Tom,Hanks,15
I was trying to use
insert into AuditRecords select * from toMerge.AuditRecords;
to combine two db into one but failed as the first column is a unique id. Two db have now the same ids but with different or the same data and merging is failing.
I would like to find unique rows in both db (all values different ignoring id) and merge results to one full db.
Result should be something like this:
1,Adam,Smith,68
2,John,Snow,76
3,Jim,Green,88
4,Jim,Green,91
5,Tom,Hanks,15
6,Chris,Prat,99
7,Tom,Hanks,09
I can extract each value one by one and compare but want to avoid it as I might have longer rows in the future with more columns.
Sorry if it is obvious and easy questions, I'm still learning. I tried to find the answer but failed, please point me to answer if it already exists somewhere else. Thank you very much for your help.
You need to define the approach to resolve duplicated rows. Will consider the max score? The min? The first one?
Considering the table AuditRecords has all the lines of both User A and B, you can use GROUP BY to deduplicate rows and use an aggregation function to resolve the score:
insert into
AuditRecords
select
id,
first_name,
last_name,
max(score) as score
from
toMerge.AuditRecords
group by
id,
first_name,
last_name;
For this requirement you should have defined a UNIQUE constraint for the combination of the columns first, last and score:
CREATE TABLE AuditRecords(
id INTEGER PRIMARY KEY AUTOINCREMENT,
first TEXT,
last TEXT,
score INTEGER,
UNIQUE(first, last, score)
);
Now you can use INSERT OR IGNORE to merge the tables:
INSERT OR IGNORE INTO AuditRecords(first, last, score)
SELECT first, last, score
FROM toMerge.AuditRecords;
Note that you must explicitly define the list of the columns that will receive the values and in this list the id is missing because its value will be autoincremented by each insertion.
Another way to do it without defining the UNIQUE constraint is to use EXCEPT:
INSERT INTO AuditRecords(first, last, score)
SELECT first, last, score FROM toMerge.AuditRecords
EXCEPT
SELECT first, last, score FROM AuditRecords
From my list, I am looking to insert row by row, but I need to perform a checking before allowing to insert in the database.
let's imagine that is my list
unique_hrefs = [
https://www.linkedin.com/in/123456,
https://www.linkedin.com/in/789013556,
https://www.linkedin.com/in/888888888,
https://www.linkedin.com/in/082b62112,
https://www.linkedin.com/in/5625a1a,
https://www.linkedin.com/in/123456,
https://www.linkedin.com/in/0000000341454,
https://www.linkedin.com/in/55555555,
https://www.linkedin.com/in/55555555,
https://www.linkedin.com/in/66666666,
https://www.linkedin.com/in/666677777
]
I need to check my table for the same string before allowing to inserting and if the string does not exist inserting.
I have my code to inset but struggling to find how to check before inserting it?
query= "INSERT INTO name_links (ulr_name) VALUES (%s)
cursor.executemany(query,[(r,) for r in unique_hrefs])
mydb.commit()
I have a set of data in a list, embedded into a dictionary as:
{'list1': ['Freddy','36','fred','123f','2017/04/25'],
'list2':['Tara','25','mtara','123t','2018/03/22']}
Ref notations:
{ key1: [name, age,nickname, userid, account_created_date],
..key2:[name, age,nickname, userid, account_created_date] }
All the data is inserted in variables in a Python function, one for each, as described above. When I call the function I would get the output right-away as
Output:
Freddy
Tara
But when I try to insert the data into a sqlite database, I get the output in the following manner:
Output:
F
R
E
D
D
Y
T
A
R
A
Code:
conn = sqlite3.connect(dbPath)
cur = conn.cursor()
results = cur.executemany("INSERT INTO Field (Field1) VALUES (?)", name)
conn.commit()
Requesting your assistance on this issue. Your help is much appreciated. Thank You.
New Answer
To suit your dataset more directly. You need a list of dictionaries, not a dictionary of lists because each dictionary has the column name as the key and a value associated with that column (key).
names = [{'name': "Freddy"}, {'name': "Joan"}]
results = c.executemany("INSERT INTO names ('name') VALUES (:name)", names)
The rest of the SQLite3 code working, this outputs:
Freddy
Joan
Option to Use a List of Tuples
You can also use a list of tuples. If you're not using a dictionary, you need to make sure each row is in tuple form. You need each of these tuples in a list. You need to call this list in the executemany function after the SQL query.
Essential code:
names = [("Freddy",), ("Joan",)]
results = c.executemany("INSERT INTO names (name) VALUES (?)", names)
Output:
Freddy
Joan
So make sure each entry is in tuple or dictionary form. Then put these tuples or dictionaries into a list. If it's in dictionary form, you cannot use the '?' placeholder. Use ':key_name' for each placeholder. (There might be a different option here, but I haven't found it yet).
Original Response
I suspect that executemany(...) should be execute(...). I haven't used SQLite3 in a while, so I'll test it now to make sure and then get back to you.
Update 1:
I remember the "?'s" needing to be sent in tuple form. I.e.
cur.executemany("INSERT INTO Field (Field1) VALUES (?)", name)
needs to be at least
cur.executemany("INSERT INTO Field (Field1) VALUES (?)", (name, ...))
But I've only gotten the execute() (and not executemany()) to work, with this, else it's back to the same error. So, here's my working code so far:
cur.execute("INSERT INTO Field (Field1) VALUES (?)", (name,))
It gave F, R, E, D, D, Y because it was reading each letter as values in the tuple. If it was in tuple form, it would read "Freddy" all in one go. But I'm still having issues with the executemany(), which I think works differently than we think it does.
Update 2:
Here's what I got to work. You need to make sure each row is in tuple form. You need each of these tuples in a list. You need to call this list in the executemany function after the SQL query.
Essential code:
names = [("Freddy",), ("Joan",)]
results = c.executemany("INSERT INTO names (name) VALUES (?)", names)
Full code:
import sqlite3
conn = sqlite3.connect('test.db')
c = conn.cursor()
# Create table
c.execute('''CREATE TABLE IF NOT EXISTS names
(name text)''')
names = [("Freddy",), ("Joan",)]
results = c.executemany("INSERT INTO names (name) VALUES (?)", names)
conn.commit()
for row in c.execute('SELECT * FROM names'):
print(row[0])
This prints out:
Freddy
Joan
Update 3:
I've updated it to suit your dataset more directly. You need a list of dictionaries, not a dictionary of lists because each dictionary has the column name as the key and a value associated with that column (key).
Here's some more code, I've gotten to work.
names = [{'name': "Freddy"}, {'name': "Joan"}]
results = c.executemany("INSERT INTO names ('name') VALUES (:name)", names)
With the rest of the code as it was, it outputs:
Freddy
Joan
(Inspired from this answer.)
you might consider to change
results = cur.executemany("INSERT INTO Field (Field1) VALUES (?)", name)
to
results = cur.executemany("INSERT INTO Field (Field1) VALUES (?)", (name))
since the expected parameter is a tuple, the string is not considered as a whole but splited