I have run into this a few times now, where I'm trying to insert (or bulk insert) into a MySQL table using VALUES without defining the columns explicitly, but there is an auto_increment field I want to let auto_increment, or a generated column that I can't insert a value for.
Specifically, let's say I have a table with three columns, two for numbers and one generated column that's the sum of those numbers:
CREATE TABLE `addition` (
`num_1` int DEFAULT NULL,
`num_2` int DEFAULT NULL,
`sum` int GENERATED ALWAYS AS ((`num_1` + `num_2`)) VIRTUAL
)
If I want to insert values to this database with a MySQLdb cursor object cur, I can't do:
cur.execute('INSERT INTO addition VALUES %s', [(2, 2, 'DEFAULT')])
...because you can't define the value for the generated field "sum", and 'DEFAULT' here is interpreted as the literal string. You'll get MySQL error 3105: The value specified for generated column 'sum' in table 'addition' is not allowed.
But the same error occurs for any value I could think to put in place of 'DEFAULT', for example None or False.
So is there any way to pass a value in the data section (i.e. [(2, 2, <something>)]) to tell MySQL to use the default value for the sum column? Or is the only way to define it in the SQL itself, i.e.
cur.execute('INSERT INTO addition VALUES (%s, %s, DEFAULT)', [2, 2])
This would be helpful when the table structure isn't known, or is prone to change, and you don't want to hard-code which fields should insert as DEFAULT.
--Edit--
Some clarification post-discussion in the comments, if I were to try cur.execute('INSERT INTO addition VALUES %s', [(2, 2, 'DEFAULT')]), this tries to insert the literal string 'DEFAULT', similarly for None, or any other value I could think of. So the question is really a Python question, is there a field (e.g. MySQLdb.DEFAULT()) that I can pass to accomplish this. So the final result would look something like cur.execute('INSERT INTO addition VALUES %s', [(2, 2, MySQLdb.DEFAULT())])
If you want a computed sum column, than handle it on the database side via a generated column:
CREATE TABLE addition (
num_1 INT,
num_2 INT,
sum AS (num_1 + num_2)
);
Then, when you insert two numbers, MySQL will handle the math for you:
INSERT INTO addition (num_1, num2) VALUES (2, 2);
Note that generated columns in MySQL are virtual by default, meaning that the sum won't actually be persisted, by rather would happen at the time you do a select.
Related
I am trying to write a row of observations into my database, but I have some unique variable called list_variable which is a list of strings that can be of length 1-3. So sometimes ['string1'] but sometimes also ['string1','string2'] or ['string1','string2','string3'].
When I try to add this to my database by:
def add_to_cockroach_db():
cur.execute(f"""
INSERT INTO database (a, b, c)
VALUES ({time.time()}, {event},{list_variable}; <--- this one
""")
conn.commit()
I would get the following error (values have been changed for readability):
SyntaxError: at or near "[": syntax error
DETAIL: source SQL:
INSERT INTO database (a, b, c)
VALUES (a_value, b_value, ['c_value_1', 'c_value_2'])
^
HINT: try \h VALUES
It seems that having a variable that is a list is not allowed, how could I make this work out?
Thanks in advance!
**edit
list_variable looks e.g., like this = ['value1','value2']
You can either cast it to string using
str(['c_value_1', 'c_value_2'])
which looks like this:
"['c_value_1', 'c_value_2']"
or join the elements of your list with a delimiter you choose. This for example generates a comma separated string.
",".join(['c_value_1', 'c_value_2'])
which looks like this:
'c_value_1,c_value_2'
Like Maurice Meyer has already pointed out in the comments, it is better to pass your values as a list or as a tuple instead of formatting the query yourself.
Your command could look like this depending on the solution you choose:
cur.execute("INSERT INTO database (a, b, c) VALUES (%s, %s, %s)", (time.time(), event, ",".join(list_variable)))
There are a few ways you could accomplish this.
The simplest way is to call str on the list and insert the result into a string (VARCHAR) column. While this works, it's not easy to work with the values in database queries, and when it's retrieved from the database it's a string, not a list.
Using a VARCHAR[] column type - an array of string values - reflects the actual data type, and enables use of PostgreSQL's array functions in queries.
Finally, you could use a JSONB column type. This allows storage of lists or dicts, or nested combinations of both, so it's very flexible, and PostgreSQL provides functions for working with JSON objects too. However it might be overkill if you don't need the flexibility, or if you want to be strict about the data.
This script shows all three methods in action:
import psycopg2
from psycopg2.extras import Json
DROP = """DROP TABLE IF EXISTS t73917632"""
CREATE = """\
CREATE TABLE t73917632 (
s VARCHAR NOT NULL,
a VARCHAR[] NOT NULL,
j JSONB NOT NULL
)
"""
INSERT = """INSERT INTO t73917632 (s, a, j) VALUES (%s, %s, %s)"""
SELECT = """SELECT s, a, j FROM t73917632"""
v = ['a', 'b', 'c']
with psycopg2.connect(dbname='test') as conn:
with conn.cursor() as cur:
cur.execute(DROP)
cur.execute(CREATE)
conn.commit()
cur.execute(INSERT, (str(v), v, Json(v)))
conn.commit()
cur.execute(SELECT)
for row in cur:
print(row)
Output:
("['a', 'b', 'c']", ['a', 'b', 'c'], ['a', 'b', 'c'])
It's worth observing that if the array of strings represents some kind of child relationship to the table - for example the table records teams, and the string array contains the names of team members - it is usually a better design to insert each element in the array into a separate row in a child table, and associate them with the parent row using a foreign key.
I have created a python list with 41 columns and 50 rows.
Now I want to insert this into an SQLite database.
When I execute the database export, I got the error message:
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 41, and there are 40 supplied.
Most of the list fields should have data. Perhaps one or two don't have any.
Can I write into the sqlite database with a prompt like:
insert if data available, otherwise write none
Or something like this?
My code is like:
c.execute("""CREATE TABLE IF NOT EXISTS statstable (
spielid integer PRIMARY KEY,
41x field descr. (real, integer and text)
UNIQUE (spielid)
)
""")
c.executemany("INSERT OR REPLACE INTO statstable VALUES (41x ?)", all_data)
Append the appropriate number of None values to the nested lists to make them all 41 elements:
c.executemany("INSERT OR REPLACE INTO statstable VALUES (41x ?)",
[l + [None] * (41 - len(l)) for l in all_data])
This assumes the missing elements are always at the end of the list of columns. If they can be different columns in each row, I don't see how you can implement any automatic solution.
If the elements of all_data were dictionaries whose keys correspond to column names, you could determine which keys are missing. Then turn it into a list with the None placeholders in the appropriate places for those columns.
I create a table with primary key and autoincrement.
with open('RAND.xml', "rb") as f, sqlite3.connect("race.db") as connection:
c = connection.cursor()
c.execute(
"""CREATE TABLE IF NOT EXISTS race(RaceID INTEGER PRIMARY KEY AUTOINCREMENT,R_Number INT, R_KEY INT,\
R_NAME TEXT, R_AGE INT, R_DIST TEXT, R_CLASS, M_ID INT)""")
I want to then insert a tuple which of course has 1 less number than the total columns because the first is autoincrement.
sql_data = tuple(b)
c.executemany('insert into race values(?,?,?,?,?,?,?)', b)
How do I stop this error.
sqlite3.OperationalError: table race has 8 columns but 7 values were supplied
It's extremely bad practice to assume a specific ordering on the columns. Some DBA might come along and modify the table, breaking your SQL statements. Secondly, an autoincrement value will only be used if you don't specify a value for the field in your INSERT statement - if you give a value, that value will be stored in the new row.
If you amend the code to read
c.executemany('''insert into
race(R_number, R_KEY, R_NAME, R_AGE, R_DIST, R_CLASS, M_ID)
values(?,?,?,?,?,?,?)''',
sql_data)
you should find that everything works as expected.
From the SQLite documentation:
If the column-name list after table-name is omitted then the number of values inserted into each row must be the same as the number of columns in the table.
RaceID is a column in the table, so it is expected to be present when you're doing an INSERT without explicitly naming the columns. You can get the desired behavior (assign RaceID the next autoincrement value) by passing an SQLite NULL value in that column, which in Python is None:
sql_data = tuple((None,) + a for a in b)
c.executemany('insert into race values(?,?,?,?,?,?,?,?)', sql_data)
The above assumes b is a sequence of sequences of parameters for your executemany statement and attempts to prepend None to each sub-sequence. Modify as necessary for your code.
I select 1 column from a table in a database. I want to iterate through each of the results. Why is it when I do this it’s a tuple instead of a single value?
con = psycopg2.connect(…)
cur = con.cursor()
stmt = "SELECT DISTINCT inventory_pkg FROM {}.{} WHERE inventory_pkg IS NOT NULL;".format(schema, tableName)
cur.execute(stmt)
con.commit()
referenced = cur.fetchall()
for destTbl in referenced:#why is destTbl a single element tuple?
print('destTbl: '+str(referenced))
stmt = "SELECT attr_name, attr_rule FROM {}.{} WHERE ppm_table_name = {};".format(schema, tableName, destTbl)#this fails because the where clause gets messed up because ‘destTbl’ has a comma after it
cur.execute(stmt)
Because that's what the db api does: always returns a tuple for each row in the result.
It's pretty simple to refer to destTbl[0] wherever you need to.
Because you are getting rows from your database, and the API is being consistent.
If your query asked for * columns, or a specific number of columns that is greater than 1, you'd also need a tuple or list to hold those columns for each row.
In other words, just because you only have one column in this query doesn't mean the API suddenly will change what kind of object it returns to model a row.
Simply always treat a row as a sequence and use indexing or tuple assignment to get a specific value out. Use:
inventory_pkg = destTbl[0]
or
inventory_pkg, = destTbl
for example.
I'm using python 2.7.6 with sqlite 3.3.6.
I have a table with an INT column. When I select max() or min() over the column, some random number from the row is returned that is neither the min nor max. I've tried to reproduce this behavior, manually creating table and inserting the same values, min and max work perfectly. However, after my script, a manual query table gives weird results again.
Here is create table statement
CREATE TABLE customer_score ( DATE INT , SPAUBM INT , "CUSTOMER" TEXT, "SCORE" INT )
I use executemany method from sqlite to fill the table.
cursor.executemany("INSERT INTO customer_score VALUES ( ?, ?, ? , ? )", list_to_put)
Here is example of the row, that is put into table from file
4203 6JASTYMPT 987335
Example of the issue :
sqlite> select * from customer_score where customer='AAA' and date in (20140219);
20140219|9214|AAA|5017262
20140219|9213|AAA|3409363
20140219|9207|AAA|2288238
20140219|9208|AAA|809365
sqlite> select max(score) from customer_score where customer='AAA' and date in (20140219);
809365
sqlite> select min(score) from customer_score where customer='AAAL' and date in (20140219);
2288238
Could it be that executemany method screws up the INT data somehow before insertion into the table?
I have no idea how can I check this.
UPDATE
SQlite manual says that MIN() MAX() are shorcut to ORDER BY LIMIT 1. In my case , ORDER BY also does not work as expected. Rows position looks completely random.
sqlite> select * from customer_score where customer='AAA' and date in (20140218, 20140219) order by score desc ;
20140219|9208|AAA|809365
20140218|9208|AAA|629937
20140219|9214|AAA|5017262
20140218|9214|AAA|3911855
20140219|9213|AAA|3409363
20140219|9207|AAA|2288238
20140218|9213|AAA|2127092
20140218|9207|AAA|1489895
The values in the score column are not numbers but strings.
You have to ensure that the values in list_to_put are actually numbers.
(Please note that SQLite pretty much ignores the column data type; it does not matter whether you declare it as INT or TEXT or FLUFFY BUNNIES.)