I have a list called
fullpricelist=[373.97, 381.0, 398.98, 402.98, 404.98, 457.97, 535.99, 550.97, 566.98]
I would like to write this list into a slqlite database column, I found the following code from another question and changed it to my situation.
cursor.executemany("""INSERT INTO cardata (fullprice) VALUES (?)""",
zip(fullpricelist))
My current script is this
for name, name2, image in zip(car_names, car_names2, images):
cursor.execute(
"insert into cardata (carname, carmodel, imageurl, location, Fro, T, companyid) values (?, ?, ?, ?, ?, ?, ?)",
(name.text, name2.text, image.get_attribute('src'), location, pickup_d, return_d, Rental_ID)
)
But now I am confused how to add these codes together
In your second piece of code, execute() is called and one specific object is stored in the database each loop iteration. This is slow and inefficient.
for price in fullpricelist:
cursor.execute("""INSERT INTO cardata (fullprice) VALUES (?)""", price)
executemany() reads from an iterable and adds each element of the iterable to the database as a distinct row. If you add many elements to a database and care about efficiency, you want to use executemany()
cursor.executemany("""INSERT INTO cardata (fullprice) VALUES (?)""", fullpricelist)
If you want to include the other columns in your question, your code will be
cursor.executemany("""INSERT INTO cardata (carname, carmodel, imageurl, location, Fro, T, companyid) values (?, ?, ?, ?, ?, ?, ?)""",
[
[name.text for name in car_names],
[name.text for name in car_names2],
[image.get_attribute('src') for image in images],
[location]*len(car_names),
[pickup_d]*len(car_names),
[return_d]*len(car_names),
[Rental_ID]*len(car_names)
]
)
This assumes all values for location, pickup_d, return_d and Rental_ID are the same, as you did not provide a list of the values.
Related
I am trying to fetch all the duplicates from tokens and load into tokn_Duplicates table. Getting Type error as below, if the sequence/format of the values passed to the insert query is causing this. Tried different options from the forum to pass the values as tuple/list, havent been successful
Code snippet below and error
with conn.cursor() as cur:
cur.execute("""
select case when tokn_type_cd IS NULL Then 'Address' else tokn_type_cd end as tokn_type_cd,case when tokn_val_tx is null then '0' else tokn_val_tx end tokn_val_tx,case when pg_duplicate_ct is null then 0 else pg_duplicate_ct end pg_duplicate_ct from ((SELECT 0 DUMMY ) DM LEFT OUTER JOIN (SELECT tokn_type_cd, tokn_val_tx, COUNT(*) AS pg_duplicate_ct
FROM devt.tokens
GROUP BY tokn_type_cd, tokn_val_tx
HAVING COUNT(*) > 1) on 1=1)
""")
rows2 = cur.fetchall()
cnxn = pyodbc.connect(server='connect to server')
crsr = cnxn.cursor()
if len(rows2):
crsr.executemany("INSERT INTO duplicates (tokn_type_cd, tokn_val_tx, pg_duplicate_ct) VALUES (?, ?, ?)", [str(r) for r in rows2])
cnxn.commit()
print("... Successfully completed...")
Error received below
crsr.executemany(“INSERT INTO duplicates (tokn_type_cd, tokn_val_tx, pg_duplicate_ct) VALUES (?, ?, ?)“, [str(r) for r in rows2])
TypeError: (‘Params must be in a list, tuple, or Row’, ‘HY000’)
Tried the below change from the forum:
crsr.executemany(“INSERT INTO duplicates (tokn_type_cd, tokn_val_tx, pg_duplicate_ct) VALUES (?, ?, ?)“, [str(tokn_type_cd), str(tokn_val_tx) ,pg_duplicate_ct])
received the error below after the change
crsr.executemany(“INSERT INTO secret_tokn_duplicates (tokn_type_cd, tokn_val_tx, pg_duplicate_ct) VALUES (?, ?, ?)“, [str(tokn_type_cd), str(tokn_val_tx) ,pg_duplicate_ct])
NameError: name ‘tokn_type_cd’ is not defined
Any help would be appreciated.
Thanks
rows2 is the result of a .fetchall() which means that it is a list of pyodbc.Row objects. Such a list is suitable without modification for use as parameters for a subsequent .executemany() operation. In other words, don't mess with rows2; just pass it to the .executemany() for your INSERT:
if len(rows2):
crsr.executemany(
"INSERT INTO duplicates (tokn_type_cd, tokn_val_tx, pg_duplicate_ct) VALUES (?, ?, ?)",
rows2
)
cnxn.commit()
Your initial attempt failed because [str(r) for r in rows2] returns a list of string objects (scalars) instead of a list of Row objects.
Try
crsr.executemany(
"INSERT INTO secret_tokn_duplicates (tokn_type_cd, tokn_val_tx, pg_duplicate_ct) VALUES (?, ?, ?)",
[(str(row.tokn_type_cd), str(row.tokn_val_tx), row.pg_duplicate_ct) for row in rows2])
or if that doesn't work:
crsr.executemany(
"INSERT INTO secret_tokn_duplicates (tokn_type_cd, tokn_val_tx, pg_duplicate_ct) VALUES (?, ?, ?)",
[(str(row[0]), str(row[1]), row[2]) for row in rows2])
The list [(str(row.tokn_type_cd), str(row.tokn_val_tx), row.pg_duplicate_ct) for row in rows2] creates a list of 3-tuples from the rows in rows2. I don't have access to your database, so I can't test that it works, but it's my best guess at will work given what's in your question.
Your first attempt
crsr.executemany("INSERT INTO duplicates (tokn_type_cd, tokn_val_tx, pg_duplicate_ct) VALUES (?, ?, ?)", [str(r) for r in rows2])
didn't work because the list you passed to crsr.executemany contained strings, which were what you got if you attempted to convert each row read from the cursor to a string. The elements in the list passed to crsr.executemany must be lists, tuples or Rows: you need to pass it a list of lists, a list of tuples or a list of Rows.
Your second attempt
crsr.executemany("INSERT INTO duplicates (tokn_type_cd, tokn_val_tx, pg_duplicate_ct) VALUES (?, ?, ?)", [str(tokn_type_cd), str(tokn_val_tx) ,pg_duplicate_ct])
didn't work because tokn_type_cd, tokn_val_tx and pg_duplicate_ct aren't variables that Python knows about. However, even if you had defined them, you would again end up passing a list of strings to crsr.executemany and would get the first error again.
However, having said all the above, most RDBMSs support INSERT INTO ... SELECT ... statements, which will copy the data directly, so why not just execute the following?
cur.execute("""
INSERT INTO duplicates (tokn_type_cd, tokn_val_tx, pg_duplicate_ct)
select case when tokn_type_cd IS NULL Then 'Address' else tokn_type_cd end as tokn_type_cd,case when tokn_val_tx is null then '0' else tokn_val_tx end tokn_val_tx,case when pg_duplicate_ct is null then 0 else pg_duplicate_ct end pg_duplicate_ct from ((SELECT 0 DUMMY ) DM LEFT OUTER JOIN (SELECT tokn_type_cd, tokn_val_tx, COUNT(*) AS pg_duplicate_ct
FROM devt.tokens
GROUP BY tokn_type_cd, tokn_val_tx
HAVING COUNT(*) > 1) on 1=1)
""")
From this answer:
cursor.execute("INSERT INTO booking_meeting (room_name,from_date,to_date,no_seat,projector,video,created_date,location_name) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", (rname, from_date, to_date, seat, projector, video, now, location_name ))
I'd like to shorten it to something like:
simple_insert(booking_meeting, rname, from_date, to_date, seat, projector, video, now, location_name)
The first parameter is the table name which can be read to get list of column names to format the first section of the SQLite3 statement:
cursor.execute("INSERT INTO booking_meeting (room_name,from_date,to_date,no_seat,projector,video,created_date,location_name)
Then the values clause (second part of the insert statement):
VALUES (?, ?, ?, ?, ?, ?, ?, ?)"
can be formatted by counting the number of column names in the table.
I hope I explained the question properly and you can appreciate the time savings of such a function. How to write this function in python? ...is my question.
There may already a simple_insert() function in SQLite3 but I just haven't stumbled across it yet.
If you're inserting into all the columns, then you don't need to specify the column names in the INSERT query. For that scenario, you could write a function like this:
def simple_insert(cursor, table, *args):
query = f'INSERT INTO {table} VALUES (' + '?, ' * (len(args)-1) + '?)'
cursor.execute(query, args)
For your example, you would call it as:
simple_insert(cursor, 'booking_meeting', rname, from_date, to_date, seat, projector, video, now, location_name)
Note I've chosen to pass cursor to the function, you could choose to just rely on it as a global variable instead.
I've got an MS Access table (SearchAdsAccountLevel) which needs to be updated frequently from a python script. I've set up the pyodbc connection and now I would like to UPDATE/INSERT rows from my pandas df to the MS Access table based on whether the Date_ AND CampaignId fields match with the df data.
Looking at previous examples I've built the UPDATE statement which uses iterrows to iterate through all rows within df and execute the SQL code as per below:
connection_string = (
r"Driver={Microsoft Access Driver (*.mdb, *.accdb)};"
r"c:\AccessDatabases\Database2.accdb;"
)
cnxn = pyodbc.connect(connection_string, autocommit=True)
crsr = cnxn.cursor()
for index, row in df.iterrows():
crsr.execute("UPDATE SearchAdsAccountLevel SET [OrgId]=?, [CampaignName]=?, [CampaignStatus]=?, [Storefront]=?, [AppName]=?, [AppId]=?, [TotalBudgetAmount]=?, [TotalBudgetCurrency]=?, [DailyBudgetAmount]=?, [DailyBudgetCurrency]=?, [Impressions]=?, [Taps]=?, [Conversions]=?, [ConversionsNewDownloads]=?, [ConversionsRedownloads]=?, [Ttr]=?, [LocalSpendAmount]=?, [LocalSpendCurrency]=?, [ConversionRate]=?, [Week_]=?, [Month_]=?, [Year_]=?, [Quarter]=?, [FinancialYear]=?, [RowUpdatedTime]=? WHERE [Date_]=? AND [CampaignId]=?",
row['OrgId'],
row['CampaignName'],
row['CampaignStatus'],
row['Storefront'],
row['AppName'],
row['AppId'],
row['TotalBudgetAmount'],
row['TotalBudgetCurrency'],
row['DailyBudgetAmount'],
row['DailyBudgetCurrency'],
row['Impressions'],
row['Taps'],
row['Conversions'],
row['ConversionsNewDownloads'],
row['ConversionsRedownloads'],
row['Ttr'],
row['LocalSpendAmount'],
row['LocalSpendCurrency'],
row['ConversionRate'],
row['Week_'],
row['Month_'],
row['Year_'],
row['Quarter'],
row['FinancialYear'],
row['RowUpdatedTime'],
row['Date_'],
row['CampaignId'])
crsr.commit()
I would like to iterate through each row within my df (around 3000) and if the ['Date_'] AND ['CampaignId'] match I UPDATE all other fields. Otherwise I want to INSERT the whole df row in my Access Table (create new row). What's the most efficient and effective way to achieve this?
Consider DataFrame.values and pass list into an executemany call, making sure to order columns accordingly for the UPDATE query:
cols = ['OrgId', 'CampaignName', 'CampaignStatus', 'Storefront',
'AppName', 'AppId', 'TotalBudgetAmount', 'TotalBudgetCurrency',
'DailyBudgetAmount', 'DailyBudgetCurrency', 'Impressions',
'Taps', 'Conversions', 'ConversionsNewDownloads', 'ConversionsRedownloads',
'Ttr', 'LocalSpendAmount', 'LocalSpendCurrency', 'ConversionRate',
'Week_', 'Month_', 'Year_', 'Quarter', 'FinancialYear',
'RowUpdatedTime', 'Date_', 'CampaignId']
sql = '''UPDATE SearchAdsAccountLevel
SET [OrgId]=?, [CampaignName]=?, [CampaignStatus]=?, [Storefront]=?,
[AppName]=?, [AppId]=?, [TotalBudgetAmount]=?,
[TotalBudgetCurrency]=?, [DailyBudgetAmount]=?,
[DailyBudgetCurrency]=?, [Impressions]=?, [Taps]=?, [Conversions]=?,
[ConversionsNewDownloads]=?, [ConversionsRedownloads]=?, [Ttr]=?,
[LocalSpendAmount]=?, [LocalSpendCurrency]=?, [ConversionRate]=?,
[Week_]=?, [Month_]=?, [Year_]=?, [Quarter]=?, [FinancialYear]=?,
[RowUpdatedTime]=?
WHERE [Date_]=? AND [CampaignId]=?'''
crsr.executemany(sql, df[cols].values.tolist())
cnxn.commit()
For the insert, use a temp, staging table with exact structure as final table which you can create with make-table query: SELECT TOP 1 * INTO temp FROM final. This temp table will be regularly cleaned out and inserted with all data frame rows. The final query migrates only new rows from temp into final with NOT EXISTS, NOT IN, or LEFT JOIN/NULL. You can run this query anytime and never worry about duplicates per Date_ and CampaignId columns.
# CLEAN OUT TEMP
sql = '''DELETE FROM SearchAdsAccountLevel_Temp'''
crsr.executemany(sql)
cnxn.commit()
# APPEND TO TEMP
sql = '''INSERT INTO SearchAdsAccountLevel_Temp (OrgId, CampaignName, CampaignStatus, Storefront,
AppName, AppId, TotalBudgetAmount, TotalBudgetCurrency,
DailyBudgetAmount, DailyBudgetCurrency, Impressions,
Taps, Conversions, ConversionsNewDownloads, ConversionsRedownloads,
Ttr, LocalSpendAmount, LocalSpendCurrency, ConversionRate,
Week_, Month_, Year_, Quarter, FinancialYear,
RowUpdatedTime, Date_, CampaignId)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?);'''
crsr.executemany(sql, df[cols].values.tolist())
cnxn.commit()
# MIGRATE TO FINAL
sql = '''INSERT INTO SearchAdsAccountLevel
SELECT t.*
FROM SearchAdsAccountLevel_Temp t
LEFT JOIN SearchAdsAccountLevel f
ON t.Date_ = f.Date_ AND t.CampaignId = f.CampaignId
WHERE f.OrgId IS NULL'''
crsr.executemany(sql)
cnxn.commit()
When using:
import datetime
import sqlite3
db = sqlite3.connect('mydb.sqlite', detect_types=sqlite3.PARSE_DECLTYPES)
c = db.cursor()
db.text_factory = str
c.execute('create table if not exists mytable (date timestamp, title str, \
custom str, x float, y float, z char default null, \
postdate timestamp default null, id integer primary key autoincrement, \
url text default null)')
c.execute('insert into mytable values(?, ?, ?, ?, ?)', \
(datetime.datetime(2018,4,23,23,00), 'Test', 'Test2', 2.1, 11.1))
I have:
sqlite3.OperationalError: table mytable has 9 columns but 5 values were supplied
Why doesn't SQlite take default values (specified during table creation) in consideration to populate a new row?
(Also, as I'm reopening a project I wrote a few years ago, I don't find the datatypes str, char anymore in the sqlite3 doc, is it still relevant?)
Because you are saying that you want to insert all columns by not specifying the specific columns.
Change 'insert into mytable values(?, ?, ?, ?, ?)'
to 'insert into mytable (date, title, custom, x, y) values(?, ?, ?, ?, ?)'
Virtually any value for column type can be specified, the value will follow a set of rules and be converted to TEXT, INTEGER, REAL, NUMERIC or BLOB. However, you can store any type of value in any column.
STR will resolve to NUMERIC,
TIMESTAMP will resolve to NUMERIC,
FLOAT will resolve to REAL,
CHAR to TEXT.
Have a read of Datatypes In SQLite or perhaps have a look at How flexible/restricive are SQLite column types?
If you're going to only supply values for some columns, you need to specify which columns. Otherwise the engine won't know where to put them. This line needs to be changed:
c.execute('insert into mytable values(?, ?, ?, ?, ?)', \
(datetime.datetime(2018,4,23,23,00), 'Test', 'Test2', 2.1, 11.1))
To this:
c.execute('insert into mytable (date, title, custom, x, y)values(?, ?, ?, ?, ?)', \
(datetime.datetime(2018,4,23,23,00), 'Test', 'Test2', 2.1, 11.1))
Example Solution
cursor.execute('CREATE TABLE vehicles_record (id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT, timestamp DATETIME DEFAULT CURRENT_TIMESTAMP)')
SQLite3 Query Above
cursor.execute("INSERT INTO vehicles_record(name) VALUES(?)", (name))
Result
id would be 1, name would be value of name var, current timestamp for last column.
Is there a way I can assign an SQLite rowid through a variable in Python? I am using Tkinters 'get()' function to retrieve the contents of the entry.
Here is the code:
def insertdata():
c.execute("INSERT INTO Students VALUES (?, ?, ?, ?, ?, ?, ?, ?);", (surnamelabelentry.get(), forenamelabelentry.get(), dateofbirthlabelentry.get(), homeaddresslabelentry.get(), homephonenumberentry.get(), genderlabelentry.get(), tutorgrouplabelentry.get(), emaillabelentry.get()))
c.execute('INSERT INTO Students(rowid) VALUES',(studentidentry.get()))
conn.commit()
rootE.destroy()
Here is the error:
File "R:/Documents/PYTHON/Login And Pw.py", line 124, in insertdata
c.execute('INSERT INTO Students(rowid) VALUES',(studentidentry.get()))
sqlite3.OperationalError: near "VALUES": syntax error
Thanks in advance.
When creating your table, declare one column as INTEGER PRIMARY KEY.
CREATE TABLE Students (
StudentID INTEGER PRIMARY KEY,
Surname VARCHAR NOT NULL,
Forename VARCHAR NOT NULL,
...
);
This makes that column an alias for the ROWID, and you can then use a normal INSERT statement to set it.
If for some bizarre reason you want to keep ROWID as a hidden column but still set an explicit value for it, then you can use an explicit column list for INSERT.
c.execute("INSERT INTO Students(ROWID, Surname, Forename) VALUES (?, ?, ?)", (5678, 'Tables', 'Robert'))