I want to copy all values that has an ID that is contained in a python list into a new table in SQLite like this:
cur.execute("INSERT INTO newtable SELECT * FROM oldtable where userid IN " + userids)
Userids is a list that comes from a file:
userids = [line.strip() for line in open('inputfile.txt')]
But I get the following error:
TypeError: cannot concatenate 'str' and 'list' objects
And I have the growing suspicion that this list with about 15000 elements would be too long for the query as well(?). How would I do this without querying once for each id in the list?
Try something like:
join the list in a string.
userIdsStr = ', '.join(userids)
Then do something like:
cur.execute('INSERT INTO newtable SELECT * FROM oldtable where userid IN (%s)' % userIdsStr)
I hope this helps
Related
I'm using the below Python script to loop around a SQL Server table and save the values to a list.
However, I don't want the brackets \ commas etc. When I run the below the records in the list look like this:
('TN12345', )
I just want the below - how it exists in the SQL table:
TN12345
This is my script:
import pyodbc
connstr = 'DRIVER={SQL Server};SERVER=XXXXXXXXXXX;DATABASE=XXXXXXXXXXX;Trusted_Connection=yes;'
conn = pyodbc.connect(connstr)
cursor = conn.cursor()
cursor.execute("""SELECT TN_Number FROM Table""")
records = cursor.fetchall()
insertObject = []
columnNames = [column[0] for column in cursor.description]
for record in records:
insertObject.append( dict( zip( columnNames , record ) ) )
for i in range(len(records)):
print(records[i])
This
('TN12345', )
is tuple with one element, representing row in database, if you can guarantee that your query will always return exactly 1 column then you might use [0] to access single element in said tuple, in your case replace
for i in range(len(records)):
print(records[i])
using
for i in range(len(records)):
print(records[i][0])
alternatively you might use for loop directly (without caring about index) in following way
for record in records:
print(record[0])
If you want to get all elements one by one in your tuples:
for i in range(len(records)):
for j in records[i]:
print(j)
Advantage is it will return every entry in your tuples no matter the length of the tuple or the length of the list of tuples.
I am retrieving data from a DBF file and I need to generate a SQL script to load the data into a database. I already have this but the values are stored in a tuple and before i create the SQL script I want to strip each item of the tuple. For example, I am getting this:
INSERT INTO my_table (col1,col2,col3) VALUES('Value 1 ', 'TESTE123', ' ADAD ')
And I need to get this:
INSERT INTO my_table (col1,col2,col3) VALUES('Value 1', 'TESTE123', 'ADAD')
For that I am trying with this code:
with dbf.Table(filename) as table:
for record in table:
fields = dbf.field_names(table)
fields = ','.join(fields)
place_holders = ','.join(['?'] * len(fields))
values = tuple(record.strip())
sql = "insert into %s (%s) values(%s)" & ('my_table', fields, values)
And I am getting the following error:
dbf.FieldMissingError: 'STRIP' no such field in table
What do you purpose?
dbf.Record is not a str, and doesn't have string methods.
If every field in the record is text (e.g. Character or Memo, not Numeric or Date) then you can:
values = [v.strip() for v in record]
I've been working most of today on a function that creates a record in an sqlite table. I believe I am finally making some progress and it currently looks like this:
shopping = True
while shopping:
itemToAdd = input("Please enter the ID of the item to add to the basket: ")
basket.append(itemToAdd)
print(basket)
continueShop = input("Continue shopping?(y/n): ")
if continueShop == "n":
conn.execute("INSERT INTO Orders (UserID) VALUES (?)", (results[0][0],))
lastID = conn.execute("SELECT last_insert_rowid()")
conn.commit()
counter = 0
for items in basket:
createOrderItems = "INSERT INTO OrderItems (OrderID, ProductID) VALUES (?,?)"
conn.execute(createOrderItems, (lastID, basket[counter]))
counter = +1
conn.commit()
However, I am now encountering this error to do with lastID if I am reading the error correctly.
line 107, in
conn.execute(createOrderItems, (lastID, basket[counter])) sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
I have 2 tables that I am currently attempting to use:
Orders - contains only an orderID and userID
OrderItems- contains OrderItemsID, OrderID (which i need to be the same as the OrderID Just added) and ProductID(which needs to be taken from the list created and looped to make a record for each item in the "basket".
When you do
lastID = conn.execute("SELECT last_insert_rowid()")
lastID gets bound to an sqlite3.Cursor:
To retrieve data after executing a SELECT statement, you can either treat the cursor as an iterator, call the cursor’s fetchone() method to retrieve a single matching row, or call fetchall() to get a list of the matching rows.
The cursor cannot be converted automatically to the raw ID. You'll have to pull it out yourself, e.g. by doing something like
row = lastID.fetchone()
row_id = row[0]
Then use row_id in your query instead of lastID.
I have a SQL Server database that has a table that lists other tables along with some meta data on them. I can pull this out through Python into a List. What I want to do then though is query each table for the number of rows in it and then append the result into my list.
So for example, I run the first part of the script and I get a List of items, each one containing a list of 3 items (name,activity, Table Name). I then want to cycle through my list, pick up the third item, use it in my SQL query and then append the result into a 4th item in the list.
It starts off
[[table1, act1, Table_1],[table2, act2, Table_2],[table3, act3, Table_3]]
The second part, first takes Table_1, counts the rows and then appends it the list
[[table1, act1, Table_1,10],[table2, act2, Table_2],[table3, act3, Table_3]]
and then for list 2 etc
[[table1, act1, Table_1,10],[table2, act2, Table_2,16],[table3, act3, Table_3]]
Tried a few things but not got any further!
Thanks in advance.
import pyodbc
conn = pyodbc.connect(connetStr)
cursor = conn.cursor()
wffList=[]
cursor.execute('SELECT C_NAME,C_ACTIVE, C_TABLE_NAME from T_FORM_HEAD')
for row in cursor:
wffList.append(row)
for row in wffList:
tabName=row[2]
quer=('SELECT Count(*) FROM '+ tabName)
cursor.execute(quer)
rowCount=cursor.fetchone()
You can creat new list and append row with all four values
new_results = []
for row in wffList:
tabName = row[2]
quer = ('SELECT Count(*) FROM '+ tabName)
cursor.execute(quer)
rowCount = cursor.fetchone()
row.append(rowCount)
new_results.append(row)
print(new_results)
Or you can use enumerate to get row's number
for number, row in enumerate(wffList):
tabName = row[2]
quer = ('SELECT Count(*) FROM '+ tabName)
cursor.execute(quer)
rowCount = cursor.fetchone()
wffList[number].append(rowCount)
print(wfflist)
But probably you could also write one SQL query to get all at once.
But it could be to complex for me at this moment.
Normally, if i want to insert values into a table, i will do something like this (assuming that i know which columns that the values i want to insert belong to):
conn = sqlite3.connect('mydatabase.db')
conn.execute("INSERT INTO MYTABLE (ID,COLUMN1,COLUMN2)\
VALUES(?,?,?)",[myid,value1,value2])
But now i have a list of columns (the length of list may vary) and a list of values for each columns in the list.
For example, if i have a table with 10 columns (Namely, column1, column2...,column10 etc). I have a list of columns that i want to update.Let's say [column3,column4]. And i have a list of values for those columns. [value for column3,value for column4].
How do i insert the values in the list to the individual columns that each belong?
As far as I know the parameter list in conn.execute works only for values, so we have to use string formatting like this:
import sqlite3
conn = sqlite3.connect(':memory:')
conn.execute('CREATE TABLE t (a integer, b integer, c integer)')
col_names = ['a', 'b', 'c']
values = [0, 1, 2]
conn.execute('INSERT INTO t (%s, %s, %s) values(?,?,?)'%tuple(col_names), values)
Please notice this is a very bad attempt since strings passed to the database shall always be checked for injection attack. However you could pass the list of column names to some injection function before insertion.
EDITED:
For variables with various length you could try something like
exec_text = 'INSERT INTO t (' + ','.join(col_names) +') values(' + ','.join(['?'] * len(values)) + ')'
conn.exec(exec_text, values)
# as long as len(col_names) == len(values)
Of course string formatting will work, you just need to be a bit cleverer about it.
col_names = ','.join(col_list)
col_spaces = ','.join(['?'] * len(col_list))
sql = 'INSERT INTO t (%s) values(%s)' % (col_list, col_spaces)
conn.execute(sql, values)
I was looking for a solution to create columns based on a list of unknown / variable length and found this question. However, I managed to find a nicer solution (for me anyway), that's also a bit more modern, so thought I'd include it in case it helps someone:
import sqlite3
def create_sql_db(my_list):
file = 'my_sql.db'
table_name = 'table_1'
init_col = 'id'
col_type = 'TEXT'
conn = sqlite3.connect(file)
c = conn.cursor()
# CREATE TABLE (IF IT DOESN'T ALREADY EXIST)
c.execute('CREATE TABLE IF NOT EXISTS {tn} ({nf} {ft})'.format(
tn=table_name, nf=init_col, ft=col_type))
# CREATE A COLUMN FOR EACH ITEM IN THE LIST
for new_column in my_list:
c.execute('ALTER TABLE {tn} ADD COLUMN "{cn}" {ct}'.format(
tn=table_name, cn=new_column, ct=col_type))
conn.close()
my_list = ["Col1", "Col2", "Col3"]
create_sql_db(my_list)
All my data is of the type text, so I just have a single variable "col_type" - but you could for example feed in a list of tuples (or a tuple of tuples, if that's what you're into):
my_other_list = [("ColA", "TEXT"), ("ColB", "INTEGER"), ("ColC", "BLOB")]
and change the CREATE A COLUMN step to:
for tupl in my_other_list:
new_column = tupl[0] # "ColA", "ColB", "ColC"
col_type = tupl[1] # "TEXT", "INTEGER", "BLOB"
c.execute('ALTER TABLE {tn} ADD COLUMN "{cn}" {ct}'.format(
tn=table_name, cn=new_column, ct=col_type))
As a noob, I can't comment on the very succinct, updated solution #ron_g offered. While testing, though I had to frequently delete the sample database itself, so for any other noobs using this to test, I would advise adding in:
c.execute('DROP TABLE IF EXISTS {tn}'.format(
tn=table_name))
Prior the the 'CREATE TABLE ...' portion.
It appears there are multiple instances of
.format(
tn=table_name ....)
in both 'CREATE TABLE ...' and 'ALTER TABLE ...' so trying to figure out if it's possible to create a single instance (similar to, or including in, the def section).