Why my code only inserts one line?
thewholeenchilada = ("SELECT SUBSTR(email, (SELECT INSTR(email,'#'))) AS org, SUM(count) as count FROM Em GROUP BY org ORDER BY count DESC")
for salida in cur.execute(thewholeenchilada):
cur.execute('''INSERT INTO Counts (org, count)
VALUES (?, ?)''', (salida[0],row[1]))
print((str(salida[0]), salida[1]))
conn.commit()
Avoid the loop and run one INSERT INTO ... SELECT query. Right now you re-use same cursor outside and inside loop causing issues with processing. Either use two different cursors or efficiently combine and have database engine run action query:
sql = '''INSERT INTO Counts (org, [count])
SELECT SUBSTR(email, INSTR(email, '#')+1) AS org,
SUM(count) as [count]
FROM Em
GROUP BY org
ORDER BY count DESC
'''
cur.execute(sql)
conn.commit()
Related
I need to execute a SQL query that deletes the duplicated rows based on one column and keep the last record. Noting that it's a large table so Django ORM takes very long time so I need SQL query instead. the column name is customer_number and table name is pages_dataupload. I'm using sqlite.
Update: I tried this but it gives me no such column: row_num
cursor = connection.cursor()
cursor.execute(
'''WITH cte AS (
SELECT
id,
customer_number ,
ROW_NUMBER() OVER (
PARTITION BY
id,
customer_number
ORDER BY
id,
customer_number
) row_num
FROM
pages.dataupload
)
DELETE FROM pages_dataupload
WHERE row_num > 1;
'''
)
You can work with an Exists subquery [Django-doc] to determine efficiently if there is a younger DataUpload:
from django.db.models import Exists, OuterRef
DataUpload.objects.filter(Exists(
DataUpload.objects.filter(
pk__gt=OuterRef('pk'), customer_number=OuterRef('customer_number')
)
)).delete()
This will thus check for each DataUpload if there exists a DataUpload with a larger primary key that has the same customer_number. If that is the case, we will remove that DataUpload.
I have solved the problem with the below query, is there any way to reset the id field after removing the duplicate?
cursor = connection.cursor()
cursor.execute(
'''
DELETE FROM pages_dataupload WHERE id not in (
SELECT Max(id) FROM pages_dataupload Group By Dial
)
'''
)
cHandler = myDB.cursor()
cHandler.execute('select UserId,C1,LogDate from DeviceLogs_12_2019') // data from remote sql server database
curs = connection.cursor()
curs.execute("""select * from biometric""") //data from my database table
lst = []
result= cHandler.fetchall()
for row in result:
lst.append(row)
lst2 = []
result2= curs.fetchall()
for row in result2:
lst2.append(row)
t = []
r = [elem for elem in lst if not elem in lst2]
for i in r:
print(i)
t.append(i)
for i in t:
frappe.db.sql("""Insert into biometric(UserId,C1,LogDate) select '%s','%s','%s' where not exists(select * from biometric where UserID='%s' and LogDate='%s')""",(i[0],i[1],i[2],i[0],i[2]),as_dict=1)
I am trying above code to insert data into my table if record not exists but getting error :
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '1111'',''in'',''2019-12-03 06:37:15'' where not exists(select * from biometric ' at line 1")
Is there anything I am doing wrong or any other way to achieve this?
It appears you have potentially four problems:
There is a from clause missing between select and where not exists.
When using a prepared statement you do not enclose your placeholder arguments, %s, within quotes. Your SQL should be:
Your loop:
Loop:
t = []
r = [elem for elem in lst if not elem in lst2]
for i in r:
print(i)
t.append(i)
If you are trying to only include rows from the remote site that will not be duplicates, then you should explicitly check the two fields that matter, i.e. UserId and LogDate. But what is the point since your SQL is taking care of making sure that you are excluding these duplicate rows? Also, what is the point of copying everything form r to t?
SQL:
Insert into biometric(UserId,C1,LogDate) select %s,%s,%s from DUAL where not exists(select * from biometric where UserID=%s and LogDate=%s
But here is the problem even with the above SQL:
If the not exists clause is false, then the select %s,%s,%s from DUAL ... returns no columns and the column count will not match the number of columns you are trying to insert, namely three.
If your concern is getting an error due to duplicate keys because (UserId, LogDate) is either a UNIQUE or PRIMARY KEY, then add the IGNORE keyword on the INSERT statement and then if a row with the key already exists, the insertion will be ignored. But there is no way of knowing since you have not provided this information:
for i in t:
frappe.db.sql("Insert IGNORE into biometric(UserId,C1,LogDate) values(%s,%s,%s)",(i[0],i[1],i[2]))
If you do not want multiple rows with the same (UserId, LogDate) combination, then you should define a UNIQUE KEY on these two columns and then the above SQL should be sufficient. There is also an ON DUPLICATE KEY SET ... variation of the INSERT statement where if the key exists you can do an update instead (look this up).
If you don't have a UNIQUE KEY defined on these two columns or you need to print out those rows which are being updated, then you do need to test for the presence of the existing keys. But this would be the way to do it:
cHandler = myDB.cursor()
cHandler.execute('select UserId,C1,LogDate from DeviceLogs_12_2019') // data from remote sql server database
rows = cHandler.fetchall()
curs = connection.cursor()
for row in rows:
curs.execute("select UserId from biometric where UserId=%s and LogDate=%s", (ros[0], row[2])) # row already in biometric table?
biometric_row = curs.fetchone()
if biometric_row is None: # no, it is not
print(row)
frappe.db.sql("Insert into biometric(UserId,C1,LogDate) values(%s, %s, %s)", (row[0],row[1],row[2]))
I would like to calculate the average in specific column of my table grouping by another column, but it doesn't work, I have a problem whith the group by.
My code:
import sqlite3
conn = sqlite3.connect("ma_base.db")
cur = conn.cursor()
cur.execute("UPDATE test_centrale set avg_price = avg(prix) group by test_centrale.version ")
conn.commit()
print('done')
cur.close()
If I got your question right. Use subquery to find the second average and update using following query
cur.execute("UPDATE test_centrale set avg_price = (select avg(prix) from test_centrale group by test_centrale.version )")
I have three python lists with about 1.5 million entries each and would like to insert these into a new SQLite table. When doing this I get the error:
OperationalError: no such column: days
This is the code I have:
con = sqlite3.connect('database.db')
cur = con.cursor()
...
cur.execute("DROP TABLE IF EXISTS days")
cur.execute("CREATE TABLE IF NOT EXISTS days(DAYS_NEEDED integer, RAISED_TIME text, POSTED_TIME text)")
cur.execute("INSERT INTO days (DAYS_NEEDED, RAISED_TIME, POSTED_TIME) VALUES (days, rt_list, pt_list)")
con.commit()
"days" is a list of integers, rt_list and pt_list are both lists of strings. Does anyone know what I'm doing wrong here?
Any help is much appreciated!
That's not the way you can insert list of values in SQL. First, you must give a valid SQL instruction using ? as placeholders. Then if you want to insert more than one row at a time, you will need the executemany method. It is a true improvement because the SQL in only parsed and prepared once.
So you should have written:
cur.execute("DROP TABLE IF EXISTS days")
cur.execute("CREATE TABLE IF NOT EXISTS days(DAYS_NEEDED integer, RAISED_TIME text, POSTED_TIME text)")
cur.executemany("INSERT INTO days (DAYS_NEEDED, RAISED_TIME, POSTED_TIME) VALUES (?,?,?)",
zip(days, rt_list, pt_list))
con.commit()
BTW, the direct usage of zip is a Sqlite3 module extension, the DB-API 2.0 Python interface normally requires a sequence where zip returns an iterator, so the more portable way (any DB engine) would be:
cur.executemany("INSERT INTO days (DAYS_NEEDED, RAISED_TIME, POSTED_TIME) VALUES (?,?,?)",
tuple(zip(days, rt_list, pt_list)))
You have to use ? placeholders inside your VALUES() and then provide the actual values to the execute method.
Something along the lines should do the job:
con = sqlite3.connect('database.db')
cur = con.cursor()
...
cur.execute("DROP TABLE IF EXISTS days")
cur.execute("CREATE TABLE IF NOT EXISTS days(DAYS_NEEDED integer, RAISED_TIME text, POSTED_TIME text)")
def insert(days_needed, rt, pt):
cur.execute("INSERT INTO days (DAYS_NEEDED, RAISED_TIME, POSTED_TIME) VALUES (?, ?, ?)", (days_needed, rt, pt))
for d, rt, pt in zip(days, rt_list, pt_list):
insert(d, rt, pt)
con.commit()
Hello, I connected two MySql tables with foreign key and I want to insert data into them through python. here is the piece of code that works but there should be an alternative and professional way to do so otherwise I don't need foreign key and I just insert ID of first table customer_id column of the second table. thanks for helping.
Product = str(self.Text.GetValue())
Product2 = str(self.Description.GetValue())
db=MySQLdb.connect('127.0.0.1', 'root','password', 'database')
cursor = db.cursor()
cursor.execute("INSERT INTO customer (Address) VALUES (%s)", (Product))
cursor.execute("SELECT id FROM customer ORDER BY id DESC LIMIT 1")
rows = cursor.fetchall()
the_id= rows[0][0]
cursor.execute("INSERT INTO product_order (customer_id, description) VALUES (%s,%s)", (the_id,Product2))
cursor.execute("commit")
use db.insert_id() to get the last inserted id/customer_id
Err... the_id = cursor.lastrowid.