I am using the dotenv framework to connect my python script to a postgres database.
I have a list of ids and want to delete all the rows containing thoses ids
ids_to_delete = df["food_id"].tolist()
conn = create_conn()
with conn.cursor() as cursor:
sql = "DELETE FROM food_recommandations.food_categorisation
WHERE food_categorisation.food_id = %(ids)s "
cursor.execute(sql, {"ids":ids_to_delete} )
cursor.close()
conn.close()
This must delete all the rows containing thoses ids
You can not use = with list because your columns doesn't have lists stored in it. Single cell contains one integer id. So what you are looking for is SQL in operator
sql = "DELETE FROM food_recommandations.food_categorisation
WHERE food_categorisation.food_id in %(ids)s "
cursor.execute(sql, {"ids":tuple(ids_to_delete)} )
Apparently your obscurification manager (dotenv framework) translates the structure {"ids":tuple(ids_to_delete)} to an array before transmitting to Postgres. That then required a slight alteration in you query. The in expects a delimited list which is close tp the same to you and I is vary different to Postgres. With an array use the predicate = ANY. So the query from #WasiHaider becomes:
sql = "DELETE FROM food_recommandations.food_categorisation
WHERE food_categorisation.food_id = ANY %(ids)s "
cursor.execute(sql, {"ids":tuple(ids_to_delete)} )
Note: Not tested - no data.
If successful credit to #WasiHaider.
Related
I want to put the result of each column of the result of my request and store them into separate variables, so I can exploit its results.
I precise this is with a SELECt * and not separate requests.
So, If I do for example:
with connection.cursor() as cursor:
# Read a single record
sql = 'SELECT * FROM table'
cursor.execute(sql)
result = cursor.fetchall()
print(result)
I want to do :
a = [results from column1]
b = [results from column2]
The results should be turned into a row and not be left as a column, to make it a dictionary.
It's probably very simple but I'm new with Python / SQL, thank you.
I have a sqlite database named StudentDB which has 3 columns Roll number, Name, Marks. Now I want to fetch only the columns that user selects in the IDE. User can select one column or two or all the three. How can I alter the query accordingly using Python?
I tried:
import sqlite3
sel={"Roll Number":12}
query = 'select * from StudentDB Where({seq})'.format(seq=','.join(['?']*len(sel))),[i for k,i in sel.items()]
con = sqlite3.connect(database)
cur = con.cursor()
cur.execute(query)
all_data = cur.fetchall()
all_data
I am getting:
operation parameter must be str
You should control the text of the query. The where clause shall allways be in the form WHERE colname=value [AND colname2=...] or (better) WHERE colname=? [AND ...] if you want to build a parameterized query.
So you want:
query = 'select * from StudentDB Where ' + ' AND '.join('"{}"=?'.format(col)
for col in sel.keys())
...
cur.execute(query, tuple(sel.values()))
In your code, the query is now a tuple instead of str and that is why the error.
I assume you want to execute a query like below -
select * from StudentDB Where "Roll number"=?
Then you can change the sql query like this (assuming you want and and not or) -
query = "select * from StudentDB Where {seq}".format(seq=" and ".join('"{}"=?'.format(k) for k in sel.keys()))
and execute the query like -
cur.execute(query, tuple(sel.values()))
Please make sure in your code the provided database is defined and contains the database name and studentDB is indeed the table name and not database name.
I am pulling data from a MSSQL db using pyodbc which returns my data set in a list. This data then needs to be transferred into a MySQL db. I have written the following stored procedure in MySQL.
CREATE DEFINER=`root`#`localhost` PROCEDURE `sp_int_pmt`(
IN pmtamt DECIMAL(16,10),
IN pmtdt DATETIME,
IN propmtref VARCHAR(128),
IN rtdinv_id INT(11)
)
BEGIN
INSERT INTO ay_financials.payment
(
pmtamt,
pmtdt,
propmtref,
rtdinv_id
)
VALUES
(
pmtamt,
pmtdt,
propmtref,
rtdinv_id
);
END
The procedure works fine if I am inserting one record at the time. So, for now, I am iterating over the list from my MSSQL query and call the procedure for each record. I am using this code:
cursor = cnxn.cursor()
cursor.execute(""" SELECT *
FROM [%s].[dbo].[pmt]
WHERE pmtdt BETWEEN '2018-01-01' AND '2018-12-31'""" %(database))
a = cursor.fetchmany(25)
cnxn.close()
import pymysql
# MySQL configurations
un = 'ssssssss'
pw = '****************'
db = 'ay_fnls'
h = '100.100.100.100'
conn = pymysql.connect(host=h, user=un, password=pw, db=db, cursorclass=pymysql.cursors.DictCursor)
cur = conn.cursor()
for ay in a:
cur.callproc('sp_int_pmt',(ay.pmtamt,ay.pmtdt,ay.propmtref,ay.rtdinv_id))
conn.commit()
The problem I will have in production is this list will contain 10,000-100,000 every day. Iterating over that data doesn't seem like an optimized way to handle this.
How can I use the full list from the MSSQL query, call the MySQL procedure one time and insert all the relevant data?
How can I use the full list from the MSSQL query, call the MySQL procedure one time and insert all the relevant data?
You can't do that with your stored procedure as written. It will only insert one row at a time, so to insert n rows you would have to call it n times.
Also, as far as I know you can't modify the stored procedure to insert n rows without using a temporary table or some other workaround because MySQL does not support table-valued parameters to stored procedures.
You can, however, insert multiple rows at once if you use a regular INSERT statement and .executemany. pymysql will bundle the inserts into one or more multi-row inserts
mssql_crsr = mssql_cnxn.cursor()
mssql_stmt = """\
SELECT 1 AS id, N'Alfa' AS txt
UNION ALL
SELECT 2 AS id, N'Bravo' AS txt
UNION ALL
SELECT 3 AS id, N'Charlie' AS txt
"""
mssql_crsr.execute(mssql_stmt)
mssql_rows = []
while True:
row = mssql_crsr.fetchone()
if row:
mssql_rows.append(tuple(row))
else:
break
mysql_cnxn = pymysql.connect(host='localhost', port=3307,
user='root', password='_whatever_',
db='mydb', autocommit=True)
mysql_crsr = mysql_cnxn.cursor()
mysql_stmt = "INSERT INTO stuff (id, txt) VALUES (%s, %s)"
mysql_crsr.executemany(mysql_stmt, mssql_rows)
The above code produces the following in the MySQL general_log
190430 10:00:53 4 Connect root#localhost on mydb
4 Query INSERT INTO stuff (id, txt) VALUES (1, 'Alfa'),(2, 'Bravo'),(3, 'Charlie')
4 Quit
Note that pymysql cannot bundle calls to a stored procedure in the same way, so if you were to use
mysql_stmt = "CALL stuff_one(%s, %s)"
instead of a regular INSERT then the general_log would contain
190430 9:47:10 3 Connect root#localhost on mydb
3 Query CALL stuff_one(1, 'Alfa')
3 Query CALL stuff_one(2, 'Bravo')
3 Query CALL stuff_one(3, 'Charlie')
3 Quit
New to python, trying to use psycopg2 to read Postgres
I am reading from a database table called deployment and trying to handle a Value from a table with three fields id, Key and Value
import psycopg2
conn = psycopg2.connect(host="localhost",database=database, user=user, password=password)
cur = conn.cursor()
cur.execute("SELECT \"Value\" FROM deployment WHERE (\"Key\" = 'DUMPLOCATION')")
records = cur.fetchall()
print(json.dumps(records))
[["newdrive"]]
I want this to be just "newdrive" so that I can do a string comparison in the next line to check if its "newdrive" or not
I tried json.loads on the json.dumps output, didn't work
>>> a=json.loads(json.dumps(records))
>>> print(a)
[['newdrive']]
I also tried to print just the records without json.dump
>>> print(records)
[('newdrive',)]
The result of fetchall() is a sequence of tuples. You can loop over the sequence and print the first (index 0) element of each tuple:
cur.execute("SELECT \"Value\" FROM deployment WHERE (\"Key\" = 'DUMPLOCATION')")
records = cur.fetchall()
for record in records:
print(record[0])
Or simpler, if you are sure the query returns no more than one row, use fetchone() which gives a single tuple representing returned row, e.g.:
cur.execute("SELECT \"Value\" FROM deployment WHERE (\"Key\" = 'DUMPLOCATION')")
row = cur.fetchone()
if row: # check whether the query returned a row
print(row[0])
I have a database table with nearly 1 million records.
I have added a new column, called concentration.
I then have a function which calculates 'concentration' for each record.
Now, I want to update the records in batch, so I have been looking at the following questions/answers: https://stackoverflow.com/a/33258295/596841, https://stackoverflow.com/a/23324727/596841 and https://stackoverflow.com/a/39626473/596841, but I am not sure how to do this using unnest...
This is my Python 3 function to do the updates:
def BatchUpdateWithConcentration(tablename, concentrations):
connection = psycopg2.connect(dbname=database_name, host=host, port=port, user=username, password=password);
cursor = connection.cursor();
sql = """
update #tablename# as t
set
t.concentration = s.con
FROM unnest(%s) s(con, id)
WHERE t.id = s.id;
"""
cursor.execute(sql.replace('#tablename#',tablename.lower()), (concentrations,))
connection.commit()
cursor.close()
connection.close()
concentrations is an array of tuples:
[(3.718244705238561e-16, 108264), (...)]
The first value is a double precision and the second is an integer, representing the concentration and rowid, respectively.
The error I'm getting is:
psycopg2.ProgrammingError: a column definition list is required for functions returning "record"
LINE 5: FROM unnest(ARRAY[(3.718244705238561e-16, 108264), (...
^
Since a Python tuple is adapted by Psycopg to a Postgresql anonymous record it is necessary to specify the data types:
from unnest(%s) s(con numeric, id integer)