I have a sqlite database named StudentDB which has 3 columns Roll number, Name, Marks. Now I want to fetch only the columns that user selects in the IDE. User can select one column or two or all the three. How can I alter the query accordingly using Python?
I tried:
import sqlite3
sel={"Roll Number":12}
query = 'select * from StudentDB Where({seq})'.format(seq=','.join(['?']*len(sel))),[i for k,i in sel.items()]
con = sqlite3.connect(database)
cur = con.cursor()
cur.execute(query)
all_data = cur.fetchall()
all_data
I am getting:
operation parameter must be str
You should control the text of the query. The where clause shall allways be in the form WHERE colname=value [AND colname2=...] or (better) WHERE colname=? [AND ...] if you want to build a parameterized query.
So you want:
query = 'select * from StudentDB Where ' + ' AND '.join('"{}"=?'.format(col)
for col in sel.keys())
...
cur.execute(query, tuple(sel.values()))
In your code, the query is now a tuple instead of str and that is why the error.
I assume you want to execute a query like below -
select * from StudentDB Where "Roll number"=?
Then you can change the sql query like this (assuming you want and and not or) -
query = "select * from StudentDB Where {seq}".format(seq=" and ".join('"{}"=?'.format(k) for k in sel.keys()))
and execute the query like -
cur.execute(query, tuple(sel.values()))
Please make sure in your code the provided database is defined and contains the database name and studentDB is indeed the table name and not database name.
I am using sqlite3 in python 3 I want to get only the updated data from the database. what I mean by that can be explained as follows: the database already has 2 rows of data and I add 2 more rows of data. How can I read only the updated rows instead of total rows
Note: indexing may not help here because the no of rows updating will change.
def read_all():
cur = con.cursor()
cur.execute("SELECT * FROM CVT")
rows = cur.fetchall()
# print(rows[-1])
assert cur.rowcount == len(rows)
lastrowids = range(cur.lastrowid - cur.rowcount + 1, cur.lastrowid + 1)
print(lastrowids)
If you insert rows "one by one" like that
cursor.execute('INSERT INTO foo (xxxx) VALUES (xxxx)')
You then can retrieve the last inserted rows id :
last_inserted_id = cursor.lastrowid
BUT it will work ONLY if you insert a single row with execute. It will return None if you try to use it after a executemany.
If you are trying to get multiple ids of rows that were inserted at the same time see that answer that may help you.
I am trying search records in sqlite3 table to get last record inserted with where condition, but I can do it with only one condition WHERE CODE = df = "DS3243". But what I want to do is with multiple WHERE conditions jf = "QS2134", df = "DS3243", sf = "MS5787", so that I can get the last record inserted with the codes provided.
DEMONTSTRATION
CODE POINT
QS2134 1000
DS3244 2000
MS5787 3000
QS2134 130
QS2134 200 # want to get this because it last with such code
DS3244 300
MS5787 4500
DS3244 860 # want to get this because it last with such code
MS5787 567
MS5787 45009 # want to get this because it last with such code
Am able to do for only one variable cur.execute("SELECT * FROM PROFILE WHERE CODE=? ORDER BY POINT ASC LIMIT 1 ",(df,)) but i want to do for multiple varaiables.
import sqlite3
jf = "QS2134"
df = "DS3243"
sf = "MS5787"
con = sqlite3.connect("TEST.db")
cur = con.cursor()
cur.execute("SELECT * FROM PROFILE WHERE CODE=? ORDER BY POINT ASC LIMIT 1 ",(df,)) # limit one means last one
rows = cur.fetchall()
for row in rows:
print(row)
con.commit()
con.close()
I'm not sure I understand your question, but is it possible that you meant that you want to group the results?
Is it "group by" clause that you're looking for?
Something like:
select CODE, MAX(POINT) group by CODE;
I think you are simply trying to extend your query, in which case, why don't you try string formatting?
x = "SELECT * FROM my_table where col1 = '{0}' or col2 ='{1}';".format(var_1, var_2)
cur.execute(x)
That way you can extend your query with as many conditions as you like.
Is there a way to produce this output using SQL for all tables in a given database (using MySQL) without having to specify individual table names and columns?
Table Column Count
---- ---- ----
Table1 Col1 0
Table1 Col2 100
Table1 Col3 0
Table1 Col4 67
Table1 Col5 0
Table2 Col1 30
Table2 Col2 0
Table2 Col3 2
... ... ...
The purpose is to identify columns for analysis based on how much data they contain (a significant number of columns are empty).
The 'workaround' solution using python (one table at a time):
# Libraries
import pymysql
import pandas as pd
import pymysql.cursors
# Connect to mariaDB
connection = pymysql.connect(host='localhost',
user='root',
password='my_password',
db='my_database',
charset='latin1',
cursorclass=pymysql.cursors.DictCursor)
# Get column metadata
sql = """SELECT *
FROM `INFORMATION_SCHEMA`.`COLUMNS`
WHERE `TABLE_SCHEMA`='my_database'
"""
with connection.cursor() as cursor:
cursor.execute(sql)
result = cursor.fetchall()
# Store in dataframe
df = pd.DataFrame(result)
df = df[['TABLE_NAME', 'COLUMN_NAME']]
# Build SQL string (one table at a time for now)
my_table = 'my_table'
df_my_table = df[df.TABLE_NAME==my_table].copy()
cols = list(df_my_table.COLUMN_NAME)
col_strings = [''.join(['COUNT(', x, ') AS ', x, ', ']) for x in cols]
col_strings[-1] = col_strings[-1].replace(',','')
sql = ''.join(['SELECT '] + col_strings + ['FROM ', my_table])
# Execute
with connection.cursor() as cursor:
cursor.execute(sql)
result = cursor.fetchall()
The result is a dictionary of column names and counts.
Basically, no. See also this answer.
Also, note that the closest match of the answer above is actually the method you're already using, but less efficiently implemented in reflective SQL.
I'd do the same as you did - build a SQL like
SELECT
COUNT(*) AS `count`,
SUM(IF(columnName1 IS NULL,1,0)) AS columnName1,
...
SUM(IF(columnNameN IS NULL,1,0)) AS columnNameN
FROM tableName;
using information_schema as a source for table and column names, then execute it for each table in MySQL, then disassemble the single row returned into N tuple entries (tableName, columnName, total, nulls).
It is possible, but it's not going to be quick.
As mentioned in a previous answer you can work your way through the columns table in the information_schema to build queries to get the counts. It's then just a question of how long you are prepared to wait for the answer because you end up counting every row, for every column, in every table. You can speed things up a bit if you exclude columns that are defined as NOT NULL in the cursor (i.e. IS_NULLABLE = 'YES').
The solution suggested by LSerni is going to be much faster, particularly if you have very wide tables and/or high row counts, but would require more work handling the results.
e.g.
DELIMITER //
DROP PROCEDURE IF EXISTS non_nulls //
CREATE PROCEDURE non_nulls (IN sname VARCHAR(64))
BEGIN
-- Parameters:
-- Schema name to check
-- call non_nulls('sakila');
DECLARE vTABLE_NAME varchar(64);
DECLARE vCOLUMN_NAME varchar(64);
DECLARE vIS_NULLABLE varchar(3);
DECLARE vCOLUMN_KEY varchar(3);
DECLARE done BOOLEAN DEFAULT FALSE;
DECLARE cur1 CURSOR FOR
SELECT `TABLE_NAME`, `COLUMN_NAME`, `IS_NULLABLE`, `COLUMN_KEY`
FROM `information_schema`.`columns`
WHERE `TABLE_SCHEMA` = sname
ORDER BY `TABLE_NAME` ASC, `ORDINAL_POSITION` ASC;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done := TRUE;
DROP TEMPORARY TABLE IF EXISTS non_nulls;
CREATE TEMPORARY TABLE non_nulls(
table_name VARCHAR(64),
column_name VARCHAR(64),
column_key CHAR(3),
is_nullable CHAR(3),
rows BIGINT,
populated BIGINT
);
OPEN cur1;
read_loop: LOOP
FETCH cur1 INTO vTABLE_NAME, vCOLUMN_NAME, vIS_NULLABLE, vCOLUMN_KEY;
IF done THEN
LEAVE read_loop;
END IF;
SET #sql := CONCAT('INSERT INTO non_nulls ',
'(table_name,column_name,column_key,is_nullable,rows,populated) ',
'SELECT \'', vTABLE_NAME, '\',\'', vCOLUMN_NAME, '\',\'', vCOLUMN_KEY, '\',\'',
vIS_NULLABLE, '\', COUNT(*), COUNT(`', vCOLUMN_NAME, '`) ',
'FROM `', sname, '`.`', vTABLE_NAME, '`');
PREPARE stmt1 FROM #sql;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;
END LOOP;
CLOSE cur1;
SELECT * FROM non_nulls;
END //
DELIMITER ;
call non_nulls('sakila');
I have a table in my SQL Server that is being updated every minute.
Currently, I get the data from my table using this lines of code:
conn = pymssql.connect(server, user, password, "tempdb")
def print_table():
cursor = conn.cursor(as_dict=True)
cursor.execute('SELECT * FROM EmotionDisturbances WHERE name=%s', 'John Doe')
for row in cursor:
#Show the data:
print("rate=%d, emotion=%s" % (row['rate'], row['emotion']))
conn.close()
In my application, I run this the function every 10 seconds.
How do I update the function so that I only print the last appended data from my table?
Thanks
Assuming you have an auto-incrementing index in column id you'd do:
SELECT * FROM EmotionDisturbances WHERE name = % ORDER BY id DESC LIMIT 1
EDIT: If you want all data that was added after a certain time, then you'll need to migrate your schema to have a created date column if it doesn't have one already, then you can do:
SELECT *
FROM EmotionDisturbances
WHERE name = % AND created >= DATEADD(second, -10, GETDATE())
This would get all of the records created over the last 10 seconds, since you said this function runs every 10 seconds.