Develop Reference

Is there a formal way to check if a file selected is a database in sqlite3 for python?

I am using sqlite3 in python. Currently I am using this following code
try:
con = sqlite3.connect(filename)
cur = con.cursor()
# run a fake query to test if the file selected is really a database
cur.execute("SELECT DISTINCT id FROM handle ORDER BY id DESC")
except:
eel.invalidFile()
else:
ExamineTable.printPhoneNum(con)
and it works fine. I am running a fake query to test for a real database, because I found that without this line users could select files that aren't databases and no exception would be thrown when sqlite3 tries to connect. Which might be by design, but I'm wondering if there's a more formal way of checking without running a dummy query?

PRAGMA integrity_check or PRAGMA quick_check can be used to verify that the file is a valid SQLite database. This will raise sqlite3.DatabaseError if the file is not a valid SQLite database or if it has any of the problems described in the documentation.
con = sqlite3.connect(filename)
cur = con.cursor()
try:
cur.execute("PRAGMA integrity_check")
except sqlite3.DatabaseError:
eel.invalidFile()
con.close()
else:
ExamineTable.printPhoneNum(con)
See also: How to tell if sqlite database file is valid or not?

psycopg2.errors.UndefinedTable: [duplicate]

I'm trying to figure out why I can't access a particular table in a PostgreSQL database using psycopg2. I am running PostgreSQL 11.5
If I do this, I can connect to the database in question and read all the tables in it:
import psycopg2
try:
connection = psycopg2.connect(user = "postgres", #psycopg2.connect() creates connection to PostgreSQL database instance
password = "battlebot",
host = "127.0.0.1",
port = "5432",
database = "BRE_2019")
cursor = connection.cursor() #creates a cursor object which allows us to execute PostgreSQL commands through python source
#Print PostgreSQL version
cursor.execute("""SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public'""")
for table in cursor.fetchall():
print(table)
The results look like this :
('geography_columns',)
('geometry_columns',)
('spatial_ref_sys',)
('raster_columns',)
('raster_overviews',)
('nc_avery_parcels_poly',)
('Zone5e',)
('AllResidential2019',)
#....etc....
The table I am interested in is the last one, 'AllResidential2019'
So I try to connect to it and print the contents by doing the following:
try:
connection = psycopg2.connect(user = "postgres",
#psycopg2.connect() creates connection to PostgreSQL database instance
password = "battlebot",
host = "127.0.0.1",
port = "5432",
database = "BRE_2019")
cursor = connection.cursor() #creates a cursor object which allows us to execute PostgreSQL commands through python source
cursor.execute("SELECT * FROM AllResidential2019;") #Executes a database operation or query. Execute method takes SQL query as a parameter. Returns list of result
record = cursor.fetchall()
print(record)
except (Exception, psycopg2.Error) as error:
print("Error while connecting to PostgreSQL: ", error)
And I get the following error:
Error while connecting to PostgreSQL: relation "allresidential2019" does not exist
LINE 1: SELECT * FROM AllResidential2019;
However, I can successfully connect and get results when attempting to connect to another table in another database I have (this works! and the results are the data in this table):
try:
connection = psycopg2.connect(user = "postgres", #psycopg2.connect() creates connection to PostgreSQL database instance
password = "battlebot",
host = "127.0.0.1",
port = "5432",
database = "ClimbingWeatherApp") . #different database name
cursor = connection.cursor()
cursor.execute("SELECT * FROM climbing_area_info ;")
record = cursor.fetchall()
print(record)
except (Exception, psycopg2.Error) as error:
print("Error while connecting to PostgreSQL: ", error)
I can't figure out why I can retrieve information from one table but not another, using exactly the same code (except names are changes). And I am also not sure how to troubleshoot this. Can anyone offer suggestions?

Your table name is case-sensitive and you have to close it in double quotes:
SELECT * FROM "AllResidential2019";
In Python program it may look like this:
cursor.execute('SELECT * FROM "AllResidential2019"')
or you can use the specialized module SQL string composition:
from psycopg2 import sql
# ...
cursor.execute(sql.SQL("SELECT * FROM {}").format(sql.Identifier('AllResidential2019')))
Note that case-sensitive Postgres identifiers (i.e. names of a table, column, view, function, etc) unnecessarily complicate simple matters. I would advise you not to use them.

Likely, the reason for your issue is Postgres' quoting rules which adheres to the ANSI SQL standard regarding double quoting identifiers. In your table creation, you likely quoted the table:
CREATE TABLE "AllResidential2019" (
...
)
Due to case sensitivity of at least one capital letter, this requires you to always quote the table when referencing the table. Do remember: single and double quotes have different meanings in SQL as opposed to being mostly interchangeable in Python.
SELECT * FROM "AllResidential2019"
DELETE FROM "AllResidential2019" ...
ALTER TABLE "AllResidential2019" ...
It is often recommended, if your table, column, or other identifier does not contain special characters, spaces, or reserved words, to always use lower case or no quotes:
CREATE TABLE "allresidential2019" (
...
)
CREATE TABLE AllResidential2019 (
...
)
Doing so, any combination of capital letters will work
SELECT * FROM ALLRESIDENTIAL2019
SELECT * FROM aLlrEsIdEnTiAl2019
SELECT * FROM "allresidential2019"
See further readings on the subject:
Omitting the double quote to do query on PostgreSQL
PostgreSQL naming conventions
Postgres Docs - 4.1.1. Identifiers and Key Words
Don’t use double quotes in PostgreSQL
What is the difference between single and double quotes in SQL?

I was facing the same error in Ubuntu. But in my case, I accidentally added the tables to the wrong database, which was in turn owned by the root postgres user instead of the new postgres user that I had created for my flask app.
I'm using a SQL file to create and populate the tables. This is the command that I used to be able to create these tables using a .sql file. This allows you to specify the owner of the tables as well as the database in which they should be created:
sudo -u postgres psql -U my_user -d my_database -f file.sql -h localhost
You will then be prompted for my_users's password.
sudo -u postgres is only necessary if you are running this from a terminal as a the root user. It basically runs the psql ... command as the postgres user.

How to commit stored procedure execution by using pyodbc

I am trying to execute stored procedure by using pyodbc in databricks, after executing SP I tried to commit the connection but, commit is not happening. Here I am giving my code, please help me out from this issue.
import pyodbc
#### Connecting Azure SQL
def db_connection():
try:
username = "starsusername"
password = "password-db"
server = "server-name"
database_name = "db-name2"
port = "db-port"
conn=pyodbc.connect('Driver={ODBC Driver 17 for SQL server};SERVER=tcp:'+server+','+port+';DATABASE='+ database_name +';UID='+ username +';PWD='+ password)
cursor=conn.cursor()
return cursor, conn
except Exception as e:
print("Faild to Connect AZURE SQL: \n"+str(e))
cursor, conn = db_connection()
# conn1.autocommit=True
cursor.execute("delete from db.table_name")
cursor.execute("insert into db.table_name(BUSINESS_DATE) values('2021-10-02')")
cursor.execute("exec db.SP_NAME '20211023'")
conn.commit()
conn.close()
here I am commiting connection after SP excution. deletion and insertion is not happening at all. and I tried with cursor.execute("SET NOCOUNT ON; exec db.SP_NAME '20211023'") but it's also not working.
Thanks in Advance

If you check this document on pyodbc, you will find that -
To call a stored procedure right now, pass the call to the execute method using either a format your database recognizes or using the ODBC call escape format. The ODBC driver will then reformat the call for you to match the given database.
Note that after connection is set up or done, try doing conn.autocommit = True before calling your SP and it will help. By default it is false.
Executing the Stored Procedure.
You will be able to execute your stored procedure if you follow the below code snippet.
cursor = conn.cursor()
conn.autocommit = True
executesp = """EXEC yourstoredprocedure """
cursor.execute(executesp)
conn.commit()
Delete the Records in SQL Server
You can delete record as shown in the below example.
...#just an example
cursor.execute('''
DELETE FROM product
WHERE product_id in (5,6)
''')
conn.commit()
Don’t forget to add conn.commit() at the end of the code, to ensure that the command would get executed.
Insert record in SQL Server
The below snippet show how we can do the same.
...#just an example
cursor.execute("INSERT INTO EMP (EMPNO, ENAME, JOB, MGR) VALUES (535, 'Scott', 'Manager', 545)")
conn.commit()
I will suggest you to read the for following document for more information.
Delete Record Documentation.
Insert Record Document

Prepared statement pymysql who is correct?

I have created a test database called test inside it has a table called testTable with an autoincrement id value and a name field that takes a varchar(30).
The PREPARE statement queries (4 of them) execute fine when copied into phpmyadmin but I get the error 👍 2021-01-08 18:26:53,022 (MainThread) [ERROR] (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'SET\n #name = 'fred';\nEXECUTE\n statement USING #name;\nDEALLOCATE\nPREPARE\n ' at line 5")
The test code:
import pymysql
import logging
class TestClass():
def __init__(self):
# mysqlconnections
self.mySQLHostName = "localhost"
self.mySQLHostPort = 3306
self.mySQLuserName = "userName"
self.mySQLpassword = "pass"
self.MySQLauthchandb = "mysql"
def QueryMYSQL (self, query):
try:
#logging.info("QueryMYSQL : " + str( query)) # Uncomment to print all mysql queries sent
conn = pymysql.connect(host=self.mySQLHostName, port=self.mySQLHostPort, user=self.mySQLuserName, passwd=self.mySQLpassword, db=self.MySQLauthchandb, charset='utf8')
conn.autocommit(True)
cursor = conn.cursor()
if cursor:
returnSuccess = cursor.execute(query)
if cursor:
returnValue = cursor.fetchall()
#logging.info ("return value : " + str(returnValue)) # Uncomment to print all returned mysql queries
if cursor:
cursor.close()
if conn:
conn.close()
return returnValue
except Exception as e:
logging.error("Problem in ConnectTomySQL")
logging.error(query)
logging.error(e)
return False
# Default error logging log file location:
logging.basicConfig(format='%(asctime)s (%(threadName)-10s) [%(levelname)s] %(message)s', filename= 'ERROR.log',filemode = "w", level=logging.DEBUG)
logging.info("Logging Started")
test = TestClass()
result = test.QueryMYSQL("Describe test.testTable")
print(result)
query = """
PREPARE
statement
FROM
'INSERT INTO test.testTable (id, name) VALUES (NULL , ?)';
SET
#name = 'fred';
EXECUTE
statement USING #name;
DEALLOCATE
PREPARE
statement;
"""
result = test.QueryMYSQL(query)
print(result)
I'm assuming this is a library issue rather than a mysql issue? I am trying to use prepared statements to prevent code injection from user input as I understand this prepared statements are the best way to do this rather than trying to pre filter user input and missing something.
I asked this question on the github but one of the authors (methane Inada Naoki) replied with this:
========
Multistatement can be used by attacker when there is a query injection vulnerability. So it is disabled by default.
as I understand this prepared statements are the best way
You are totally wrong. Your use of prepared statement doesn't protect you from SQL injection at all. If you enable multistatement, your "prepared statement" can be attacked by SQL injection.
But I am not free tech support nor free teacher for you. OSS maintainers are not. Please don't ask here.
and he closed the issue.
Is he correct?
The author book I am reading Robin Nixon,"Learning PHP, MySQL and JavaScript" O'Reilly 5th edition. He appears to be under the misconception and I quote "Let me introduce the best and recommended way to interact with MySQL, which is pretty much bulletproof in terms of Security" Its in the Using Placeholders section pg 260. Is he wrong?
Because I bought this book to improve my security practices and now I'm not sure what is correct.

I found out from the developer of pymysql that the library does not support the PREPARE mysql statement. Also the pymysql library by default does not execute multi-statements.
I understand that my first attempt at substituting values into the INSERT statement is inherently unsafe if multi-statements are enabled. This can be done by using the client_flag=pymysql.constants.CLIENT.MULTI_STATEMENTS in the connect constructor.
The pymysql library does however allow for placeholders to be used in MySQL queries using the cursor.execute(query, (tuple)) method.
To demonstrate this I wrote the following test code example.
import pymysql
import logging
class TestClass():
def __init__(self):
# mysqlconnections
self.mySQLHostName = "localhost"
self.mySQLHostPort = 3306
self.mySQLuserName = "name"
self.mySQLpassword = "pw"
self.MySQLauthchandb = "mysql"
def QueryMYSQL (self, query, data = ()):
try:
logging.info("QueryMYSQL : " + str( query)) # Uncomment to print all mysql queries sent
conn = pymysql.connect(host=self.mySQLHostName, port=self.mySQLHostPort, user=self.mySQLuserName, passwd=self.mySQLpassword, db=self.MySQLauthchandb, charset='utf8', client_flag=pymysql.constants.CLIENT.MULTI_STATEMENTS) #code injection requires multistatements to be allowed this is off in pymysql by default and has to be set on manually.
conn.autocommit(True)
cursor = conn.cursor()
if cursor:
if data:
returnSuccess = cursor.execute(query, data)
else:
returnSuccess = cursor.execute(query)
if cursor:
returnValue = cursor.fetchall()
logging.info ("return value : " + str(returnValue)) # Uncomment to print all returned mysql queries
if cursor:
cursor.close()
if conn:
conn.close()
return returnValue
except Exception as e:
logging.error("Problem in ConnectTomySQL")
logging.error(e)
logging.error(query)
if data:
logging.error("Data {}".format(str(data)))
return False
# Default error logging log file location:
logging.basicConfig(format='%(asctime)s (%(threadName)-10s) [%(levelname)s] %(message)s', filename= 'ERROR.log',filemode = "w", level=logging.DEBUG)
logging.info("Logging Started")
def usePlaceholder(userInput):
query = "INSERT INTO test.testTable (id, name) VALUES (NULL , %s)"
data = (userInput,)
result = test.QueryMYSQL(query,data)
print(result)
def useSubstitution(userInput):
query = "INSERT INTO test.testTable (id, name) VALUES (NULL , '{}')".format(userInput) # this is unsafe.
result = test.QueryMYSQL(query)
print(result)
test = TestClass()
#Create the test database and testTable.
query = "CREATE DATABASE test"
test.QueryMYSQL(query)
query = "CREATE TABLE `test`.`testTable` ( `id` INT NOT NULL AUTO_INCREMENT , `name` VARCHAR(256) NULL DEFAULT NULL , PRIMARY KEY (`id`)) ENGINE = InnoDB;"
test.QueryMYSQL(query)
#Simulated user input.
legitUserEntry = "Ringo"
injectionAttempt = "333' ); INSERT INTO test.testTable (id, name) VALUES (NULL , 666);#" #A simulated user sql injection attempt.
useSubstitution(legitUserEntry) # this will also insert Ringo - but could be unsafe.
usePlaceholder(legitUserEntry) # this will insert Ringo - but is safer.
useSubstitution(injectionAttempt) # this will inject the input code and execute it.
usePlaceholder(injectionAttempt) # this will insert the input into the database without executing the injected code.
So from this exercise, I shall henceforth improve my security by keeping multi-statements set to off (the default) AND using the placeholders and data tuple rather than substitution.

I use python mysql API to insert data into mysql but it didn't work when i didn't add statement 'with conn:'

#!/usr/bin/python
#Program:
# insert data into mysql and the display them
import MySQLdb as mdb
conn = mdb.connect(host = 'localhost', user = 'root', passwd = '8023xue0526', db ='contact')
cur = conn.cursor()
cur.execute("insert into contact values('123221', 'ni')")
cur.execute("select * from contact")
row_num = int(cur.rowcount)
for i in range(row_num):
row = cur.fetchone()
print row
I use those code to insert a data into mysql, the program worked. but after that, i check it in mysqlclient, the data didn't exist.
But when I add a statement 'with conn:' before 'cur = conn.cursor(), the data really insert into mysql. the code like this
#!/usr/bin/python
#Program:
# to get some information from mysql
import MySQLdb as mdb
import sys
conn = mdb.connect(host = 'localhost', user = 'root', passwd = '8023xue0526', db = 'contact')
with conn:
cur = conn.cursor()
cur.execute("insert into contact values('122221', 'ni')")
cur.execute("select * from contact")
row_num = int(cur.rowcount)
for i in range(row_num):
row = cur.fetchone()
print row

with conn: (using the connection object as a context manager) ensures that the transaction is committed if no exceptions occurred within the code block governed by the with statement.
Without the context manager, use conn.commit() to explicitly commit the transaction.

Martijn Pieters answer is the right one. Just to develop a little bit more, you have to understand that databases are designated both with "concurrent access" and "possibility of failure" in mind.
In that case, is would be unacceptable if someone started to make changes to the DB, showing that (incomplete) changes to other DB users, and suddenly, for some reason (bug, kill, etc.) aborting its modifications thus leaving the DB in an inconsistent state.
To prevent that, when your DB run in a decent isolation level you have to explicitly state that your changes are ready to publish. That is the purpose of the commit statement.
In Python you either have to explicitly call conn.commit()yourself. Or let the context manager with conn: do it for you if there is no exception. The two fragments below does globally the same thing:
>>> with conn:
... c = conn.cursor()
... c.doSomething()
... # implicit commit here
>>> conn = sqlite3.connect(....)
>>> c = conn.cursor()
>>> c.doSomething()
>>> conn.commit() # explicit commit here
Please note that, in either cases, the commit operation might fail. For example, if a concurrent transaction has already committed incompatible changes to the database.