Develop Reference

Dynamic SQL Error - SQL error code = -104 Token unknown happening on "select * from table". Python Firebird

I seem to be getting an error while trying to dump all the data from the fdb file.
That was the error. 'BLOB' is the name of the table
("Error while preparing SQL statement:\n- SQLCODE: -104\n- Dynamic SQL Error\n- SQL error code = -104\n- Token unknown - line 1, column 15\n- 'BLOB'", -104, 335544569)
the code
def js(val):
if type(val) == int:
return val
if type(val) == str:
return val
if val is None:
return val
if type(val) == decimal.Decimal:
return str(val)
if type(val) == datetime.datetime:
return val.isoformat()
raise Exception(type(val))
con = fdb.connect(dsn='202204.fdb', user='sysdba', password='masterkey')
cur = con.cursor()
cur.execute(
"SELECT a.RDB$RELATION_NAME FROM RDB$RELATIONS a WHERE RDB$SYSTEM_FLAG=0")
tables = [row[0].strip() for row in cur.fetchall()]
db = {}
for table in tables:
db[table] = {}
cur.execute(
f"select rdb$field_name from rdb$relation_fields where rdb$relation_name='{table}' order by rdb$field_position")
db[table]['cols'] = [head[0].strip() for head in cur.fetchall()]
print(db)
cur.execute(f"select * from '{table}'") # code breaks here
db[table]['rows'] = [[js(field) for field in row]
for row in cur.fetchall()]
the expected structure
{"BLOB": {"cols": ["GUID", "UPDATE", "DATA"], "rows": []}}
Python 3.9
Firebird 2.5

The interpolated string "select * from '{table}'" will not produce a valid query. Things enclosed in single quotes are string literals, and you cannot select from a string literal. If you intended this as a quoted identifier, you should enclose it in double quotes ("), not single quotes (').
That is, right now you produce a statement select * from 'BLOB', which is why the error refers to the unknown token 'BLOB', as Firebird expects a (quoted or unquoted) identifier. Change your code to produce select * from "BLOB".
Also, please be aware that string interpolation like this makes your code vulnerable to SQL injection, although that is less of a problem in this case, as you're selecting from system tables, there are edge cases with table names which contain a double quote (third query), or a single quote (second query). Your second query should use parameters, not string interpolation (this is not possible for the third).

Postgres Api-rest error: SyntaxError: EOF while scanning triple-quoted string liter

I need to do a pokemon api in postgres from this url "https://pokeapi.co/"
The details of database (hidden.py) and the utils (myutils.py) they work right, the error is when i do in the console:
python3 pokeapi.py
File "/home/mcala88/pokeapi.py", line 67
cur.close()
^
SyntaxError: EOF while scanning triple-quoted string literal
import psycopg2
import hidden
import time
import myutils
import requests
import json
# Load the secrets
secrets = hidden.secrets()
conn = psycopg2.connect(host=secrets['host'],
port=secrets['port'],
database=secrets['database'],
user=secrets['user'],
password=secrets['pass'],
connect_timeout=3)
cur = conn.cursor()
defaulturl = 'https://pokeapi.co/api/v2/pokemon/1/'
print('If you want to restart the spider, run')
print('DROP TABLE IF EXISTS pokeapi CASCADE;')
print(' ')
sql = '''
CREATE TABLE IF NOT EXISTS pokeapi (id INTEGER, body JSONB);
print(sql)
cur.execute(sql)
# Check to see if we have ids in the table, if not add starting points
# for each of the object trees
sql = 'SELECT COUNT(id) FROM pokeapi;'
count = myutils.queryValue(cur, sql)
if count < 1:
objects = ['pokemon', 'region', 'type']
for obj in objects:
sql = f"INSERT INTO pokeapi (id) VALUES ( 'https://pokeapi.co/api/v2/{obj}/1/' )";
print(sql)
cur.execute(sql, (defaulturl))
conn.commit()
cur.close()

You forgot to close off your triple quotes.
sql = '''
CREATE TABLE IF NOT EXISTS pokeapi (id INTEGER, body JSONB);
should be
sql = '''
CREATE TABLE IF NOT EXISTS pokeapi (id INTEGER, body JSONB);
'''

Safely Inserting Strings Into a SQLite3 UNION Query Using Python

I'm aware that the best way to prevent sql injection is to write Python queries of this form (or similar):
query = 'SELECT %s %s from TABLE'
fields = ['ID', 'NAME']
cur.execute(query, fields)
The above will work for a single query, but what if we want to do a UNION of 2 SQL commands? I've set this up via sqlite3 for sake of repeatability, though technically I'm using pymysql. Looks as follows:
import sqlite3
conn = sqlite3.connect('dummy.db')
cur = conn.cursor()
query = 'CREATE TABLE DUMMY(ID int AUTO INCREMENT, VALUE varchar(255))'
query2 = 'CREATE TABLE DUMMy2(ID int AUTO INCREMENT, VALUE varchar(255)'
try:
cur.execute(query)
cur.execute(query2)
except:
print('Already made table!')
tnames = ['DUMMY1', 'DUMMY2']
sqlcmds = []
for i in range(0,2):
query = 'SELECT %s FROM {}'.format(tnames[i])
sqlcmds.append(query)
fields = ['VALUE', 'VALUE']
sqlcmd = ' UNION '.join(sqlcmds)
cur.execute(sqlcmd, valid_fields)
When I run this, I get a sqlite Operational Error:
sqlite3.OperationalError: near "%": syntax error
I've validated the query prints as expected with this output:
INSERT INTO DUMMY VALUES(%s) UNION INSERT INTO DUMMY VALUES(%s)
All looks good there. What is the issue with the string substitutions here? I can confirm that running a query with direct string substitution works fine. I've tried it with both selects and inserts.
EDIT: I'm aware there are multiple ways to do this with executemany and a few other. I need to do this with UNION for the purposes I'm using this for because this is a very, very simplified example fo the operational code I'm using

The code below executes few INSERTS at once
import sqlite3
conn = sqlite3.connect('dummy.db')
cur = conn.cursor()
query = 'CREATE TABLE DUMMY(ID int AUTO INCREMENT NOT NULL, VALUE varchar(255))'
try:
cur.execute(query)
except:
print('Already made table!')
valid_fields = [('ya dummy',), ('stupid test example',)]
cur.executemany('INSERT INTO DUMMY (VALUE) VALUES (?)',valid_fields)

PYTHON error in SQL syntax while writing json to MYSQL database

I want to receive json data from MQTT and store it in my database.
When am executing my code am receiving this error:
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '%s)' at line 1
There is my code :
import mysql.connector
import json
mydb = mysql.connector.connect(
host="localhost",
user="***",
passwd="***",
database="database"
)
mycursor = mydb.cursor()
def DATA_REPARTITION(Topic, jsonData):
if Topic == "test":
#print ("Start")
INSERT_DEBIT(jsonData)
def INSERT_DEBIT(jsonData):
#Read json from MQTT
print("Start read data to insert")
json_Dict = json.loads(jsonData)
debit = json_Dict['debit']
#Insert into DB Table
sql = ("INSERT INTO debit (data_debit) VALUES (%s)")
val=(debit)
mycursor.execute(sql,val)
mydb.commit()
print(mycursor.rowcount, "record inserted.")
mycursor.close()
mydb.close()
Thanks for your help, am working on this problem for the last 2 days.

You've written your parameterized query properly for MySQL:
sql = ("INSERT INTO debit (data_debit) VALUES (%s)")
The problem is that you're passing in the arguments wrong:
val=(debit)
mycursor.execute(sql,val)
The parentheses don't make debit into a tuple of 1 value. They don't do anything at all; val is just the same value as debit.
But execute wants a sequence of separate values, not 1 value.
To fix this, you need to add a comma. Commas are what create tuples in Python:
val = debit,
If you're wondering why this raises a SQL error, instead of a Python error about val not being iterable… Most likely, val is a string. And strings are iterable. They just iterate their characters. If val is, say, '$100', then you're passing the arguments '$', '1', '0', and '0', to fit a parameterized query with only one parameter.

How to store python dictionary in to mysql DB through python

I am trying to store the the following dictionary into mysql DB by converting the dictionary into a string and then trying to insert, but I am getting following error. How can this be solved, or is there any other way to store a dictionary into mysql DB?
dic = {'office': {'component_office': ['Word2010SP0', 'PowerPoint2010SP0']}}
d = str(dic)
# Sql query
sql = "INSERT INTO ep_soft(ip_address, soft_data) VALUES ('%s', '%s')" % ("192.xxx.xx.xx", d )
soft_data is a VARCHAR(500)
Error:
execution exception (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to
use near 'office': {'component_office': ['Word2010SP0', 'PowerPoint2010SP0' at line 1")
Any suggestions or help please?

First of all, don't ever construct raw SQL queries like that. Never ever. This is what parametrized queries are for. You've asking for an SQL injection attack.
If you want to store arbitrary data, as for example Python dictionaries, you should serialize that data. JSON would be good choice for the format.
Overall your code should look like this:
import MySQLdb
import json
db = MySQLdb.connect(...)
cursor = db.cursor()
dic = {'office': {'component_office': ['Word2010SP0', 'PowerPoint2010SP0']}}
sql = "INSERT INTO ep_soft(ip_address, soft_data) VALUES (%s, %s)"
cursor.execute(sql, ("192.xxx.xx.xx", json.dumps(dic)))
cursor.commit()

Change your code as below:
dic = {'office': {'component_office': ['Word2010SP0', 'PowerPoint2010SP0']}}
d = str(dic)
# Sql query
sql = """INSERT INTO ep_soft(ip_address, soft_data) VALUES (%r, %r)""" % ("192.xxx.xx.xx", d )

Try this:
dic = { 'office': {'component_office': ['Word2010SP0', 'PowerPoint2010SP0'] } }
"INSERT INTO `db`.`table`(`ip_address`, `soft_data`) VALUES (`{}`, `{}`)".format("192.xxx.xx.xx", str(dic))
Change db and table to the values you need.

It is a good idea to sanitize your inputs, and '.format' is useful when needing to use the same variable multiple times within a query. (Not that you to for this example)
dic = {'office': {'component_office': ['Word2010SP0', 'PowerPoint2010SP0']}}
ip = '192.xxx.xx.xx'
with conn.cursor() as cur:
cur.execute("INSERT INTO `ep_soft`(`ip_address`, `soft_data`) VALUES ({0}, '{1}')".format(cur.escape(ip),json.dumps(event)))
conn.commit()
If you do not use cur.escape(variable), you will need to enclose the placeholder {} in quotes.

This answer has some pseudo code regarding the connection object and the flavor of mysql is memsql, but other than that it should be straightforward to follow.
import json
#... do something
a_big_dict = getAHugeDict() #build a huge python dict
conn = getMeAConnection(...)
serialized_dict = json.dumps(a_big_dict) #serialize dict to string
#Something like this to hold the serialization...
qry_create = """
CREATE TABLE TABLE_OF_BIG_DICTS (
ROWID BIGINT NOT NULL AUTO_INCREMENT,
SERIALIZED_DICT BLOB NOT NULL,
UPLOAD_DT TIMESTAMP NULL DEFAULT CURRENT_TIMESTAMP,
KEY (`ROWID`) USING CLUSTERED COLUMNSTORE
);
"""
conn.execute(qry_create)
#Something like this to hold em'
qry_insert = """
INSERT INTO TABLE_OF_BIG_DICTS (SERIALIZED_DICT)
SELECT '{SERIALIZED_DICT}' as SERIALIZED_DICT;
"""
#Send it to db
conn.execute(qry_insert.format(SERIALIZED_DICT=serialized_dict))
#grab the latest
qry_read = """
SELECT a.SERIALIZED_DICT
from TABLE_OF_BIG_DICTS a
JOIN
(
SELECT MAX(UPLOAD_DT) AS MAX_UPLOAD_DT
FROM TABLE_OF_BIG_DICTS
) b
ON a.UPLOAD_DT = b.MAX_UPLOAD_DT
LIMIT 1
"""
#something like this to read the latest dict...
df_dict = conn.sql_to_dataframe(qry_read)
dict_str = df_dict.iloc[df_dict.index.min()][0]
#dicts never die they just get rebuilt
dict_better = json.loads(dict_str)