Develop Reference

Is there a different way of coding the select query in python psycopg2?

can't figure out what the error w.r.t select query here in python psycopg2 (postgresql query)
cursor = con.cursor()
postgreSQL_select_Query = """select ticker, timestamp, close from usstockseod u where u.ticker = %s;"""
cursor.execute(postgreSQL_select_Query,[ticker,])
close_prices = cursor.fetchall()
c_f = pd.DataFrame(close_prices, columns=['ticker','timestamp','close'])
return c_f```

Ticker is a single value. I am trying to learn how to write parameterized queries using Psycopg2 as the syntax isnt obvious, but couldnt find good sources of info.
The issue resolved itself when i re-ran the code in the notebook - so the prior variables data might have caused the issue in the first place.
Thanks...

Pass BOTH single bind variable and list variable to SQL query cx_Oracle Python

I have a Oracle SQL query:
SELECT * from table1 WHERE deliveredDate = ? AND productID IN (?,?,?,...);
I would like to pass a single variable to deliveredDate and a list with length unknown to the productID using cx_Oracle and Python
From the Oracle Using Bind guide (https://cx-oracle.readthedocs.io/en/latest/user_guide/bind.html) I understand that you can bind either the single variable or list of items, but I'm not sure if we can bind both.
Please help me with this issue.
Thank you.

Of course you can, but convert the notation for bind variables from ? to :-preceeded integers such as
import pandas as pd
import cx_Oracle
import datetime
conn = cx_Oracle.connect('un/pwd#ip:port/db')
cur = conn.cursor()
sql = """
SELECT *
FROM table1
WHERE deliveredDate = :0 AND productID IN (:1,:2)
"""
cur.execute(sql,[datetime.datetime(2022, 5, 3),1,2])
res = cur.fetchall()
print(res)

The key part of your question was the 'unknown length' for the IN clause. The cx_Oracle documentation Binding Multiple Values to a SQL WHERE IN Clause shows various solutions each with some pros & cons depending on size of the list and the number of times the statement will be executed. For most cases you will not want to bind to a single placeholder in your statement IN list because of performance implications. If there is an upper bound on the size of the IN list, then put that many placeholders and bind None for all unknown values. The doc example explains it better:
cursor.execute("""
select employee_id, first_name, last_name
from employees
where last_name in (:name1, :name2, :name3, :name4, :name5)""",
name1="Smith", name2="Taylor", name3=None, name4=None, name5=None)
for row in cursor:
print(row)
(This uses keyword parameters to match the bind placeholders, but you can use a list instead).
Other solutions are shown in that doc link.

python pyodbc SQLite sql injections

I use pyodbc in my python flask Project for the SQLite DB connection.
I know and understand SQL Injections but this is my first time dealing with it.
I tried to execute some
I have a function which concatenates the SQL String in my database.py file:
def open_issue(self, data_object):
cursor = self.conn.cursor()
# data_object is the issue i get from the user
name = data_object["name"]
text = data_object["text"]
rating_sum = 0
# if the user provides an issue
if name:
# check if issue is already in db
test = cursor.execute(f'''SELECT name FROM issue WHERE name = "{name}"''')
data = test.fetchall()
# if not in db insert
if len(data) == 0:
# insert the issue
cursor.executescript(f'''INSERT INTO issue (name, text, rating_sum)
VALUES ("{name}", "{text}", {rating_sum})''')
else:
print("nothing inserted!")
In the api.py file the open_issue() function gets called:
#self.app.route('/open_issue')
def insertdata():
# data sent from client
# data_object = flask.request.json
# unit test dictionary
data_object = {"name": "injection-test-table",
"text": "'; CREATE TABLE 'injected_table-1337';--"}
DB().open_issue(data_object)
The "'; CREATE TABLE 'injected_table-1337';--" sql injection has not created the injected_table-1337, instead it got inserted normally like a string into the text column of the injection-test-table.
So i don't really know if i am safe for the standard ways of SQL injection (this project will only be hosted locally but good security is always welcome)
And secondary: are there ways with pyodbc to check if a string contains sql syntax or symbols, so that nothing will get inserted in my example or do i need to check the strings manually?
Thanks a lot

As it turns out, with SQLite you are at much less risk of SQL injection issues because by default neither Python's built-in sqlite3 module nor the SQLite ODBC driver allow multiple statements to be executed in a single .execute call (commonly known as an "anonymous code block"). This code:
thing = "'; CREATE TABLE bobby (id int primary key); --"
sql = f"SELECT * FROM table1 WHERE txt='{thing}'"
crsr.execute(sql)
throws this for sqlite3
sqlite3.Warning: You can only execute one statement at a time.
and this for SQLite ODBC
pyodbc.Error: ('HY000', '[HY000] only one SQL statement allowed (-1) (SQLExecDirectW)')
Still, you should follow best practices and use a proper parameterized query
thing = "'; CREATE TABLE bobby (id int primary key); --"
sql = "SELECT * FROM table1 WHERE txt=?"
crsr.execute(sql, (thing, ))
because this will also correctly handle parameter values that would cause errors if injected directly, e.g.,
thing = "it's good to avoid SQL injection"

Problem executing raw SQL query in Python: sqlalchemy.exc.programmingerror syntax error at or near

I have a query in a python script that creates a materialized view after some tables get created.
Script is something like this:
from sqlalchemy import create_engine, text
sql = '''CREATE MATERIALIZED VIEW schema1.view1 AS 
       SELECT t1.a,
         t1.b,
         t1.c,
         t2.x AS d
       FROM schema1.t1 t1
LEFT JOIN schema1.t2 t2 ON t1.f = t2.f
       UNION ALL
       SELECT t3.a, 
         t3.b, 
         t3.c, 
         t3.d
       FROM schema1.t3 t3;'''
con=create_engine(db_conn)
con.execute(sql)
The query successfully executes when I run on the database directly.
But when running the script in python, I get an error:
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.SyntaxError) syntax error at or near "CREATE MATERIALIZED VIEW schema"
I can't for the life of me figure out what it has an issue with - any ideas?

This was the weirdest thing. I had copied my query text out of another tool that I use to navigate around my pg DB into VS Code. The last part of the answer by #EOhm gave me the idea to just type the whole thing out in VS Code instead of copy/pasting.
And everything worked.
Even though the pasted text and what I typed appear identical in every way. So apparently there was some invisible formatting causing this issue.

I don't know wether SQLAlchemy suports MView-Creation, but if it should be similiar or done with specific Metadata functions (https://docs.sqlalchemy.org/en/13/core/schema.html).
The text function is designed for database indepenendent DML, not DDL. Maybe it works for DDL (I don't know about SQLAlchemy) but by design the syntax is different than when You would execute directly on the database as SQLAlchemy shall abstract the details of databases from user.
If SQLAlchemy does no offer some convenient way for that and You nevertheless have valid reasons to use SQLAlchemy at all, You can just execute the plain SQL Statememt in the dialect the database backend understands, so just omit the sqlalchemies text function for the SQL statement, like:
from sqlalchemy import create_engine, text
sql = '''CREATE MATERIALIZED VIEW schema.view1 AS
SELECT t1.a,
t1.b,
t1.c
t2.x AS d
FROM schema.t1 t1
LEFT JOIN schema.t2 t2 ON t1.f = t2.f
UNION ALL
SELECT t3.a,
t3.b,
t3.c,
t3.d
FROM schema.t3 t3;'''
con=create_engine(db_conn)
con.raw_connection().cursor().execute(sql)
(But of course You have to take care for the backend type then opposed to the SQLAlchemy wrapped statements.)
I tested on my pg server without any issues using psycopg2 directly.
postgres=# create schema schema;
CREATE TABLE
postgres=# create table schema.t1 (a varchar, b varchar, c varchar, f integer);
CREATE TABLE
postgres=# create table schema.t2 (x varchar, f integer);
CREATE TABLE
postgres=# create table schema.t3 (a varchar, b varchar, c varchar, d varchar);
CREATE TABLE
postgres=# commit;
With the following script:
#!/usr/bin/python3
import psycopg2;
conn = psycopg2.connect("dbname=postgres")
cur = conn.cursor()
cur.execute("""
CREATE MATERIALIZED VIEW schema.view1 AS
SELECT t1.a,
t1.b,
t1.c,
t2.x AS d
FROM schema.t1 t1
LEFT JOIN schema.t2 t2 ON t1.f = t2.f
UNION ALL
SELECT t3.a,
t3.b,
t3.c,
t3.d
FROM schema.t3 t3;
""")
conn.commit()
cur.close()
conn.close()
I tested with quite current versions of python3.7/2.7 and current version of psycopg2 module and current libraries (I have 11.5 pg client and 2.8.3 psycopg2) from pgdg installed on a quite recent linux? Can You try to execute directly on psycopg2 like I did?
Also did You make sure Your dots are plain ascii dots as all the other characters in the statement are in this case? (Also keep in mind there can be invisible codepoints in unicode that can cause such sort of problems.) Maybe You can convert Your string to ASCII binary and back to Unicode-String if You are on Python. If it does not raise an error on .encode('ASCII') it should be clean.

Why can't IPython return records with multiple fields when submitting a query to sqlite?

I am trying to write a simple query to an sqlite database in a python script. To test if my parameters were correct, I tried running the query from the ipython command line. It looked something like this:
import sqlite3
db = 'G:\path\to\db\file.sqlite'
conn = sqlite3.connect(db)
results = conn.execute('SELECT * FROM studies').fetchall()
for some reason, my results came back totally empty. Then I tried another test query:
results = conn.execute('SELECT id FROM studies').fetchall()
Which returned correctly. I figured there was a problem with the asterisk [WRONG, SEE SECOND UPDATE BELOW], so I tried the 'SELECT * FROM studies' query from a default python command line. Lo and behold, it returned correctly. I tried all the normal ways to escape the asterisk only to be met by a wide variety of error messages. Is there any way to run this query in IPython?
EDIT: Sorry, I incorrectly assumed IronPython and IPython were the same. What I meant was the IPython command line, not the IronPython framework.
EDIT2: Okay, it turns out the asterisk DOES work as shown by this successful query:
'SELECT COUNT(*) FROM studies'
From the suggestions posted here, it turns out the error results from trying to return records with multiple fields, i.e.:
'SELECT field1,field2 FROM studies'
which still results in to records being returned. I have changed the title of the question accordingly.

This is SQL. IronPython has little or nothing to do with the processing of the query. Are you using an unusual character encoding? (IE not UTF-8 or ASCII)?
What happens if you SELECT id,fieldname,fieldname FROM studies (In other words, simulating what '*' does.)

Some more debugging you could try:
s = 'SELEECT * from studies'
print s
conn.execute(s).fetchall()
or:
s = 'SELECT ' + chr(42) + ' from studies'
conn.execute(s).fetchall()
You might also try:
conn.execute('select count(*) from studies').fetchall()
if that comes back as [(0,)] then something really weird is going on :-)
Some more things you could try:
conn.execute('select id from (select * from studies)').fetchall()
or:
cur = conn.cursor()
cur.execute('select * from studies').fetchall()

I've tried all the things you've mentioned in IPython and sqlite without any problems (ipython 0.9.1, python 2.5.2).
Is there a chance this is some kind of version mismatch issue? Maybe your shells are referencing different libraries?
For example, does
import sqlite3; print sqlite3.version
return the same thing from both shells (i.e. ipython and the regular one where the sql query works)?
How about
conn.execute('select sqlite_version()').fetchall()
Does that return the same thing?

Just a wild guess, but please try to escape backslashes in the path to the database file. In other words instead of
db = 'G:\path\to\db\file.sqlite'
try
db = 'G:\\path\\to\\db\\file.sqlite'

I had a similar problem and found the only way to get the correct output was by assigning the results of the execute statement to another cursor object and calling fetchall on that:
In [1]: import sqlite3
In [2]: conn = sqlite3.connect('file.db')
In [3]: cur = conn.cursor()
In [4]: cur2 = cur.execute("<STATEMENT>")
In [5]: cur2.fetchall()