I am trying to run a SQL "SELECT" query in Postgres from Python using Psycopg2. I am trying to compose the query string as below, but getting error message, using psycopg2 version 2.9.
from psycopg2 import sql
tablename = "mytab"
schema = "public"
query = sql.SQL("SELECT table_name from information_schema.tables where table_name = {tablename} and table_schema = {schema};")
query = query.format(tablename=sql.Identifier(tablename), schema=sql.Identifier(schema))
cursor.execute(query)
result = cursor.fetchone()[0]
Error:
psycopg2.error.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block
Can someone please help. Thanks.
In the (a bit strange) query
select table_name
from information_schema.tables
where table_name = 'mytab'
and table_schema = 'public';
'mytab' and 'public' are literals, not identifiers. For comparison, mytab is an identifier here:
select *
from mytab;
Thus your format statement should look like this:
query = query.format(tablename=sql.Literal(tablename), schema=sql.Literal(schema))
Note that the quoted error message is somewhat misleading as it is about executing a query other than what is shown in the question.
Since this query is only dealing with dynamic values it can be simplified to:
import psycopg2
con = psycopg2.connect(<params>)
cursor = con.cursor()
tablename = "mytab"
schema = "public"
# Regular placeholders
query = """SELECT
table_name
from
information_schema.tables
where
table_name = %s and table_schema = %s"""
cursor.execute(query, [tablename, schema])
result = cursor.fetchone()[0]
# Named placeholders
query = """SELECT
table_name
from
information_schema.tables
where
table_name = %(table)s and table_schema = %(schema)s"""
cursor.execute(query, {"table": tablename, "schema": schema})
result = cursor.fetchone()[0]
Related
Am new to Postgres. Anyone can tell how to have it work?
What I want to do is to write Pandas datataframe to PostgreSQL database. I have already created a database 'customer' and table 'users'.
I am creating a simple Pandas dataframe as follows:
data = {'Col1':[1,2,3,4,5], 'Col2':[1,2,3,4,5]}
df = pd.DataFrame(data)
After that I am creating Postgres database connection to my 'customer' database follows:
conn = psycopg2.connect(
database="customer", user='postgres', password='password', host='127.0.0.1', port= '5432')
Then, I am using the following command to insert records from dataframe into table 'users':
df.to_sql('users', conn, if_exists='replace')
conn.commit()
conn.close()
Error that I am getting is:
pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': syntax error at or near ";"
LINE 1: ...ELECT name FROM sqlite_master WHERE type='table' AND name=?;
^
df.to_sql() does not work for "conn" in psycopg2. It is for "engine" in sqlalchemy. For psycopg2, try insert instead:
Step 1: Creation of an empty table
First you need to create a cursor and then create a table:
cursor = conn.cursor()
cursor.execute("CREATE TABLE users_table (col1 integer, col2 integer)")
conn.commit()
Step 2: Insert pandas df to the users_table
tuples = [tuple(x) for x in df.to_numpy()]
cols = ','.join(list(df.columns))
query = "INSERT INTO %s(%s) VALUES(%%s,%%s)" % (users_table, cols) #two columns
cursor.executemany(query, tuples)
conn.commit()
If you want to use df.to_sql():
from sqlalchemy import create_engine
engine = create_engine('postgresql+psycopg2://user:password#hostname/database_name')
df.to_sql('users', engine)
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_sql.html
I'm aware that the best way to prevent sql injection is to write Python queries of this form (or similar):
query = 'SELECT %s %s from TABLE'
fields = ['ID', 'NAME']
cur.execute(query, fields)
The above will work for a single query, but what if we want to do a UNION of 2 SQL commands? I've set this up via sqlite3 for sake of repeatability, though technically I'm using pymysql. Looks as follows:
import sqlite3
conn = sqlite3.connect('dummy.db')
cur = conn.cursor()
query = 'CREATE TABLE DUMMY(ID int AUTO INCREMENT, VALUE varchar(255))'
query2 = 'CREATE TABLE DUMMy2(ID int AUTO INCREMENT, VALUE varchar(255)'
try:
cur.execute(query)
cur.execute(query2)
except:
print('Already made table!')
tnames = ['DUMMY1', 'DUMMY2']
sqlcmds = []
for i in range(0,2):
query = 'SELECT %s FROM {}'.format(tnames[i])
sqlcmds.append(query)
fields = ['VALUE', 'VALUE']
sqlcmd = ' UNION '.join(sqlcmds)
cur.execute(sqlcmd, valid_fields)
When I run this, I get a sqlite Operational Error:
sqlite3.OperationalError: near "%": syntax error
I've validated the query prints as expected with this output:
INSERT INTO DUMMY VALUES(%s) UNION INSERT INTO DUMMY VALUES(%s)
All looks good there. What is the issue with the string substitutions here? I can confirm that running a query with direct string substitution works fine. I've tried it with both selects and inserts.
EDIT: I'm aware there are multiple ways to do this with executemany and a few other. I need to do this with UNION for the purposes I'm using this for because this is a very, very simplified example fo the operational code I'm using
The code below executes few INSERTS at once
import sqlite3
conn = sqlite3.connect('dummy.db')
cur = conn.cursor()
query = 'CREATE TABLE DUMMY(ID int AUTO INCREMENT NOT NULL, VALUE varchar(255))'
try:
cur.execute(query)
except:
print('Already made table!')
valid_fields = [('ya dummy',), ('stupid test example',)]
cur.executemany('INSERT INTO DUMMY (VALUE) VALUES (?)',valid_fields)
Here i am trying to enclose an appended value coming from function into a sql string in python
but somehow i am not able to it.
getname is returning Puru
def getValue(getname):
sql= '''SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_CATALOG = '''+getname
I want to return it in this form in single quotes
SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_CATALOG = 'Puru'
Here are three ways to achieve what you want:
string concatenation
sql = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_CATALOG = '"+getname+"'"
string.format()
sql = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_CATALOG = '{}'".format(getname)
format string (requires python 3.6+)
sql = f"SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_CATALOG = '{getname}'"
For your understanding other than str.format() python also provides Template class from string module.
Templates provide simpler string substitutions in python. Instead of the normal %-based substitutions, Templates support $-based substitutions.
for ex:
>>> from string import Template
>>> sql = Template("SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES "
"WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_CATALOG = '$getname'")
>>> sql.substitute(getname='Puru')
"SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_CATALOG = 'Puru'"
The $getname names a substitution placeholder matching a mapping key of "getname". But I would recommend you to use str.format() method which is more readable and widely used for string formatting in python.
You need to replace the ''' with ". Otherwise Python will not know how to interpret the ' in the middle. If you don't want the line to get too long, you can wrap in in parentheses:
sql = ("SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES "
"WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_CATALOG = ") + getname
Edit: I have to point out the dangers of SQL injection and creating these queries on the fly. There are numerous libraries that will do it for you in a safe manner. But I have a feeling/hope this won't end up in production code anyway.
I want to fetch data using my python code like we do with describe [tableName] statement. I want to do that on Redshift and DB2.
I tried to do that using Pandas and cursors, I tried the following chunks of commands:
"set search_path to SCHEMA; select * from pg_table_def where schemaname = 'schema' and LOWER(tablename) = 'TableName';
describe schema.tableName
select column_name, data_type, character_manimum_length from information_schema.columns where table_schema = 'Schema' and table_name = 'TableName';
\d or \d+
.
import psycopg2
import numpy as np
import pandas as pd
con=psycopg2.connect(dbname= 'DBNAME', host="Host-Link",
port= '5439', user= 'username', password= 'password')
print(con)
cur = con.cursor()
query = "set search_path to Schema; select * from pg_table_def where schemaname = 'Schema' and LOWER(tablename) = 'TableName';"
cur.execute(query)
temp = cur.fetchall()
print(temp)
data_frame = pd.read_sql("set search_path to Schema; select * from pg_table_def where schemaname = 'Schema' and LOWER(tablename) = 'TableName';", con)
print(data_frame)
con.close()
I want the output as following:
COLUMN_NAME DATA_TYPE PK NULLABLE DEFAULT AUTOINCREMENT COMPUTED REMARKS POSITION
col1 varchar(10) YES NO NO NO 1
col2 varchar(50) NO NO NO NO 2
col3 smallint NO NO NO NO 3
A lot of this data is included in the SVV_COLUMNS system view. You can query that table using the table_name and table_schema columns.
https://docs.aws.amazon.com/redshift/latest/dg/r_SVV_COLUMNS.html
I have a list that contains the name of columns I want to retrieve from a table in the database.
My question is how to make the cursor select columns specified in the list. Do I have to convert nameList to a string variable before include it in the select statement? Thanks
nameList = ['A','B','C','D',...]
with sqlite3.connect(db_fileName) as conn:
cursor = conn.cursor()
cursor.execute("""
select * from table
""")
As long as you can be sure your input is sanitized -- to avoid SQL injection attack -- you can do:
...
qry = "select {} from table;"
qry.format( ','.join(nameList) )
cursor.execute(qry)
If you're on a really old version of Python do instead:
...
qry = "select %s from table;"
qry % ','.join(nameList)
cursor.execute(qry)
nameList = ["'A(pct)'",'B','C','D',...]
with sqlite3.connect(db_fileName) as conn:
cursor = conn.cursor()
cursor.execute("""
select {} from table
""".format(", ".join(nameList)))