I am using Python 3.6 with Flask. I am trying to connect to Amazon Redshift database using SqlAlchemy. The query has IN operation in the Where clause and the values for it are sent by binding it to the query. For some reason it does not work? It does not error out, but does not fetch any results. If I hard code the values in the query, it works fine.
I have tried a few suggested options but no luck -
1. binding the values as a list or just as comma separated string
2. removing the brackets in the query
3. changing the code to
stmt = text(file.read())
stmt = stmt.bindparams(search = ids)
df = pd.read_sql_query(stmt, connection)
dtv_script.txt
Select * from tbl1 where id IN (:search)
def get_dt(id_list):
engine = create_engine('postgresql://xxxxxxxxxx')
connection = engine.connect()
ids = list(id_list.split(","))
#dtv_script.txt has the sql
file = open('dtv_script.txt')
sql = text(file.read())
df = pd.read_sql_query(sql, connection, params={'search' : ids})
connection.close()
return df
The ids are posted from a form on the index.html.
Sample ids = 2011592,2021593,2033591.
The flask route page captures it in the get_dt() function and returns the dataframe back to the results.html page for display
#app.route('/result', methods=['POST'])
def result():
if request.method == 'POST':
id_list = request.form.get('ids')
df_dt = dofri.get_dt(id_list)
return render_template('result.html', **locals())
else:
flash('There was some error. Check the logs')
return index()
Below is the solution. Make sure to
add - from sqlalchemy import bindparam
remove - brackets from the query
add - expanding=True
dtv_script.txt
Select * from tbl1 where id IN :search
def get_dt(id_list):
engine = create_engine('postgresql://xxxxxxxxxx')
connection = engine.connect()
ids = list(id_list.split(","))
#dtv_script.txt has the sql
file = open('dtv_script.txt')
sql = file.read()
t = text(sql)
t = t.bindparams(bindparam('search', expanding=True))
df = pd.read_sql_query(t, connection, params={'search' : ids })
connection.close()
return df
Related
So I am fairly new to flask and I am currently trying to create a flask api for a project I am working on. However, there are a couple of issues I am facing.
So for my 1st issue, I can't get my dataframe from the 1st function to work in my second function. I am just wondering how I can get the data_1 to work in the second function.
Code:
from flask import Flask
from sqlalchemy import create_engine
import sqlite3 as sql
import pandas as pd
import datetime
import os
app = Flask(__name__)
#app.route('/', methods=['GET'])
def get_data():
...
data_1 = ...
#print(data_1.head(n=10))
return "hello"
#app.route('/table1', methods=['GET'])
def store_table1_data_df():
db_path = os.path.join(os.path.dirname(__file__),'table1.db')
engine = create_engine('sqlite:///{}'.format(db_path), echo=True)
sqlite_connection = engine.connect()
sqlite_table = 'table1'
data_1.to_sql(sqlite_table,sqlite_connection, if_exists='append')
sqlite_connection.close()
return "table1"
For my second issue, is there a better way of storing a dataframe within flask api using sqlalchemy or sqlite3?
More context as to what kind of data_1 is: data_1 can only hold the past 15 days/records like from 6/15/2021-6/30/2021. However, tomorrow, if I fetch the newest data_1 it will contain 6/16/2021-7/01/2021. How can I just append 07/01/2021 to the old data_1 without creating duplicate records from 06/16/2021, creating two more functions, and an extra db file?
#app.route('/table1', methods=['GET'])
def store_table1_data_df():
db_path = os.path.join(os.path.dirname(__file__),'table1.db')
engine = create_engine('sqlite:///{}'.format(db_path), echo=True)
sqlite_connection = engine.connect()
sqlite_table = 'table1'
data_1.to_sql(sqlite_table,sqlite_connection, if_exists='append')
sqlite_connection.close()
return "table1"
#app.route('/table2', methods=['GET'])
def store_table2_data_df():
db_path2 = os.path.join(os.path.dirname(__file__),'table2.db')
engine2 = create_engine('sqlite:///{}'.format(db_path2), echo=True)
sqlite_connection2 = engine2.connect()
sqlite_table2 = 'table2'
data_1.to_sql(sqlite_table2,sqlite_connection2, if_exists='append')
sqlite_connection2.close()
return "table2"
# What I probably have down below is not the correct way to solve this problem
#app.route('/table1', methods=['GET'])
conn = sql.connect("table1.db")
cur = conn.cursor()
#cur.execute
cur.execute("ATTACH 'table2.db' as 'table2' ")
conn.commit()
table_3 = pd.read_sql_query("SELECT DISTINCT date, value FROM table1 UNION SELECT DISTINCT date, value from table2 ORDER BY date", conn)
cur.execcute("SELECT DISTINCT date, value FROM table1 UNION SELECT DISTINCT date, value from table2 ORDER BY date")
conn.commit()
results3 = cur.fetchall()
sqlite_table='table1'
table_3.to_sql(sqlite_table, conn, if_exists='replace')
cur.close()
conn.close()
return "work"
Any help is greatly appreciated.
For your 1st problem. You may do either of these:
If the size of data-1 is small(than 200kb) you may use flask-session to store the data and access it across routes.
You create a function that returns data_1. Call that function in any route you want. Hint:
def getdata1(val1, val2):
#calculation here
return data_1
Just call this wherever you need data_1.
Store the data frame in a DB and fetch it.
For the second part, a simple for loop will work. Hint on that:
sql_table = ["Fetch your sql table here with the dataframe. Considering dates in one column"]
data_1 = ["Your dataframe"]
for i in data_1['Dates']:
if i != sql_table['dates']:
#insert this key:value pair in sql table
If your data frame and sql is getting loaded in order by date, even better. You just need to check the last elements of each.
I am trying to send a query to a Teradata Database through a python script using the jaydebapi package. I have had no issues pulling data in the past until I tried to use a wild card in a like statement
SELECT .... FROM TABLE WHERE column LIKE 'value%'
JayDeBeApi says that this package provides a Python DB-API v2.0 connection to a database. Looking at the parameter usage (and trying many many different combinations of formatting the parameter strings) I am not pulling back any records.
When I run this query in Teradata SQL Assistant, however, I do find records.
How do I properly send a query with a parameter that has a wildcard character?
Here is my code:
import jaydebeapi
import pandas as pd
from tabulate import tabulate
jars = ['C:\\Users\\<path to TD jars>\\tdgssconfig.jar', 'C:\\Users\\<path to TD jars>\\terajdbc4.jar']
user = 'USER_NAME'
password = "PASSWORD"
jclassname = 'com.teradata.jdbc.TeraDriver'
url = "jdbc:teradata://<URL_TO_DB>/LOGMECH=LDAP,TMODE=ANSI,CHARSET=UTF8"
driver_args = [url, user, password]
class Teradata(object):
def __init__(self):
self.conn = jaydebeapi.connect(jclassname, driver_args, jars)
self.cursor = self.conn.cursor()
def search_by_base_address(account):
td = Teradata()
address1 = account.base_address
city = account.city
state = account.state
q = '''
SELECT
BUSINESS_NAME,
SECONDARY_NAME,
STREET_ADDR,
STREET_ADDR2,
CITY,
STATE,
POSTAL_CD,
FROM DB.TABLE
WHERE STREET_ADDR LIKE ?
AND CITY = '{}'
AND STATE = '{}';
'''.format(city, state)
if len(address1) > 1:
result = pd.read_sql(q, td.conn, params=(address1+"%",))
print(tabulate(result, headers='keys', tablefmt='psql'))
else:
print('No base address to search')
in the DB I have two events:
{1: {'title': 'APPLE'}, 2: {'title': 'BANANA'}}
as you can see bellow, I am calling the deletefromDB function on refresh with the parameter of the second event and I am printing out the data (the json above). My issue is, that the data will change, only after I restart my flask server. Till that point it's the same json output. Can somebody explain what I am doing wrong?
this is my code:
import sqlite3
from flask import Flask, render_template, request
app = Flask(__name__)
data = {}
conn = sqlite3.connect('events.db')
cursor = conn.cursor()
# INIT
def initDB():
with conn:
cursor.execute("SELECT * FROM Events")
rows = cursor.fetchall()
for row in rows:
temp = {}
temp["title"] = row[1]
data[row[0]] = temp
print(data)
initDB()
def deletefromDB(eventID):
query = 'DELETE FROM Events WHERE EventId = {}'.format(eventID)
with conn:
cursor.execute(query)
print(query)
initDB()
#app.route('/')
def index():
deletefromDB(2)
return 'Index Page'
if __name__ == "__main__":
app.run()
You are caching the tables result into the data dict on database initialization. Then you delete a row from the table without committing the changes to the data dict. Either you reload the entire data dict after deletion or you remove the entry directly from it:
def deletefromDB(eventID):
query = 'DELETE FROM Events WHERE EventId = {}'.format(eventID)
with conn:
cursor.execute(query)
del data[eventID]
print(query)
initDB()
Be careful, you create a formatted SQL string without escaping the parameter (SQL Injection alert). Use the cursor execute to substitute the variables.
query = 'DELETE FROM Events WHERE EventId = %d'
# ...
cursor.execute(query, (eventID,))
Do it whenever you substitute something into a SQL query!
I want to return result in to custom model(class) but I got this error message:
SQL expression, column, or mapped entity expected
I don't know where I went wrong.
connect = Connector()
Session = sessionmaker(bind=connect.ConnectorMySql())
ses = Session()
query = u"""
SELECT
`reports`.ID As 'ID',
reports.Title AS 'ReportTitle',
`reports`.Text as 'ReporText',
`reports`.Status as 'Status',
`user`.ID AS 'ReporterID',
`user`.Name as 'ReporterName' ,
`user`.Username as 'ReporterUserName',
`user`.ImageProfile as 'ReporterAvatar',
`Clinet`.ID AS 'ClinetID',
`Clinet`.SiteUserName AS 'ClinetUserName',
`Clinet`.ImageProfile as 'ClinetAvatar'
FROM reports
JOIN Clinet on reports.ClinetID = `Clinet`.ID
JOIN users user on reports.UserID = `user`.ID
where
:pClinetID IS NULL OR reports.ClinetID=:pClinetID
AND
:pStatus IS NULL OR reports.Status=:pStatus;
"""
QueryResult=CustomModel()
QueryResult=ses.query(CustomModel).from_statement(query).all()
return QueryResult
To use query strings in SQLAlchemy, you have to craft them with text function.
So you should use
QueryResult=ses.query(CustomModel).from_statement(text(query)).all()
When I try to pass a tuple to the IN argument of a WHERE clause, it gets double-quoted so my query fails. For example, if I do this,
# Connect to DB
import MySQLdb
cnxn = MySQLdb.connect(connectString)
curs = cnxn.cursor()
# Setup query
accounts = ('Hyvaco','TLC')
truck_type = 'fullsize'
query_args = (truck_type, accounts)
sql ='SELECT * FROM archive.incoming WHERE LastCapacity=%s AND Account IN %s'
# Run query and print
curs.execute(sql, query_args)
print(curs._executed)
then I get zero rows back, and the query prints out as
SELECT * FROM archive.incoming WHERE LastCapacity='fullsize'
AND Account IN ("'Hyvaco'", "'TLC'")
Switching accounts from a tuple to a list does not affect the result. How should I be passing these arguments?
How about you create the accounts as a string and then do this:
accounts = "('Hyvaco','TLC')"
sql ='SELECT * FROM archive.incoming WHERE LastCapacity=%s AND Account IN '+ accounts