I'm trying to make an API in flask, but when I load up the API and then change the data in the database and reload the page the data doesn't change unless I restart the program. How do I make it so when the program is running and I change the data in the database I'll see it when I refresh the API in the browser? Thanks in Advance.
Here's the code:
from flask import Flask, jsonify, request
import pymysql.cursors
connection = pymysql.connect(host='127.0.0.1',
user='root',
password='myroot',
db='test',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
app = Flask(__name__)
#app.route('/', methods=['GET'])
def test():
with connection.cursor() as cursor:
connection.connect()
sql = "SELECT * FROM importantData"
cursor.execute(sql)
result = cursor.fetchall()
return jsonify(result)
#app.route('/locations', methods=['GET'])
def test1():
with connection.cursor() as cursor:
connection.connect()
sql = "SELECT * FROM locations"
cursor.execute(sql)
result = cursor.fetchall()
return jsonify(result)
if __name__ == '__main__':
app.run(port=8080)
EDIT:
The solution was that I had to open the database connection in every route function, edited the code so now it works as it's supposed to.
I have never used pymysql, and your code looks fine except that part:
result = cursor.fetchone()
Why fetchone() when you are clearly select * selecting everything? What do you mean by I change data in database, did you change a row, or did you insert a new row?
In anycase, looks like you are selecting only the first row with fetchone(), so even if you changed something else, or if you added a new row, you will never see the new modifications
The solution was that I had to open the database connection in every route function, edited the code so now it works as it's supposed to
Related
Let me start off by saying I am extremely new to Python and Postgresql so I feel like I'm in way over my head. My end goal is to get connected to the dvdrental database in postgresql and be able to access/manipulate the data. So far I have:
created a .config folder and a database.ini is within there with my login credentials.
in my src i have a config.py folder and use config parser, see below:
def config(filename='.config/database.ini', section='postgresql'):
# create a parser
parser = ConfigParser()
# read config file
parser.read(filename)
# get section, default to postgresql
db = {}
if parser.has_section(section):
params = parser.items(section)
for param in params:
db[param[0]] = param[1]
else:
raise Exception('Section {0} not found in the {1} file'.format(section, filename))
return db
then also in my src I have a tasks.py file that has a basic connect function, see below:
import pandas as pd
from clients.config import config
import psycopg
def connect():
""" Connect to the PostgreSQL database server """
conn = None
try:
# read connection parameters
params = config()
# connect to the PostgreSQL server
print('Connecting to the PostgreSQL database...')
conn = psycopg.connect(**params)
# create a cursor
cur = conn.cursor()
# execute a statement
print('PostgreSQL database version:')
cur.execute('SELECT version()')
# display the PostgreSQL database server version
db_version = cur.fetchone()
print(db_version)
# close the communication with the PostgreSQL
cur.close()
except (Exception, psycopg.DatabaseError) as error:
print(error)
finally:
if conn is not None:
conn.close()
print('Database connection closed.')
if __name__ == '__main__':
connect()
Now this runs and prints out the Postgresql database version which is all well & great but I'm struggling to figure out how to change the code so that it's more generalized and maybe just creates a cursor?
I need the connect function to basically just connect to the dvdrental database and create a cursor so that I can then use my connection to select from the database in other needed "tasks" -- for example I'd like to be able to create another function like the below:
def select_from_table(cursor, table_name, schema):
cursor.execute(f"SET search_path TO {schema}, public;")
results= cursor.execute(f"SELECT * FROM {table_name};").fetchall()
return results
but I'm struggling with how to just create a connection to the dvdrental database & a cursor so that I'm able to actually fetch data and create pandas tables with it and whatnot.
so it would be like
task 1 is connecting to the database
task 2 is interacting with the database (selecting tables and whatnot)
task 3 is converting the result from 2 into a pandas df
thanks so much for any help!! This is for a project in a class I am taking and I am extremely overwhelmed and have been googling-researching non-stop and seemingly end up nowhere fast.
The fact that you established the connection is honestly the hardest step. I know it can be overwhelming but you're on the right track.
Just copy these three lines from connect into the select_from_table method
params = config()
conn = psycopg.connect(**params)
cursor = conn.cursor()
It will look like this (also added conn.close() at the end):
def select_from_table(cursor, table_name, schema):
params = config()
conn = psycopg.connect(**params)
cursor = conn.cursor()
cursor.execute(f"SET search_path TO {schema}, public;")
results= cursor.execute(f"SELECT * FROM {table_name};").fetchall()
conn.close()
return results
I want to connect to MySql database using Python through PythonAnywhere, without creating a Flask/Django application.
I have seemingly managed to connect through MySQLdb, using the code below, but I do not receive a response when I run the code. Any solutions?
import MySQLdb
db = MySQLdb.connect(
host = "myuser.mysql.pythonanywhere-services.com",
user = "myuser",
passwd = XXX,
db = "myuser$db_name"
)
cursor = db.cursor()
cursor.execute("SELECT * FROM table_name")
for x in cursor:
print(x)
cursor.close()
db.close()
You retrieve all rows in the table, without error.
cursor.execute("SELECT * FROM table_name")
for x in cursor:
print(x)
Yet you see no output. This is normal for a table that contains zero rows.
Consider doing one or more INSERTs, and a COMMIT,
prior to the query.
I tried a lot however I am unable to copy data available as json file in S3 bucket(I have read only access to the bucket) to Redshift table using python boto3. Below is the python code which I am using to copy the data. Using the same code I was able to create the tables in which I am trying to copy.
import configparser
import psycopg2
from sql_queries import create_table_queries, drop_table_queries
def drop_tables(cur, conn):
for query in drop_table_queries:
cur.execute(query)
conn.commit()
def create_tables(cur, conn):
for query in create_table_queries:
cur.execute(query)
conn.commit()
def main():
try:
config = configparser.ConfigParser()
config.read('dwh.cfg')
# conn = psycopg2.connect("host={} dbname={} user={} password={} port={}".format(*config['CLUSTER'].values()))
conn = psycopg2.connect(
host=config.get('CLUSTER', 'HOST'),
database=config.get('CLUSTER', 'DB_NAME'),
user=config.get('CLUSTER', 'DB_USER'),
password=config.get('CLUSTER', 'DB_PASSWORD'),
port=config.get('CLUSTER', 'DB_PORT')
)
cur = conn.cursor()
#drop_tables(cur, conn)
#create_tables(cur, conn)
qry = """copy DWH_STAGE_SONGS_TBL
from 's3://udacity-dend/song-data/A/A/A/TRAAACN128F9355673.json'
iam_role 'arn:aws:iam::xxxxxxx:role/MyRedShiftRole'
format as json 'auto';"""
print(qry)
cur.execute(qry)
# execute a statement
# print('PostgreSQL database version:')
# cur.execute('SELECT version()')
#
# # display the PostgreSQL database server version
# db_version = cur.fetchone()
# print(db_version)
print("Executed successfully")
cur.close()
conn.close()
# close the communication with the PostgreSQL
except Exception as error:
print("Error while processing")
print(error)
if __name__ == "__main__":
main()
I don't see any error in the Pycharm console but I see Aborted status in the redshift query console. I don't see any reason why it has been aborted(or I don't know where to look for that)
Other thing that I have noticed is when I run the copy statement in Redshift query editor , it runs fine and data gets moved into the table. I tried to delete and recreate the cluster but no luck. I am not able to figure what I am doing wrong. Thank you
Quick read - it looks like you haven't committed the transaction and the COPY is rolled back when the connection closes. You need to either change the connection configuration to be in "autocommit" or add an explicit "commit()".
It's more of a theoratical question but i have been trying to find a correct answer of it for hours and yet i have't arrived at a solution. I have a big flask app and it contains multiple routes.
#app.route('/try'):
#app.route('/new'):
and many others. I am using MySQLdb for database purpose. Before i was having this in the starting of the application.
import MySQLdb as mysql
db = mysql.connect('localhost', 'root', 'password', 'db')
cursor = db.cursor()
It works fine but after a time, it generates a error "Local Variable 'cursor' referenced before assignment.". This may be due to the reason that after a time mysql closes a connection. So, i entered
cursor=db.cursor() in every route function and close it afer i have done the processing like this:
db = mysql.connect('localhost', 'root', 'password', 'db')
#app.route('/')
def home():
cursor=db.cursor()
...some processing...
cursor.close()
return render_template('home.html')
#app.route('/new')
def home_new():
cursor=db.cursor()
...some processing...
cursor.close()
return render_template('homenew.html')
Now i want to ask is this approach right? Should i define a cursor for each request and close it?
This is how I have my MySQLdb setup
def requestConnection():
"Create new connection. Return connection."
convt = cv.conversions.copy()
convt[3] = int
convt
conn = db.connect(host=c.SQL_HOST, port=c.SQL_PORT, user=c.SQL_USER, passwd=c.SQL_PASSWD, db=c.SQL_DB, conv=convt, use_unicode=True, charset="utf8")
return conn
def requestCursor(conn):
return conn.cursor(db.cursors.DictCursor)
Then, at the start of every SQL function I do this:
def executeQuery(query):
"Execute a given query. Used for debug purpouses."
conn = requestConnection()
cur = requestCursor(conn)
cur.execute(query)
r = cur.fetchall()
cur.close()
conn.close()
return r
I change conversions because I had to change int values in DB from Float to int due to my work, but you can skip this step.
If not, you need to import this:
import MySQLdb as db # https://github.com/farcepest/MySQLdb1
import MySQLdb.converters as cv
Hope it helps!
I'm trying to run a Python script which leaves a connection open permanently, and responds to changes made outside of the script.
So for example:
Data script: Accepts form posts and commits form data to the database
Worker script: Monitors the database for new form posts and takes action accordingly
The relevant code in the worker script is:
import pymysql
conn = pymysql.connect(host='127.0.0.1', port=3306, user='dbuser', passwd='dbpass', db='my_db')
def processForms(Formdat):
c = conn.cursor(pymysql.cursors.DictCursor)
myform.sendEmail(c)
conn.commit()
c.close()
def doForms():
while True:
... get data and store in 'myforms' ...
futures = [executor.submit(processForms, myform) for myform in myforms]
time.sleep(30)
doForms()
Now I don't understand why this is not picking up new forms... If I create a new connection in each iteration of doForms(), the new forms are picked up, but I don't want to be creating and destroying connections all the time.
For example, this modification works:
conn = None
def doForms():
while True:
global conn
conn = pymysql.connect(host='127.0.0.1', port=3306, user='root', passwd='', db='mw_py')
... get data and store in 'myforms' ...
futures = [executor.submit(processForms, myform) for myform in myforms]
conn.close()
time.sleep(30)
Is there a way for me to use the open connection and have it poll the latest data?
Open 1 connection in the beginning of your script. It is not a cheap operation to connect.
Remember ID of last row fetched.
On every iteration select rows with ID greater than last seen.