I'm trying to create REST API endpoints using flask framework. This is my fully working script:
from flask import Flask, jsonify
from flask_restful import Resource, Api
from flask_restful import reqparse
from sqlalchemy import create_engine
from flask.ext.httpauth import HTTPBasicAuth
from flask.ext.cors import CORS
conn_string = "mssql+pyodbc://x:x#x:1433/x?driver=SQL Server"
auth = HTTPBasicAuth()
#auth.get_password
def get_password(username):
if username == 'x':
return 'x'
return None
app = Flask(__name__)
cors = CORS(app)
api = Api(app)
class Report(Resource):
decorators = [auth.login_required]
def get(self):
parser = reqparse.RequestParser()
parser.add_argument('start', type = str)
parser.add_argument('end', type = str)
args = parser.parse_args()
e = create_engine(conn_string)
conn = e.connect()
stat = """
select x from report
"""
query = conn.execute(stat)
json_dict = []
for i in query.cursor.fetchall():
res = {'x': i[0], 'xx': i[1]}
json_dict.append(res)
conn.close()
e.dispose()
return jsonify(results=json_dict)
api.add_resource(Report, '/report')
if __name__ == '__main__':
app.run(host='0.0.0.0')
The issue is that I get results when I call this API only for a day or so after which I stop getting results unless I restart my script (or sometimes even my VM) after which I get results again. I reckon there is some issue with the database connection pool or something but I'm closing the connection and disposing it as well. I have no idea why the API gives me results only for some time being because of which I have to restart my VM every single day. Any ideas?
Per my experience, the issue was caused by coding create_engine(conn_string) to create db pool inside the Class Report so that always do the create & destory operations of db pool for per restful request. It's not correct way for using SQLAlchemy ORM, and be cause IO resouce clash related to DB connection, see the engine.dispose() function description below at http://docs.sqlalchemy.org/en/rel_1_0/core/connections.html#sqlalchemy.engine.Engine:
To resolve the issue, you just need to move e = create_engine(conn_string) to the below of the code conn_string = "mssql+pyodbc://x:x#x:1433/x?driver=SQL Server" and remove the code e.dispose() both in the Class Report, see below.
conn_string = "mssql+pyodbc://x:x#x:1433/x?driver=SQL Server"
e = create_engine(conn_string) # To here
In the def get(delf) function：
args = parser.parse_args()
# Move: e = create_engine(conn_string)
conn = e.connect()
and
conn.close()
# Remove: e.dispose()
return jsonify(results=json_dict)
Related
I have a code where, I am establishing connection with MongoDB. This code is
ConnectMongoDB.py:
import pymongo
from pymongo import MongoClient
from flask import Flask, render_template, request,redirect,url_for
app = Flask(__name__)
# Connection to MongoDB
class ConnectMdb:
#staticmethod
def connect2mongodb():
global client
try:
client = pymongo.MongoClient("mongodb") # modified to avoid showing actual string. Kindly ignore this part.
print("Connected to Avengers MongoClient Successfully!!!")
print (type(client))
print(client)
except:
print("Connection to MongoClient Failed!!!")
#db = client.avengers_hack_db
return("Connection established")
if __name__ == '__main__':
ConnectMdb.connect2mongodb()
I import this script in another program which has some business logic. Here is some part of the code which is relevant to this issue:
ProcessData.py:
import pymongo
from pymongo import MongoClient
import datetime
import sys
from flask import Flask, render_template, request
#import ProcessTaskData
#import werkzeug
import ConnectMongoDB as cDB
app = Flask(__name__)
CMdb = cDB.ConnectMdb.connect2mongodb()
db = CMdb.client.avengers_hack_db
#app.route('/')
def index():
return render_template('index.html')
#app.route('/Avengers',methods = ['POST'])
def Avengers():
ip = request.remote_addr
Project_ID = request.form['pid']
Name = request.form['pname']
Resource_Names = request.form['rsrc']
db.ppm_master_db_collection.insert_many([
{"Org_Id":"",
"Name": Name,
"last_modified": datetime.datetime.utcnow()}
])
return render_template('ptasks.html', user_ip=ip)
#app.route('/ProjectTasks',methods = ['POST'])
def ProjectTask():
ip = request.remote_addr
Task_ID = request.form['tid']
TAlert = request.form['talrt']
TResource_Names = request.form['trsrc']
db.ppm_tasks_data_collection.insert_many([
{"Task_ID": Task_ID,
"Resource_Names": TResource_Names,
"last_modified": datetime.datetime.utcnow()}
])
return render_template('ptasks.html', user_ip=ip)
if __name__ == '__main__':
app.run(debug = True)
If I put the code from ConnectMongoDB.py directly in the ProcessData.py rather than importing, it works good. But from separate file it errors.
Also, client is of type:
<class 'pymongo.mongo_client.MongoClient'>
Ideally it is expected to behave normally (establish connection to db as well) like when the code is in the ProcessData.py. Not sure where am I missing.
Changing
db = CMdb.client.avengers_hack_db
to
db = cDB.client.avengers_hack_db
should make your error go away, you are referencing the wrong thing. The return value of your staticmethod is a string, and it has no client attribute.
A bit better approach would be if your connect2mongodb method would return client:
class ConnectMdb:
#staticmethod
def connect2mongodb():
try:
client = pymongo.MongoClient("mongodb") # modified to avoid showing actual string. Kindly ignore this part.
print("Connected to Avengers MongoClient Successfully!!!")
print (type(client))
print(client)
except:
raise Exception("Connection to MongoClient Failed!!!")
return client
This way db = CMdb.client.avengers_hack_db would work.
I have a Flask API based on Flask RestPlus extension and is hosted on Google App Engine. The API does a basic job of fetching data from a Google Cloud SQL PostgreSQL. The API is working fine otherwise but sometimes it starts returning InterfaceError: cursor already closed.
Strangely, when I do a gcloud app deploy, the API starts working fine again.
Here's a basic format of the API:
import simplejson as json
import psycopg2
from flask import Flask, jsonify
from flask_restplus import Api, Resource, fields
from psycopg2.extras import RealDictCursor
app = Flask(__name__)
app.config['SWAGGER_UI_JSONEDITOR'] = True
api = Api(app=app,
doc='/docs',
version="1.0",
title="Title",
description="description")
app.config['SWAGGER_UI_JSONEDITOR'] = True
ns_pricing = api.namespace('cropPricing')
db_user = "xxxx"
db_pass = "xxxx"
db_name = "xxxxx"
cloud_sql_connection_name = "xxxxxx"
conn = psycopg2.connect(user=db_user,
password=db_pass,
host='xxxxx',
dbname=db_name)
#ns_pricing.route('/list')
class States(Resource):
def get(self):
"""
list all the states for which data is available
"""
cur = conn.cursor(cursor_factory=RealDictCursor)
query = """
SELECT
DISTINCT state
FROM
db.table
"""
conn.commit()
cur.execute(query)
states = json.loads(json.dumps(cur.fetchall()))
if len(states) == 0:
return jsonify(data=[],
status="Error",
message="Requested data not found")
else:
return jsonify(status="Success",
message="Successfully retreived states",
data=states)
What should I fix to not see the error anymore?
It would be good to use the ORMs such as SQLAlchemy / Flask-SQLAlchemy which would handle the establishing / re-establishing the connection part.
Though, if using psycopg2. you can use try except to catch the exception and re-establish the connection again.
try:
cur.execute(query)
except psycopg2.InterfaceError as err:
print err.message
conn = psycopg2.connect(....)
cur = conn.cursor()
cur.execute(query)
I am using flask to create simple api. The api simply returns values from mongoDB. Everything works great if i do the connection within same function. I am not doing connection simply at start of file because i am using uwsgi and nginx server on ubuntu. If i do that then there will be a problem of fork.
However, I have to use this connection with other api so thought to make a seperate class for connection and each api will simply call it . I m using this functionality to make codes manageable. However when i try the these codes it always shows internal server error. I tried making this function static too , still the error exists.
Note - I have replaced mongodb address with xxx as i am using mongodbatlas account here
from flask import Flask
from flask import request, jsonify
from flask_pymongo import pymongo
from pymongo import MongoClient
from flask_restful import Resource, Api, reqparse
app = Flask(__name__)
api = Api(app)
#client = MongoClient("xxx")
#db = client.get_database('restdb')
#records = db.stars
class dbConnect():
def connect(self):
client = MongoClient("xxx")
db = client.get_database('restdb')
records = db.stars
return records
class Order(Resource):
def get(self):
#client = MongoClient("xxx")
#db = client.get_database('restdb')
#records = db.stars
#star = records
star = dbConnect.connect
output = []
for s in star.find():
output.append({'name' : s['name'], 'distance' : s['distance']})
return jsonify({'result' : output})
api.add_resource(Order, '/')
if __name__ == "__main__":
app.run(host='0.0.0.0')
ERROR {"message": "Internal Server Error"}
Preliminary investigation suggests that you haven't instantiated your dbConnect class. Also, you haven't called the method connect properly.
class Order(Resource):
def get(self):
db = dbConnect() # This was missing
star = db.connect() # This is how you make method call properly.
output = []
for s in star.find():
output.append({'name' : s['name'], 'distance' : s['distance']})
return jsonify({'result' : output})
Also class dbConnect() should be declared as class dbConnect:.
I'm writing a simple internal REST API for our solution using Flask, serving JSON objects through get calls (including authentication). We have multiple backends to fetch data from. From what I understand these should be connected to in a function decorated with #app.before_request and assigned to the g global for use in the specific route being requested. It's not a pattern I'm used to.
Here is a toy example of what I'm doing:
#app.before_request
def before_request():
g.some_conn_a = create_connection('a')
g.some_conn_b = create_connection('b')
g.some_client = create_client()
#app.route('/get_some_data')
#requires_auth
def get_some_data():
# Fetch something from all connections in g
payload = ... # Construct payload using above connections
return jsonify(payload)
#app.route('/get_some_other_data')
#requires_auth
def get_some_other_data():
# Fetch something from maybe just g.some_conn_b
payload = ... # Construct payload using g.some_conn_b
return jsonify(payload)
This seems wasteful to me if the user makes a request for data residing in only one or two of these connections/clients, like in the get_some_other_data route example.
I'm considering just making the connections/clients in the route functions instead, or load it lazily. What's the "correct" way? I hope it isn't to make a new module, that seems extreme for what I'm doing.
Riffing on the Flask docs Database Connections example
you could modify get_db() to accept an argument for each of your multiple connections.
def get_db(conn):
"""Open specificied connection if none yet for the current app context. """
if conn == 'some_conn_a':
if not hasattr(g, 'some_conn_a'):
g.some_conn_a = create_connection('a')
db = g.some_conn_a
elif conn == 'some_conn_b':
if not hasattr(g, 'some_conn_b'):
g.some_conn_b = create_connection('b')
db = g.some_conn_b
elif conn == 'some_client':
if not hasattr(g, 'some_client'):
g.some_client = create_client()
db = g.some_client
else:
raise Exception("Unknown connection: %s" % conn)
return db
#app.teardown_appcontext
def close_db(error):
"""Closes the db connections. """
if hasattr(g, 'some_conn_a'):
g.some_conn_a.close()
if hasattr(g, 'some_conn_b'):
g.some_conn_b.close()
if hasattr(g, 'some_client'):
g.some_client.close()
Then you could query each connection as needed:
#app.route('/get_some_data')
def get_some_data():
data_a = get_db('some_conn_a').query().something()
data_b = get_db('some_conn_b').query().something()
data_c = get_db('some_client').query().something()
payload = {'a': data_a, 'b': data_b, 'c': data_c}
return jsonify(payload)
The get_db() pattern is preferred over the before_request pattern for lazy loading database connections. The docs examples for Flask 0.11 and up utilize the get_db() pattern to a larger extent.
I want to create a REST API using python flask. So anytime I do this:
localhost/customers?cust_country=USA
I want to fetch every row from table 'customers' where everyone is from USA.
This is the script I've so far:
from flask import Flask, request
from flask_restful import Resource, Api
from sqlalchemy import create_engine
from flask_restful import reqparse
e = create_engine("mssql+pyodbc://....")
app = Flask(__name__)
api = Api(app)
parser = reqparse.RequestParser()
parser.add_argument('cust_country', type = 'string')
class Dep(Resource):
def get(self):
conn = e.connect()
args = parser.parse_args()
query = conn.execute("select * from customers where cust_country = ?", [args['cust_country']])
print(query)
return {'custid': [i[0] for i in query.cursor.fetchall()]}
api.add_resource(Dep, '/customers')
if __name__ == '__main__':
app.run()
I'm getting this error:
C:\Users\x>curl 127.0.0.1:5000/customers?cust_country=USA 404 Not
Found Not Found The requested URL was not found on
the server. If you entered the URL manually please check your
spelling and try again.
======================
part 2: sending multiple parameters:
from flask import Flask
from flask_restful import Resource, Api
from flask_restful import reqparse
from sqlalchemy import create_engine
e = create_engine("x")
parser = reqparse.RequestParser()
parser.add_argument('cust_country', type = str)
parser.add_argument('cust_name', type = str)
app = Flask(__name__)
api = Api(app)
class Dep(Resource):
def get(self):
args = parser.parse_args()
conn = e.connect()
query = conn.execute("select cust_id from customers where cust_country = ? and cust_name = ?", [args['cust_country'], args['cust_name']])
return {'custid': [i[0] for i in query.cursor.fetchall()]}
api.add_resource(Dep, '/customers')
if __name__ == '__main__':
app.run()
this is what I'm using in my curl:
curl "127.0.0.1:5000/customers?cust_country=USA&cust_name=Wascals"
One error that you may be facing is by using
parser.add_argument('cust_country', type = 'string') which is wrong.
It should be parser.add_argument('cust_country', type = str).
Also I would suggest to use app.run(debug=True) as it will help you debug simple errors.