I have created a python webapp with Flask and it seems like I am having connection issues with the database. I think this is because I don't close my sessions somewhere in my code.
I have
db = SQLAlchemy(app)
for the database and use
#views.route('/test/', methods=['GET', 'POST'])
def test():
db.session.add(something)
db.session.commit()
#views.route('/another_page/', methods=['GET', 'POST'])
def page():
some_records = User.query.get(some_ids)
for adding records to the database.
When do I have to close my session in this case? Is there a way to close the connection after the user leaves? Should I close every time a page is done with the database? Do I need to close my connection after a query?
The documentation says next:
As in the declarative approach, you need to close the session after each request or application context shutdown. Put this into your application module:
from yourapplication.database import db_session
#app.teardown_appcontext
def shutdown_session(exception=None):
db_session.remove()
UPD: In case of Flask-SQLAlchemy this staff is hardcoded, thus you don't need to care about it when develop.
Related
From what I have already learned, the Database sessions isolated in the request threads - I mean, when user send request to the Flask server, then the route function create session, manipulate the database and then commit/rollback database (which ends session). I would like to know how to manipulate one session with multiple requests. Something like:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
db = SQLAlchemy(app)
#app.route("/create_session")
def create_session():
session = db.session # create session
...
return response
#app.route("/query")
def query():
... # perform some operations with the session
return response
#app.route("/end_session")
def end_session():
session.commit() # commit session
...
return response
where:
First I am calling the request to the '/create_session' route to create session
Then I am calling the request to the '/query' route to do some operations with session
On the end I am calling the request to the '/end_session' route to end session
All routes manage only one, shared transaction.
Thanks for any response!
I have a Flask app, which performs operations on a mysql database. It uses sqlalchemy, and creates a session per request. In my testing framework, I set up a session, then insert appropriate data and ensure removal afterward. However, in my test, when I attempt to make sure that something was deleted (via a DELETE request to my Flask app), the row in my session is unaffected by the external thread. Do I need to close and reopen my session to ensure proper deletion, or can I refresh it somehow, or am I doing this all wrong somehow that I'm missing?
class Endpoint(flask_restful.Resource):
# in the webapp thread
def delete(self,id):
db.session.query(db.Table).filter(id=id).delete()
db.session.commit()
class Test(unittest.TestCase):
# in the testing thread
def setUp(self):
self.data = db.session.add(db.Table())
db.session.commit()
def tearDown(self):
self.data.delete()
db.session.commit()
def test_delete(self):
requests.delete(flask.url_for(Endpoint, id=self.data.id))
assert db.session.query(db.Table).filter(id = self.data.id).count() == 0
So I am using Amazon Web Services RDS to run a MySQL server and using Python's Flask framework to run the application server and Flask-SQLAlchemy to interface with the RDS.
My app config.py
SQLALCHEMY_DATABASE_URI = '<RDS Host>'
SQLALCHEMY_POOL_RECYCLE = 60
My __ init __.py
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
application = Flask(__name__)
application.config.from_object('config')
db = SQLAlchemy(application)
I have my main application.py
from flask import Flask
from application import db
import flask.ext.restless
from application.models import Person
application = Flask(__name__)
application.debug=True
db.init_app(application)
#application.route('/')
def index():
return "Hello, World!"
manager = flask.ext.restless.APIManager(application, flask_sqlalchemy_db=db)
manager.create_api(Person, methods=['GET','POST', 'DELETE'])
if __name__ == '__main__':
application.run(host='0.0.0.0')
The models.py
class Person(db.Model):
__bind_key__= 'people'
id = db.Column(db.Integer, primary_key=True)
firstName = db.Column(db.String(80))
lastName = db.Column(db.String(80))
email = db.Column(db.String(80))
def __init__(self, firstName=None, lastName=None, email=None):
self.firstName = firstName
self.lastName = lastName
self.email = email
I then have a script to populate the database for testing purposes after db creation and app start:
from application import db
from application.models import Person
person = Person('Bob', 'Jones', 'bob#website.net')
db.session.add(person)
db.session.commit()
Once I've reset the database with db.drop_all() and db.create_all() I start the application.py and then the script to populate the database.
The server will respond with correct JSON but if I come back and check it hours later, I get the error that I need to rollback or sometimes the 2006 error that the MySQL server has gone away.
People suggested that I change timeout settings on the MySQL server but that hasn't fixed anything. Here are my settings:
innodb_lock_wait_timeout = 3000
max_allowed_packet = 65536
net_write_timeout = 300
wait_timeout = 300
Then when I look at the RDS monitor, it shows the MySQL server kept the connection open for quite a while until the timeout. Now correct me if I'm wrong but isn't the connection supposed to be closed after it's finished? It seems that the application server keeps making sure that the database connection exists and then when the MySQL server times out, Flask/Flask-SQLAlchemy throws an error and brings down the app server with it.
Any suggestions are appreciated, thanks!
I think what did it was adding
db.init_app(application)
in application.py, haven't had the error since.
Everytime checking rollback or not is troublesome..
I made insert, update functions which need commit.
#app.teardown_request
def session_clear(exception=None):
Session.remove()
if exception and Session.is_active:
Session.rollback()
It seems not to be a problem with the transactions at the first place, but this is probably caused by an MySQL Error like Connection reset by peer beforehand. That means your connection is lost, probably because your application context was not setup correctly.
In general it is preferrable to use the factory pattern to create your app. This has a lot of advantages, your code is
easier to read and setup
easier to test
avoid circular imports
To prevent the invalid transaction error (that is probably caused by an OperationalError: Connection reset by peer) you should ensure that you are handling the database connection right.
The following example is based on this article which gives a nice explanation of the flask application context and how to use it with database connections or any other extensions.
application.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
def create_app():
"""Construct the core application."""
application = Flask(__name__)
application.config.from_object('config') # Set globals
db = SQLAlchemy()
with application.app_context():
# Initialize globals/extensions in app context
db.init_app(app)
# import routes here
from . import routes
return application
if __name__ == "__main__":
app = create_app()
app.run(host="0.0.0.0")
routes.py
from flask import current_app as application
#application.route('/', methods=['GET'])
def index():
return "Hello, World!"
If you still run into disconnect-problems you should also check the SQLAlchemy documentation on dealing with disconnects and have a look at this question.
Here you missing pool recycle as MySql closes session after some time so you need to add pool recycle so that connections in pool get reconnect after pool recycle time.
app.config['SQLALCHEMY_POOL_RECYCLE'] = 3600
This error usually appears when you create sqlalchemy the engine as a singleton. In that case after the connection is invalidated (in my case it was 3600sec) you get the InvalidTransaction error.
Best advice would be to initialise the db session at the time of application initialisation
db.init_app(app)
and import this db session when ever you have to do some CRUD operation.
Never faced this issue post this change on my application.
Alternatively, use this at the end of the script that populates your database:
db.session.close()
That should prevent those annoying "MySQL server has gone away" errors.
I am new to flask framework and I have just created app in it but now i am struggling with DB connections in flask. I want to connect my app with MySQL. For that I have follow this link http://flask.pocoo.org/snippets/11/ , but I am not able to connect with DB.
My code is as follows :
from flask import Flask , render_template,g
from torndb import Connection
app=Flask(__name__)
#app.before_request
def connect_db():
g.db = Connection(DB_HOST="localhost",
DB_NAME="flask",
DB_USER="root",
DB_PASSWD="ghrix321")
#app.route('/')
def home():
rows = g.db.iter("select * from user")
return render_template('home.html',rows=rows)
TypeError: init() got an unexpected keyword argument 'DB_NAME'.
So please suggest me some way so that I can connect with DB and fetch data from there.
Thanks
The snippet you refer to does not use keyword parameters.
The documentation of torndb is at http://torndb.readthedocs.org/en/latest/. If you use keyword parameters, you have to name them like they are in the function definition.
This is the correct call:
g.db = Connection('localhost','flask', user='root', password='ghrix321')
As an aside, use dedicated users in your database, and don't hardcode your password into the app, use a configuration file for that.
I am developing a web application that needs to login to the database with credentials that are provided by the end user; the application itself does not have a login to the database.
The problem is how to create one connection per user session.
One approach is:
Request user's credentials
Check if credentials are valid against db backend
Store credentials in session-level variable
Problem with this approach is, on each subsequent request for that session; you would need to create a new connection; and this will quickly exhaust the max connections to the server.
I am using Flask with Oracle.
In Flask, there is a g object, which stores request-scoped objects. This snippet though, does not work:
app = Flask(__name__)
app.config.from_object(__name__)
def login_required(f):
#wraps(f)
def decorated_function(*args, **kwargs):
if g.db is None:
return redirect(url_for('login', next=request.url))
return f(*args, **kwargs)
return decorated_function
class LoginForm(Form):
username = TextField('Username', [validators.Length(min=4, max=25)])
password = PasswordField('Password', [validators.Required()])
#app.route('/app', methods=['GET','POST'])
#login_required
def index():
return 'Index'
#app.route('/', methods=['GET','POST'])
def login():
form = LoginForm(request.form)
if request.method == 'POST':
if form.validate():
try:
dsn = cx_Oracle.makedsn(app.config['DB_HOST'],
app.config['DB_PORT'], app.config['DB_SID'])
g.db = cx_Oracle.connect(form.username.data,
form.password.data, dsn)
except cx_Oracle.DatabaseError as e:
flash(unicode(e), 'error')
return render_template('login.html', form=form)
return redirect(url_for('index'))
else:
return render_template('login.html', form=form)
else:
return render_template('login.html', form=form)
AttributeError: '_RequestGlobals' object has no attribute 'db'
The reason why this snippet is not working is this line
if g.db is None:
You are accessing an attribute which does not belong to g. Add these lines of code:
#app.before_request
def before_request():
g.db = None
Functions marked with before_request() are called before a request and passed no arguments.
Regarding Connection Pooling
There is an ORM and SQL Toolkit called SQLAlchemy in Python which does connection pooling automatically for you.
It also manages g object of flask and creates prepared SQL statements which are much more secure.
It has two parts:
1. Core which is a SQL abstraction toolkit.
2. ORM is an optional package which builds on top of Core.
So if you don't want to use ORM then simply use its Core with all the features intact.
Check here.
It supports Oracle via cx_Oracle and it can be set up as mentioned in SQLAlchemy Docs here.
Check this question for some more discussion on connection pooling in python.
Sounds like you need to implement connection pooling. Instead of requesting a new connection for each request (which does not scale at all), you simply check out a connection from the pool with the required attributes. If no suitable connection is available, the pool would create a new one. The pool would also need to handle the opposite case of course in closing connections that haven't been used for some time.
You might want to check if such a pool is available for python. For java oracle supports this with UCP and for OCI with the session pool. If I'm not mistaken a connection pool is even provided for .Net.