before rendering the template I want to check if the database that I am using is available. The database is only available with specific IP, therefore, if someone tries to access the page, I don't want to error everything out, but redirect to a different template without the need for a database.
The error message:
(2003, "Can't connect to MySQL server on 'db-test' (110)")
(Background on this error at: https://sqlalche.me/e/14/e3q8)
There is no issue with the error itself, I just want to be able to check if the database is accessible without waiting until the error occurs while loading the page.
storage.py:
from flask_sqlalchemy import SQLAlchemy
db_req = SQLAlchemy()
class Requests(db_req.Model):
__tablename__ = 'requests'
id = db_req.Column(db_req.Integer, primary_key=True, nullable=False)
created = db_req.Column(db_req.DateTime, nullable=False)
name = db_req.Column(db_req.String, nullable=False)
app.py:
from flask import Flask
from models import db_req
from views.requests import requests
app = Flask(__name__)
app.register_blueprint(requests)
app.config.from_object("config.Config")
db_req.init_app(app)
if __name__ == '__main__':
app.run(debug=True)
requests.py:
from flask import Blueprint, render_template, request
from query_request_db import get_fields
requests = Blueprint("requests", __name__)
#requests.route("/", methods=["GET", "POST"])
def requests_page():
# need to check before everything if I can access database not to error out
return render_template("requests_page.html", names=get_fields.get_field_data("name"))
I am trying to find a way to check if the database is available without erroring out. Any help is welcome!
Hope I understand you right? that u want to check ip Addresse of the viewer of your Page.
Then u can use
request.remote_addr
to validate the IP Adress that has access to database ohterwise just do a redirect.
Related
I am beyond confused at this point. I have read so much documentation and there seems to be sparse examples of what to do for running raw SQL statements on my flask app using Flask_sqlalchemy or flask_mysqldb.
I have started by downloading XAMPP and creating a database on MySQL server through my localhost. I then created my flask application and created an initial database from the terminal using
>>> from yourapplication import db
>>> db.create_all()
The code from my flask app is as follows based on the documentation here
from flask import Flask, render_template, request, redirect, session
from flask_session import Session
from flask_sqlalchemy import SQLAlchemy
from datetime import datetime
from werkzeug.security import check_password_hash, generate_password_hash
# Set up Flask instance
app = Flask(__name__)
# Configure db
app.config["SQLALCHEMY_DATABASE_URI"] = "mysql+pymysql://root#localhost/hoook"
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
db = SQLAlchemy(app)
# Reload templates when changed (take down when in production)
app.config["TEMPLATES_AUTO_RELOAD"] = True
# Configure session to use filesystem (instead of signed cookies)
app.config["SESSION_PERMANENT"] = False
app.config["SESSION_TYPE"] = "filesystem"
Session(app)
# Create db model
class Users(db.Model):
id = db.Column(db.Integer, primary_key=True)
first_name = db.Column(db.Text, nullable=False)
last_name = db.Column(db.Text, nullable=False)
email = db.Column(db.Text, unique=True, nullable=False)
password = db.Column(db.Text, nullable=False)
date = db.Column(db.DateTime, default=datetime.utcnow)
def __repr__(self):
return "<Users %r>" % self.id
great, now I can see my table in myPhpAdmin. I then ran a test statement to see if information would be added as follows
db.engine.execute("INSERT INTO users (first_name, last_name, email, password) VALUES ('chris', 'blah', 'blah#gmail.com', 'something')")
works! but then looking at the previous stackoverflow answer and then the subsequent documentation I find this method is depreciated therefore I cant use this going forward. So I try to use session.execute instead (since connection.execute also shows its depreciated) as I see that for some reason there are three different methods all with the same function execute() that can be used...???? source. So using the following statement I try to add another row to my table but it failed.
db.session.execute("INSERT INTO users (first_name, last_name, email, password) VALUES ('jeremy', 'blah', 'something#gmail.com', 'whatever')")
there were no error messages, just when I check my database, nothing new was added. So if I got this right engine.execute didnt need a connection but session does? Does that mean this line
app.config["SQLALCHEMY_DATABASE_URI"] = "mysql+pymysql://root#localhost/hoook"
is actually not connecting the session method to my database then? What about the pymysql connector in the URI? do I also need to import pymysql to be able to use this connector? What is the correct method for generating queries and being able to add tables etc from within your flask app... Please clarify as this is confusing and from my point of view, all this documentation and abstraction needs to be cleaned up.
I am working on to build a web application and using Oracle database with flask sqlalchemy. The connection is successful with my below code, but when i am running the code i am getting a warning as
SAWarning: Attribute 'CATEGORY' on class appears to be a non-schema 'sqlalchemy.sql.column()' object; this won't be part of the declarative mapping
My code is working and i am able to run the html, but i feel its the schema name which is missing in my connection. Below is my code. ( i have not mentioned the credentials and host name) does any one know where the schema needs to be declared or if any one can help resolve the warning.
below is the code i have created.
'''from flask import Flask, render_template, request
from flask_sqlalchemy import SQLAlchemy
from datetime import date, datetime
from sqlalchemy import create_engine
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI']= 'oracle://username:password#host:port/sid'
db = SQLAlchemy(app)
class Master_dates(db.Model):
DATA_DATE = db.Column(db.Integer, primary_key=True)
LOCATN = db.Column(db.String(80))
CATEGORY = db.column(db.String(120))
#app.route("/")
def home():
return render_template('index.html') '''
Using sqlalchemy i got the same error with the following code
class users(db.Model):
_id=db.Column("id", db.Integer, primary_key=True)
name=db.Column(db.String(100))
address=db.Column(db.String(100))
neighborhood=db.Column(db.String(100))
city=db.Column(db.String(100))
phone=db.Column(db.String(100))
document=db.Column(db.String(100))
passwoord=db.Column(db.String(100))
verifyPass=db.Column(db.String(100))
rol=db.Column(db.String(100))`
it got fixed when inside of the same class (in your case will be Master_dates(db.Model)) when i added a constructor to it. like this:
def __init__(self, name, address, neighborhood, city, phone, document, passwoord, rol):
self.name=name
self.address=address
self.neighborhood=neighborhood
self.city=city
self.phone=phone
self.document=document
self.passwoord=passwoord
self.rol=rol
you can try to add this piece of code on your Master_dates(db.Model) class
def __init__(self, DATA_DATE, LOCATN, CATEGORY):
self.DATA_DATE=DATA_DATE
self.LOCATN=LOCATN
self.CATEGORY=CATEGORY
I hope not to reach out here too late
This question already has answers here:
flask make_response with large files
(2 answers)
Closed 3 years ago.
I'm fairly new to this task so please help me identify the concepts I'm missing.
I'm trying to stream data from an API to my SQLite db and let a Flask app consume the data. I defined the models in models.py like this
# models.py
import os
from flask_sqlalchemy import SQLAlchemy
# Create sqlite db
db = SQLAlchemy()
class MyDataModel(db.Model):
# Manual table name choice
__tablename__ = 'table1'
id = db.Column(db.Integer, primary_key=True)
created_at = db.Column(db.Text)
text = db.Column(db.Text)
def __init__(self, created_at, text):
self.created_at = created_at
self.text = text
def __repr__(self):
return f"Data: {self.text} ... created at {self.created_at}"
In app.py I have a simple view function that counts the rows and return a Server Sent Event to the frontend for realtime tracking.
# app.py
import os
import time
from flask import Flask, render_template, url_for, redirect, Response
from flask_migrate import Migrate
from models import db, MyDataModel
from settings import *
app = Flask(__name__)
logging.basicConfig(level=logging.DEBUG)
basedir = os.path.abspath(os.path.dirname(__file__))
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir, 'data.sqlite')
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
db.init_app(app)
Migrate(app, db)
#app.route('/')
def index():
return render_template('index.html')
#app.route('/getcount')
def getcount():
def count_stream():
count = db.session.query(MyDataModel).count()
while True:
yield f"data:{str(count)}\n\n"
time.sleep(0.5)
return Response(count_stream(), mimetype='text/event-stream')
if __name__ == "__main__":
app.run(debug=True, port=PORT)
Now I have another python script stream_to_db.py that gets data from an API as a stream, approximately like this
# stream_to_db.py
import logging
import os
from models import db, MyDataModel
from settings import *
from SomeExternalAPI import SomeAPI
def stream_to_db():
api = SomeAPI(
API_KEY, API_SECRET_KEY, ACCESS_TOKEN, ACCESS_TOKEN_SECRET
)
r = api.request()
for item in r:
created_at, text = item['created_at'], item['text']
logging.info(text)
datum = MyDataModel(created_at, text)
db.session.add(datum)
db.session.commit()
# Stream data to sqlite
stream_to_db()
When I try to run this python stream_to_db.py I get error
RuntimeError: No application found. Either work inside a view function or push an application context. See http://flask-sqlalchemy.pocoo.org/contexts/.
I looked at the documentation of application context but am still confused. Say I go without SQLAlchemy and use Python and SQL directly for data insertion, this stream_to_db.py script should be independent of the Flask app. But if I still want to leverage SQLAlchemy for its syntax and model definitions from models.py, how should I do it?
Conceptually I feel the streaming to db part is independent of the Flask app and should be a script that's essentially a while True loop and goes forever. The Flask app just reads the db, send data to frontend and does nothing else. I tried put the stream_to_db() function into __main__ in app.py but that doesn't make sense, app.run() and stream_to_db() are both essentially while True loops and can't be put together.
I'm lost and miss key concepts here. Please help and suggest the right way/best practices to do it. I feel this is a really basic task which should have a best practice and a set of dedicated tools already. Thanks in advance!
EDIT
To further experiment, I imported app into stream_to_db.py and added
with app.app_context():
stream_to_db()
Now I can run python stream_to_db.py with no problem, but if I start the Flask app at the same time, I get several
Debugging middleware caught exception in streamed response at a point where response headers were already sent.
and also
Traceback (most recent call last):
File "/Users/<username>/webapps/<appname>/venv/lib/python3.6/site-packages/werkzeug/wsgi.py", line 507, in __next__
return self._next()
File "/Users/<username>/webapps/<appname>/venv/lib/python3.6/site-packages/werkzeug/wrappers/base_response.py", line 45, in _iter_encoded
for item in iterable:
File "/Users/<username>/webapps/<appname>/app.py", line 33, in count_stream
count = db.session.query(CardanoTweet).count()
File "/Users/<username>/webapps/<appname>/venv/lib/python3.6/site-packages/sqlalchemy/orm/scoping.py", line 162, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/Users/<username>/webapps/<appname>/venv/lib/python3.6/site-packages/sqlalchemy/util/_collections.py", line 1012, in __call__
return self.registry.setdefault(key, self.createfunc())
File "/Users/<username>/webapps/<appname>/venv/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 3214, in __call__
return self.class_(**local_kw)
File "/Users/<username>/webapps/<appname>/venv/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py", line 136, in __init__
self.app = app = db.get_app()
File "/Users/<username>/webapps/<appname>/venv/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py", line 982, in get_app
'No application found. Either work inside a view function or push'
RuntimeError: No application found. Either work inside a view function or push an application context. See http://flask-sqlalchemy.pocoo.org/contexts/.
Looks like SQLAlchemy can't find the application. Not sure if it's a problem about the db.init_app(app) in app.py and db = SQLAlchemy() in models.py. I avoided having db = SQLAlchemy(app) in models.py because I can't import app into models for the circular dependency reason.
EDIT2
This time I moved all the code from models.py to app.py and used db = SQLAlchemy(app), removed db.init_app(app), kept the import app to stream_to_db.py and the app context in it, and it worked!
My questions
How to correctly move the model definitions to models.py with no circular dependency?
What's the implication of having app context in stream_to_db.py? If I have gunicorn running many workers, there will essentially be multiple Flask app instances. Would that cause a problem?
EDIT3
Thanks for all the replies. I don't think this is a duplicate to flask make_response with large files.
The problem is not streaming data FROM Flask to the client, it is to stream data from external API to DB, and have Flask consume some statistics from the data in DB. So, I don't see why conceptually the streaming job should be related to Flask at all, they are independent. The problem is that I'm using SQLAlchemy in the streaming job for data model and db transaction, and SQLAlchemy needs Flask app defined. This part is where I get confused about.
What is the right way to write this ever-running background job that streams data to db, with data models defined with SQLAlchemy? Should I strip out the SQLAlchemy code from streaming code and just use SQL, and manually make sure the schema agrees if there are further migration down the road?
Please try following change in models.py where you've created db.
db = SQLAlchemy(app)
I have a heroku app that I am trying to add a database to. I am using Flask-SLAlchemy, PostgreSQL (and psql). I've already created a table in the database, but I cannot add any rows to it. Here is what I believe to be all relevant code:
import flask
import keys
import requests_oauthlib
import json
import os
import psycopg2
import urlparse
from flask import (Flask, jsonify, render_template, redirect, url_for, request, make_response)
from flask.ext.sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'heroku-url-here'
db = SQLAlchemy(app)
class Page (db.Model):
__tablename__ = "pages"
title = db.Column('Title', db.String)
date = db.Column('Date', db.String, primary_key=True)
writing = db.Column('Writing', db.String)
def __init__(self, title, date, writing):
self.title = title
self.date = date
self.writing = writing
def __repr__(self):
return '<Page %r>' % self.date
app.secret_key = keys.secret_key
# db.create_all()
# this created the database already after I ran it once, it made a psycopg2 error after that first time.
#app.route('/db', methods=['GET', 'POST'])
def db():
if request.method == 'POST':
title = request.form['title']
date = request.form['date']
writing = request.form['writing']
newest = Page(title, date, writing)
print newest
db.session.add(newest)
db.session.commit()
else:
title = None
date = None
writing = None
return flask.redirect(flask.url_for('home'))
In my heroku logs, there are no errors shown. The code runs to the the print newest line, and the newly created Page is printed as <Page u'whatever-the-date-was'>. When a form is submitted in my html template, it calls the function by using the action {{url_for('db')}}.
This is my first time using heroku and flask and basically doing any back-end stuff, so please explain thoroughly if you have an answer. Thanks in advance!
Take advantage of your Page model here.
db.session.add(Page(
title = request.form['title']
date = request.form['date']
writing = request.form['writing']
))
db.session.commit()
You'll probably also run into trouble with your conditional - if the method isn't POST then nothing will happen, and there won't be any message logged about it. If you remove the 'GET' from the methods in the route declaration you won't need that conditional at all.
I'd recommend taking a look at the Flask-WTF extension, as well as breaking out your form validation and redirect steps into separate functions. Flask works best by breaking down elements to their smallest usable components and then reassembling them in many different ways.
For more info on form handling, check out Miguel Grinberg's Flask Mega-Tutorial (if you haven't already).
So I am using Amazon Web Services RDS to run a MySQL server and using Python's Flask framework to run the application server and Flask-SQLAlchemy to interface with the RDS.
My app config.py
SQLALCHEMY_DATABASE_URI = '<RDS Host>'
SQLALCHEMY_POOL_RECYCLE = 60
My __ init __.py
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
application = Flask(__name__)
application.config.from_object('config')
db = SQLAlchemy(application)
I have my main application.py
from flask import Flask
from application import db
import flask.ext.restless
from application.models import Person
application = Flask(__name__)
application.debug=True
db.init_app(application)
#application.route('/')
def index():
return "Hello, World!"
manager = flask.ext.restless.APIManager(application, flask_sqlalchemy_db=db)
manager.create_api(Person, methods=['GET','POST', 'DELETE'])
if __name__ == '__main__':
application.run(host='0.0.0.0')
The models.py
class Person(db.Model):
__bind_key__= 'people'
id = db.Column(db.Integer, primary_key=True)
firstName = db.Column(db.String(80))
lastName = db.Column(db.String(80))
email = db.Column(db.String(80))
def __init__(self, firstName=None, lastName=None, email=None):
self.firstName = firstName
self.lastName = lastName
self.email = email
I then have a script to populate the database for testing purposes after db creation and app start:
from application import db
from application.models import Person
person = Person('Bob', 'Jones', 'bob#website.net')
db.session.add(person)
db.session.commit()
Once I've reset the database with db.drop_all() and db.create_all() I start the application.py and then the script to populate the database.
The server will respond with correct JSON but if I come back and check it hours later, I get the error that I need to rollback or sometimes the 2006 error that the MySQL server has gone away.
People suggested that I change timeout settings on the MySQL server but that hasn't fixed anything. Here are my settings:
innodb_lock_wait_timeout = 3000
max_allowed_packet = 65536
net_write_timeout = 300
wait_timeout = 300
Then when I look at the RDS monitor, it shows the MySQL server kept the connection open for quite a while until the timeout. Now correct me if I'm wrong but isn't the connection supposed to be closed after it's finished? It seems that the application server keeps making sure that the database connection exists and then when the MySQL server times out, Flask/Flask-SQLAlchemy throws an error and brings down the app server with it.
Any suggestions are appreciated, thanks!
I think what did it was adding
db.init_app(application)
in application.py, haven't had the error since.
Everytime checking rollback or not is troublesome..
I made insert, update functions which need commit.
#app.teardown_request
def session_clear(exception=None):
Session.remove()
if exception and Session.is_active:
Session.rollback()
It seems not to be a problem with the transactions at the first place, but this is probably caused by an MySQL Error like Connection reset by peer beforehand. That means your connection is lost, probably because your application context was not setup correctly.
In general it is preferrable to use the factory pattern to create your app. This has a lot of advantages, your code is
easier to read and setup
easier to test
avoid circular imports
To prevent the invalid transaction error (that is probably caused by an OperationalError: Connection reset by peer) you should ensure that you are handling the database connection right.
The following example is based on this article which gives a nice explanation of the flask application context and how to use it with database connections or any other extensions.
application.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
def create_app():
"""Construct the core application."""
application = Flask(__name__)
application.config.from_object('config') # Set globals
db = SQLAlchemy()
with application.app_context():
# Initialize globals/extensions in app context
db.init_app(app)
# import routes here
from . import routes
return application
if __name__ == "__main__":
app = create_app()
app.run(host="0.0.0.0")
routes.py
from flask import current_app as application
#application.route('/', methods=['GET'])
def index():
return "Hello, World!"
If you still run into disconnect-problems you should also check the SQLAlchemy documentation on dealing with disconnects and have a look at this question.
Here you missing pool recycle as MySql closes session after some time so you need to add pool recycle so that connections in pool get reconnect after pool recycle time.
app.config['SQLALCHEMY_POOL_RECYCLE'] = 3600
This error usually appears when you create sqlalchemy the engine as a singleton. In that case after the connection is invalidated (in my case it was 3600sec) you get the InvalidTransaction error.
Best advice would be to initialise the db session at the time of application initialisation
db.init_app(app)
and import this db session when ever you have to do some CRUD operation.
Never faced this issue post this change on my application.
Alternatively, use this at the end of the script that populates your database:
db.session.close()
That should prevent those annoying "MySQL server has gone away" errors.