I tried to use Flask-Restful + SqlAlchemy (with Automap) + MySql SGDB but I don't understand why this error occured in my code:
I sent the request in my controller and it worked normally, but after 10s an error is generated about a loss of connection with a SGDB.
itens = session.query(estados).filter(estados.ativo == True)
But stranger is that if I use SQL string syntax, the problem does not occur.
itens = engine.execute("SELECT `TBEstados`.`id`, `TBEstados`.`nome`, `TBEstados`.`ativo` FROM `intbr_webapp`.`TBEstados`;")
I'm using SqlAlchemy 1.2 but I did try also the 1.1 version. I did try also use the pre-ping=true and I didn't obtain success.
Does someone know anything about this? I don't understand why using ORM structure doesn't work, but with SQL syntax it does work. The connection is same, but the result is not.
My code is below:
estado.py
from flask import jsonify
from flask_restful import Resource
from json import dumps
from resources.database import Base, session, engine
#from resources.dataEncoder import JsonModel
from models.TBEstados import TBEstadosSchema
class Estados(Resource):
def get(self):
estados = Base.classes.TBEstados
itens = session.query(estados).filter(estados.ativo == True)
result = TBEstadosSchema(many=True).dump(itens)
return jsonify(result.data)
database.py (imported in EstadoModel)
from flask import Flask, g
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import create_engine
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm import sessionmaker
from flask_marshmallow import Marshmallow
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = my conn string
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['SQLALCHEMY_POOL_RECYCLE'] = 3600
app.config['SQLALCHEMY_POOL_TIMEOUT'] = 30
Base = automap_base()
engine = create_engine(app.config['SQLALCHEMY_DATABASE_URI'], pool_pre_ping=True)
# reflect the tables
Base.prepare(engine, reflect=True)
Session = sessionmaker(bind=engine)
session = Session()
ma = Marshmallow()
run.py
from flask import Flask, g
from flask_restful import Resource, Api
import resources.database
from controllers.Estados import Estados
app = Flask(__name__)
api = Api(app)
api.add_resource(Estados, '/estados')
if __name__ == '__main__':
app.run(debug=True, port=9002)
The exact error:
sqlalchemy.exc.OperationalError
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: 'SELECT TBEstados.id AS TBEstados_id, TBEstados.nome AS TBEstados_nome, TBEstados.ativo AS TBEstados_ativo \nFROM TBEstados \nWHERE TBEstados.ativo = true']
I solved my problem!
I changed value of the MySql variable wait_timeout of the "10s" to 100s (to test) and error in SqlAlchemy using PyMySql provider doesn't occurred again.
https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_wait_timeout
Thanks for help!
Related
how can I start a connection to an oracle database using flask-sqlalchemy and cx-oracle by sending these parameters:
alter session set nls_comp=linguistic;
alter session set nls_sort=Latin_AI;
In order to be able to make queries and sorts that do not distinguish, for example, between "Jose" and "José"
That's how I usually make the connection:
import os
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from config import config
def create_app(config_name=(os.getenv('FLASK_CONFIG') or 'default')):
app = Flask(__name__)
app.config.from_object(config[config_name])
db.init_app(app)
from commons import commons_bp
from admin import admin_bp
from inventario import inventario_bp
from incidencias import incidencias_bp
from inconformidades import inconformidades_bp
with app.app_context():
app.register_blueprint(commons_bp)
app.register_blueprint(admin_bp)
app.register_blueprint(inventario_bp)
app.register_blueprint(incidencias_bp)
app.register_blueprint(inconformidades_bp)
return app
and the configuration:
class DevelopmentConfig(Config):
TESTING = True
DEBUG = True
SQLALCHEMY_ECHO = os.environ.get('SQLALCHEMY_ECHO') or False
SQLALCHEMY_POOL_SIZE = None
SQLALCHEMY_POOL_TIMEOUT = None
SQLALCHEMY_BINDS = {
'adminapps': 'oracle://ADMINAPPS:ADMINAPPS#SERVIDOR53/ENTECO',
'incidencias': 'oracle://INCIDENCIAS:INCIDENCIAS#SERVIDOR53/ENTECO',
'incidencias': 'oracle://INCIDENCIAS:INCIDENCIAS#SERVIDOR53/ENTECO',
'inventario': 'oracle://INVENTARIO:INVENTARIO#SERVIDOR53/ENTECO',
'inconformidades': 'oracle://INCONFORMIDADES:INCONFORMIDADES#SERVIDOR53/ENTECO',
'documentacion': 'oracle://DOCUMENTACION:DOCUMENTACION#SERVIDOR53/ENTECO',
'auditoria': 'oracle://AUDITORIA:AUDITORIA#SERVIDOR53/ENTECO',
'navision': 'mssql+pyodbc://DB:DB.2020#SERVIDOR52',
}
I've tried among other things:
'adminapps': ['oracle://ADMINAPPS:ADMINAPPS#SERVIDOR53/ENTECO', 'alter session set "nls_comp"="linguistic"', 'alter session set "nls_sort"="Latin_AI"'],
But I haven't been successful, I can't figure out how to do it, thank you for your answers!
I have a Flask API based on Flask RestPlus extension and is hosted on Google App Engine. The API does a basic job of fetching data from a Google Cloud SQL PostgreSQL. The API is working fine otherwise but sometimes it starts returning InterfaceError: cursor already closed.
Strangely, when I do a gcloud app deploy, the API starts working fine again.
Here's a basic format of the API:
import simplejson as json
import psycopg2
from flask import Flask, jsonify
from flask_restplus import Api, Resource, fields
from psycopg2.extras import RealDictCursor
app = Flask(__name__)
app.config['SWAGGER_UI_JSONEDITOR'] = True
api = Api(app=app,
doc='/docs',
version="1.0",
title="Title",
description="description")
app.config['SWAGGER_UI_JSONEDITOR'] = True
ns_pricing = api.namespace('cropPricing')
db_user = "xxxx"
db_pass = "xxxx"
db_name = "xxxxx"
cloud_sql_connection_name = "xxxxxx"
conn = psycopg2.connect(user=db_user,
password=db_pass,
host='xxxxx',
dbname=db_name)
#ns_pricing.route('/list')
class States(Resource):
def get(self):
"""
list all the states for which data is available
"""
cur = conn.cursor(cursor_factory=RealDictCursor)
query = """
SELECT
DISTINCT state
FROM
db.table
"""
conn.commit()
cur.execute(query)
states = json.loads(json.dumps(cur.fetchall()))
if len(states) == 0:
return jsonify(data=[],
status="Error",
message="Requested data not found")
else:
return jsonify(status="Success",
message="Successfully retreived states",
data=states)
What should I fix to not see the error anymore?
It would be good to use the ORMs such as SQLAlchemy / Flask-SQLAlchemy which would handle the establishing / re-establishing the connection part.
Though, if using psycopg2. you can use try except to catch the exception and re-establish the connection again.
try:
cur.execute(query)
except psycopg2.InterfaceError as err:
print err.message
conn = psycopg2.connect(....)
cur = conn.cursor()
cur.execute(query)
I need help fixing an error that occurs when I run an sql query using pymssql, below is the code I am using.
config.py:
import pymssql
class devConfig():
server = 'servername'
usr = 'user'
pwd = 'password'
db_name = 'database'
init.py:
from flask import Flask
from sqlalchemy import create_engine
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy.orm import sessionmaker, scoped_session
from config import devConfig
import pymssql
app = Flask(__name__)
app.config.from_object('config')
Session = scoped_session(sessionmaker())
_sql_engine = None
_current_session = None
def session():
global _current_session
if _current_session is None:
session = Session.configure(bind=sql_engine())
_current_session = Session()
return _current_session
def sql_engine():
global _sql_engine
if _sql_engine:
return _sql_engine
else:
server = devConfig.server
db_name = devConfig.db_name
usr = devConfig.usr
pwd = devConfig.pwd
_sql_engine = create_engine("mssql+pymssql://filex.com\{}:{}#{}/{}".format(usr, pwd, server, db_name)) #create_engine("mssql+pymssql://{0}:{1}#{2}/{3}".format(username, password, dns, db))
return _sql_engine
from app import views
views.py:
from flask import render_template, flash, redirect
from app import app
from .forms import LoginForm
from . import session
from sqlalchemy.sql import text
#app.config.from_object('config.DevConfig')
#attributes = inspect.getmembers('config.DevConfig')
#app.route('/')
#app.route('/index')
def index():
select_sql = 'Select top 5 * from vendor'
result = session().execute(text(select_sql))
return render_template('index.html',
title='Home',
data=result)
session().close()
When I run this, I get this error:
File "pymssql.pyx", line 464, in pymssql.Cursor.execute (pymssql.c:7491)
sqlalchemy.exc.ProgrammingError: (pymssql.ProgrammingError) (102, b"Incorrect syntax near 'from\xef\xbb\xbfvendor'.DB-Lib error message 20018, severity 15:\nGeneral SQL Server error: Check messages from the SQL Server\n") [SQL: 'Select top 5 * from\ufeffvendor']
I don't understand why my query becomes 'from\xef\xbb\xbfvendor' when those characters don't exist on my query. What is causing this and how do I fix this?
You are using the wrong syntax for your SQL-Query. The top keyword is not available in myssql. Use limit 0,5 instead:
select_sql = 'Select * from vendor LIMIT 0,5'
Check this from W3C schools too.
By the way: please tell me this is not your real password in the config.py
Edit: I misread the import statement, please ignore this answer, it is wrong and not helping.
Next try:
When you check the Unicode character that is inserted there, it seems that it is a "Zero with no Break Space" character, not a usual space. You can see this here: fileformat.info
Can you check your sourcecode and insert an Space at this position? Make sure that the statement
select_sql 'Select top 5 * from vendor'
holds only proper Spaces. This may have happened by Copy&Pasting Code.
I'm trying to create REST API endpoints using flask framework. This is my fully working script:
from flask import Flask, jsonify
from flask_restful import Resource, Api
from flask_restful import reqparse
from sqlalchemy import create_engine
from flask.ext.httpauth import HTTPBasicAuth
from flask.ext.cors import CORS
conn_string = "mssql+pyodbc://x:x#x:1433/x?driver=SQL Server"
auth = HTTPBasicAuth()
#auth.get_password
def get_password(username):
if username == 'x':
return 'x'
return None
app = Flask(__name__)
cors = CORS(app)
api = Api(app)
class Report(Resource):
decorators = [auth.login_required]
def get(self):
parser = reqparse.RequestParser()
parser.add_argument('start', type = str)
parser.add_argument('end', type = str)
args = parser.parse_args()
e = create_engine(conn_string)
conn = e.connect()
stat = """
select x from report
"""
query = conn.execute(stat)
json_dict = []
for i in query.cursor.fetchall():
res = {'x': i[0], 'xx': i[1]}
json_dict.append(res)
conn.close()
e.dispose()
return jsonify(results=json_dict)
api.add_resource(Report, '/report')
if __name__ == '__main__':
app.run(host='0.0.0.0')
The issue is that I get results when I call this API only for a day or so after which I stop getting results unless I restart my script (or sometimes even my VM) after which I get results again. I reckon there is some issue with the database connection pool or something but I'm closing the connection and disposing it as well. I have no idea why the API gives me results only for some time being because of which I have to restart my VM every single day. Any ideas?
Per my experience, the issue was caused by coding create_engine(conn_string) to create db pool inside the Class Report so that always do the create & destory operations of db pool for per restful request. It's not correct way for using SQLAlchemy ORM, and be cause IO resouce clash related to DB connection, see the engine.dispose() function description below at http://docs.sqlalchemy.org/en/rel_1_0/core/connections.html#sqlalchemy.engine.Engine:
To resolve the issue, you just need to move e = create_engine(conn_string) to the below of the code conn_string = "mssql+pyodbc://x:x#x:1433/x?driver=SQL Server" and remove the code e.dispose() both in the Class Report, see below.
conn_string = "mssql+pyodbc://x:x#x:1433/x?driver=SQL Server"
e = create_engine(conn_string) # To here
In the def get(delf) function：
args = parser.parse_args()
# Move: e = create_engine(conn_string)
conn = e.connect()
and
conn.close()
# Remove: e.dispose()
return jsonify(results=json_dict)
app.py
from flask import Flask, render_template, request,jsonify,json,g
import mysql.connector
app = Flask(__name__)
**class TestMySQL():**
#app.before_request
def before_request():
try:
g.db = mysql.connector.connect(user='root', password='root', database='mysql')
except mysql.connector.errors.Error as err:
resp = jsonify({'status': 500, 'error': "Error:{}".format(err)})
resp.status_code = 500
return resp
#app.route('/')
def input_info(self):
try:
cursor = g.db.cursor()
cursor.execute ('CREATE TABLE IF NOT EXISTS testmysql (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, name VARCHAR(40) NOT NULL, \
email VARCHAR(40) NOT NULL UNIQUE)')
cursor.close()
test.py
from app import *
class Test(unittest.TestCase):
def test_connection1(self):
with patch('__main__.mysql.connector.connect') as mock_mysql_connector_connect:
object=TestMySQL()
object.before_request() """Runtime error on calling this"
I am importing app into test.py for unit testing.On calling 'before_request' function into test.py ,it is throwing RuntimeError: working outside of application context
same is happening on calling 'input_info()'
Flask has an Application Context, and it seems like you'll need to do something like:
def test_connection(self):
with app.app_context():
#test code
You can probably also shove the app.app_context() call into a test setup method as well.
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///todo.db'
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
db = SQLAlchemy(app)
app.app_context().push()
Run in terminal
>python
>>>from app import app
>>>from app import db
>>>db.create_all()
Now it should work
I followed the answer from #brenns10 when I ran into a similar problem when using pytest.
I followed the suggestion of putting it into test setup, this works:
import pytest
from src.app import app
#pytest.fixture
def app_context():
with app.app_context():
yield
def some_test(app_context):
# <test code that needs the app context>
I am using python3.8 and had to use a small variation to the answers already posted. I included the the below in pytests and didn't have to change anything else in the rest of the test file.
from flask import Flask
#pytest.fixture(autouse=True)
def app_context():
app = Flask(__name__)
with app.app_context():
yield
This can also be used with a context manager as well.
The main different to note here is the creation of the Flask app within the test file rather than it being imported from the main application file.