Develop Reference

Why do my Flask psycopg2 app returns sync error with server?

I'm facing issue with my flask app that uses psycopg2 for postgres. Sometimes my API returns 500 server error in which it said in the log file:
[SQL: SELECT gallery.id AS gallery_id, gallery.caption AS gallery_caption, gallery.image AS gallery_image, gallery.created_date AS gallery_created_date, gallery.updated_date AS gallery_updated_date
FROM gallery ORDER BY gallery.id]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
[UID:1549][94744] Child process with pid: 98159 was killed by signal: 15, core dump: 0
[ERROR] [UID:1549][97456] wsgiAppHandler pApp->start_response() return NULL.
/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py:873: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future. Set it to True or False to suppress this warning.
'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
[2021-12-27 11:22:11,886] ERROR in app: Exception on /gallery/find-all [GET]
Traceback (most recent call last):
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
psycopg2.OperationalError: lost synchronization with server: got message type "G", length 1935229752
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/home/myapp/api/myapplication/auth.py", line 32, in decorated
return f(auth_data, *args, **kwargs)
File "/home/myapp/api/myapplication/gallery.py", line 71, in fetch_image
image_data = Gallery.query.filter().order_by(Gallery.id).all()
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2759, in all
return self._iter().all()
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2897, in _iter
execution_options={"_sa_orm_load_options": self.load_options},
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1692, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 326, in _execute_on_connection
self, multiparams, params, execution_options
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1491, in _execute_clauseelement
cache_hit=cache_hit,
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
e, statement, parameters, cursor, context
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2027, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/home/myapp/virtualenv/api/3.7/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) lost synchronization with server: got message type "G", length 1935229752
This API just fetch all the data from the table and returns them to the client. The table only have max 12 rows and contains base-64 string of image data. However, this also happens at other API that fetch a single data from other table and returns only text data-type.
These APIs works well in my local development env but went error in production (CPanel shared hosting)

Unable to send POST request using Flask test client

I'm following the TestDriven.io course, after setting up restful routes I encounter this error even after adding the route handler in project/api/users.py:
test_app_is_development (test_config.TestDevelopmentConfig) ... ok
test_app_is_production (test_config.TestProductionConfig) ... ok
test_app_is_testing (test_config.TestTestingConfig) ... ok
test_add_user (test_users.TestUserService)
Ensure a new user can be added to the database ... ERROR
test_users (test_users.TestUserService)
Ensure the /ping route behaves correctly. ... ok
======================================================================
ERROR: test_add_user (test_users.TestUserService)
Ensure a new user can be added to the database
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1283, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
cursor.execute(statement, parameters)
psycopg2.ProgrammingError: can't adapt type 'now'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/src/app/project/tests/test_users.py", line 22, in test_add_user
response = self.client.post(
File "/usr/local/lib/python3.8/site-packages/werkzeug/test.py", line 1016, in post
return self.open(*args, **kw)
File "/usr/local/lib/python3.8/site-packages/flask/testing.py", line 222, in open
return Client.open(
File "/usr/local/lib/python3.8/site-packages/werkzeug/test.py", line 970, in open
response = self.run_wsgi_app(environ.copy(), buffered=buffered)
File "/usr/local/lib/python3.8/site-packages/werkzeug/test.py", line 861, in run_wsgi_app
rv = run_wsgi_app(self.application, environ, buffered=buffered)
File "/usr/local/lib/python3.8/site-packages/werkzeug/test.py", line 1096, in run_wsgi_app
app_rv = app(environ, start_response)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2464, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2450, in wsgi_app
response = self.handle_exception(e)
File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 272, in error_router
return original_handler(e)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1867, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 38, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 272, in error_router
return original_handler(e)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 38, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 468, in wrapper
resp = resource(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/flask/views.py", line 89, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 583, in dispatch_request
resp = meth(*args, **kwargs)
File "/usr/src/app/project/api/users.py", line 28, in post
db.session.commit()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/scoping.py", line 163, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1042, in commit
self.transaction.commit()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2523, in flush
self._flush(objects)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2664, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2624, in _flush
flush_context.execute()
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 239, in save_obj
_emit_insert_statements(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1135, in _emit_insert_statements
result = cached_connections[connection].execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1133, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1323, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1517, in _handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1283, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) can't adapt type 'now'
[SQL: INSERT INTO users (username, email, active, created_date) VALUES (%(username)s, %(email)s, %(active)s, %(created_date)s) RETURNING users.id]
[parameters: {'username': 'michael', 'email': 'michael#mherman.org', 'active': True, 'created_date': <sqlalchemy.sql.functions.now at 0x7f5369329f40; now>}]
(Background on this error at: http://sqlalche.me/e/f405)
----------------------------------------------------------------------
Ran 5 tests in 0.307s
FAILED (errors=1)
<unittest.runner.TextTestResult run=5 errors=1 failures=0>
As you can see the error has to do with the self.client.post request done in this function:
def test_add_user(self):
"""Ensure a new user can be added to the database"""
with self.client:
response = self.client.post(
'/users',
data=json.dumps({
'username': 'michael',
'email': 'michael#mherman.org'
}),
content_type='application/json',
)
print (response)
data = json.loads(response.data.decode())
self.assertEqual(response.status_code, 201)
self.assertIn('michael#mherman.org was added!', data['message'])
self.assertIn('success', data['status'])
Everything seems correct in the logic and syntax of the code but I can't figure out why the test isn't passing?

Judging by the error log, the problem is in the INSERT INTO statement. The created_date parameter is not a date but a function sqlalchemy.sql.functions.now. It looks like you forgot to call it.

Table fields not found when trying to select distinct values of a column

I have created a table with a dynamic name as follows:
def create_data_table(table_name):
meta = Base.metadata
my_table= Table(
table_name, meta,
Column('id', BigInteger, primary_key=True, nullable=False),
Column('file_name', String(250), nullable=False)
)
meta.create_all(engine, [my_table])
I can query all records from it by using session.query(table_data).all() (although, curiously, if I try to print it, it will print all records twice).
I am trying to retrieve a list with all distinct file_names. If I run print(Base.metadata.tables[table_name].columns.keys()) I will get ['id', 'file_name'] back, so clearly the field file_name exists. However, I get an error saying that it does not exist when I try to query for distinct file names. Here is the complete function:
def retrieve_classes(table_name):
names = []
Base.metadata.reflect(engine)
with Session() as s:
table_data = Base.metadata.tables[table_name]
for info in s.query(table_data).distinct(table_data.file_name).group_by(table_data.file_name):
names.append(info.file_name)
return names
Any help is very appreciated. Thanks!
--
Update: Please notice that the question listed above, which already has an answer, did not solve my problem.
As suggested, I changed the line to the following:
for info in s.query(table_data).distinct(table_data.c.file_name).group_by(table_data.c.file_name)
This also did resulted in an error. Full traceback is listed below.
127.0.0.1 - - [04/Sep/2019 15:17:30] "POST /predict HTTP/1.1" 500 -
Traceback (most recent call last):
File "C:\git\a\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1249, in _execute_context
cursor, statement, parameters, context
File "C:\git\a\venv\lib\site-packages\sqlalchemy\engine\default.py", line 552, in do_execute
cursor.execute(statement, parameters)
File "C:\git\a\venv\lib\site-packages\MySQLdb\cursors.py", line 209, in execute
res = self._query(query)
File "C:\git\a\venv\lib\site-packages\MySQLdb\cursors.py", line 315, in _query
db.query(q)
File "C:\git\a\venv\lib\site-packages\MySQLdb\connections.py", line 226, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1055, '')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\git\a\venv\lib\site-packages\flask\app.py", line 2463, in __call__
return self.wsgi_app(environ, start_response)
File "C:\git\a\venv\lib\site-packages\flask\app.py", line 2449, in wsgi_app
response = self.handle_exception(e)
File "C:\git\a\venv\lib\site-packages\flask\app.py", line 1866, in handle_exception
reraise(exc_type, exc_value, tb)
File "C:\git\a\venv\lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "C:\git\a\venv\lib\site-packages\flask\app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "C:\git\a\venv\lib\site-packages\flask\app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\git\a\venv\lib\site-packages\flask\app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "C:\git\a\venv\lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "C:\git\a\venv\lib\site-packages\flask\app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "C:\git\a\venv\lib\site-packages\flask\app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "C:\git\a\model\model.py", line 87, in retrieve_names
for info in s.query(table_data).distinct(table_data.c.file_name).group_by(table_data.c.file_name):
File "C:\git\a\venv\lib\site-packages\sqlalchemy\orm\query.py", line 3334, in __iter__
return self._execute_and_instances(context)
File "C:\git\a\venv\lib\site-packages\sqlalchemy\orm\query.py", line 3359, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "C:\git\a\venv\lib\site-packages\sqlalchemy\engine\base.py", line 988, in execute
return meth(self, multiparams, params)
File "C:\git\a\venv\lib\site-packages\sqlalchemy\sql\elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "C:\git\a\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1107, in _execute_clauseelement
distilled_params,
File "C:\git\a\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1253, in _execute_context
e, statement, parameters, cursor, context
File "C:\git\a\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1473, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "C:\git\a\venv\lib\site-packages\sqlalchemy\util\compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "C:\git\a\venv\lib\site-packages\sqlalchemy\util\compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "C:\git\a\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1249, in _execute_context
cursor, statement, parameters, context
File "C:\git\a\venv\lib\site-packages\sqlalchemy\engine\default.py", line 552, in do_execute
cursor.execute(statement, parameters)
File "C:\git\a\venv\lib\site-packages\MySQLdb\cursors.py", line 209, in execute
res = self._query(query)
File "C:\git\a\venv\lib\site-packages\MySQLdb\cursors.py", line 315, in _query
db.query(q)
File "C:\git\a\venv\lib\site-packages\MySQLdb\connections.py", line 226, in query
_mysql.connection.query(self, query)sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1055, '')
[SQL: SELECT DISTINCT data_model1.id AS data_model1_id, data_model1.file_name AS data_model1_file_name, data_model1.field2 AS data_model1_field2
FROM data_model1 GROUP BY data_model1.file_name]
(Background on this error at: http://sqlalche.me/e/e3q8)

Flask-sqlalchemy broken pipe error 32 with mysql and flask-login

Broken pipe (error 32) from mysql to flask-sqlalchemy on initial page load (login screen using flask-login), second request works. Site will continue to work for hours, then fail again.
Connection is through mysql+pymysql url
Database is local mysql 5.7 server on a ubuntu 18.04 machine
Site is hosted through gunicorn
Site is using blueprints.
Flask==1.0.2
Flask-SQLAlchemy==2.4.0
Flask-Login==0.4.1
site was is built from:
https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world
In config.py
SQLALCHEMY_POOL_RECYCLE=300
from mysql db:
wait_timeout = 28800
From /etc/mysql/mysql.conf.d/mysqld.cnf
max_allowed_packet = 100M
From other threads on here, sqlalchemy is performing the session closure, so I do not have anything in a after_request function.
I have gone to the third page of google with the following searches:
flask-sqlalchemy broken pipe
mysql sqlalchemy broken pipe
flask-login mysql broken pipe
2019-06-15 10:48:43,061 ERROR: Exception on /user/login [POST] [in /srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py:1761]
11 Traceback (most recent call last):
12 File "/srv/troopnet/venv/lib/python3.6/site-packages/pymysql/connections.py", line 713, in _write_bytes
13 self._sock.sendall(data)
14 BrokenPipeError: [Errno 32] Broken pipe
Simplified site:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_login import LoginManager
db = SQLAlchemy()
login = LoginManager()
login.login_view = 'user.login'
def create_app(config_class=Config):
app = Flask(__name__)
app.config.from_object(config_class)
db.init_app(app)
login.init_app(app)
from app.main import mod_main
app.register_blueprint(mod_main, url_prefix='/')
from app.user import mod_user
app.register_blueprint(mod_user, url_prefix='/user')
I had this site on AWS running on a 16.04 VM for 2 years, then I recently migrated to a 18.04 server and upgraded all my flask/python modules.
I have yet to go through each of the module versions to confirm why this behaviour now occurs.
EDIT
Full stack trace (removed email address and hash where 'ABC' and 'XYZ' are)
2019-06-15 10:48:43,061 ERROR: Exception on /user/login [POST] [in /srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py:1761]
Traceback (most recent call last):
File "/srv/troopnet/venv/lib/python3.6/site-packages/pymysql/connections.py", line 713, in _write_bytes
self._sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
cursor.execute(statement, parameters)
File "/srv/troopnet/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/srv/troopnet/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/srv/troopnet/venv/lib/python3.6/site-packages/pymysql/connections.py", line 516, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/srv/troopnet/venv/lib/python3.6/site-packages/pymysql/connections.py", line 771, in _execute_command
self._write_bytes(packet)
File "/srv/troopnet/venv/lib/python3.6/site-packages/pymysql/connections.py", line 718, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/srv/troopnet/app/user/routes.py", line 519, in login
user = User().authenticate(form.email.data, form.password.data).first()
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3215, in first
ret = list(self[0:1])
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3007, in __getitem__
return list(res)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
return self._execute_and_instances(context)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3342, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
return meth(self, multiparams, params)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
distilled_params,
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
e, statement, parameters, cursor, context
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
raise value.with_traceback(tb)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
cursor.execute(statement, parameters)
File "/srv/troopnet/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/srv/troopnet/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/srv/troopnet/venv/lib/python3.6/site-packages/pymysql/connections.py", line 516, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/srv/troopnet/venv/lib/python3.6/site-packages/pymysql/connections.py", line 771, in _execute_command
self._write_bytes(packet)
File "/srv/troopnet/venv/lib/python3.6/site-packages/pymysql/connections.py", line 718, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")
[SQL: SELECT users.id AS users_id, users.email AS users_email, users.password AS users_password
FROM users
WHERE users.email = %(email_1)s AND users.password = %(password_1)s
LIMIT %(param_1)s]
[parameters: {'email_1': 'ABC', 'password_1': 'XYZ', 'param_1': 1}]
(Background on this error at: http://sqlalche.me/e/e3q8)
2019-06-15 10:48:47,064 ERROR: Exception on /user/login [POST] [in /srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py:1761]
Traceback (most recent call last):
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1177, in _execute_context
conn = self._revalidate_connection()
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 463, in _revalidate_connection
"Can't reconnect until invalid "
sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid transaction is rolled back
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/srv/troopnet/app/user/routes.py", line 519, in login
user = User().authenticate(form.email.data, form.password.data).first()
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3215, in first
ret = list(self[0:1])
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3007, in __getitem__
return list(res)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
return self._execute_and_instances(context)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3342, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
return meth(self, multiparams, params)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
distilled_params,
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
e, util.text_type(statement), parameters, None, None
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
raise value.with_traceback(tb)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1177, in _execute_context
conn = self._revalidate_connection()
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 463, in _revalidate_connection
"Can't reconnect until invalid "
sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
[SQL: SELECT users.id AS users_id, users.email AS users_email, users.password AS users_password
FROM users
WHERE users.email = %(email_1)s AND users.password = %(password_1)s
LIMIT %(param_1)s]
[parameters: [{}]]
2019-06-15 10:48:50,638 ERROR: Exception on /user/login [POST] [in /srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py:1761]
Traceback (most recent call last):
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1177, in _execute_context
conn = self._revalidate_connection()
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 463, in _revalidate_connection
"Can't reconnect until invalid "
sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid transaction is rolled back
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/srv/troopnet/app/user/routes.py", line 519, in login
user = User().authenticate(form.email.data, form.password.data).first()
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3215, in first
ret = list(self[0:1])
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3007, in __getitem__
return list(res)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
return self._execute_and_instances(context)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3342, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
return meth(self, multiparams, params)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
distilled_params,
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
e, util.text_type(statement), parameters, None, None
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
raise value.with_traceback(tb)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1177, in _execute_context
conn = self._revalidate_connection()
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 463, in _revalidate_connection
"Can't reconnect until invalid "
sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
[SQL: SELECT users.id AS users_id, users.email AS users_email, users.password AS users_password
FROM users
WHERE users.email = %(email_1)s AND users.password = %(password_1)s
LIMIT %(param_1)s]
[parameters: [{}]]
2019-06-15 10:48:56,354 ERROR: Exception on /user/login [POST] [in /srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py:1761]
Traceback (most recent call last):
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1177, in _execute_context
conn = self._revalidate_connection()
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 463, in _revalidate_connection
"Can't reconnect until invalid "
sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid transaction is rolled back
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
raise value
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/srv/troopnet/venv/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/srv/troopnet/app/user/routes.py", line 519, in login
user = User().authenticate(form.email.data, form.password.data).first()
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3215, in first
ret = list(self[0:1])
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3007, in __getitem__
return list(res)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
return self._execute_and_instances(context)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3342, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
return meth(self, multiparams, params)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
distilled_params,
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
e, util.text_type(statement), parameters, None, None
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
raise value.with_traceback(tb)
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1177, in _execute_context
conn = self._revalidate_connection()
File "/srv/troopnet/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 463, in _revalidate_connection
"Can't reconnect until invalid "
sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
[SQL: SELECT users.id AS users_id, users.email AS users_email, users.password AS users_password
FROM users
WHERE users.email = %(email_1)s AND users.password = %(password_1)s
LIMIT %(param_1)s]
[parameters: [{}]]

Based on your comment of the recent upgrade of dependencies, I think that the problem is with the version of Flask-Login.
I had the same issue on a different, but similar Flask build (using the same dependencies). There is an issue of how users sessions are handled by flask_login version 0.4.1 (issue tracker here).
The problem went away when I upgraded to the below:
Flask-Login==0.5.0
Flask==1.1.2
note you may also need the below (don't install the latest 1.0.1 as it didn't work in my case):
Werkzeug==0.16.1
If the above doesn't help you may want to go through the below list (didn't work in my case):
https://www.cnblogs.com/lesliexong/p/8654615.html <- you may have to use google translate
Good luck!

pymongo gridfs mongodb authentication fails

I am using gridfs to store and get files from my MongoDB database
fs = gridfs.GridFS(MongoClient('mongodb://username:password#XX.XX.XX.XXX:27017/').mydb)
and to get a file I am using
file = fs.get_last_version(file_name).read()
On this line I get Authentication Failed exception.
Traceback (most recent call last):
File "C:\Python36\lib\site-packages\flask\app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "C:\Python36\lib\site-packages\flask\app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "C:\Python36\lib\site-packages\flask\app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "C:\Python36\lib\site-packages\flask\_compat.py", line 33, in reraise
raise value
File "C:\Python36\lib\site-packages\flask\app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "C:\Python36\lib\site-packages\flask\app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Python36\lib\site-packages\flask\app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "C:\Python36\lib\site-packages\flask\_compat.py", line 33, in reraise
raise value
File "C:\Python36\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Python36\lib\site-packages\flask\app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "D:\git\EEG\ABM\BLabPythonService\app.py", line 1039, in start_processing
sys.stderr.flush()
File "D:\git\EEG\ABM\BLabPythonService\process_studies.py", line 271, in start_processing
overlay_artifacts(study)
File "D:\git\EEG\ABM\BLabPythonService\process_studies.py", line 49, in overlay_artifacts
edf_file = fs.get_last_version(file_name).read()
File "C:\Python36\lib\site-packages\gridfs\__init__.py", line 200, in get_last_version
return self.get_version(filename=filename, **kwargs)
File "C:\Python36\lib\site-packages\gridfs\__init__.py", line 184, in get_version
grid_file = next(cursor)
File "C:\Python36\lib\site-packages\pymongo\cursor.py", line 1132, in next
if len(self.__data) or self._refresh():
File "C:\Python36\lib\site-packages\pymongo\cursor.py", line 1055, in _refresh
self.__collation))
File "C:\Python36\lib\site-packages\pymongo\cursor.py", line 892, in __send_message
**kwargs)
File "C:\Python36\lib\site-packages\pymongo\mongo_client.py", line 950, in _send_message_with_response
exhaust)
File "C:\Python36\lib\site-packages\pymongo\mongo_client.py", line 961, in _reset_on_error
return func(*args, **kwargs)
File "C:\Python36\lib\site-packages\pymongo\server.py", line 99, in send_message_with_response
with self.get_socket(all_credentials, exhaust) as sock_info:
File "C:\Python36\lib\contextlib.py", line 81, in __enter__
return next(self.gen)
File "C:\Python36\lib\site-packages\pymongo\server.py", line 168, in get_socket
with self.pool.get_socket(all_credentials, checkout) as sock_info:
File "C:\Python36\lib\contextlib.py", line 81, in __enter__
return next(self.gen)
File "C:\Python36\lib\site-packages\pymongo\pool.py", line 852, in get_socket
sock_info.check_auth(all_credentials)
File "C:\Python36\lib\site-packages\pymongo\pool.py", line 570, in check_auth
auth.authenticate(credentials, self)
File "C:\Python36\lib\site-packages\pymongo\auth.py", line 486, in authenticate
auth_func(credentials, sock_info)
File "C:\Python36\lib\site-packages\pymongo\auth.py", line 466, in _authenticate_default
return _authenticate_scram_sha1(credentials, sock_info)
File "C:\Python36\lib\site-packages\pymongo\auth.py", line 237, in _authenticate_scram_sha1
res = sock_info.command(source, cmd)
File "C:\Python36\lib\site-packages\pymongo\pool.py", line 477, in command
collation=collation)
File "C:\Python36\lib\site-packages\pymongo\network.py", line 116, in command
parse_write_concern_error=parse_write_concern_error)
File "C:\Python36\lib\site-packages\pymongo\helpers.py", line 210, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Authentication failed.
Everything worked fine when my mongo database didnt have authentication.
Other use of database is working fine. I have this python service that is connecting to database using flask_pymogno and a c# application that is also connecting to database using c#, and all connections are with the same credentials.

Solved it. The problem was actually in my password witch contained '+' character. It is a reserved URI character. Fixed it by replacing '+' with '%2B' and it now works.
Reference:
https://en.wikipedia.org/wiki/Percent-encoding