We are creating an application which consists of a frontend (flask api) and a backend that uses celery. The API starts a celery task and retrieves the result like this:
result = data_source_tasks.add_data_point.delay(tok, uuid, source_type, datum, request_counter)
return result.get(timeout=5)
We use RabbitMQ as broker and result backend:
celery_broker_url = pyamqp://guest#localhost//
celery_result_backend = rpc://
After everything runs fine for a while (multiple thousand api calls) I get the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.4/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.4/dist-packages/connexion/decorators/decorator.py", line 66, in wrapper
response = function(request)
File "/usr/local/lib/python3.4/dist-packages/connexion/decorators/validation.py", line 122, in wrapper
response = function(request)
File "/usr/local/lib/python3.4/dist-packages/connexion/decorators/validation.py", line 293, in wrapper
return function(request)
File "/usr/local/lib/python3.4/dist-packages/connexion/decorators/decorator.py", line 42, in wrapper
response = function(request)
File "/usr/local/lib/python3.4/dist-packages/connexion/decorators/parameter.py", line 219, in wrapper
return function(**kwargs)
File "/mynedata/lib/api/apicalls.py", line 747, in store_datum
return result.get(timeout=5)
File "/usr/local/lib/python3.4/dist-packages/celery/result.py", line 224, in get
on_message=on_message,
File "/usr/local/lib/python3.4/dist-packages/celery/backends/async.py", line 188, in wait_for_pending
for _ in self._wait_for_pending(result, **kwargs):
File "/usr/local/lib/python3.4/dist-packages/celery/backends/async.py", line 255, in _wait_for_pending
on_interval=on_interval):
File "/usr/local/lib/python3.4/dist-packages/celery/backends/async.py", line 56, in drain_events_until
yield self.wait_for(p, wait, timeout=1)
File "/usr/local/lib/python3.4/dist-packages/celery/backends/async.py", line 65, in wait_for
wait(timeout=timeout)
File "/usr/local/lib/python3.4/dist-packages/celery/backends/rpc.py", line 63, in drain_events
return self._connection.drain_events(timeout=timeout)
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 301, in drain_events
return self.transport.drain_events(self.connection, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/kombu/transport/pyamqp.py", line 103, in drain_events
return connection.drain_events(**kwargs)
File "/usr/local/lib/python3.4/dist-packages/amqp/connection.py", line 471, in drain_events
while not self.blocking_read(timeout):
File "/usr/local/lib/python3.4/dist-packages/amqp/connection.py", line 476, in blocking_read
frame = self.transport.read_frame()
File "/usr/local/lib/python3.4/dist-packages/amqp/transport.py", line 226, in read_frame
frame_header = read(7, True)
File "/usr/local/lib/python3.4/dist-packages/amqp/transport.py", line 401, in _read
s = recv(n - len(rbuf))
ConnectionResetError: [Errno 104] Connection reset by peer
I can see in the console where I started the celery worker that the task (and all following tasks) succeeded, however, result.get results in a timeout for this and all following tasks. Did my connection to the result backend somehow break? If I restart the API, neither restarting the celery worker nor rabbitmq, everything works fine again.
Related
I am writing an application and whenever I do one request at a time, everything is fine. but when my vuejs frontend does a call to my flask backend, using 2 requests at the same time, I get a big error, see below. This is presumably because flask is still processing the previous request and has a lock on the database. Is there a way to make my MySQL either asynchronous or to make them queue up? Even changing to a better database solution is OK. As long as I can have multiple requests in flask at the same time.
app.py
session = classes.Session()
# configuration
DEBUG = True
# instantiate the app
app = Flask(__name__)
app.config.from_object(__name__)
classes.py
ssl_args = {'ssl_ca': './ca.crt'}
engine = create_engine("URL", connect_args=ssl_args)
Session = scoped_session(sessionmaker(autocommit=False,
autoflush=False,
bind=engine))
Base = declarative_base()
Base.query = Session.query_property()
The error:
Traceback (most recent call last):
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\pymysql\connections.py", line 756, in _write_bytes
self._sock.sendall(data)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\Lib\ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\Lib\ssl.py", line 1173, in send
return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2477)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\engine\base.py", line 1706, in _execute_context
self.dialect.do_execute(
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\engine\default.py", line 716, in do_execute
cursor.execute(statement, parameters)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\pymysql\cursors.py", line 148, in execute
result = self._query(query)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\pymysql\cursors.py", line 310, in _query
conn.query(q)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\pymysql\connections.py", line 547, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\pymysql\connections.py", line 814, in _execute_command
self._write_bytes(packet)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\pymysql\connections.py", line 759, in _write_bytes
raise err.OperationalError(
pymysql.err.OperationalError: (2006, "MySQL server has gone away (SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2477)'))")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\flask\app.py", line 2464, in __call__
return self.wsgi_app(environ, start_response)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\flask\app.py", line 2450, in wsgi_app
response = self.handle_exception(e)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\flask_cors\extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\flask\app.py", line 1867, in handle_exception
reraise(exc_type, exc_value, tb)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\flask\app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\flask\app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\flask_cors\extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\flask\app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\flask\_compat.py", line 39, in reraise
raise value
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\flask\app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\flask\app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "E:\Documents\GitHub\thought4food\app\app.py", line 160, in recipeById
return json.loads(str(recipe))
File "E:\Documents\GitHub\thought4food\app\classes\recipe.py", line 21, in __repr__
ingredients = json.loads(str(self.recipe_ingredients))
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\orm\attributes.py", line 465, in __get__
return self.impl.get(state, dict_)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\orm\attributes.py", line 911, in get
value = self.callable_(state, passive)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\orm\strategies.py", line 878, in _load_for_state
return self._emit_lazyload(
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\orm\strategies.py", line 1041, in _emit_lazyload
result = session.execute(
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\orm\session.py", line 1670, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\engine\base.py", line 1521, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\sql\lambdas.py", line 479, in _execute_on_connection
return connection._execute_clauseelement(
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\engine\base.py", line 1390, in _execute_clauseelement
ret = self._execute_context(
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\engine\base.py", line 1749, in _execute_context
self._handle_dbapi_exception(
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\engine\base.py", line 1930, in _handle_dbapi_exception
util.raise_(
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\util\compat.py", line 211, in raise_
raise exception
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\engine\base.py", line 1706, in _execute_context
self.dialect.do_execute(
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\sqlalchemy\engine\default.py", line 716, in do_execute
cursor.execute(statement, parameters)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\pymysql\cursors.py", line 148, in execute
result = self._query(query)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\pymysql\cursors.py", line 310, in _query
conn.query(q)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\pymysql\connections.py", line 547, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\pymysql\connections.py", line 814, in _execute_command
self._write_bytes(packet)
File "E:\Documents\GitHub\thought4food\app\env\Lib\site-packages\pymysql\connections.py", line 759, in _write_bytes
raise err.OperationalError(
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2477)'))")
[SQL: SELECT ingredients.id AS ingredients_id, ingredients.name AS ingredients_name, ingredients.amount AS ingredients_amount, ingredients.unit AS ingredients_unit
FROM ingredients, recipe_ingredients
WHERE %(param_1)s = recipe_ingredients.recipe_id AND ingredients.id = recipe_ingredients.ingredient_id]
[parameters: {'param_1': 0}]
(Background on this error at: http://sqlalche.me/e/14/e3q8)
I found the solution.
I changed session = classes.Session() to session = classes.Session and I added:
#app.teardown_request
def remove_session(ex=None):
session.remove()
This made sure that the session was no longer in use and gave control of the session to flask.
I am using gridfs to store and get files from my MongoDB database
fs = gridfs.GridFS(MongoClient('mongodb://username:password#XX.XX.XX.XXX:27017/').mydb)
and to get a file I am using
file = fs.get_last_version(file_name).read()
On this line I get Authentication Failed exception.
Traceback (most recent call last):
File "C:\Python36\lib\site-packages\flask\app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "C:\Python36\lib\site-packages\flask\app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "C:\Python36\lib\site-packages\flask\app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "C:\Python36\lib\site-packages\flask\_compat.py", line 33, in reraise
raise value
File "C:\Python36\lib\site-packages\flask\app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "C:\Python36\lib\site-packages\flask\app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Python36\lib\site-packages\flask\app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "C:\Python36\lib\site-packages\flask\_compat.py", line 33, in reraise
raise value
File "C:\Python36\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Python36\lib\site-packages\flask\app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "D:\git\EEG\ABM\BLabPythonService\app.py", line 1039, in start_processing
sys.stderr.flush()
File "D:\git\EEG\ABM\BLabPythonService\process_studies.py", line 271, in start_processing
overlay_artifacts(study)
File "D:\git\EEG\ABM\BLabPythonService\process_studies.py", line 49, in overlay_artifacts
edf_file = fs.get_last_version(file_name).read()
File "C:\Python36\lib\site-packages\gridfs\__init__.py", line 200, in get_last_version
return self.get_version(filename=filename, **kwargs)
File "C:\Python36\lib\site-packages\gridfs\__init__.py", line 184, in get_version
grid_file = next(cursor)
File "C:\Python36\lib\site-packages\pymongo\cursor.py", line 1132, in next
if len(self.__data) or self._refresh():
File "C:\Python36\lib\site-packages\pymongo\cursor.py", line 1055, in _refresh
self.__collation))
File "C:\Python36\lib\site-packages\pymongo\cursor.py", line 892, in __send_message
**kwargs)
File "C:\Python36\lib\site-packages\pymongo\mongo_client.py", line 950, in _send_message_with_response
exhaust)
File "C:\Python36\lib\site-packages\pymongo\mongo_client.py", line 961, in _reset_on_error
return func(*args, **kwargs)
File "C:\Python36\lib\site-packages\pymongo\server.py", line 99, in send_message_with_response
with self.get_socket(all_credentials, exhaust) as sock_info:
File "C:\Python36\lib\contextlib.py", line 81, in __enter__
return next(self.gen)
File "C:\Python36\lib\site-packages\pymongo\server.py", line 168, in get_socket
with self.pool.get_socket(all_credentials, checkout) as sock_info:
File "C:\Python36\lib\contextlib.py", line 81, in __enter__
return next(self.gen)
File "C:\Python36\lib\site-packages\pymongo\pool.py", line 852, in get_socket
sock_info.check_auth(all_credentials)
File "C:\Python36\lib\site-packages\pymongo\pool.py", line 570, in check_auth
auth.authenticate(credentials, self)
File "C:\Python36\lib\site-packages\pymongo\auth.py", line 486, in authenticate
auth_func(credentials, sock_info)
File "C:\Python36\lib\site-packages\pymongo\auth.py", line 466, in _authenticate_default
return _authenticate_scram_sha1(credentials, sock_info)
File "C:\Python36\lib\site-packages\pymongo\auth.py", line 237, in _authenticate_scram_sha1
res = sock_info.command(source, cmd)
File "C:\Python36\lib\site-packages\pymongo\pool.py", line 477, in command
collation=collation)
File "C:\Python36\lib\site-packages\pymongo\network.py", line 116, in command
parse_write_concern_error=parse_write_concern_error)
File "C:\Python36\lib\site-packages\pymongo\helpers.py", line 210, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Authentication failed.
Everything worked fine when my mongo database didnt have authentication.
Other use of database is working fine. I have this python service that is connecting to database using flask_pymogno and a c# application that is also connecting to database using c#, and all connections are with the same credentials.
Solved it. The problem was actually in my password witch contained '+' character. It is a reserved URI character. Fixed it by replacing '+' with '%2B' and it now works.
Reference:
https://en.wikipedia.org/wiki/Percent-encoding
Hello I am trying to connect to a neo4j database with py2neo.
This code works so far:
graph = Graph(bolt=True, host='***',
bolt_port=***,
http_port=***,
user='***',
password='***')
But when I protect my IP address with a password over HTTP then I can't connect and I don't know how to authorize py2neo to connect.
Does anyone know how I can solve this? :)
Edit :
When I open the ip in a web browser i need to input a user and a password before i can see the neo4j browser - and I do not now how to input these credentias with py2neo (because it seems that this is my problem)
Stacktrace:
Traceback (most recent call last):
File "WebApp35\lib\site-packages\flask\app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "WebApp35\lib\site-packages\flask\app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "WebApp35\lib\site-packages\flask_restful\__init__.py", line 271, in error_router
return original_handler(e)
File "WebApp35\lib\site-packages\flask\app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "WebApp35\lib\site-packages\flask\_compat.py", line 32, in reraise
raise value.with_traceback(tb)
File "WebApp35\lib\site-packages\flask\app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "WebApp35\lib\site-packages\flask\app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "WebApp35\lib\site-packages\flask_restful\__init__.py", line 271, in error_router
return original_handler(e)
File "WebApp35\lib\site-packages\flask\app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "WebApp35\lib\site-packages\flask\_compat.py", line 32, in reraise
raise value.with_traceback(tb)
File "WebApp35\lib\site-packages\flask\app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "WebApp35\lib\site-packages\flask\app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "WebApp35\lib\site-packages\flask_restful\__init__.py", line 477, in wrapper
resp = resource(*args, **kwargs)
File "WebApp35\lib\site-packages\flask\views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "WebApp35\lib\site-packages\flask_restful\__init__.py", line 587, in dispatch_request
resp = meth(*args, **kwargs)
File "app\handlers\nodeHandlers.py", line 70, in get
return DataManager.get_suggestion(suggestion_string), 201, {
File "app\adapter\dataManager.py", line 44, in get_suggestion
return cls.adapter.get_suggestion(suggestion_string)
File "app\adapter\neoAdapter.py", line 337, in get_suggestion
for node in cls.cypher.run(query):
File "WebApp35\lib\site-packages\py2neo\database\__init__.py", line 676, in run
return self.begin(autocommit=True).run(statement, parameters, **kwparameters)
File "WebApp35\lib\site-packages\py2neo\database\__init__.py", line 351, in begin
return self.transaction_class(self, autocommit)
File "WebApp35\lib\site-packages\py2neo\database\__init__.py", line 1171, in __init__
self.session = driver.session()
File "WebApp35\lib\site-packages\py2neo\packages\neo4j\v1\session.py", line 148, in session
session = Session(self)
File "WebApp35\lib\site-packages\py2neo\packages\neo4j\v1\session.py", line 461, in __init__
self.connection = connect(driver.host, driver.port, driver.ssl_context, **driver.config)
File "WebApp35\lib\site-packages\py2neo\packages\neo4j\v1\connection.py", line 399, in connect
raise error
py2neo.packages.neo4j.v1.exceptions.ProtocolError: Cannot establish secure connection; [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:720)
I am fairly new to redis and using configuring redis with flask app.
i have edited redis.conf file and uncommented requirepass
requirepass foobared
and I am able to get into redid-cli using this password.
like this ./redis-cli -h 127.0.0.1 -p 6379 -a foobared
but when I try to initialise an object of Redis class by passing in the password,
>>> import os
>>> import redis
>>> from flask import Flask
>>> from redis import Redis
>>> app = Flask('weatherApp')
>>> app.config["REDIS_HOST"] = os.getenv("REDIS_HOST")
>>> app.config["REDIS_PORT"] = os.getenv("REDIS_PORT")
>>> app.config["REDIS_PWD"] = os.getenv("REDIS_PWD")
>>> app
<Flask 'weatherApp'>
>>> app.config
>>> redis = Redis(app.config["REDIS_HOST"], port=app.config["REDIS_PORT"],
... db=0, password=app.config["REDIS_PWD"])
>>> redis.client_list()
I keep getting the error saying:
Traceback (most recent call last):
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/flask/app.py", line 2000, in __call__
return self.wsgi_app(environ, start_response)
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/flask/app.py", line 1991, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/flask/app.py", line 1567, in handle_exception
reraise(exc_type, exc_value, tb)
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
response = self.full_dispatch_request()
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/Users/ciasto-piekarz/Development/web/python/weatherApp/src/views.py", line 192, in dispatch_request
print redis.client_list()
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/redis/client.py", line 703, in client_list
return self.execute_command('CLIENT LIST')
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/redis/client.py", line 667, in execute_command
connection.send_command(*args)
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/redis/connection.py", line 610, in send_command
self.send_packed_command(self.pack_command(*args))
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/redis/connection.py", line 585, in send_packed_command
self.connect()
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/redis/connection.py", line 493, in connect
self.on_connect()
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/redis/connection.py", line 561, in on_connect
if nativestr(self.read_response()) != 'OK':
File "/Users/ciasto-piekarz/Development/web/python/flask/flsk-dimnd/venv/lib/python2.7/site-packages/redis/connection.py", line 629, in read_response
raise response
ResponseError: Client sent AUTH, but no password is set
Any help will be greatly appreciated!
I want to send bytes in a string to Celery's task using Redis as broker, but I'm receiving error shown below:
[2017-06-17 21:27:13,826] ERROR in app: Exception on /endpoint_method [POST]
Traceback (most recent call last):
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/Users/developer/my_project/application.py", line 23, in endpoint_method
task = my_task.execute.delay(request.data)
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/celery/app/task.py", line 412, in delay
return self.apply_async(args, kwargs)
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/celery/app/task.py", line 535, in apply_async
**options
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/celery/app/base.py", line 737, in send_task
amqp.send_task_message(P, name, message, **options)
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/celery/app/amqp.py", line 558, in send_task_message
**properties
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/kombu/messaging.py", line 169, in publish
compression, headers)
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/kombu/messaging.py", line 252, in _prepare
body) = dumps(body, serializer=serializer)
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/kombu/serialization.py", line 54, in _reraise_errors
reraise(wrapper, wrapper(exc), sys.exc_info()[2])
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/Users/developer/my_project/venv/lib/python2.7/site-packages/kombu/utils/json.py", line 72, in dumps
**dict(default_kwargs, **kwargs))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 250, in dumps
sort_keys=sort_keys, **kw).encode(obj)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
EncodeError: 'utf8' codec can't decode byte 0x80 in position 5: invalid start byte
Shortened (mostly it's up to 1 mln characters string) example data which I want to send to:
'RIFF$\x80\r\x00WAVEfmt \x10\x00\x00\x00\x01\x00\x02\x00D\xac\x00\x00\x10\xb1\x02\x00\x04\x00\x10\x00data\x00\x80\r\x00z\xefz\xef\xd5\xec\xd5\xec\xc1\xee\xc1\xee\xe6\xf3\xe6\xf3\xb4\xfa\xb4\xfa\x92\x03\x92\x03\xab\x0e\xab\x0e\xf9\x18\xf9\x18D\x1eD\x1ev\x1dv\x1d#\x19#\x19\x86\x13\x86\x13w\nw\n\xf0\xfd\xf0\xfd\xf9\xf2\xf9\xf2\xd0\xed\xd0\xedK\xedK\xed{\xed{\xed%\xee%\xeeP\xf3P\xf3\xeb\xfc\xeb\xfc!\x05!\x05\xa4\x08\xa4\x08\xe5\t\xe5\t\x84\x0b\x84\x0bF\x0bF\x0b\xfb\x04\xfb\x040\xfa0\xfa\x86\xf1\x86\xf1T\xeeT'
Part of the code responsible for calling Celery's task:
task = my_task.execute.delay(request.data)
This is how tasks/my_task.py file looks like:
from __future__ import absolute_import, unicode_literals
from my_module.celery import app
#app.task
def execute(request_data):
return <some operations on request_data>
and celery.py:
from __future__ import absolute_import, unicode_literals
from celery import Celery
app = Celery(
'my_module',
broker='redis://127.0.0.1:6379/1',
backend='redis://127.0.0.1:6379/2',
include=[
'my_module.tasks.my_task'
]
)
app.conf.update(
result_expires=3600,
)
if __name__ == '__main__':
app.start()
Maybe is there some clever way to encode that string before sending it to Redis and decode it in while running the task?
Finally, I came up with a solution using binascii methods binascii.b2a_base64(data) and binascii.a2b_base64(string). This is what I had to change to make this working:
task = my_task.execute.delay(binascii.b2a_base64(request.data))
and in tasks/my_task.py I had to convert data back to be able to make proper operations on it:
binascii.a2b_base64(request_data)