I'm getting the following error in one of my Celery workers:
2015-07-21T15:02:04.010066+00:00 app[worker.1]: Traceback (most recent call last):
2015-07-21T15:02:04.010069+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 296, in trace_task
2015-07-21T15:02:04.010070+00:00 app[worker.1]: on_chord_part_return(task, state, R)
2015-07-21T15:02:04.010073+00:00 app[worker.1]: deps.delete()
2015-07-21T15:02:04.010074+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/result.py", line 773, in delete
2015-07-21T15:02:04.010071+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 587, in on_chord_part_return
2015-07-21T15:02:04.010078+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 329, in delete_group
2015-07-21T15:02:04.010076+00:00 app[worker.1]: (backend or self.app.backend).delete_group(self.id)
2015-07-21T15:02:04.010079+00:00 app[worker.1]: return self._delete_group(group_id)
2015-07-21T15:02:04.010081+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 499, in _delete_group
2015-07-21T15:02:04.010082+00:00 app[worker.1]: self.delete(self.get_key_for_group(group_id))
2015-07-21T15:02:04.010083+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 172, in delete
2015-07-21T15:02:04.010084+00:00 app[worker.1]: self.client.delete(key)
2015-07-21T15:02:04.010085+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 824, in delete
2015-07-21T15:02:04.010087+00:00 app[worker.1]: return self.execute_command('DEL', *names)
2015-07-21T15:02:04.010088+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 565, in execute_command
2015-07-21T15:02:04.010089+00:00 app[worker.1]: return self.parse_response(connection, command_name, **options)
2015-07-21T15:02:04.010090+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 579, in parse_response
2015-07-21T15:02:04.010091+00:00 app[worker.1]: return self.response_callbacks[command_name](response, **options)
2015-07-21T15:02:04.010093+00:00 app[worker.1]: ValueError: invalid literal for int() with base 10: 'QUEUED'
What I find weird is that I see no call to int in the last line of the stack trace. QUEUED probably came in as a worker's status. I'm using it as a custom worker status like this:
#before_task_publish.connect
def update_sent_state(sender=None, body=None, **kwargs):
# the task may not exist if sent using `send_task` which
# sends tasks by name, so fall back to the default result backend
# if that is the case.
task = current_app.tasks.get(sender)
backend = task.backend if task else current_app.backend
logging.debug("Setting status for %s" % body["id"])
backend.store_result(body['id'], None, "QUEUED")
What could be the issue here?
In case it's relevant, here's the code for my task. I only call fetch directly is fetch.
#app.task
def fetch(url_or_urls, subscribe=None):
"""This fetches a (list of) podcast(s) and stores it in the db. It assumes that it only gets called
by Podcast.get_by_url, or some other method that knows whether a given podcast has
already been fetched.
If *subscribe* is given, it should be a User instance to be subscribed to the given podcasts."""
if isinstance(url_or_urls, basestring):
url_or_urls = [url_or_urls]
body = _store_podcasts.s()
if subscribe:
body.link(_subscribe_user.s(user=subscribe))
return chord([_fetch_podcast_data.s(url) for url in url_or_urls])(body)
#app.task
def _fetch_podcast_data(url):
return do_fetch(url) # This function returns a dict of podcast data.
#app.task
def _store_podcasts(podcasts_data):
"""Given a list of dictionaries representing podcasts, store them all in the database."""
podcasts = [Podcast(**pdata) for pdata in podcasts_data]
return Podcast.objects.insert(podcasts)
#app.task
def _subscribe_user(podcasts, user):
"""Subscribe the given users to all the podcasts in the list."""
return user.subscribe_multi(podcasts)
Is there anything else that could be relevant here?
Library versions as shown by pip freeze:
redis==2.10.3
celery==3.1.18
It is hard to debug such a bug without working code. Here is what i think it could be.
Lets start here:
http://celery.readthedocs.org/en/latest/_modules/celery/backends/base.html#BaseBackend.store_result
def store_result(self, task_id, result, status,
traceback=None, request=None, **kwargs):
"""Update task state and result."""
result = self.encode_result(result, status)
self._store_result(task_id, result, status, traceback,
request=request, **kwargs)
return result
It calls ecnode_result. Lets check that out
def encode_result(self, result, status):
if status in self.EXCEPTION_STATES and isinstance(result, Exception):
return self.prepare_exception(result)
else:
return self.prepare_value(result)
It looks like "status" is expected to be something from predefined STATE constants.
Its code is here
http://celery.readthedocs.org/en/latest/_modules/celery/states.html#state
And docs here
http://celery.readthedocs.org/en/latest/reference/celery.states.html
That does not look like they expect to see something like "QUEUED" there. Try one of the predefined.
The redis python packages expects the response from the DEL action to always be an integer, which I assume is the count of deleted rows.
The call to int happens in the last line (return self.response_callbacks[command_name](response, **options)) where self.response_callbacks['DEL'] is equal to int.
As a workaround, you could subclass the redis.client.StrictRedis and set the DEL response callback to something other than int, just make sure you're familiar with the implications.
I got the same error these days. And founded my QUEUED response comes from redis MULTI commands. See https://redis.io/topics/transactions#usage.
It maybe that your are reading response from wrong connection. Maybe in multi-processing / multi-threading / eventlet, etc. Not sure.
Related
Im working on a slack bot using the new slack 2.0 python library. I am new to python decorators and I suspect that is part of my problem.
Here is my code...
#!/opt/rh/rh-python36/root/usr/bin/python
import os
import slack
# instantiate Slack client
slack_token = os.environ['SLACK_BOT_TOKEN']
rtmclient = slack.RTMClient(token=slack_token)
webclient = slack.WebClient(token=slack_token)
# get the id of my user
bot_id = webclient.auth_test()['user_id']
print('Bot ID: {0}'.format(bot_id))
def get_user_info(user_id):
user_info = webclient.users_info(user=user_id)['ok']
return user_info
#slack.RTMClient.run_on(event='message')
def parse_message(**payload):
data = payload['data']
user_id = data['user']
print(get_user_info(user_id))
rtmclient.start()
It outputs the Bot ID(using the webclient) when started but then crashes with RuntimeError: This event loop is already running when I make another call to webclient.
[root#slackbot-01 bin]# scl enable rh-python36 /root/slackbot/bin/slackbot.py
Bot ID: UBT547D31
Traceback (most recent call last):
File "/root/slackbot/bin/slackbot.py", line 24, in <module>
rtmclient.start()
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/rtm/client.py", line 197, in start
return self._event_loop.run_until_complete(future)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/asyncio/base_events.py", line 467, in run_until_complete
return future.result()
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/rtm/client.py", line 339, in _connect_and_read
await self._read_messages()
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/rtm/client.py", line 390, in _read_messages
await self._dispatch_event(event, data=payload)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/rtm/client.py", line 440, in _dispatch_event
self._execute_in_thread(callback, data)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/rtm/client.py", line 465, in _execute_in_thread
future.result()
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/root/slackbot/bin/slackbot.py", line 22, in parse_message
print(get_user_info(user_id))
File "/root/slackbot/bin/slackbot.py", line 15, in get_user_info
user_info = webclient.users_info(user=user_id)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/web/client.py", line 1368, in users_info
return self.api_call("users.info", http_verb="GET", params=kwargs)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/slack/web/base_client.py", line 154, in api_call
return self._event_loop.run_until_complete(future)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/asyncio/base_events.py", line 454, in run_until_complete
self.run_forever()
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/asyncio/base_events.py", line 408, in run_forever
raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
The really confusing part to me is that if I comment out the line that makes the first call to webclient.auth_test(), I have no issues at all. My call to webclient.users_info() works every time rtmclient sends me data.
#!/opt/rh/rh-python36/root/usr/bin/python
import os
import slack
# instantiate Slack client
slack_token = os.environ['SLACK_BOT_TOKEN']
rtmclient = slack.RTMClient(token=slack_token)
webclient = slack.WebClient(token=slack_token)
# get the id of my user
#bot_id = webclient.auth_test()['user_id']
#print('Bot ID: {0}'.format(bot_id))
def get_user_info(user_id):
user_info = webclient.users_info(user=user_id)['ok']
return user_info
#slack.RTMClient.run_on(event='message')
def parse_message(**payload):
data = payload['data']
user_id = data['user']
print(get_user_info(user_id))
rtmclient.start()
[root#slackbot-01 bin]# scl enable rh-python36 /root/slackbot/bin/slackbot.py
True
True
^C[root#slackbot-01 bin]#
I need to get the bot id so that I can make sure it doesnt answer it's own messages. I don't why my code doesnt work after I get the bot id outside of the parse message function with a decorator.
What am I doing wrong here?
The python event loop is a tricky thing to program libraries around and there are some issues with the way the event queue is managed in the 2.0 version of SlackClient. It looks like some improvements were made with 2.1 but it appears to be a work in progress, and I still encounter this. I'd expect there will be future updates to make it more robust.
In the meantime, the following code at the top of your file (use pip to install) usually resolves it for me:
import nest_asyncio
nest_asyncio.apply()
Keep in mind this will alter the way the rest of your application is handling the event queue, if that's a factor.
If you're using RTM, the RTMClient creates a WebClient for you. The handle for it should be getting passed to you in the payload when you handle an event. You can check your ID by looking for the 'open' event which is always dispatched after RTM successfully connects and doing the lookup inside your 'open' event handler.
I have a textbased interface (asciimatics module) for my program that uses asyncio and discord.py module and occasionally when my wifi adapter goes down I get an exception like so:
Task exception was never retrieved
future: <Task finished coro=<WebSocketCommonProtocol.run() done, defined at /home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py:428> exception=ConnectionResetError(104, 'Connection reset by peer')>
Traceback (most recent call last):
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 434, in run
msg = yield from self.read_message()
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 456, in read_message
frame = yield from self.read_data_frame(max_size=self.max_size)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 511, in read_data_frame
frame = yield from self.read_frame(max_size)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/protocol.py", line 546, in read_frame
self.reader.readexactly, is_masked, max_size=max_size)
File "/home/mike/.local/lib/python3.5/site-packages/websockets/framing.py", line 86, in read_frame
data = yield from reader(2)
File "/usr/lib/python3.5/asyncio/streams.py", line 670, in readexactly
block = yield from self.read(n)
File "/usr/lib/python3.5/asyncio/streams.py", line 627, in read
yield from self._wait_for_data('read')
File "/usr/lib/python3.5/asyncio/streams.py", line 457, in _wait_for_data
yield from self._waiter
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/selector_events.py", line 662, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
This exception is non-fatal and the program is able to re-connect despite it - what I want to do is prevent this exception from dumping to stdout and mucking up my text interface.
I tried using ensure_future to handle it but it doesn't seem to work. Am I missing something:
#asyncio.coroutine
def handle_exception():
try:
yield from WebSocketCommonProtocol.run()
except Exception:
print("SocketException-Retrying")
asyncio.ensure_future(handle_exception())
#start discord client
client.run(token)
Task exception was never retrieved - is not actually exception propagated to stdout, but a log message that warns you that you never retrieved exception in one of your tasks. You can find details here.
I guess, most easy way to avoid this message in your case is to retrieve exception from task manually:
coro = WebSocketCommonProtocol.run() # you don't need any wrapper
task = asyncio.ensure_future(coro)
try:
#start discord client
client.run(token)
finally:
# retrieve exception if any:
if task.done() and not task.cancelled():
task.exception() # this doesn't raise anything, just mark exception retrieved
The answer provided by Mikhail is perfectly acceptable, but I realized it wouldn't work for me since the task that is raising the exception is buried deep in some module so trying to retrieve it's exception is kind've difficult. I found that instead if I simply set a custom exception handler for my asyncio loop (loop is created by the discord client):
def exception_handler(loop,context):
print("Caught the following exception")
print(context['message'])
client.loop.set_exception_handler(exception_handler)
client.run(token)
I've created a task that adds a value to session so then i can access it through browser , my celery is running perfectly, here are my codes:
from flask import session, g
from ..extensions import celery
CELERYBEAT_SCHEDULE = {
'add-every-10-seconds': {
'task': 'tasks.ruble',
'schedule': timedelta(seconds=20)
}
}
#celery.task(name='tasks.ruble')
def ruble():
url = 'https://finance.google.com/finance/converter?a=1&from=KGS&to=RUB&meta=ei%3DmSr0WeHCCYvBsAH8n6OIBA'
urlHandler = urllib2.urlopen(url)
html = urlHandler.read()
bsoup = BeautifulSoup(html, 'lxml')
num = bsoup.find('span').text.split()[0]
g.get('ruble', float(num))
session['get_ruble'] = float(num)
but when the times come to execute the task, it ends up with this error:
[2017-10-30 19:19:29,372: ERROR/PoolWorker-4] Task tasks.ruble[ff7617ad-764b-455c-b541-96f3ba78a87b] raised unexpected: RuntimeError('Working outside of request context.\n\nThis typically means that you attempted to use functionality that needed\nan active HTTP request. Consult the documentation on testing for\ninformation about how to avoid this problem.',)
Traceback (most recent call last):
File "/home/xakep/Desktop/work/optomvse/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/xakep/Desktop/work/optomvse/celery_worker.py", line 21, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/home/xakep/Desktop/work/optomvse/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
return self.run(*args, **kwargs)
File "/home/xakep/Desktop/work/optomvse/develop/tasks.py", line 34, in ruble
session['get_ruble'] = float(num)
File "/home/xakep/Desktop/work/optomvse/venv/local/lib/python2.7/site-packages/werkzeug/local.py", line 350, in __setitem__
self._get_current_object()[key] = value
File "/home/xakep/Desktop/work/optomvse/venv/local/lib/python2.7/site-packages/werkzeug/local.py", line 306, in _get_current_object
return self.__local()
File "/home/xakep/Desktop/work/optomvse/venv/local/lib/python2.7/site-packages/flask/globals.py", line 37, in _lookup_req_object
raise RuntimeError(_request_ctx_err_msg)
RuntimeError: Working outside of request context.
This typically means that you attempted to use functionality that needed
an active HTTP request. Consult the documentation on testing for
information about how to avoid this problem.
Another thing i forgot to mention is that i also tried to use g instead of session , by just assigning the float(num) value to it so g.ruble = float(num), but inside my template if i typed {{g.ruble}} nothing get printed out .
If the value keeps changing store it somewhere (Redis/ any database) and when the request from the user comes just update the session data in the API call.
session['get_ruble'] = get_stored_value()
You can not update the session data asynchronously since you don't have the request context in the async task.
Just a suggestion:
If the value is so dynamic instead of storing it in the session, you can provide a separate API to fetch the latest data.
Trying to get the GET parameters from the URL. I have it working in my __init__.py file, but in a different file its not working.
I tried to use with app.app_context(): but I am still getting the same issue.
def log_entry(entity, type, entity_id, data, error):
with app.app_context():
zip_id = request.args.get('id')
RuntimeError: working outside of request context
Any suggestions?
Additional Info:
This is using Flask web framework which is setup as a service (API).
Example URL the user would hit http://website.com/api/endpoint?id=1
As mentioned above using `zip_id = request.args.get('id') works fine in the main file but I am in runners.py (just another file with definitions in)
Full traceback:
Debugging middleware caught exception in streamed response at a point where response headers were already sent.
Traceback (most recent call last):
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/werkzeug/wsgi.py", line 703, in __next__
return self._next()
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/werkzeug/wrappers.py", line 81, in _iter_encoded
for item in iterable:
File "/Users/ereeve/Documents/TechSol/pi-automation-api/automation_api/runners.py", line 341, in create_agencies
log_entry("test", "created", 1, "{'data':'hey'}", "")
File "/Users/ereeve/Documents/TechSol/pi-automation-api/automation_api/runners.py", line 315, in log_entry
zip_id = request.args.get('id')
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/werkzeug/local.py", line 343, in __getattr__
return getattr(self._get_current_object(), name)
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/werkzeug/local.py", line 302, in _get_current_object
return self.__local()
File "/Users/ereeve/.virtualenvs/pi-automation-api/lib/python2.7/site-packages/flask/globals.py", line 20, in _lookup_req_object
raise RuntimeError('working outside of request context')
RuntimeError: working outside of request context
Def in the same file calling the log_entry def
def create_agencies(country_code, DB, session):
document = DB.find_one({'rb_account_id': RB_COUNTRIES_new[country_code]['rb_account_id']})
t2 = new_t2(session)
log_entry("test", "created", 1, "{'data':'hey'}", "")
I'm encountering a circumstance where a demonstrably known hash in db2 of our redis cache dies when being requested with .hgetall(key). I'm hoping for some insight! Thank you.
Right, so... first, a sliver of code:
def from_cache(self, cachekey):
""" pull oft needed material from our persistent redis memory cache, ensuring of course that we have a connection """
try:
log.debug('trying to get \'%s\' from cache' % cachekey)
return self.redis.hgetall(cachekey)
except Exception, e:
self.connect_to_cache()
return self.redis.get(cachekey)
resulting in:
2013-05-21 14:45:26,035 23202 DEBUG trying to get 'fax:1112223333' from cache
2013-05-21 14:45:26,036 23202 DEBUG initializing connection to redis/cache memory localhost, port 6379, db 2...
2013-05-21 14:45:26,039 23202 ERROR stopping with an exception
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/simpledaemon/base.py", line 165, in start
self.run()
File "newgov.py", line 51, in run
if self.ready_for_queue(fax):
File "newgov.py", line 61, in ready_for_queue
if self.too_many_already_queued(fax):
File "newgov.py", line 116, in too_many_already_queued
rules = self.from_cache(key)
File "newgov.py", line 142, in from_cache
return self.redis.get(cachekey)
File "/usr/lib/python2.6/site-packages/redis/client.py", line 588, in get
return self.execute_command('GET', name)
File "/usr/lib/python2.6/site-packages/redis/client.py", line 378, in execute_command
return self.parse_response(connection, command_name, **options)
File "/usr/lib/python2.6/site-packages/redis/client.py", line 388, in parse_response
response = connection.read_response()
File "/usr/lib/python2.6/site-packages/redis/connection.py", line 309, in read_response
raise response
ResponseError: Operation against a key holding the wrong kind of value
And here is what is in redis:
$ redis-cli
redis 127.0.0.1:6379> SELECT 2
OK
redis 127.0.0.1:6379[2]> type fax:1112223333
hash
redis 127.0.0.1:6379[2]> hgetall fax:1112223333
1) "delay"
2) "0"
3) "concurrent"
4) "20"
5) "queued"
6) "20"
7) "exclude"
8) ""
Look at your Python stack trace: it fails on "return self.execute_command('GET', name)". It means that:
the hgetall command failed (probably because the connection was not established before)
an exception was raised and caught in your method
the Redis connection is established (I suppose by calling connect_to_cache())
then you try to run "self.redis.get(cachekey)"
it fails of course because the content of cachekey is the key of a hash (not a string)
(here I imagine you should use hgetall instead)
an other exception is raised - the Redis error is a type error (Operation against a key holding the wrong kind of value)
With redis-cli, try to run "GET fax:1112223333", you will have the same error.
There are different types of data when you set and hset. In hset your value is hash map. And when you try get hash map you have same Error. Just try hgetall and redis will return hashmap, or try hget with cachekey and any key of setted hashmap and enjoy