Develop Reference

Why does celery worker keep trying to connect to amqp even though the broker is sqs?

I tried to configure broker via settings and directly from the celery file .
Settings that apply to celery below.
AWS_SQS_SECRET = os.environ.get("AWS_SQS_SECRET")
broker_url = 'sqs://%s:%s#' % (AWS_SQS_ACCESS, AWS_SQS_SECRET)
task_default_queue = os.environ.get("DEFAULT_QUEUE")
AWS_SQS_REGION = os.environ.get("AWS_REGION")
broker_backend = "SQS"
broker_transport_options = {
"region": AWS_SQS_REGION,
# 'queue_name_prefix': '%s-' % 'dev' , # os.environ.get('ENVIRONMENT', 'development'),
'visibility_timeout': 7200,
'polling_interval': 1,
}
accept_content = ['application/json']
result_serializer = 'json'
task_serializer = 'json'
Also, as I mentioned, I tried to configure directly from the celery file.
import os
from celery import Celery
from celery.schedules import crontab
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'MyApp.settings')
AWS_SQS_ACCESS = os.environ.get("AWS_SQS_ACCESS")
AWS_SQS_SECRET = os.environ.get("AWS_SQS_SECRET")
app = Celery('MyApp') #,, broker='sqs://%s:%s#' % (AWS_SQS_ACCESS, AWS_SQS_SECRET), backend='django-db'
# app.config_from_object('django.conf:settings') #, namespace='CELERY'
CELERY_CONFIG = {
"CELERY_TASK_SERIALIZER": "json",
"CELERY_ACCEPT_CONTENT": ["json"],
"CELERY_RESULT_SERIALIZER": "json",
"CELERY_RESULT_BACKEND": None,
"CELERY_TIMEZONE": "America/Sao_Paulo",
"CELERY_ENABLE_UTC": True,
"CELERY_ENABLE_REMOTE_CONTROL": False,
}
BROKER_URL = 'sqs://%s:%s#' % (AWS_SQS_ACCESS, AWS_SQS_SECRET)
CELERY_CONFIG.update(
**{
"BROKER_URL": BROKER_URL,
"BROKER_TRANSPORT": "sqs",
"BROKER_TRANSPORT_OPTIONS": {
"region": "sa-east-1",
"visibility_timeout": 3600,
"polling_interval": 60,
},
}
)
app.conf.update(**CELERY_CONFIG)
app.autodiscover_tasks()
During deployment on elastik beanstalk , in the service I am running the command:
$PYTHONPATH/celery -A celery worker -Q default-dev -n default-worker \
--logfile=/var/log/celery/celery-stdout-error.log --loglevel=DEBUG --concurrency=1
Tried to run before:
$PYTHONPATH/celery -A MyApp worker -Q default-dev -n default-worker \
--logfile=/var/log/celery/celery-stdout-error.log --loglevel=DEBUG --concurrency=1
But getting error , celery "unable to load app MyApp".
In the log file I get the following error:
[2022-06-10 15:58:25,678: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**#127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 6.00 seconds... (3/100)
My celery version is 5.2.7

If I understand the configuration docs correctly, we're supposed to set config as follows:
app.conf.broker_url = BROKER_URL
# or, alternatively
app.conf.update(broker_url=BROKER_URL)
i.e., use the lowercase names instead of the uppercase names.

If u faced with the same issue , just try to restart ur celery services.

periodic tasks in Celery 4.0

As I know, since Celery 3.1 decorator #periodic_task is depricated.
So I am trying to run an example from celery docs, and can't realise, what am I doing wrong.
I have the following code in task_planner.py:
from celery import Celery
from kombu import Queue, Exchange
class Config(object):
CELERY_QUEUES = (
Queue(
'try',
exchange=Exchange('try'),
routing_key='try',
),
)
celery = Celery('tasks',
backend='redis://',
broker='redis://localhost:6379/0')
celery.config_from_object(Config)
celery.conf.beat_schedule = {
'planner': {
'task': 'some_task',
'schedule': 5.0,
},
}
#celery.task(queue='try')
def some_task():
print('Hooray')
And when I run: celery -A task_planner worker -l info -B, I recieve only the following: [2016-11-27 19:06:56,119: INFO/Beat] Scheduler: Sending due task planner (some_task) every 5 sec.
But I am expecting the output 'Hooray'.
So, what am I missing?

Have found the solution.
I had the task:
#celery.task(queue='try')
def some_task():
print('Hooray')
I printed it's name:
print(some_task)
Got the following:
<#task: task_planner.some_task of tasks:0x7fceaaf5b9e8>
So I just changed the name of the task from some_task to task_planner.some_task here:
celery.conf.beat_schedule = {
'planner': {
'task': 'task_planner.some_task',
'schedule': 5.0,
},
}
And it worked!
[2016-11-29 10:09:57,697: WARNING/PoolWorker-3] Hooray
Note. You should run beat with worker (if task in the same module as beat) and loglevel 'info' in order to see the results:
celery -A task_planner worker -B -l info

Django/Celery multiple queues on localhost - routing not working

I followed celery docs to define 2 queues on my dev machine.
My celery settings:
CELERY_ALWAYS_EAGER = True
CELERY_TASK_RESULT_EXPIRES = 60 # 1 mins
CELERYD_CONCURRENCY = 2
CELERYD_MAX_TASKS_PER_CHILD = 4
CELERYD_PREFETCH_MULTIPLIER = 1
CELERY_CREATE_MISSING_QUEUES = True
CELERY_QUEUES = (
Queue('default', Exchange('default'), routing_key='default'),
Queue('feeds', Exchange('feeds'), routing_key='arena.social.tasks.#'),
)
CELERY_ROUTES = {
'arena.social.tasks.Update': {
'queue': 'fs_feeds',
},
}
i opened two terminal windows, in virtualenv of my project, and ran following commands:
terminal_1$ celery -A arena worker -Q default -B -l debug --purge -n deafult_worker
terminal_2$ celery -A arena worker -Q feeds -B -l debug --purge -n feeds_worker
what i get is that all tasks are being processed by both queues.
My goal is to have one queue to process only the one task defined in CELERY_ROUTES and default queue to process all other tasks.
I also followed this SO question, rabbitmqctl list_queues returns celery 0, and running rabbitmqctl list_bindings returns exchange celery queue celery [] twice. Restarting rabbit server didn't change anything.

Ok, so i figured it out. Following is my whole setup, settings and how to run celery, for those who might be wondering about same thing as my question did.
Settings
CELERY_TIMEZONE = TIME_ZONE
CELERY_ACCEPT_CONTENT = ['json', 'pickle']
CELERYD_CONCURRENCY = 2
CELERYD_MAX_TASKS_PER_CHILD = 4
CELERYD_PREFETCH_MULTIPLIER = 1
# celery queues setup
CELERY_DEFAULT_QUEUE = 'default'
CELERY_DEFAULT_EXCHANGE_TYPE = 'topic'
CELERY_DEFAULT_ROUTING_KEY = 'default'
CELERY_QUEUES = (
Queue('default', Exchange('default'), routing_key='default'),
Queue('feeds', Exchange('feeds'), routing_key='long_tasks'),
)
CELERY_ROUTES = {
'arena.social.tasks.Update': {
'queue': 'feeds',
'routing_key': 'long_tasks',
},
}
How to run celery?
terminal - tab 1:
celery -A proj worker -Q default -l debug -n default_worker
this will start first worker that consumes tasks from default queue. NOTE! -n default_worker is not a must for the first worker, but is a must if you have any other celery instances up and running. Setting -n worker_name is the same as --hostname=default#%h.
terminal - tab 2:
celery -A proj worker -Q feeds -l debug -n feeds_worker
this will start second worker that consumers tasks from feeds queue. Notice -n feeds_worker, if you are running with -l debug (log level = debug), you will see that both workers are syncing between them.
terminal - tab 3:
celery -A proj beat -l debug
this will start the beat, executing tasks according to the schedule in your CELERYBEAT_SCHEDULE.
I didn't have to change the task, or the CELERYBEAT_SCHEDULE.
For example, this is how looks my CELERYBEAT_SCHEDULE for the task that should go to feeds queue:
CELERYBEAT_SCHEDULE = {
...
'update_feeds': {
'task': 'arena.social.tasks.Update',
'schedule': crontab(minute='*/6'),
},
...
}
As you can see, no need for adding 'options': {'routing_key': 'long_tasks'} or specifying to what queue it should go. Also, if you were wondering why Update is upper cased, its because its a custom task, which are defined as sub classes of celery.Task.
Update Celery 5.0+
Celery made a couple changes since version 5, here is an updated setup for routing of tasks.
How to create the queues?
Celery can create the queues automatically. It works perfectly for simple cases, where celery default values for routing are ok.
task_create_missing_queues=True or, if you're using django settings and you're namespacing all celery configs under CELERY_ key, CELERY_TASK_CREATE_MISSING_QUEUES=True. Note, that it is on by default.
Automatic scheduled task routing
After configuring celery app:
celery_app.conf.beat_schedule = {
"some_scheduled_task": {
"task": "module.path.some_task",
"schedule": crontab(minute="*/10"),
"options": {"queue": "queue1"}
}
}
Automatic task routing
Celery app still has to be configured first and then:
app.conf.task_routes = {
"module.path.task2": {"queue": "queue2"},
}
Manual routing of tasks
In case and you want to route the tasks dynamically, then when sending the task specify the queue:
from module import task
def do_work():
# do some work and launch the task
task.apply_async(args=(arg1, arg2), queue="queue3")
More details re routing can be found here:
https://docs.celeryproject.org/en/stable/userguide/routing.html
And regarding calling tasks here:
https://docs.celeryproject.org/en/stable/userguide/calling.html

In addition to accepted answer, if anyone comes here and still wonders why his settings aren't working (as I did just moments ago), here's why: celery documentation isn't listing settings names properly.
For celery 5.0.5 settings CELERY_DEFAULT_QUEUE, CELERY_QUEUES, CELERY_ROUTES should be named CELERY_TASK_DEFAULT_QUEUE, CELERY_TASK_QUEUESand CELERY_TASK_ROUTES instead. These are settings that I've tested, but my guess is the same rule applies for exchange and routing key aswell.

celery scheduling task periodically but not executing them

My django app have two task out of which one is periodic task.
normal task: AddScore
periodic task: CalculateTopScore
class CalculateTopScore(celery.Task):
default_retry_delay = settings.DEFAULT_RETRY_DELAY
max_retries = settings.DEFAULT_MAX_RETRIES
name = 'games.tasks.CalculateTopScore'
def run(self):
try:
# Code to run
except Exception, err:
logger.exception("Error in running task calculate_top_score")
self.retry(exc=err)
return True
def on_failure(self, exc, task_id):
failure = "%s task for calculate_top_score failed permanently." % task_id
logger.error(failure)
def on_success(self):
task_info = 'calculate_top_score task successfully'
logger.info(task_info)
I want to execute this task periodically every 30 minute.
Here are the settings i'm using:
#celery/settings.py
import djcelery
import kombu
from celery.schedules import crontab
from config.celery import exchanges
djcelery.setup_loader()
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
CELERYBEAT_SCHEDULE = {
"calcluate_score": {
"task": "games.tasks.CalculateTopScore",
"schedule": crontab(minute='*/30'),
"args": (),
},
}
CELERY_QUEUES = (
kombu.Queue('add_score',
exchange=exchanges.add_score_exchange,
routing_key='add.scores'),
kombu.Queue('calcluate_score',
exchange=exchanges.calcluate_score_exchange,
routing_key='calculate.scores'),
)
CELERY_ROUTES = ('config.celery.routers.CeleryTaskRouter',)
# Default delay(in seconds) for retrying tasks.
DEFAULT_RETRY_DELAY = 60
# Maximum retry count
DEFAULT_MAX_RETRIES = 6
CELERY_IGNORE_RESULT = True
exchanges.py file
#exchanges.py
from kombu import Exchange
add_score_exchange = Exchange('add_score', type='direct')
calcluate_score_exchange = Exchange('calcluate_score', type='direct')
routes.py file
ROUTES = {
'players.tasks.AddScore': {
'exchange': 'add_score',
'exchange_type': 'direct',
'routing_key': 'add.score',
},
'games.tasks.CalculateTopScore': {
'exchange': 'calculate_score',
'exchange_type': 'direct',
'routing_key': 'calculate.score',
},
}
class CeleryTaskRouter(object):
""" This is a basic celery task router.
"""
def route_for_task(self, task, arg=None, kwargs=None):
return ROUTES.get(task)
On our production server i run celeryd with following arguments: celeryd worker -B
Now in logs i observe that celerybeat schedules the task every 30 minutes but worker doesn't know at all about the task scheduled and hence its not executed.
Why? any config/settings missing? How to execute a class based task periodically?
Please help

You probably have to specify the queues you want celery to get tasks from. Default is to use the celery queue.
Try:
celeryd worker -B -Q add_score,calculate_score

Route celery task to specific queue

I have two separate celeryd processes running on my server, managed by supervisor. They are set to listen on separate queues as such:
[program:celeryd1]
command=/path/to/celeryd --pool=solo --queues=queue1
...
[program:celeryd2]
command=/path/to/celeryd --pool=solo --queues=queue2
...
And my celeryconfig looks something like this:
from celery.schedules import crontab
BROKER_URL = "amqp://guest:guest#localhost:5672//"
CELERY_DISABLE_RATE_LIMITS = True
CELERYD_CONCURRENCY = 1
CELERY_IGNORE_RESULT = True
CELERY_DEFAULT_QUEUE = 'default'
CELERY_QUEUES = {
'default': {
"exchange": "default",
"binding_key": "default",
},
'queue1': {
'exchange': 'queue1',
'routing_key': 'queue1',
},
'queue2': {
'exchange': 'queue2',
'routing_key': 'queue2',
},
}
CELERY_IMPORTS = ('tasks', )
CELERYBEAT_SCHEDULE = {
'first-queue': {
'task': 'tasks.sync',
'schedule': crontab(hour=02, minute=00),
'kwargs': {'client': 'client_1'},
'options': {'queue': 'queue1'},
},
'second-queue': {
'task': 'tasks.sync',
'schedule': crontab(hour=02, minute=00),
'kwargs': {'client': 'client_2'},
'options': {'queue': 'queue1'},
},
}
All tasks.sync tasks must be routed to a specific queue (and therefore celeryd progress). But when I try to run the task manually with sync.apply_async(kwargs={'client': 'value'}, queue='queue1') both celery workers pick up the task. How can I make the task route to the correct queue and only be run by the worker that is bound to the queue?

You are only running one celerybeat instance right?
Maybe you have old queue bindings that clash with this?
Try running rabbitmqctl list_queues and rabbitmqctl list_bindings,
maybe reset the data in the broker to start from scratch.
The example you have here should work, and is working for me when I just tried it.
Tip: Since you are using the same exchange and binding_key value as the queue name,
you don't have to explicitly list them in CELERY_QUEUES. When CELERY_CREATE_MISSING_QUEUES
is on (which it is by default) the queues will be automatically created exactly like you have
if you just do celeryd -Q queue1 or send a task to a queue that is undefined.