Develop Reference

Celery - how to use multiple queues?

I want to create a multiple queues for different tasks. For example emailqueue to sending emails or pipedrive queue to sync tasks with pipedrive API so email does not have to wait until all pipedrives are synced and vice versa.
I'm new in routing and I tried two approaches to create queues but none of them seemes to work.
This is a preferred approach. I tried to define queue inside #task decorator
#task(bind=True, queue='pipedrivequeue')
def backsync_lead(self,lead_id):
settings.py
CELERY_ROUTES = { # tried CELERY_TASK_ROUTES too
'pipedrive.tasks.*': {'queue': 'pipedrivequeue'},
...
}
In both cases, when I run celery worker manually, I see only one default celery queue.
(project) milano#milano-PC:~/PycharmProjects/project$ celery -A project.celery worker -l info
-------------- celery#milano-PC v4.2.2 (windowlicker)
---- **** -----
--- * *** * -- Linux-4.15.0-47-generic-x86_64-with-Ubuntu-18.04-bionic 2019-04-12 17:17:05
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: project:0x7f3b47f66cf8
- ** ---------- .> transport: redis://localhost:6379//
- ** ---------- .> results: redis://localhost/
- *** --- * --- .> concurrency: 12 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. project.apps.apis.pipedrive.tasks.backsync_all_stages
. project.apps.apis.pipedrive.tasks.backsync_lead
As you can see in this line:
-------------- [queues]
.> celery exchange=celery(direct) key=celery
There is probably just one queue. I want to use this queue only for tasks without queue specified.
Do you know where is the problem?
EDIT
(project) milano#milano-PC:~/PycharmProjects/peoject$ celery inspect active_queues
Error: No nodes replied within time constraint.

You need to run a worker with the queue named explicitly, then django will be able to feed into that queue;
celery worker -A project.celery -l info # Default queue worker
celery worker -A project.celery -l info -Q pipedrivequeue # pipedrivequeue worker
celery worker -A project.celery -l info -Q testqueue # testqueue worker

Celery not queuing to a remote broker, adding tasks to a localhost instead

My question is same like this Celery not queuing tasks to broker on remote server, adds tasks to localhost instead, but the answer is not working to me.
My celery.py
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
app = Celery('project', broker='amqp://<user>:<user_pass>#remoteserver:5672/<vhost>', backend='amqp')
# app = Celery('project')
# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
When I run:
$ celery -A worker -l info
I receive the following output:
-------------- celery#paulo-Inspiron-3420 v4.2.1 (windowlicker)
---- **** -----
--- * *** * -- Linux-4.15.0-36-generic-x86_64-with-Ubuntu-18.04-bionic 2018-10-30 13:44:07
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: mycelery:0x7ff88ca043c8
- ** ---------- .> transport: amqp://<user>:**#<remote_ip>:5672/<vhost>
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
I tried stop rabbitmq server and uninstalled it too, but celery keeps queuing to localhost.
Someone can help?

You need to add something like this to your __init__.py file in the same directory as the celery.py file:
from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app
__all__ = ('celery_app',)
Also, make sure you're starting the worker process from inside your project's virtualenv.

Django + Celery + SQS setup. Celery connects to default RabbitMQ via ampq

I am trying to setup Amazon SQS as a default message broker for Celery in Django app. Celery worker is starting but broker is set to default RabbitMQ. Below you can find the output of my worker.
Here are some configs which I have in the project. My celery.py looks like:
from __future__ import absolute_import
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'dance.settings')
app = Celery('dance')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
The essential part of the Django celery settings responsible for setup of broker url is:
BROKER_URL = 'sqs://{}:{}#'.format(AWS_ACCESS_KEY_ID, quote(AWS_SECRET_ACCESS_KEY, safe=''))
BROKER_TRANSPORT_OPTIONS = {
'region': 'eu-west-1',
'polling_interval': 3,
'visibility_timeout': 300,
'queue_name_prefix':'dev-celery-',
}
When I am trying to launch worker within virtual environment with:
celery -A dance worker -l info
I receive following output:
-------------- celery#universe v4.0.0 (latentcall)
---- **** -----
--- * *** * -- Linux-4.8.0-28-generic-x86_64-with-debian-stretch-sid 2016-12-02 14:20:40
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: dance:0x7fdc592ca9e8
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results:
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
...
task1
task2
...
Tasks are listed so I guess Celery gets and processes related Django settings. If to switch in settings from SQS to Redis, I get the same problem.
As I understand from read tutorials worker's output should look similar to.
- ** ---------- .> transport: sqs://*redacted*:**#localhost//
- ** ---------- .> results: djcelery.backends.database:DatabaseBackend
Also I am not using djcelery as far as it is outdated. Instead I am using django_celery_results as it is recommended on Celery setup pages. The last output is just a guess from side project.

The only possible solution which I have found is to explicitly specify broker and database backend.
For me it looks strange, because settings from Django settings.py are not fully loaded or probably I have missed something, otherwise it is bug of Celery.
app = Celery('dance', broker='sqs://', backend='django-db')
Real solution:
Here is why I had problems:
All the Celery variables in Django should start with CELERY so instead of using BROKER_URL and BROKER_TRANSPORT_OPTIONS I had to use CELERY_BROKER_URL and CELERY_BROKER_TRANSPORT_OPTIONS

Incorrect: you need to use CELERY_BROKER_URL when you use CELERY namespace. But some options by default go with CELERY prefix, for example, CELERY_RESULT_BACKEND. If you use CELERY namespace so you need to write CELERY_CELERY_RESULT_BACKEND.

Celery send_task doesn't send tasks

I have a server running Celery with RabbitMQ. But when I try to send tasks using send_task, it just returns with an AsyncResult object.
But the actual task is not running (even though the workers and the queues are empty)
c = Celery("tasks", broker="amqp://guest#127.0.0.1//")
c.send_task("tasks.printing.test_print", (100), queue="print_queue", routing_key="printing.test_print")
My celery configuration is:
CELERY_QUEUES = (
Queue('default', routing_key='task.#'),
Queue('print_queue', routing_key='printing.#'),
)
CELERY_DEFAULT_EXCHANGE = 'tasks'
CELERY_ROUTES = {
'tasks.printing.test_print': {
'queue': 'print_queue',
'routing_key': 'printing.test_print',
}}
BROKER_URL = 'amqp://'
I execute only one worker:
celery -A celerymain worker --loglevel=debug
This is it's initial log:
- ** ---------- [config]
- ** ---------- .> app: __main__:0x7eff96903b50
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: amqp://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues] -------------- .> default exchange=tasks(topic) key=task.#
.> print_queue exchange=tasks(topic) key=printing.#
[tasks] . test_print
This is the task:
class test_print(Task):
name = "test_print"
def run(self,a):
log.info("running")
print a
The rabbitMQ queue 'print_queue' stays empty and there is nothing new in the rabbitMQ logs.
I have 4 GB free space so it's not a disk space problem.
What can be the problem here?

I solved the problem by removing the routing_key parameter from send_task.
I don't really know why this was a problem but at least it works

#app.task(name="test_print")
class test_print(Task):
name = "test_print"
def run(self,a):
log.info("running")
print a

Celery Backend Enabled, Results Say Otherwise

I'll keep it short and to the point:
project directory
proj/__init__.py
/tasks.py
/celery_app.py
celery_app.py
from __future__ import absolute_import
from celery import Celery
app = Celery('proj',
broker='amqp://',
backend='amqp://',
include=['proj.tasks'])
app.conf.update(
CELERY_TASK_RESULT_EXPIRES=3600,
)
if __name__ == '__main__':
app.start()
tasks.py
from __future__ import absolute_import
from celery import current_app
from celery.contrib.methods import task_method
class A:
#current_app.task(filter=task_method)
def add(self,x, y):
return x + y
worker log
-------------- celery#mycomp.localdomain v3.1.17 (Cipater)
---- **** -----
--- * *** * -- Linux-2.6.32-504.8.1.el6.x86_64-x86_64-with-centos-6.6-Final
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: proj:0x1dc12d0
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: amqp://
- *** --- * --- .> concurrency: 24 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
[tasks]
. proj.tasks.add
[2015-04-08 17:45:20,788: INFO/MainProcess] Connected to amqp://guest:**#127.0.0.1:5672//
[2015-04-08 17:45:20,801: INFO/MainProcess] mingle: searching for neighbors
[2015-04-08 17:45:21,812: INFO/MainProcess] mingle: all alone
[2015-04-08 17:45:21,828: WARNING/MainProcess] celery#mycomp.localdomain ready.
[2015-04-08 17:50:25,610: INFO/MainProcess] Received task: proj.tasks.add[e0020f67-dbe7-4f6d-9547-a8ace36c2a2c]
[2015-04-08 17:50:25,635: INFO/MainProcess] Task proj.tasks.add[e0020f67-dbe7-4f6d-9547-a8ace36c2a2c] succeeded in 0.023062946042s: 4
python shell
>>> from proj.tasks import A
>>> a = A()
>>> s = a.add.delay(2,2)
>>> s
<AsyncResult: e0020f67-dbe7-4f6d-9547-a8ace36c2a2c>
>>> s.backend
<celery.backends.base.DisabledBackend object at 0x113fdd0>
As you can see, I have a backend enabled. I'm using amqp. However, when I try and get the result, it's saying I dont have an enabled backend.

By including the line from proj.celery_app import app in tasks.py, the backend started to work.
This seems like a bug, since current_app should contain that backend instance.
I opened an issue on the celery github. Hopefully this helps anyone who encounters this problem as well.
Link to the github issue