Develop Reference

How to assign multiple queues for multiple tasks in celery?

I've a python big data project in which i'm trying to do my tasks using celery and Redis server.
The thing is i need three different queue's for three different tasks which i'm applying to celery to do.
This is the configuration i've made to run the three tasks at once but they are using one single queue to perform the task one after the other so it's taking so much time.
from celery import Celery
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
app = Celery("tasks", broker="redis://localhost:6379")
#app.task()
def main(capital,gamma,alpha):
#app.task()
def gain(capital,gamma,alpha):
#app.task()
def lain(capital,gamma,alpha):
And to start the celery app i'm using this lines of code
("celery -A task worker --loglevel=info -P eventlet --concurrency=10 -n worker1#%h" , shell=True)
("celery -A task worker --loglevel=info -P eventlet --concurrency=10 -n worker2#%h" , shell=True)
("celery -A task worker --loglevel=info -P eventlet --concurrency=10 -n worker3#%h" , shell=True)
These code perfectly runs my app with three tasks but i need to make three independent queue's for each of these tasks so that all the three tasks can run concurrently or parallelly at once.
This is how my celery app looks like with three tasks but it's using only one queue which is default queue named celery.
-------------- worker1#EC2AMAZ-RTM8UD8 v5.2.3 (dawn-chorus)
--- ***** -----
-- ******* ---- Windows-10-10.0.20348-SP0 2022-03-10 12:59:07
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: tasks:0x248adf81e80
- ** ---------- .> transport: redis://localhost:6379//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 10 (eventlet)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. task.gain
. task.lain
. task.main
[2022-03-10 12:59:07,461: INFO/MainProcess] Connected to redis://localhost:6379//
[2022-03-10 12:59:07,477: INFO/MainProcess] mingle: searching for neighbors
[2022-03-10 12:59:08,493: INFO/MainProcess] mingle: all alone
[2022-03-10 12:59:08,493: INFO/MainProcess] pidbox: Connected to redis://localhost:6379//.
[2022-03-10 12:59:08,493: INFO/MainProcess] worker1#EC2AMAZ-RTM8UD8 ready.
So please can anyone help me defining three different queue's for three different tasks in celery app and any help would be very much appreciated :)

It should be as simple as adding -Q <queue name> to those three Celery workers that you run.

how to see the celery messages in redis?

I have a celery worker running redis as a broker.
Starting the worker processes gives me this:
celery -A celeryworker worker --loglevel=INFO
-------------- celery#cd38f5e26c28 v5.2.1 (dawn-chorus)
--- ***** -----
-- ******* ---- Linux-5.10.25-linuxkit-x86_64-with-glibc2.28 2021-12-14 00:22:02
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: myapp:0x7f96dd51af10
- ** ---------- .> transport: redis://redis-container:6379/1
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 6 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> 0 exchange=0(direct) key=0
[tasks]
. app.tasks.bye
. app.tasks.printme
[2021-12-14 00:22:02,708: INFO/MainProcess] Connected to redis://redis-container:6379/1
[2021-12-14 00:22:02,717: INFO/MainProcess] mingle: searching for neighbors
[2021-12-14 00:22:03,740: INFO/MainProcess] mingle: all alone
[2021-12-14 00:22:03,762: INFO/MainProcess] celery#cd38f5e26c28 ready.
[2021-12-14 00:22:23,332: INFO/MainProcess] Task app.task.bye[7e28e6a0-8aaa-4609-bd85-9312e91cb355] received
[2021-12-14 00:23:23,326: INFO/ForkPoolWorker-3] Task app.tasks.bye[7e28e6a0-8aaa-4609-bd85-9312e91cb355] succeeded in 60.061842500006605s: 'the text was byebye!!'
This is what I can see in redis right after starting the celery workers:
127.0.0.1:6379[1]> keys *
1) "_kombu.binding.0"
2) "_kombu.binding.celery.pidbox"
3) "_kombu.binding.celeryev"
Even if I set a long timer on my tasks (sleep(60)) the tasks will take 60 seconds to run, but I still don't see anything in my redis container.
mget <key> returns nil for all keys above.
I was expecting to see messages incoming in form of ID or something into Redis (I can see messages if I use SQS as broker, but not for redis).

Your messages are picked immediately by your worker.
To actually see where redis stores them, stop your worker process and then publish it (You can execute task.delay(*args, **kwargs) from python shell).
You'll find your messages under celery key in your redis.
Celery Keys in redis
Note: Check your redis broker url and which logical database it is using

Received(queue) or Executing multiple tasks at same time in celery

I added 10 tasks in celery process. It picks one by one (default celery process). After the completion of first task it picks the second task received and executed .
Those tasks are not depending on each other. so I want to run simultaneously or I want to received all the 10 tasks in queue(celery console) and executing one by one.
-------------- celery#rana-04 v5.1.2 (sun-harmonics)
--- ***** -----
-- ******* ---- Windows-10-10.0.18362-SP0 2021-08-16 13:18:28
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: rana:0x2507baac198
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results:
- *** --- * --- .> concurrency: 8 (solo)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
Celery Command :
celery -A rana worker --loglevel=INFO --without-gossip --without-mingle --without-heartbeat -Ofair --pool=solo
Any command changes or configurations needed ? Thanks in advance

I think you should not use --pool=solo in your command. Because the pool of worker just can hold one task at the same time.

Celery: Which are the queues consumed if the -Q option is not specified

According to the Celery documentation, the -Q/--queues command line option can be used for:
-Q, --queues
List of queues to enable for this worker, separated by comma. By default all configured queues are enabled. Example: -Q video,image
However I don't understand what does it mean with configured queues here. Does this mean all queues known to Celery, including the default one? Or only the ones defined in the task_queues config option? Does the task_create_missing_queues option affect this?

If you haven't configured anything, it will consume from celery queue and you can as you can see from logs
celery worker -A t
-------------- celery#pavilion v4.0.2 (latentcall)
---- **** -----
--- * *** * -- Linux-4.4.0-79-generic-x86_64-with-Ubuntu-16.04-xenial 2017-06-09 10:39:14
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: tasks:0x7f15cf9cdfd0
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: rpc://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
You can also configure celery to consume a set of queues by default like this
from celery import Celery
from kombu import Queue
app = Celery(broker='amqp://guest#localhost//', backend='rpc')
app.conf.task_queues = (Queue('foo'), Queue('bar'))
Now all workers will consume foo, bar queues by default.
-------------- [queues]
.> bar exchange=celery(direct) key=celery
.> foo exchange=celery(direct) key=celery

I was facing an issue where when I would execute
$ celery -A myCeleryConfig worker -Q myQueue2
I would get error
celery.exceptions.ImproperlyConfigured: Trying to select queue subset of ['myQueue2'], but queue 'myQueue2' isn't defined in the `task_queues` setting.
The documentation for the task_queues setting was unclear to me. It does state
If you really want to configure advanced routing, this setting should be a list of kombu.Queue objects the worker will consume from.
I wasn't sure what it meant by this. No code examples are provided in the documentation. But, thanks to #Chillar's response, I came to find that configuring
app.conf.task_queues = (Queue('myQueue1'), Queue('myQueue2'))
solved the issue. I now see
-------------- [queues]
.> myQueue1 exchange=myQueue1 key=myQueue1
.> myQueue2 exchange=myQueue2 key=myQueue2
when I start the worker, indicating the queue is now registered.

Celery send_task doesn't send tasks

I have a server running Celery with RabbitMQ. But when I try to send tasks using send_task, it just returns with an AsyncResult object.
But the actual task is not running (even though the workers and the queues are empty)
c = Celery("tasks", broker="amqp://guest#127.0.0.1//")
c.send_task("tasks.printing.test_print", (100), queue="print_queue", routing_key="printing.test_print")
My celery configuration is:
CELERY_QUEUES = (
Queue('default', routing_key='task.#'),
Queue('print_queue', routing_key='printing.#'),
)
CELERY_DEFAULT_EXCHANGE = 'tasks'
CELERY_ROUTES = {
'tasks.printing.test_print': {
'queue': 'print_queue',
'routing_key': 'printing.test_print',
}}
BROKER_URL = 'amqp://'
I execute only one worker:
celery -A celerymain worker --loglevel=debug
This is it's initial log:
- ** ---------- [config]
- ** ---------- .> app: __main__:0x7eff96903b50
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: amqp://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues] -------------- .> default exchange=tasks(topic) key=task.#
.> print_queue exchange=tasks(topic) key=printing.#
[tasks] . test_print
This is the task:
class test_print(Task):
name = "test_print"
def run(self,a):
log.info("running")
print a
The rabbitMQ queue 'print_queue' stays empty and there is nothing new in the rabbitMQ logs.
I have 4 GB free space so it's not a disk space problem.
What can be the problem here?

I solved the problem by removing the routing_key parameter from send_task.
I don't really know why this was a problem but at least it works

#app.task(name="test_print")
class test_print(Task):
name = "test_print"
def run(self,a):
log.info("running")
print a