I want to implement task priority in my celery workers. I can do this by creating different queues for high priority tasks and low priority tasks. But also I need to send broadcast tasks to all workers with a broadcast queue and its not working. Here is tasks.py file:
from celery import Celery
from kombu.common import Broadcast, Queue, Exchange
app = Celery('tasks')
app.conf.update(
CELERY_RESULT_BACKEND='amqp',
CELERY_ACCEPT_CONTENT=['json'],
CELERY_TASK_SERIALIZER='json',
CELERY_RESULT_SERIALIZER='json',
BROKER_URL='amqp://',
CELERY_QUEUES=(Queue('default',
Exchange('default'),
routing_key='default'),
Queue('low_priority',
Exchange('low_priority'),
routing_key='low_priority'),
Broadcast('broadcast_tasks'), ),
CELERY_ROUTES={'tasks.broadcast':
{'queue': 'broadcast_tasks'},
'tasks.low_task':
{'queue': 'low_priority'},
},
CELERY_DEFAULT_QUEUE = 'default',
CELERY_DEFAULT_EXCHANGE = 'default',
CELERY_DEFAULT_ROUTING_KEY = 'default'
)
#app.task
def broadcast():
print "Broadcast called"
#app.task
def low_task():
print "Low priority called"
#app.task
def def_task():
print "Default called"
When I run celery workers with this command:
celery -A tasks -Q default worker --loglevel=info
celery -A tasks -Q default,low_priority worker --loglevel=info
Task priority works but broadcast tasks are not acknowledged.
When I run the same command without a queue argument, broadcast works but task priority does not:
celery -A tasks worker --loglevel=info
celery -A tasks worker --loglevel=info
As I understand, it happens because broadcast queues have unique names, like bcast.0b5dbce0-9bcb-48a5-8554-cbb7f32a6703 for each worker.
Does anyone have a good workaround? Thanks in advance!
You must explicitly consume a broadcast queue, so modify your commandline invocation as the comment above per ANDY_VAR.
A similar question was asked here:
start celery worker and enable it for broadcast queue
Related
I have a python file called tasks.py in which I am defining 4 single tasks. I would like to configure celery in order to use 4 queues because each queue would have a different number of workers assigned. I was reading I should use route_task property but I tried several options and not a success.
I was following this doc celery route_tasks docs
My goal would be run 4 workers, one for each task, and don't mix tasks from different workers in different queues. It's possible? It's a good approach?
If I am doing something wrong I would be happy to change my code to make it work
Here is my config so far
tasks.py
app = Celery('tasks', broker='pyamqp://guest#localhost//')
app.conf.task_default_queue = 'default'
app.conf.task_queues = (
Queue('queueA', routing_key='tasks.task_1'),
Queue('queueB', routing_key='tasks.task_2'),
Queue('queueC', routing_key='tasks.task_3'),
Queue('queueD', routing_key='tasks.task_4')
)
#app.task
def task_1():
print "Task of level 1"
#app.task
def task_2():
print "Task of level 2"
#app.task
def task_3():
print "Task of level 3"
#app.task
def task_4():
print "Task of level 4"
Run celery one worker for each queue
celery -A tasks worker --loglevel=debug -Q queueA --logfile=celery-A.log -n W1&
celery -A tasks worker --loglevel=debug -Q queueB --logfile=celery-B.log -n W2&
celery -A tasks worker --loglevel=debug -Q queueC --logfile=celery-C.log -n W3&
celery -A tasks worker --loglevel=debug -Q queueD --logfile=celery-D.log -n W4&
There is no need to get into complex routing for submitting tasks into different queues. Define your tasks as usual.
from celery import celery
app = Celery('tasks', broker='pyamqp://guest#localhost//')
#app.task
def task_1():
print "Task of level 1"
#app.task
def task_2():
print "Task of level 2"
Now while queuing the tasks, put the tasks in proper queue. Here is an example on how to do it.
In [12]: from tasks import *
In [14]: result = task_1.apply_async(queue='queueA')
In [15]: result = task_2.apply_async(queue='queueB')
This will put the task_1 in queue named queueA and task_2 in queueB.
Now you can just start your workers to consume them.
celery -A tasks worker --loglevel=debug -Q queueA --logfile=celery-A.log -n W1&
celery -A tasks worker --loglevel=debug -Q queueB --logfile=celery-B.log -n W2&
Note: task and message are used interchangeably in the answer. It is basically a payload that the producer sends to RabbitMQ
You can either follow approach suggested by Chillar or you can define and use the task_routes configuration to route the messages to appropriate queue. This way you don't need to specify queue name every time you call apply_async.
Example: Route task1 to QueueA and route task2 to QueueB
app = Celery('my_app')
app.conf.update(
task_routes={
'task1': {'queue': 'QueueA'},
'task2': {'queue': 'QueueB'}
}
)
Sending a task to multiple queue is a bit tricky. You will have to declare an exchange, and then route your task with appropriate routing_key. You can get more information about type of exchange here. Let's go with direct for purpose of illustration.
Create Exchange
from kombu import Exchange, Queue, binding
exchange_for_queueA_and_B = Exchange('exchange_for_queueA_and_B', type='direct')
Create bindings on Queues to that exchange
app.conf.update(
task_queues=(
Queue('QueueA', [
binding(exchange_for_queueA_and_B, routing_key='queue_a_and_b')
]),
Queue('QueueB', [
binding(exchange_for_queueA_and_B, routing_key='queue_a_and_b')
])
)
)
Define the task_route to send task1 to the exchange
app.conf.update(
task_routes={
'task1': {'exchange': 'exchange_for_queueA_and_B', 'routing_key': 'queue_a_and_b'}
}
)
You can also declare these options of exchange and routing_key in your apply_async method as suggested by Chillar in the above answer.
After that, you can define your workers on same machine or different machines, to consume from those queues.
celery -A my_app worker -n consume_from_QueueA_and_QueueB -Q QueueA,QueueB
celery -A my_app worker -n consume_from_QueueA_only -Q QueueA
I'd like to apply queue settings to chained tasks. Here's sample tasks.
from celery import chain
from task.calc import *
def execute(some, some2):
return chain(add_1.s(some, some2, 10), add_2.s).apply_async()
Here's celeryconfig specify queue settings.
class MyRouter():
def route_for_task(self, task, args=None, kwargs=None):
if 'task.calc' in task:
return {
'queue': 'calc',
'exchange': 'calc',
'exchange_type': 'topic',
'routing_key': 'direct'
}
CELERY_QUEUES = (
Queue('calc', Exchange('calc', type='topic'), routing_key='calc'),
)
CELERY_ROUTES = (
MyRouter()
)
When I runs celery worker with below command, chained task is executed properly.
$ celery worker -A task -P eventlet -Q calc
But when I runs celery worker with concurrent workers, chained task is not executed, but executed just first task(task.calc.add_1).
$ celery worker -A task -P eventlet -Q calc -c 32
Is there any tips to use celery workers with concurrent mode and queue.
If I have a celery task such as the following:
#celery.task(name='tasks.webrequest')
def webrequest(*args):
try:
webrequest = requests.get('{0}'.format(args[0]), auth=(args[1], args[2]), verify=False, timeout=240)
except Exception as e:
print e
webrequest='cant talk to server'
return webrequest
and a celery worker with only one core, so only 1 worker thread. Is there a way and how would you have that worker perform two or more of these tasks at once?
Currently I am executing the working like so:
celery -A app.celery worker -l DEBUG
When I call it with the concurrency (thanks Ale) it allows me to have more threads than cpu's.
celery -A app.celery worker -c 30 -l DEBUG
I have one queue with several task types and I need to run worker for specific task.
Something like: 'celery worker --routing_key task.type1 --app=app'
Queue configuration:
CELERY_QUEUES = (
Queue('myqueue', routing_key='task.#'),
)
CELERY_DEFAULT_EXCHANGE_TYPE = 'topic'
Using pika task is easy to solve: http://www.rabbitmq.com/tutorials/tutorial-five-python.html but how to do it with celery?
Np, you can't bind a worker to a routing_key.
Workers consume queues not routing_key.
Producers send messages with a routing_key, that rabbitmq route to queues.
It's not possible with pika also.
In the tutorial the worker/consumer binds its own queue to the routing key.
The producer emit logs with routing key = 'info'
RabbitMQ will discard all of them until a queue is bound to this routing_key
The receiver create a Queue A and bind it to routing_key 'info'
Now rabbitmq routes logs with routing_key 'info' to Queue A, and the receiver consume them
You can reproduce this binding easily with celery.
In exemple you can do it in the celery configuration file:
exchange = Exchange('default', type=topic)
CELERY_QUEUES = (
Queue('all_logs', exchange, routing_key='logs.#'),
Queue('info_logs', exchange, routing_key='logs.info')
)
receive all logs:
$ celery worker -A receive_logs -Q all_logs
receive only 'info' logs (msg with routing_key=logs.info only)
$ celery worker -A receive_logs -Q info_logs
In the end you have started a worker that consume only msg with a specific routing_key, which is what you want.
note: info logs are duplicated in both Queue:all_logs and Queue:info_logs
You might be interested by:
http://docs.celeryproject.org/en/latest/configuration.html?highlight=direct#std:setting-CELERY_WORKER_DIRECT
I'm working with Celery http://celery.readthedocs.org/en/latest/index.html
I need to run a periodic tasks at a specific moment. But I only want to start my task after starting the celery worker.
For that I'm trying to create my own "PeriodicTask". But I'm dealing with a problem.
When I'm starting the worker and executing the run_tasks.py in another terminal, it seems that my periodic tasks is executed only one time.
How could I do to have my periodic task running every 3 seconds.
Here is a part of the code.
Start celery :
celery worker --app=worker_manager.celery --loglevel=info
file tasks.py
class MyPeriodicTask(PeriodicTask):
name = "periodic-task"
run_every = timedelta(seconds=3)
def run(self, **kwargs):
logger = self.get_logger(**kwargs)
logger.info("Running periodic task!")
file run_tasks.py
tasks.register(MyPeriodicTask)
wmi_collector_task = worker_app.tasks[MyPeriodicTask.name]
Thanks in advance.
To run periodic tasks you need to start celery beat. You can do this by passing -B argument when starting workers:
celery worker -B --app=worker_manager.celery --loglevel=info