I have a python file called tasks.py in which I am defining 4 single tasks. I would like to configure celery in order to use 4 queues because each queue would have a different number of workers assigned. I was reading I should use route_task property but I tried several options and not a success.
I was following this doc celery route_tasks docs
My goal would be run 4 workers, one for each task, and don't mix tasks from different workers in different queues. It's possible? It's a good approach?
If I am doing something wrong I would be happy to change my code to make it work
Here is my config so far
tasks.py
app = Celery('tasks', broker='pyamqp://guest#localhost//')
app.conf.task_default_queue = 'default'
app.conf.task_queues = (
Queue('queueA', routing_key='tasks.task_1'),
Queue('queueB', routing_key='tasks.task_2'),
Queue('queueC', routing_key='tasks.task_3'),
Queue('queueD', routing_key='tasks.task_4')
)
#app.task
def task_1():
print "Task of level 1"
#app.task
def task_2():
print "Task of level 2"
#app.task
def task_3():
print "Task of level 3"
#app.task
def task_4():
print "Task of level 4"
Run celery one worker for each queue
celery -A tasks worker --loglevel=debug -Q queueA --logfile=celery-A.log -n W1&
celery -A tasks worker --loglevel=debug -Q queueB --logfile=celery-B.log -n W2&
celery -A tasks worker --loglevel=debug -Q queueC --logfile=celery-C.log -n W3&
celery -A tasks worker --loglevel=debug -Q queueD --logfile=celery-D.log -n W4&
There is no need to get into complex routing for submitting tasks into different queues. Define your tasks as usual.
from celery import celery
app = Celery('tasks', broker='pyamqp://guest#localhost//')
#app.task
def task_1():
print "Task of level 1"
#app.task
def task_2():
print "Task of level 2"
Now while queuing the tasks, put the tasks in proper queue. Here is an example on how to do it.
In [12]: from tasks import *
In [14]: result = task_1.apply_async(queue='queueA')
In [15]: result = task_2.apply_async(queue='queueB')
This will put the task_1 in queue named queueA and task_2 in queueB.
Now you can just start your workers to consume them.
celery -A tasks worker --loglevel=debug -Q queueA --logfile=celery-A.log -n W1&
celery -A tasks worker --loglevel=debug -Q queueB --logfile=celery-B.log -n W2&
Note: task and message are used interchangeably in the answer. It is basically a payload that the producer sends to RabbitMQ
You can either follow approach suggested by Chillar or you can define and use the task_routes configuration to route the messages to appropriate queue. This way you don't need to specify queue name every time you call apply_async.
Example: Route task1 to QueueA and route task2 to QueueB
app = Celery('my_app')
app.conf.update(
task_routes={
'task1': {'queue': 'QueueA'},
'task2': {'queue': 'QueueB'}
}
)
Sending a task to multiple queue is a bit tricky. You will have to declare an exchange, and then route your task with appropriate routing_key. You can get more information about type of exchange here. Let's go with direct for purpose of illustration.
Create Exchange
from kombu import Exchange, Queue, binding
exchange_for_queueA_and_B = Exchange('exchange_for_queueA_and_B', type='direct')
Create bindings on Queues to that exchange
app.conf.update(
task_queues=(
Queue('QueueA', [
binding(exchange_for_queueA_and_B, routing_key='queue_a_and_b')
]),
Queue('QueueB', [
binding(exchange_for_queueA_and_B, routing_key='queue_a_and_b')
])
)
)
Define the task_route to send task1 to the exchange
app.conf.update(
task_routes={
'task1': {'exchange': 'exchange_for_queueA_and_B', 'routing_key': 'queue_a_and_b'}
}
)
You can also declare these options of exchange and routing_key in your apply_async method as suggested by Chillar in the above answer.
After that, you can define your workers on same machine or different machines, to consume from those queues.
celery -A my_app worker -n consume_from_QueueA_and_QueueB -Q QueueA,QueueB
celery -A my_app worker -n consume_from_QueueA_only -Q QueueA
Related
I am currently running celery 4.0.2 with a single worker like this:
celery.py:
app = Celery('project',
broker='amqp://jimmy:jimmy123#localhost/jimmy_vhost',
backend='rpc://',
include=['project.tasks'])
if __name__ == '__main__':
app.start()
app.name
tasks.py:
from .celery import app
from celery.schedules import schedule
from time import sleep, strftime
app.conf.beat_schedule = {
'planner_1': {
'task': 'project.tasks.call_orders',
'schedule': 1800,
},
'planner_2': {
'task': 'project.tasks.call_inventory',
'schedule': 900,
},
}
I used the following command to run with beat:
celery -A project worker -l info --concurrency=3 --beat -E
Right now it is only a single queue with only one worker running.
My question is how to run celery with multiple workers and single queue so that tasks are executed in parallel using multiprocessing without duplication?
I looked up on the internet, how to run celery with multiprocessing. According to this article:
celery worker -l info -P processes -c 16
will result in a single message consumer delegating work to
16 OS-level pool processes. Each OS-level process can be assigned to different CPU in a multicore environment, and as such it will process tasks
in parallel, but it will not consume messages in parallel.
can using the -p processes argument solve my problem? Also but what is meant by, "it will process tasks in parallel, but it will not consume messages in parallel"?
I'd like to apply queue settings to chained tasks. Here's sample tasks.
from celery import chain
from task.calc import *
def execute(some, some2):
return chain(add_1.s(some, some2, 10), add_2.s).apply_async()
Here's celeryconfig specify queue settings.
class MyRouter():
def route_for_task(self, task, args=None, kwargs=None):
if 'task.calc' in task:
return {
'queue': 'calc',
'exchange': 'calc',
'exchange_type': 'topic',
'routing_key': 'direct'
}
CELERY_QUEUES = (
Queue('calc', Exchange('calc', type='topic'), routing_key='calc'),
)
CELERY_ROUTES = (
MyRouter()
)
When I runs celery worker with below command, chained task is executed properly.
$ celery worker -A task -P eventlet -Q calc
But when I runs celery worker with concurrent workers, chained task is not executed, but executed just first task(task.calc.add_1).
$ celery worker -A task -P eventlet -Q calc -c 32
Is there any tips to use celery workers with concurrent mode and queue.
I want to implement task priority in my celery workers. I can do this by creating different queues for high priority tasks and low priority tasks. But also I need to send broadcast tasks to all workers with a broadcast queue and its not working. Here is tasks.py file:
from celery import Celery
from kombu.common import Broadcast, Queue, Exchange
app = Celery('tasks')
app.conf.update(
CELERY_RESULT_BACKEND='amqp',
CELERY_ACCEPT_CONTENT=['json'],
CELERY_TASK_SERIALIZER='json',
CELERY_RESULT_SERIALIZER='json',
BROKER_URL='amqp://',
CELERY_QUEUES=(Queue('default',
Exchange('default'),
routing_key='default'),
Queue('low_priority',
Exchange('low_priority'),
routing_key='low_priority'),
Broadcast('broadcast_tasks'), ),
CELERY_ROUTES={'tasks.broadcast':
{'queue': 'broadcast_tasks'},
'tasks.low_task':
{'queue': 'low_priority'},
},
CELERY_DEFAULT_QUEUE = 'default',
CELERY_DEFAULT_EXCHANGE = 'default',
CELERY_DEFAULT_ROUTING_KEY = 'default'
)
#app.task
def broadcast():
print "Broadcast called"
#app.task
def low_task():
print "Low priority called"
#app.task
def def_task():
print "Default called"
When I run celery workers with this command:
celery -A tasks -Q default worker --loglevel=info
celery -A tasks -Q default,low_priority worker --loglevel=info
Task priority works but broadcast tasks are not acknowledged.
When I run the same command without a queue argument, broadcast works but task priority does not:
celery -A tasks worker --loglevel=info
celery -A tasks worker --loglevel=info
As I understand, it happens because broadcast queues have unique names, like bcast.0b5dbce0-9bcb-48a5-8554-cbb7f32a6703 for each worker.
Does anyone have a good workaround? Thanks in advance!
You must explicitly consume a broadcast queue, so modify your commandline invocation as the comment above per ANDY_VAR.
A similar question was asked here:
start celery worker and enable it for broadcast queue
If I have a celery task such as the following:
#celery.task(name='tasks.webrequest')
def webrequest(*args):
try:
webrequest = requests.get('{0}'.format(args[0]), auth=(args[1], args[2]), verify=False, timeout=240)
except Exception as e:
print e
webrequest='cant talk to server'
return webrequest
and a celery worker with only one core, so only 1 worker thread. Is there a way and how would you have that worker perform two or more of these tasks at once?
Currently I am executing the working like so:
celery -A app.celery worker -l DEBUG
When I call it with the concurrency (thanks Ale) it allows me to have more threads than cpu's.
celery -A app.celery worker -c 30 -l DEBUG
I'm working with Celery http://celery.readthedocs.org/en/latest/index.html
I need to run a periodic tasks at a specific moment. But I only want to start my task after starting the celery worker.
For that I'm trying to create my own "PeriodicTask". But I'm dealing with a problem.
When I'm starting the worker and executing the run_tasks.py in another terminal, it seems that my periodic tasks is executed only one time.
How could I do to have my periodic task running every 3 seconds.
Here is a part of the code.
Start celery :
celery worker --app=worker_manager.celery --loglevel=info
file tasks.py
class MyPeriodicTask(PeriodicTask):
name = "periodic-task"
run_every = timedelta(seconds=3)
def run(self, **kwargs):
logger = self.get_logger(**kwargs)
logger.info("Running periodic task!")
file run_tasks.py
tasks.register(MyPeriodicTask)
wmi_collector_task = worker_app.tasks[MyPeriodicTask.name]
Thanks in advance.
To run periodic tasks you need to start celery beat. You can do this by passing -B argument when starting workers:
celery worker -B --app=worker_manager.celery --loglevel=info