I have one queue with several task types and I need to run worker for specific task.
Something like: 'celery worker --routing_key task.type1 --app=app'
Queue configuration:
CELERY_QUEUES = (
Queue('myqueue', routing_key='task.#'),
)
CELERY_DEFAULT_EXCHANGE_TYPE = 'topic'
Using pika task is easy to solve: http://www.rabbitmq.com/tutorials/tutorial-five-python.html but how to do it with celery?
Np, you can't bind a worker to a routing_key.
Workers consume queues not routing_key.
Producers send messages with a routing_key, that rabbitmq route to queues.
It's not possible with pika also.
In the tutorial the worker/consumer binds its own queue to the routing key.
The producer emit logs with routing key = 'info'
RabbitMQ will discard all of them until a queue is bound to this routing_key
The receiver create a Queue A and bind it to routing_key 'info'
Now rabbitmq routes logs with routing_key 'info' to Queue A, and the receiver consume them
You can reproduce this binding easily with celery.
In exemple you can do it in the celery configuration file:
exchange = Exchange('default', type=topic)
CELERY_QUEUES = (
Queue('all_logs', exchange, routing_key='logs.#'),
Queue('info_logs', exchange, routing_key='logs.info')
)
receive all logs:
$ celery worker -A receive_logs -Q all_logs
receive only 'info' logs (msg with routing_key=logs.info only)
$ celery worker -A receive_logs -Q info_logs
In the end you have started a worker that consume only msg with a specific routing_key, which is what you want.
note: info logs are duplicated in both Queue:all_logs and Queue:info_logs
You might be interested by:
http://docs.celeryproject.org/en/latest/configuration.html?highlight=direct#std:setting-CELERY_WORKER_DIRECT
Related
Problem Statement
After booting the GUnicorn worker processes, I want the worker processes still be able to receive data from another process. Currently, I'm trying to use multiprocessing.Queue to achieve this. Specifically, I start a data management process before forking the workers and use two queues to connect it with the workers. One queue is for the workers to request data from the data management process, the other to receive the data. In the post_fork hook, a worker sends out a request to the request queue and receives a response on the response queue, and only then proceeds to serving the application.
This works fine at first. However, when I manually terminate the workers and gunicorn restarts it, it will get stuck in the post_fork method and never receive a response from the data management process.
Minimal Example
The following code shows a minimal example (config.py):
import logging
import os
import multiprocessing
logging.basicConfig(level=logging.INFO)
bind = "localhost:8080"
workers = 1
def s(req_q: multiprocessing.Queue, resp_q: multiprocessing.Queue):
while True:
logging.info("Waiting for messages")
other_pid = req_q.get()
logging.info("Got a message from %d", other_pid)
resp_q.put(os.getpid())
m = multiprocessing.Manager()
q1 = m.Queue()
q2 = m.Queue()
proc = multiprocessing.Process(target=s, args=(q1, q2), daemon=True)
proc.start()
def post_fork(server, worker):
logging.info("Sending request")
q1.put(os.getpid())
logging.info("Request sent")
other_pid = q2.get()
logging.info("Got response from %d", other_pid)
My application module (app.py) is:
from flask import Flask
app = Flask(__name__)
And I start the server via
$ gunicorn -c config.py app:app
INFO:root:Waiting for messages
[2023-01-31 14:20:46 +0800] [24553] [INFO] Starting gunicorn 20.1.0
[2023-01-31 14:20:46 +0800] [24553] [INFO] Listening at: http://127.0.0.1:8080 (24553)
[2023-01-31 14:20:46 +0800] [24553] [INFO] Using worker: sync
[2023-01-31 14:20:46 +0800] [24580] [INFO] Booting worker with pid: 24580
INFO:root:Sending request
INFO:root:Request sent
INFO:root:Got a message from 24580
INFO:root:Waiting for messages
INFO:root:Got response from 24574
The log shows that the messages were successfully exchanged. Now, we'll stop the worker process and let gunicorn restart it:
$ kill 24580
[2023-01-31 14:22:40 +0800] [24580] [INFO] Worker exiting (pid: 24580)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/util.py", line 319, in _exit_function
p.join()
File "/usr/lib/python3.6/multiprocessing/process.py", line 122, in join
assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process
[2023-01-31 14:22:40 +0800] [24553] [WARNING] Worker with pid 24574 was terminated due to signal 15
[2023-01-31 14:22:40 +0800] [29497] [INFO] Booting worker with pid: 29497
INFO:root:Sending request
INFO:root:Request sent
Question
Why doesn't s receive the message from the worker after re-starting?
Besides, why am I getting this 'can only join a child process' error thrown? Does it has something to do with the problem?
Environment
Python: 3.8.0
GUnicorn: 20.1.0
OS: Ubuntu 18.04
Related Questions
In this question, a similar problem is presented, and the solution was to use "multiprocessing.manager.queue". However, this didn't solved the issue in my case.
Side Note
I already considered the following alternative designs:
Use HTTP/gRPC/... to share the data: The data that I need to share isn't serializable
Use threading.Thread instead of multiprocessing.Process for the data management process: The data management process initializes an object that will throw an error when it is forked, so I cannot initialize this object within the GUnicorn master process.
Gunicorn Issue #1621 somewhat answers my question. As far as I understand this short statement, this is because Gunicorn uses os.fork and not multiprocessing, so the utilities in multiprocessing apparently aren't guaranteed to work with Gunicorn.
So, instead of directly using multiprocessing.Queue, I replicate the behavior of Queue with another IPC library. Internally, Queue is using a ForkingPickler to serialize the data and this serialized data can also be sent via other IPC libraries, such as ZeroMQ. So, I don't necessarily need the multiprocessing module for this. Unfortunately, directly replacing the Queues with corresponding zeromq code in the above code exhibits the same behavior than in the question.
This problem can be eliminated by putting the complete multiprocessing related code into another script, so the service process s isn't a child process of Gunicorn anymore. This leads to the following code:
config.py:
import logging
import os
import pickle
import zmq
logging.basicConfig(level=logging.INFO)
bind = "localhost:8080"
workers = 1
zmq_url = "tcp://127.0.0.1:5555"
def post_fork(server, worker):
logging.info("Connecting")
context = zmq.Context()
with context.socket(zmq.REQ) as socket:
socket.connect(zmq_url)
logging.info("Sending request")
socket.send(pickle.dumps(os.getpid()))
logging.info("Waiting for a response")
other_pid = pickle.loads(socket.recv())
logging.info("Got response from %d", other_pid)
server.py:
import logging
import os
import pickle
import zmq
def serve(url):
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind(url)
while True:
logging.info("Waiting for requests on %s", url)
message = socket.recv()
logging.info("Got a message from %d", pickle.loads(message))
socket.send(pickle.dumps(os.getpid()))
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO)
serve("tcp://127.0.0.1:5555")
The startup script looks somewhat like this:
#!/usr/bin/env bash
set -euo pipefail
python server.py &
gunicorn -c config.py app:app
This worked reliably during my testing also for killed and restarting workers.
I have a SQS queue on a LocalStack server and I'm trying to consume messages from it with a Celery consumer.
It seams that the consumer is properly attached to the queue, for example the queue sqs-test-queue, but it does not receive any message when I try to send one with aws command.
My celeryconfig.py looks like this:
from kombu import (
Exchange,
Queue
)
broker_transport_options = {'region': REGION}
broker_transport = 'sqs'
accept_content = ['application/json']
result_serializer = 'json'
content_encoding = 'utf-8'
task_serializer = 'json'
worker_enable_remote_control = False
worker_send_task_events = False
result_backend = None
task_queues = (
Queue('sqs-test-queue', exchange=Exchange(''), routing_key='sqs-test-queue'),
)
and my tasks.py module looks like this:
from celery import Celery
from kombu.utils.url import quote
AWS_ACCESS_KEY = quote("AWS_ACCESS_KEY")
AWS_SECRET_KEY = quote("AWS_SECRET_KEY")
LOCALSTACK = "<IP>:<PORT>"
broker_url = "sqs://{access}:{secret}#{host}".format(access=AWS_ACCESS_KEY,
secret=AWS_SECRET_KEY,
host=LOCALSTACK)
app = Celery('tasks', broker=broker_url, backend=None)
app.config_from_object('celeryconfig')
#app.task(bind=True, name='tasks.consume', acks_late=True, ignore_result=True)
def consume(self, msg):
# DO SOMETHING WITH THE RECEIVED MESSAGE
return True
Tried to execute it with celery -A tasks worker -l INFO -Q sqs-test-queue and everything seams OK:
...
[tasks]
. tasks.consume
[... INFO/MainProcess] Connected to sqs://AWS_ACCESS_KEY:**#<IP>:<PORT>//
[... INFO/MainProcess] celery#local ready
but when I try to send a message with aws sqs send-message --endpoint-url=http://<IP>:<PORT> --queue-url=http://localhost:<PORT>/queue/sqs-test-queue --message-body="Test message", nothing happens.
What am I doing wrong? Have I missed something in the configuration maybe?
PS: If I try to run the command aws sqs receive-message --endpoint-url=http://<IP>:<PORT> --queue-url=http://localhost:<PORT>/queue/sqs-test-queue, I'm able to get the message.
NOTE:
I'm using Python 3.7.0 and my pip freeze looks like this:
boto3==1.10.16
botocore==1.13.16
celery==4.3.0
kombu==4.6.6
pycurl==7.43.0.3
...
I am going through the same thing as you. To fix it I did a couple of things:
I set the HOSTNAME_EXTERNAL and HOSTNAME env variables in localstack
Set broker_url to sqs://{access}:{secret}#{host}:{port} (as you have it)
Make sure that the celery worker's broker_transport_options does not include the config item: wait_time_seconds since this causes errors with localstack as of February 7th, 2020 (check issue here).
Once I did those two things, it started working, hope it helps.
Celery can't publish or consume arbitrary messages to/from any message queue system. Use kombu for that - that is what Celery uses behind the scenes too.
I have a python file called tasks.py in which I am defining 4 single tasks. I would like to configure celery in order to use 4 queues because each queue would have a different number of workers assigned. I was reading I should use route_task property but I tried several options and not a success.
I was following this doc celery route_tasks docs
My goal would be run 4 workers, one for each task, and don't mix tasks from different workers in different queues. It's possible? It's a good approach?
If I am doing something wrong I would be happy to change my code to make it work
Here is my config so far
tasks.py
app = Celery('tasks', broker='pyamqp://guest#localhost//')
app.conf.task_default_queue = 'default'
app.conf.task_queues = (
Queue('queueA', routing_key='tasks.task_1'),
Queue('queueB', routing_key='tasks.task_2'),
Queue('queueC', routing_key='tasks.task_3'),
Queue('queueD', routing_key='tasks.task_4')
)
#app.task
def task_1():
print "Task of level 1"
#app.task
def task_2():
print "Task of level 2"
#app.task
def task_3():
print "Task of level 3"
#app.task
def task_4():
print "Task of level 4"
Run celery one worker for each queue
celery -A tasks worker --loglevel=debug -Q queueA --logfile=celery-A.log -n W1&
celery -A tasks worker --loglevel=debug -Q queueB --logfile=celery-B.log -n W2&
celery -A tasks worker --loglevel=debug -Q queueC --logfile=celery-C.log -n W3&
celery -A tasks worker --loglevel=debug -Q queueD --logfile=celery-D.log -n W4&
There is no need to get into complex routing for submitting tasks into different queues. Define your tasks as usual.
from celery import celery
app = Celery('tasks', broker='pyamqp://guest#localhost//')
#app.task
def task_1():
print "Task of level 1"
#app.task
def task_2():
print "Task of level 2"
Now while queuing the tasks, put the tasks in proper queue. Here is an example on how to do it.
In [12]: from tasks import *
In [14]: result = task_1.apply_async(queue='queueA')
In [15]: result = task_2.apply_async(queue='queueB')
This will put the task_1 in queue named queueA and task_2 in queueB.
Now you can just start your workers to consume them.
celery -A tasks worker --loglevel=debug -Q queueA --logfile=celery-A.log -n W1&
celery -A tasks worker --loglevel=debug -Q queueB --logfile=celery-B.log -n W2&
Note: task and message are used interchangeably in the answer. It is basically a payload that the producer sends to RabbitMQ
You can either follow approach suggested by Chillar or you can define and use the task_routes configuration to route the messages to appropriate queue. This way you don't need to specify queue name every time you call apply_async.
Example: Route task1 to QueueA and route task2 to QueueB
app = Celery('my_app')
app.conf.update(
task_routes={
'task1': {'queue': 'QueueA'},
'task2': {'queue': 'QueueB'}
}
)
Sending a task to multiple queue is a bit tricky. You will have to declare an exchange, and then route your task with appropriate routing_key. You can get more information about type of exchange here. Let's go with direct for purpose of illustration.
Create Exchange
from kombu import Exchange, Queue, binding
exchange_for_queueA_and_B = Exchange('exchange_for_queueA_and_B', type='direct')
Create bindings on Queues to that exchange
app.conf.update(
task_queues=(
Queue('QueueA', [
binding(exchange_for_queueA_and_B, routing_key='queue_a_and_b')
]),
Queue('QueueB', [
binding(exchange_for_queueA_and_B, routing_key='queue_a_and_b')
])
)
)
Define the task_route to send task1 to the exchange
app.conf.update(
task_routes={
'task1': {'exchange': 'exchange_for_queueA_and_B', 'routing_key': 'queue_a_and_b'}
}
)
You can also declare these options of exchange and routing_key in your apply_async method as suggested by Chillar in the above answer.
After that, you can define your workers on same machine or different machines, to consume from those queues.
celery -A my_app worker -n consume_from_QueueA_and_QueueB -Q QueueA,QueueB
celery -A my_app worker -n consume_from_QueueA_only -Q QueueA
I want to implement task priority in my celery workers. I can do this by creating different queues for high priority tasks and low priority tasks. But also I need to send broadcast tasks to all workers with a broadcast queue and its not working. Here is tasks.py file:
from celery import Celery
from kombu.common import Broadcast, Queue, Exchange
app = Celery('tasks')
app.conf.update(
CELERY_RESULT_BACKEND='amqp',
CELERY_ACCEPT_CONTENT=['json'],
CELERY_TASK_SERIALIZER='json',
CELERY_RESULT_SERIALIZER='json',
BROKER_URL='amqp://',
CELERY_QUEUES=(Queue('default',
Exchange('default'),
routing_key='default'),
Queue('low_priority',
Exchange('low_priority'),
routing_key='low_priority'),
Broadcast('broadcast_tasks'), ),
CELERY_ROUTES={'tasks.broadcast':
{'queue': 'broadcast_tasks'},
'tasks.low_task':
{'queue': 'low_priority'},
},
CELERY_DEFAULT_QUEUE = 'default',
CELERY_DEFAULT_EXCHANGE = 'default',
CELERY_DEFAULT_ROUTING_KEY = 'default'
)
#app.task
def broadcast():
print "Broadcast called"
#app.task
def low_task():
print "Low priority called"
#app.task
def def_task():
print "Default called"
When I run celery workers with this command:
celery -A tasks -Q default worker --loglevel=info
celery -A tasks -Q default,low_priority worker --loglevel=info
Task priority works but broadcast tasks are not acknowledged.
When I run the same command without a queue argument, broadcast works but task priority does not:
celery -A tasks worker --loglevel=info
celery -A tasks worker --loglevel=info
As I understand, it happens because broadcast queues have unique names, like bcast.0b5dbce0-9bcb-48a5-8554-cbb7f32a6703 for each worker.
Does anyone have a good workaround? Thanks in advance!
You must explicitly consume a broadcast queue, so modify your commandline invocation as the comment above per ANDY_VAR.
A similar question was asked here:
start celery worker and enable it for broadcast queue
I am hoping someone can help me as I've looked on Stack Overflow and cannot find a solution to my problem. I am running a Django project and have Supervisor, RabbitMQ and Celery installed. RabbitMQ is up and running and Supervisor is ensuring my celerybeat is running, however, while it logs that the beat has started and sends tasks every 5 minutes (see below), the tasks never actually execute:
My supervisor program conf:
[program:nrv_twitter]
; Set full path to celery program if using virtualenv
command=/Users/tsantor/.virtualenvs/nrv_env/bin/celery beat -A app --loglevel=INFO --pidfile=/tmp/nrv-celerybeat.pid --schedule=/tmp/nrv-celerybeat-schedule
; Project dir
directory=/Users/tsantor/Projects/NRV/nrv
; Logs
stdout_logfile=/Users/tsantor/Projects/NRV/nrv/logs/celerybeat_twitter.log
redirect_stderr=true
autorestart=true
autostart=true
startsecs=10
user=tsantor
; if rabbitmq is supervised, set its priority higher so it starts first
priority=999
Here is the output of the log from the program above:
[2014-12-16 20:29:42,293: INFO/MainProcess] beat: Starting...
[2014-12-16 20:34:08,161: INFO/MainProcess] Scheduler: Sending due task gettweets-every-5-mins (twitter.tasks.get_tweets)
[2014-12-16 20:39:08,186: INFO/MainProcess] Scheduler: Sending due task gettweets-every-5-mins (twitter.tasks.get_tweets)
[2014-12-16 20:44:08,204: INFO/MainProcess] Scheduler: Sending due task gettweets-every-5-mins (twitter.tasks.get_tweets)
[2014-12-16 20:49:08,205: INFO/MainProcess] Scheduler: Sending due task gettweets-every-5-mins (twitter.tasks.get_tweets)
[2014-12-16 20:54:08,223: INFO/MainProcess] Scheduler: Sending due task gettweets-every-5-mins (twitter.tasks.get_tweets)
Here is my celery.py settings file:
from datetime import timedelta
BROKER_URL = 'amqp://guest:guest#localhost//'
CELERY_DISABLE_RATE_LIMITS = True
CELERYBEAT_SCHEDULE = {
'gettweets-every-5-mins': {
'task': 'twitter.tasks.get_tweets',
'schedule': timedelta(seconds=300) # 300 = every 5 minutes
},
}
Here is my celeryapp.py:
from __future__ import absolute_import
import os
from django.conf import settings
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'app.settings')
app = Celery('app')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
Here is my twitter/tasks.py:
from __future__ import absolute_import
import logging
from celery import shared_task
from twitter.views import IngestTweets
log = logging.getLogger('custom.log')
#shared_task
def get_tweets():
"""
Get tweets and save them to the DB
"""
instance = IngestTweets()
IngestTweets.get_new_tweets(instance)
log.info('Successfully ingested tweets via celery task')
return True
The get_tweets method never gets executed, however I know it works as I can execute get_tweets manually and it works fine.
I have spent two days trying to figure out why its sending due tasks, but not executing them? Any help is greatly appreciated. Thanks in advance.
user2097159 thanks for pointing me in the right direction, I was not aware I also must run a worker using supervisor. I thought it was either a worker or a beat, but now I understand that I must have a worker to handle the task and a beat to fire off the task periodically.
Below is the missing worker config for supervisor:
[program:nrv_celery_worker]
; Worker
command=/Users/tsantor/.virtualenvs/nrv_env/bin/celery worker -A app --loglevel=INFO
; Project dir
directory=/Users/tsantor/Projects/NRV/nrv
; Logs
stdout_logfile=/Users/tsantor/Projects/NRV/nrv/logs/celery_worker.log
redirect_stderr=true
autostart=true
autorestart=true
startsecs=10
user=tsantor
numprocs=1
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
I then reset the RabbitMQ queue. Now that I have both the beat and worker programs managed via supervisor, all is working as intended. Hope this helps someone else out.
You need to a start both a worker process and a beat process. You can create separate processes as described in tsantor's answer, or you can create a single process with both a worker and a beat. This can be more convenient during development (but is not recommended for production).
From "Starting the scheduler" in the Celery documentation:
You can also embed beat inside the worker by enabling the workers -B option, this is convenient if you’ll never run more than one worker node, but it’s not commonly used and for that reason isn’t recommended for production use:
$ celery -A proj worker -B
For expression in Supervisor config files see https://github.com/celery/celery/tree/master/extra/supervisord/ (linked from "Daemonization")