Can I have finer grain control over the number of celery workers running per task? I'm running pyramid applications and using pceleryd for async.
from ini file:
CELERY_IMPORTS = ('learning.workers.matrix_task',
'learning.workers.pipeline',
'learning.workers.classification_task',
'learning.workers.metric')
CELERYD_CONCURRENCY = 6
from learning.workers.matrix_task
from celery import Task
class BuildTrainingMatrixTask(Task):
....
class BuildTestMatrixTask(Task):
....
I want up to 6 BuildTestMatrixTask tasks running at a time. But I want only 1 BuiltTrainingMatrixTask running at a time. Is there a way to accomplish this?
You can send tasks to separate queues according to its type, i.e. BuildTrainingMatrixTask to first queue (let it be named as 'training_matrix') and BuildTestMatrixTask to second one (test_matrix). See Routing Tasks for details. Then you should start a worker for each queue with desirable concurrency:
$ celery worker --queues 'test_matrix' --concurrency=6
$ celery worker --queues 'training_matrix' --concurrency=1
Related
I have a Django project with celery
Due to RAM limitations I can only run two worker processes.
I have a mix of 'slow' and 'fast' tasks.
Fast tasks shall be executed ASAP. There can be many fast tasks in a short time frame (0.1s - 3s), so ideally both CPUs should handle them.
Slow tasks might run for a few minutes but the result can be delayed.
Slow tasks occur less often, but it can happen that 2 or 3 are queued up at the same time.
My idea was to have one:
1 celery worker W1 with concurrency 1, that handles only fast tasks
1 celery worker W2 with concurrency 1 that can handle fast and slow tasks.
celery has by default a task prefetch multiplier ( https://docs.celeryproject.org/en/latest/userguide/configuration.html#worker-prefetch-multiplier ) of 4, which means that 4 fast tasks could be queued behind a slow task and could be delayed by several minutes. Thus I'd like to disable prefetch for worker W2. The doc states:
To disable prefetching, set worker_prefetch_multiplier to 1. Changing
that setting to 0 will allow the worker to keep consuming as many
messages as it wants.
However what I observe is, that with a prefetch_multiplier of 1 one task is prefetched and would still be delayed by a slow task.
Is this a documentation bug? Is this an implementation bug? Or do I misunderstand the documentation?
Is there any way to implement what I want?
The commands, that I execute to start the workers are:
celery -A miniclry worker --concurrency=1 -n w2 -Q=fast,slow --prefetch-multiplier 0
celery -A miniclry worker --concurrency=1 -n w1 -Q=fast
my celery settings are default except:
CELERY_BROKER_URL = "pyamqp://*****#localhost:5672/mini"
CELERY_TASK_ROUTES = {
'app1.tasks.task_fast': {"queue": "fast"},
'app1.tasks.task_slow': {"queue": "slow"},
}
my django project's celery.py file is:
from __future__ import absolute_import
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'miniclry.settings')
app = Celery("miniclry", backend="rpc", broker="pyamqp://")
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
The __init__.py of my django project is
from .celery import app as celery_app
__all__ = ('celery_app',)
The code of my workers
import time, logging
from celery import shared_task
from miniclry.celery import app as celery_app
logger = logging.getLogger(__name__)
#shared_task
def task_fast(delay=0.1):
logger.warning("fast in")
time.sleep(delay)
logger.warning("fast out")
#shared_task
def task_slow(delay=30):
logger.warning("slow in")
time.sleep(delay)
logger.warning("slow out")
If I execute following from a management shell I see, that one fast task is only executed after the slow task finished.
from app1.tasks import task_fast, task_slow
task_slow.delay()
for i in range(30):
task_fast.delay()
Can anybody help?
I could post the entire test project if this is considered helpful. Just advise about the recommended SO way of exchanging such kind of projects
Version info:
celery==4.3.0
Django==1.11.25
Python 2.7.12
I confirm the issue, there is a bug in this section of the documentation. worker_prefetch_multiplier = 1 will just as it says, set the worker's prefetch to 1, means worker will hold one more task in addition to one that is executing at the moment.
To actually disable the prefetch you also need to use task_acks_late = True along with the prefetch setting, see this docs section
I have a few questions regarding task routing, concurrency and performance.
Here is my use case :
I've got one dedicated server to run celery tasks, so I can use all the CPUs to run celery workers on this server.
I have a lot of different python tasks, which I route using : CELERY_ROUTES and because the tasks perform really different types of python code, I created 5 different workers.
These worker are created when I deploy my project using ansible, here is an example:
[program:default_queue-celery]
command={{ venv_dir }}/bin/celery worker --app=django_coreapp --loglevel=INFO --concurrency=1 --autoscale=15,10 --queues=default_queue
environment =
SERVER_TYPE="{{ SERVER_TYPE }}",
DB_SCHEMA="{{ DB_SCHEMA }}",
DB_USER="{{ DB_USER }}",
DB_PASS="{{ DB_PASS }}",
DB_HOST="{{ DB_HOST }}"
directory={{ git_dir }}
user={{ user }}
group={{ group }}
stdout_logfile={{ log_dir }}/default_queue.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=5
redirect_stderr=true
autostart=true
autorestart=true
startsecs=10
killasgroup=true
I have also a CELERY_QUEUES in settings.py to make the bridge between CELERY_ROUTES and my celery programs (Queues)
CELERY_DEFAULT_QUEUE = 'default_queue'
And if it happens that I don't route a task it will go to my 'default_queue'
To give space to all of my queues, I set --concurrency to 1 for default_queue, and more for my most important queue.
But I am wondering, does AutoScale have impact on the same value as concurrency ? Meaning, if I set concurrency to 1 and --autoscale to 15,10 (example above)
Will my worker 'work' on CPU and process a maximum of 15 tasks on this CPU ?
Or does this mean something completely different ?
It makes no sense setting both concurrency and autoscale since both are means to control the number of worker subprocesses for a given worker instance, as explained here.
--concurrency N means you will have exactly N worker subprocesses for your worker instance (meaning the worker instance can handle N conccurent tasks).
--autoscale max, min means you will have at least min and at most max concurrent worker subprocesses for a given worker instance.
On which CPU each process (the main worker process or any of it's child subprocesses) will run is not predictable, it's an OS thing, but do not assume subprocesses will all run on the same CPU (chances are they won't - that's part of the point of having concurrent subprocesses actually).
I am using celery for my web application.
Celery executes Parent tasks which then executes further pipline of tasks
The issues with celery
I can't get dependency graph and visualizer i get with luigi to see whats the status of my parent task
Celery does not provide mechanism to restart the failed pipeline and start from where it failed.
These two thing i can easily get from luigi.
So i was thinking that once celery runs the parent task then inside that task i execute the Luigi pipleine.
Is there going to be any issue with that i.e i need to autoscale the celery workers based on queuesize . will that affect any luigi workers across multiple machines??
Never tried but I think it should be possible to call a luigi task form inside a celery task, the same way you do it from python code in general:
from foobar import MyTask
from luigi import scheduler
task = MyTask(123, 'another parameter value')
sch = scheduler.CentralPlannerScheduler()
w = worker.Worker(scheduler=sch)
w.add(task)
w.run()
About scaling your queue and celery workers: if you have too many celery workers calling luigi tasks of course it will require you to scale your luigi scheduler/daemon so it can handle the number of API requests (every time you call a task to be excecuted, you hit the luigi scheduler API, every N seconds -it dependes on your config- your tasks will hit the scheduler API to say "I'm alive", every time a task finished with -error or success- you hit the scheduler API, and so on).
So yes, take a close look at your scheduler to see if it's receiving too many http requests or if its database is being a bottle neck (luigi uses by default an sqlite but you can easily change it to mysql o postgres).
UPDATE:
Since version 2.7.0, luigi.scheduler.CentralPlannerScheduler has been renamed to luigi.scheduler.Scheduler as you may see here so the above code should now be:
from foobar import MyTask
from luigi import scheduler
task = MyTask(123, 'another parameter value')
sch = scheduler.Scheduler()
w = worker.Worker(scheduler=sch)
w.add(task)
w.run()
My task is consists of sequential requests (data from first request is needed for second and so on) to API which has rate limits for one host. So I've tried to get speed up by using Celery for managing several workers. But with several running workers tasks does not distributed evenly. So while there is free workers, task goes to worker on cool-down, which significantly reduces overall processing speed.
Code which demonstrate this problem:
from celery import Celery
app = Celery('tasks', backend='mongodb', broker='mongodb://localhost:27017/celery')
app.conf.update(
CELERY_ACKS_LATE = True,
CELERYD_CONCURRENCY = 1,
CELERYD_PREFETCH_MULTIPLIER = 1,
)
#app.task(rate_limit='10/m')
def mytask():
return 'ok'
If you start several workers (with or without -Ofair option) and run following code:
while True:
tasks.mytask.apply_async().get()
Tasks will not be distributed evenly and you can easily see that task almost every time goes to worker on coll-down.
That is right Celery configuration for this kind of problem?
Part1
I've read and tried various SO threads to purge the celery tasks using Redis, but none of them worked. Please let me know how to purge tasks in celery using Redis as the broker.
Part 2
Also, I've multiple queues. I can run it within the project directory, but when demonizing, the workers dont take task. I still need to start the celery workers manually. How can I demozize it?
Here is my celerd conf.
# Name of nodes to start, here we have a single node
CELERYD_NODES="w1 w2 w3 w4"
CELERY_BIN="/usr/local/bin/celery"
# Where to chdir at start.
CELERYD_CHDIR="/var/www/fractal/parser-quicklook/"
# Python interpreter from environment, if using virtualenv
#ENV_PYTHON="/somewhere/.virtualenvs/MyProject/bin/python"
# How to call "manage.py celeryd_multi"
#CELERYD_MULTI="/usr/local/bin/celeryd-multi"
# How to call "manage.py celeryctl"
#CELERYCTL="/usr/local/bin/celeryctl"
#CELERYBEAT="/usr/local/bin/celerybeat"
# Extra arguments to celeryd
CELERYD_OPTS="--time-limit=300 --concurrency=8 -Q BBC,BGR,FASTCOMPANY,Firstpost,Guardian,IBNLIVE,LIVEMINT,Mashable,NDTV,Pandodaily,Reuters,TNW,TheHindu,ZEENEWS "
# Name of the celery config module, don't change this.
CELERY_CONFIG_MODULE="celeryconfig"
# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.
#CELERYD_USER="nobody"
#CELERYD_GROUP="nobody"
# Set any other env vars here too!
PROJET_ENV="PRODUCTION"
# Name of the projects settings module.
# in this case is just settings and not the full path because it will change the dir to
# the project folder first.
CELERY_CREATE_DIRS=1
Celeryconfig is already provided in part1.
Here is my proj directory structure.
project
|-- main.py
|-- project
| |-- celeryconfig.py
| |-- __init__.py
|-- tasks.py
How can I demonize with the Queues? I have provided the queues in CELERYD_OPTS as well.
Is there a way in which we can dynamically demonize the number of queues in the celery? For eg:- we have CELERY_CREATE_MISSING_QUEUES = True for creating the missing queues. Is there something similar to daemonize the celery queues?
celery purge should be enough to clean up the queue in redis. However, your worker will have its own reserved tasks and it will send them back to the queue when you stop the worker. So, first, stop all the workers. Then run celery purge.
If you have several queues, celery purge will purge the default one. You can specify which queue(s) you would like to purge as such:
celery purge -A proj -Q queue1,queue2
In response to part 1, a programmatic solution to purge your queue, further documentation can be found at the following link celery.app.control.purge docs.
from celery import Celery
app = Celery()
app.control.purge()
#OR
app.control.discard_all()
This revokes all the tasks it can without terminating any processes. (To do so add terminate=True to the revoke call at your own risk.)
It takes a second or two to run, so is not suitable for high throughput code.
from myapp.celery import app as celery_app
celery_app.control.purge()
i = celery_app.control.inspect()
# scheduled(): tasks with an ETA or countdown
# active(): tasks currently running - probably not revokable without terminate=True
# reserved(): enqueued tasks - usually revoked by purge() above
for queues in (i.active(), i.reserved(), i.scheduled()):
for task_list in queues.values():
for task in task_list:
task_id = task.get("request", {}).get("id", None) or task.get("id", None)
celery_app.control.revoke(task_id)
Just .purge() then revoking .scheduled() would probably have the same effect to be honest, I haven't experimented extensively. But purge alone will not revoke tasks sat in the queues with an ETA or countdown set.
Credit to #kahlo's answer, which was the basis for this.
Starting with Celery v5, you should now use:
celery -A proj purge -Q queue1,queue2