Develop Reference

Detect and Initiate celery worker in Python Code

Normally I run the following in terminal to start the worker process
celery -A myapp worker --loglevel=info
What I want to achieve now is that with python code
I will check whether they are worker process being initiated,
if not only I run this command (with python code)
How to achieve that?

There is no need for that as Celery gives you standard way to do it...
--pidfile PIDFILE Optional file used to store the process pid. The
program won't start if this file already exists and
the pid is still alive.
So simply change how you start your worker to something like celery -A myapp worker --loglevel=info --pidfile celery1.pid
If you open another terminal and run the command I wrote above, it will not run as the PID file is already created.

Trying to get supervisor to create a worker for python-rq

I am trying to get supervisor to spawn a worker following this pattern using python-RQ, much like what is mentioned in this stackoverflow question. I can start workers manually from the terminal as follows:
$ venv/bin/rq worker
14:35:27 Worker rq:worker:fd403822518a4f21802fa0dc417e526a: started, version 1.2.2
14:35:27 *** Listening on default...
It works great. I can confirm the worker exists in another terminal:
$ venv/bin/rq info
0 queues, 0 jobs total
fd403822518a4f21802fa0dc417e526a (b'' 41735): idle default
1 workers, 0 queues
Now to start a worker using supervisor.... Here is my supervisord.conf file, located in the same directory.
[supervisord]
;[program:worker]
command=venv/bin/rq worker
process_name=%(program_name)s-%(process_num)s
numprocs=1
directory=.
stopsignal=TERM
autostart=false
autorestart=false
I can start supervisor as follows:
$ venv/bin/supervisord -n
2020-03-05 14:36:45,079 INFO supervisord started with pid 41757
However, checking for a new worker, I see it's not there.
$ venv/bin/rq info
0 queues, 0 jobs total
0 workers, 0 queues
I have tried a multitude of other ways to get this worker to start, such as...
... within the virtual environment:
$ source venv/bin/activate
(venv) $ rq worker
*** Listening on default...
... using a shell file
#!/bin/bash
source /venv/bin/activate
rq worker low
$ ./start.sh
*** Listening on default...
... using a python script
$ venv/bin/python3 worker.py
*** Listening on default...
When started manually they all work fine. Changing the command= in supervisord.conf doesn't seem to make a difference. There is no worker to be found. What am I missing? Why won't supervisor start a worker? I am running this in Mac OS and my file structure is as follows:
.
|--__pycache__
|--supervisord.conf
|--supervisord.log
|--supervisord.pid
|--main.py
|--task.py
|--venv
|--bin
|--rq
|--supervisord
|--...etc
|--include
|--lib
|--pyenv.cfg
Thanks in advance.

I had two problems with supervisord.conf, which was preventing the worker from starting. The corrected config file is as follows:
[supervisord]
[program:worker]
command=venv/bin/rqworker
process_name=%(program_name)s-%(process_num)s
numprocs=1
directory=.
stopsignal=TERM
autostart=true
autorestart=false
First, the line [program:worker] was in fact commented out. I must have taken this line from the commented out sample file and not realized. However removing the comment still didn't start the worker.... I also had to set autostart=true, as starting supervisor does not automatically start a command.

How to change default path of Celery beat service?

I installed Celery as a Windows service. My code moves *.pid and Celery log files into another directory, but three files (celerybeat-schedule.bak, celerybeat-schedule.dir, celerybeat-schedule.dat) which I am not able to move.
I used below code for changing other file's default path:
command = '"{celery_path}" -A {proj_dir} beat -f "{log_path}" -l info --pidfile="{pid_path}" '.format(
celery_path=os.path.join(PYTHONSCRIPTPATH, 'celery.exe'),
proj_dir=PROJECTDIR,
# log_path_1=os.path.join(INSTDIR,'celery_2.log')),
log_path=os.path.join(tmpdir,'celery_'+cur_date_time+'.log'),
pid_path = os.path.join(tmpdir,'celerybeat_'+cur_date_time+'.pid'))
How to change default path of Celery beat service?

If you executed celery -A your.project.app beat --help it would print you very useful CLI help where you would find the solution to your problem - the -s <path to the scheduler database file> flag.
-s SCHEDULE, --schedule SCHEDULE
Path to the schedule database. Defaults to celerybeat-
schedule. The extension '.db' may be appended to the
filename. Default is celerybeat-schedule.
All you have to do is to pass a full path to the schedule database file to your Celery beat process. Example: -s C:/services/celery/celerybeat-schedule.db

Finally I am able to change path of celery services using below code.
command = '"{celery_path}" -A {proj_dir} beat -f "{log_path}" -l info --pidfile="{pid_path}" '.format(
celery_path=os.path.join(PYTHONSCRIPTPATH, 'celery.exe'),
proj_dir=PROJECTDIR,
# log_path_1=os.path.join(INSTDIR,'celery_2.log')),
log_path=os.path.join(CELERYDIR,'celery_'+cur_date_time+'.log'),
# bak_path=os.path.join(CELERYDIR,'celerybeat-schedule'),
pid_path = os.path.join(CELERYDIR,'celerybeat_'+cur_date_time+'.pid'))

app-celery: ERROR (no such file)

I've followed the tutorial on how to implement celery on my django production server, using supervisor.
I've done this successfully, however when I try to start supervisor with sudo supervisorctl start app-celery - it returns:
app-celery: ERROR (no such file)
Here is my config in the folder /etc/supervisor/conf.d (app-celery.conf):
[program:app-celery]
command=/home/app/bin/celery worker -A draft1 --loglevel=INFO
directory=/home/app/draft1
numprocs=1
stdout_logfile=/var/log/supervisor/celery.log
stderr_logfile=/var/log/supervisor/celery.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
stopasgroup=true
; Set Celery priority higher than default (999)
; so, if rabbitmq is supervised, it will start first.
priority=1000
Any idea what the problem is?

I had the same issue. Adding the following resolved the issue for me.
environment=DJANGO_SETTINGS_MODULE="my_proj.settings"
I'm not sure why this is necessary. It's not listed in the documentation I've seen, and running the raw command either inside or outside of the virtual environment seems to be fine. Nevertheless, celery now starts and restarts without issue for me.

Celery Running as Daemon stops

I have installed celery using redis on a centos linux server.
I start a celery worker using the following command:
celery multi start worker1 -A proj -Q lopri,lopri2 -l info --pidfile="$HOME/run/celery/%n.pid" --logfile="$HOME/log/celery/%n.log"
The problem is that after a few hours, the worker no longer responds task creation. I have to restart the worker in order for it to process tasks again.
Here is the celery settings file located at /etc/default/celeryd:
# Names of nodes to start
# most people will only start one node:
CELERYD_NODES="worker1"
# but you can also start multiple and configure settings
# for each in CELERYD_OPTS (see `celery multi --help` for examples):
#CELERYD_NODES="worker1 worker2 worker3"
# alternatively, you can specify the number of nodes to start:
#CELERYD_NODES=10
# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"
# App instance to use
# comment out this line if you don't use an app
CELERY_APP="proj"
# or fully qualified:
#CELERY_APP="proj.tasks:app"
#django settings
#export DJANGO_SETTINGS_MODULE="settings"
# Where to chdir at start.
CELERYD_CHDIR="/var/www/html/proj/"
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"
# Workers should run as an unprivileged user.
# You need to create this user manually (or you can choose
# a user/group combination that already exists, e.g. nobody).
CELERYD_USER="ec2-user"
CELERYD_GROUP="ec2-user"
# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1
One problem that maybe a clue as to what is happening is that I have to run the "start" command from the project folder other wise I receive the error:
ImportError: No module named proj
Shouldn't the CELERYD_CHDIR setting take care of this? Does that mean that my start command is not using this celery default setting file?
Thanks for any light you can shed on this.

If you are running celery by command line, your settings file is pretty much useless. I mean, they are not being applied, that's why you get an error saying your project can't be imported.
This is the script to run celery as daemon:
https://github.com/celery/celery/blob/3.1/extra/generic-init.d/celeryd
As you can see here: https://github.com/celery/celery/blob/3.1/extra/generic-init.d/celeryd#L56
That's where the script imports your settings.
I'm not quite sure why your celeryd stops working after a few hours, but this shows that's you're not really running as a daemon.
Maybe setting the propert init.d script with settings can fix that, but I doubt it.