Develop Reference

Trying to get supervisor to create a worker for python-rq

I am trying to get supervisor to spawn a worker following this pattern using python-RQ, much like what is mentioned in this stackoverflow question. I can start workers manually from the terminal as follows:
$ venv/bin/rq worker
14:35:27 Worker rq:worker:fd403822518a4f21802fa0dc417e526a: started, version 1.2.2
14:35:27 *** Listening on default...
It works great. I can confirm the worker exists in another terminal:
$ venv/bin/rq info
0 queues, 0 jobs total
fd403822518a4f21802fa0dc417e526a (b'' 41735): idle default
1 workers, 0 queues
Now to start a worker using supervisor.... Here is my supervisord.conf file, located in the same directory.
[supervisord]
;[program:worker]
command=venv/bin/rq worker
process_name=%(program_name)s-%(process_num)s
numprocs=1
directory=.
stopsignal=TERM
autostart=false
autorestart=false
I can start supervisor as follows:
$ venv/bin/supervisord -n
2020-03-05 14:36:45,079 INFO supervisord started with pid 41757
However, checking for a new worker, I see it's not there.
$ venv/bin/rq info
0 queues, 0 jobs total
0 workers, 0 queues
I have tried a multitude of other ways to get this worker to start, such as...
... within the virtual environment:
$ source venv/bin/activate
(venv) $ rq worker
*** Listening on default...
... using a shell file
#!/bin/bash
source /venv/bin/activate
rq worker low
$ ./start.sh
*** Listening on default...
... using a python script
$ venv/bin/python3 worker.py
*** Listening on default...
When started manually they all work fine. Changing the command= in supervisord.conf doesn't seem to make a difference. There is no worker to be found. What am I missing? Why won't supervisor start a worker? I am running this in Mac OS and my file structure is as follows:
.
|--__pycache__
|--supervisord.conf
|--supervisord.log
|--supervisord.pid
|--main.py
|--task.py
|--venv
|--bin
|--rq
|--supervisord
|--...etc
|--include
|--lib
|--pyenv.cfg
Thanks in advance.

I had two problems with supervisord.conf, which was preventing the worker from starting. The corrected config file is as follows:
[supervisord]
[program:worker]
command=venv/bin/rqworker
process_name=%(program_name)s-%(process_num)s
numprocs=1
directory=.
stopsignal=TERM
autostart=true
autorestart=false
First, the line [program:worker] was in fact commented out. I must have taken this line from the commented out sample file and not realized. However removing the comment still didn't start the worker.... I also had to set autostart=true, as starting supervisor does not automatically start a command.

Running Celery on AWS

I've a django application which uses celery which I'm moving onto AWS. I've upgraded from Python 2.7 to 3.6 at the same time.
I'm having trouble getting celery to run on AWS, with this error:
Application deployment failed at 2018-08-06T14:02:02Z with exit status 127 and error: Hook /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh failed.
[program:django-celery-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/python /opt/python/current/app/manage.py celery worker -A myapp --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",RDS_PASSWORD="stmtrx321",RDS_DB_NAME="ebdb",RDS_USERNAME="postgres",RDS_HOSTNAME="aan9bvuq3a4wd1.cxzlzbczjyp1.us-east-1.rds.amazonaws.com"
/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh: line 47: supervisorctl: command not found
/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh: line 50: supervisorctl: command not found
/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh: line 53: supervisorctl: command not found.
Incorrect application version "app-b38e-180802_150757" (deployment 23). Expected version "app-0def-180806_140451" (deployment 26).
supervisor is installed.
This is my config file:
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh" :
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
celeryenv=${celeryenv%?}
# Create celery configuration script
celeryconf="[program:django-celery-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/python /opt/python/current/app/manage.py celery worker -A myapp --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf
# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# Reread the supervisord config
supervisorctl -c /opt/python/etc/supervisord.conf reread
# Update supervisord in cache without restarting all services
supervisorctl -c /opt/python/etc/supervisord.conf update
# Start/Restart celeryd through supervisord
supervisorctl -c /opt/python/etc/supervisord.conf restart django-celery-worker
container_commands:
01_celery_tasks_run:
command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
option_settings:
aws:elasticbeanstalk:container:python:
WSGIPath: myapp/wsgi.py
aws:elb:listener:443:
SSLCertificateId: arn:aws:acm:us-east-1:349705394408:certificate/5e94d3d2-039a-4ffc-bf7e-06995dc65fc9
ListenerProtocol: HTTPS
InstancePort: 80
What have I got wrong?

Django celery beat doesn't see periodic task on Elastic Beanstalk

I have configured celery worker and celery beat on EB. There are no errors in logs during deploy and celery worker works fine, but doesn't see periodic tasks. On local machine everything works smoothly.
Here is my config file for the celery
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
# Create required directories
sudo mkdir -p /var/log/celery/
sudo mkdir -p /var/run/celery/
# Create group called 'celery'
sudo groupadd -f celery
# add the user 'celery' if it doesn't exist and add it to the group with same name
id -u celery &>/dev/null || sudo useradd -g celery celery
# add permissions to the celery user for r+w to the folders just created
sudo chown -R celery:celery /var/log/celery/
sudo chmod -R 777 /var/log/celery/
sudo chown -R celery:celery /var/run/celery/
sudo chmod -R 777 /var/run/celery/
# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/%/%%/g' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
celeryenv=${celeryenv%?}
# Create CELERY configuration script
celeryconf="[program:celeryd]
directory=/opt/python/current/app
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A config.celery:app --loglevel=INFO --logfile="/var/log/celery/celery_worker.log" --pidfile="/var/run/celery/celery_worker_pid.pid"
user=celery
numprocs=1
stdout_logfile=/var/log/std_celery_worker.log
stderr_logfile=/var/log/std_celery_worker_errors.log
autostart=true
autorestart=true
startsecs=10
startretries=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 60
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# Create CELERY BEAT configuraiton script
celerybeatconf="[program:celerybeat]
directory=/opt/python/current/app
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A config.celery:app --loglevel=INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler --logfile="/var/log/celery/celery_beat.log" --pidfile="/var/run/celery/celery_beat_pid.pid"
user=celery
numprocs=1
stdout_logfile=/var/log/std_celery_beat.log
stderr_logfile=/var/log/std_celery_beat_errors.log
autostart=true
autorestart=true
startsecs=10
startretries=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 60
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999
environment=$celeryenv"
# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf
echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf
# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: uwsgi.conf celery.conf celerybeat.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# Enable supervisor to listen for HTTP/XML-RPC requests.
# supervisorctl will use XML-RPC to communicate with supervisord over port 9001.
# Source: https://askubuntu.com/questions/911994/supervisorctl-3-3-1-http-localhost9001-refused-connection
if ! grep -Fxq "[inet_http_server]" /opt/python/etc/supervisord.conf
then
echo "[inet_http_server]" | tee -a /opt/python/etc/supervisord.conf
echo "port = 127.0.0.1:9001" | tee -a /opt/python/etc/supervisord.conf
fi
# Reread the supervisord config
supervisorctl -c /opt/python/etc/supervisord.conf reread
# Update supervisord in cache without restarting all services
supervisorctl -c /opt/python/etc/supervisord.conf update
# Start/Restart celeryd through supervisord
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd
supervisorctl -c /opt/python/etc/supervisord.conf restart celerybeat
commands:
01_kill_other_beats:
command: "ps auxww | grep 'celery beat' | awk '{print $2}' | sudo xargs kill -9 || true"
ignoreErrors: true
02_restart_beat:
command: "supervisorctl -c /opt/python/etc/supervisord.conf restart celerybeat"
leader_only: true
03_upgrade_pip_global:
command: "if test -e /usr/bin/pip; then sudo /usr/bin/pip install --upgrade pip; fi"
04_upgrade_pip_global:
command: "if test -e /usr/local/bin/pip; then sudo /usr/local/bin/pip install --upgrade pip; fi"
05_upgrade_pip_for_venv:
command: "if test -e /opt/python/run/venv/bin/pip; then sudo /opt/python/run/venv/bin/pip install --upgrade pip; fi"
Can somebody say where is the error?
I start periodic task like this:
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
pass
Update:
supervisord logs
2018-07-10 12:56:18,683 INFO stopped: celerybeat (terminated by SIGTERM)
2018-07-10 12:56:18,691 INFO spawned: 'celerybeat' with pid 1626
2018-07-10 12:56:19,181 INFO stopped: celerybeat (terminated by SIGTERM)
2018-07-10 12:56:20,187 INFO spawned: 'celerybeat' with pid 1631
2018-07-10 12:56:30,200 INFO success: celerybeat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2018-07-10 12:56:30,466 INFO stopped: celeryd (terminated by SIGTERM)
2018-07-10 12:56:31,472 INFO spawned: 'celeryd' with pid 1638
2018-07-10 12:56:41,486 INFO success: celeryd entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2018-07-10 13:28:32,572 CRIT Supervisor running as root (no user in config file)
2018-07-10 13:28:32,573 WARN No file matches via include "/opt/python/etc/uwsgi.conf"
2018-07-10 13:28:32,573 WARN Included extra file "/opt/python/etc/celery.conf" during parsing
2018-07-10 13:28:32,573 WARN Included extra file "/opt/python/etc/celerybeat.conf" during parsing
2018-07-10 13:28:32,591 INFO RPC interface 'supervisor' initialized
2018-07-10 13:28:32,591 CRIT Server 'inet_http_server' running without any HTTP authentication checking

The source of the issue were imports when tried to set periodic tasks:
celery.py
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
from my_app.tasks.task1 import some_task
sender.some_task(
60.0, some_task.s(), name='call every 60 seconds'
)
The solution is to use task inside the celery app:
celery.py
#app.task
def temp_task():
from my_app.tasks.task1 import some_task
some_task()
so set periodic tasks will look like this
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
from my_app.tasks.task1 import some_task
sender.some_task(
60.0, temp_task.s(), name='call every 60 seconds'
)
It was difficult to get the source of the issue as there were no error logs and regular logs were empty as indeed celery was not started.

ERROR: CANT_REREAD: The directory named as part of the path /home/app/logs/celery.log does not exist

I'm following a tutorial on how to use Celery on my Django production server.
When I get to the bit where it says:
Now reread the configuration and add the new process:
sudo supervisorctl reread
sudo supervisorctl update
When I perform sudo supervisorctl reread in my server (Ubuntu 16.04) terminal, it returns this:
ERROR: CANT_REREAD:
The directory named as part of the path /home/app/logs/celery.log does not exist.
in section 'app-celery' (file: '/etc/supervisor/conf.d/app-celery.conf')
I've done all of the instructions prior to this including installing supervisor as well as creating a file named mysite-celery.conf (app-celery.conf) in the folder: /etc/supervisor/conf.d
If you're curious my app-celery.conf file looks like this:
[program:app-celery]
command=/home/app/bin/celery worker -A draft1 --loglevel=INFO
directory=/home/app/draft1
user=zorgan
numprocs=1
stdout_logfile=/home/app/logs/celery.log
stderr_logfile=/home/app/logs/celery.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
stopasgroup=true
; Set Celery priority higher than default (999)
; so, if rabbitmq is supervised, it will start first.
priority=1000
Any idea what the problem is?

Somehow supervisor is not able to create the folder - /home/app/logs/.
You can create it manually using mkdir and restart the supervisor service
mkdir /home/app/logs
sudo service supervisor restart

I added my username to the superisord.conf file under the [unix_http_server] section like so:
[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
chmod=0770 ; sockef file mode (default 0700)
chown=appuser:supervisor ;(username:group)
This seemed to work- time will tell if it continues working after I manage solve the rest of the supervisor issues.

Celeryd multi with supervisord

Trying to run supervisord (3.2.2) with celery multi.
Seems to be that supervisord can't handle it. Single celery worker works fine.
This is my supervisord configuration
celery multi v3.1.20 (Cipater)
> Starting nodes...
> celery1#parzee-dev-app-sfo1: OK
Stale pidfile exists. Removing it.
> celery2#parzee-dev-app-sfo1: OK
Stale pidfile exists. Removing it.
celeryd.conf
; ==================================
; celery worker supervisor example
; ==================================
[program:celery]
; Set full path to celery program if using virtualenv
command=/usr/local/src/imbue/application/imbue/supervisorctl/celeryd/celeryd.sh
process_name = %(program_name)s%(process_num)d#%(host_node_name)s
directory=/usr/local/src/imbue/application/imbue/conf/
numprocs=2
stderr_logfile=/usr/local/src/imbue/application/imbue/log/celeryd.err
logfile=/usr/local/src/imbue/application/imbue/log/celeryd.log
stdout_logfile_backups = 10
stderr_logfile_backups = 10
stdout_logfile_maxbytes = 50MB
stderr_logfile_maxbytes = 50MB
autostart=true
autorestart=false
startsecs=10
Im using the following supervisord variables to emulate the way I start celery:
%(program_name)s
%(process_num)d
#
%(host_node_name)s
Supervisorctl
supervisorctl
celery:celery1#parzee-dev-app-sfo1 FATAL Exited too quickly (process log may have details)
celery:celery2#parzee-dev-app-sfo1 FATAL Exited too quickly (process log may have details)
I tried changing this value in /usr/local/lib/python2.7/dist-packages/supervisor/options.py from 0 to 1:
numprocs_start = integer(get(section, 'numprocs_start', 1))
I still get:
celery:celery1#parzee-dev-app-sfo1 FATAL Exited too quickly (process log may have details)
celery:celery2#parzee-dev-app-sfo1 EXITED May 14 12:47 AM
Celery is starting but supervisord is not keeping track of it.
root#parzee-dev-app-sfo1:/etc/supervisor#
ps -ef | grep celery
root 2728 1 1 00:46 ? 00:00:02 [celeryd: celery1#parzee-dev-app-sfo1:MainProcess] -active- (worker -c 16 -n celery1#parzee-dev-app-sfo1 --loglevel=DEBUG -P processes --logfile=/usr/local/src/imbue/application/imbue/log/celeryd.log --pidfile=/usr/local/src/imbue/application/imbue/log/1.pid)
root 2973 1 1 00:46 ? 00:00:02 [celeryd: celery2#parzee-dev-app-sfo1:MainProcess] -active- (worker -c 16 -n celery2#parzee-dev-app-sfo1 --loglevel=DEBUG -P processes --logfile=/usr/local/src/imbue/application/imbue/log/celeryd.log --pidfile=/usr/local/src/imbue/application/imbue/log/2.pid)
celery.sh
source ~/.profile
CELERY_LOGFILE=/usr/local/src/imbue/application/imbue/log/celeryd.log
CELERYD_OPTS=" --loglevel=DEBUG"
CELERY_WORKERS=2
CELERY_PROCESSES=16
cd /usr/local/src/imbue/application/imbue/conf
exec celery multi start $CELERY_WORKERS -P processes -c $CELERY_PROCESSES -n celeryd#{HOSTNAME} -f $CELERY_LOGFILE $CELERYD_OPTS
Similar:
Running celeryd_multi with supervisor
How to use Supervisor + Django + Celery with multiple Queues and Workers?

Since supervisor monitors(start/stop/restart) process, the process should be run in foreground(should not be daemonized).
Celery multi daemonizes itself, so it can't be run with supervisor.
You can create separate process for each worker and group them into one.
[program:worker1]
command=celery worker -l info -n worker1
[program:worker2]
command=celery worker -l info -n worker2
[group:workers]
programs=worker1,worker2
You can also write a shell script which makes daemon process run in foreground like this.
#! /usr/bin/env bash
set -eu
pidfile="/var/run/your-daemon.pid"
command=/usr/sbin/your-daemon
# Proxy signals
function kill_app(){
kill $(cat $pidfile)
exit 0 # exit okay
}
trap "kill_app" SIGINT SIGTERM
# Launch daemon
$ celery multi start 2 -l INFO
sleep 2
# Loop while the pidfile and the process exist
while [ -f $pidfile ] && kill -0 $(cat $pidfile) ; do
sleep 0.5
done
exit 1000 # exit unexpected