Develop Reference

How do I get Celery to execute in production on Heroku?

I'm using Celery on my Django app.
I can't get the celery task to execute unless I run celery -A appname worker -l info --pool=solo in the terminal.
How do I get it execute when the site is in production?

You need to add a worker process to your Procfile, e.g.
web: gunicorn some_app.wsgi
worker: celery -A appname worker -l info --pool solo
After redeploying, scale up one or more workers, e.g. by running
heroku ps:scale worker=1

set up pipenv with supervisor

I want to deploy dev server but I have a problem with starting celery and gunicorn. I'm using scripts for my purposes
celery.sh
#!/bin/bash
cd /home/dev/app
pipenv run celery -A config worker -B -l info
and start.sh for gunicorn
#!/bin/bash
cd /home/dev/app
pipenv run gunicorn config.wsgi:application -b 127.0.0.1:8005 -w 2 -t 60 \
--env DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE \
--env DSN=$SENTRY_DSN \
--env DATABASE_URL=$DATABASE_URL \
--log-file - \
--error-logfile /home/dev/app/errors.log
Also here is my config for supervisor
[program:back]
directory=/home/dev/app/
command=/home/dev/bin/start
user=dev
autostart=true
autorestart=true
redirect_stderr=true
stopsignal=QUIT
stopasgroup=true
killasgroup=true
[program:celery]
directory=/home/dev/app/
command=/home/dev/bin/celery
user=dev
autostart=true
autorestart=true
redirect_stderr=true
stopsignal=QUIT
stopasgroup=true
killasgroup=true
When I'm running sudo supervisorctl start celery I'm getting the following error:
/home/dev/bin/celery: line 3: pipenv: command not found
Also I added the following lines as pipenv documentation says (https://pipenv.readthedocs.io/en/latest/diagnose/)
[supervisord]
environment=LC_ALL='en_US.UTF-8',LANG='en_US.UTF-8'
UPDATE
Changed my supervisor config:
[program:back]
directory=/home/dev/app/
command=pipenv run gunicorn config.wsgi:application --bind 127.0.0.1:8005
user=dev
autostart=true
autorestart=true
redirect_stderr=true
stopsignal=QUIT
stopasgroup=true
killasgroup=true
[program:celery]
directory=/home/dev/app/
command=pipenv run celery -A config:celery_app worker -B -l info
user=dev
autostart=true
autorestart=true
redirect_stderr=true
stopsignal=QUIT
stopasgroup=true
killasgroup=true
And now I'm getting an error:
back: ERROR (no such file)

You need to give explicit path of gunicorn. Though I'm not sure about pipenv, but the error you are getting is because supervisor tries to find gunicorn in the directory. You should change your config file into something like this:
[program:back]
directory=/home/dev/app/
command=/path/to/pipenv run /path/to/gunicorn config.wsgi:application --bind 127.0.0.1:8005
Then you must restart your supervisord in order to load the settings.
sudo service supervisord reload

in your config file. change command= to bash -c followed by the full path and file to execute
this should do the trick

Django celery monitor doesn't show any tasks

I cannot see the tasks in admin.
I followed the steps in https://github.com/jezdez/django-celery-monitor
I used
celery==4.1.1
django-celery-results==1.0.1
django-celery-beat==1.0.1
django_celery_monitor==1.1.2
ran manage.py migrate celery_monitor The migrations went well. ran celery -A lbb events -l info --camera django_celery_monitor.camera.Camera --frequency=2.0 and celery -A lbb worker -l info in separate shells. But still cannot see the tasks I ran in celery-monitor > tasks table.

Running celery command with -E to force event worked for me.
celery -A proj worker -l info -E

How can I configure celery to run on startup of nginx?

I have celery running locally by just running celery -A proj -l info (although I don't even know if I should be using this command in production), and I want to get celery running on my production web server every time nginx starts. The init system is systemd

Create a service file like this celery.service
[Unit]
Description=celery service
After=network.target
[Service]
PIDFile=/run/celery/pid
User=celery
Group=celery
RuntimeDirectory=/path/to/project
WorkingDirectory=/path/to/project
ExecStart=celery -A proj -l info
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
Restart=on-abort
PrivateTmp=true
[Install]
WantedBy=multi-user.target
Move the file to /etc/systemd/system/ and next time your restart server, celery will be started by systemd on boot.

How do we run below celery commands in production server?

celery -A app worker -Q priority_high -B -l debug --purge -n priority_high_worker
celery -A app worker -Q default -B -l debug --purge -n default_worker
celery -A app beat -l info
As of now we are running the three commands in screens. What is the more production way of running these commands?

The easiest way to create daemons is with supervisord. sentry, which also uses django and celery recommends using supervisord to run the workers - you can adjust the configuration to suit your set up:
[program:celery-priority-high]
directory=/www/my_app/
command=/path/to/celery -A app worker -Q priority_high -B -l debug --purge -n priority_high_worker
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=syslog
stderr_logfile=syslog
You can, of course, also run django itself using this method.
If supervisord is too much bloat for your needs, you can also create init scripts for your init system of choice (eg. systemd).