My djcelery tasks run locally but not remotely? - python

I have automated tasks working locally but not reomotely in my django app. I was watching a tutorial and the guy said to stop my worker. but before I did that I put my app in maintenance mode, that didn't work. then I ran
heroku ps:restart
that didn't work, then I ran
heroku ps:stop worker
which outputed
Warning: The dynos in your dyno formation will be automatically restarted.
then I ran
heroku ps:scale worker=1
and still nothing. I remind those who are reading this that it worked locally. What am I missing?
my procfile
web: gunicorn gettingstarted.wsgi --log-file -
worker: celery worker -A blog -l info.
While researching I'm seeing mentions of adding beat to the procfile. 2 mentions in fact but this was not discussed in the tutorial I watched. only time celery beat was mentioned is when I added this to the settings.py file
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
and just in case it makes a difference I'm using the djcelery gui to set periodic tasks not configuring the sceduler in the settings.py like I see in a majority of the examples.
If I run the task in my view and call it, it works. But it wont run if I set it up using djcelery


I read the docs and realized I had to add to my worker procfile
-B
so it now looks like this
celery -A proj worker -B -l info
after I made the change I did this
heroku ps:scale worker=0
then
git add .
git commit -am 'added -B'
git push heroku master
then I
heroku ps:scale worker=1
then so I could see the output from heroku
heroku logs -t -p worker
and created a schedule in my admin and it worked. I saw the output in the console. Hope this helps. NOTE it says it's not recomended for production not sure why but if you know or find out let me know


As you've read in the docs, using the -B option is not recommended for production use, you'd better run celery-beat as a different process.
So it's best practice to run it in the server like :
celery beat -A messaging_router --loglevel=INFO
And if you're using supervisor to keep your processes running, you'd add something like the following to your configuration file.
[program:api_beat]
command=/path/to/v_envs/v_env/bin/celery -A project beat --loglevel=info
autostart=true
autorestart=true
user=your_user (echo $USER)
directory=/path/to/project/root
stdout_logfile=/var/log/supervisor/beat.log
stderr_logfile=/var/log/supervisor/beat.log
redirect_stderr=true
environment=EVN_VAR1="whatever",
The reason for this is as the docs say
You can also start embed beat inside the worker by enabling workers -B option, this is convenient if you will never run more than one worker node, but it’s not commonly used and for that reason is not recommended for production use:
Consider you having more than 1 worker, in maintenance, you need to always be vary of which one of the celery workers you've run with the -B and that definitely can be burden.

Related

How do I get Celery to execute in production on Heroku?

I'm using Celery on my Django app.
I can't get the celery task to execute unless I run celery -A appname worker -l info --pool=solo in the terminal.
How do I get it execute when the site is in production?

You need to add a worker process to your Procfile, e.g.
web: gunicorn some_app.wsgi
worker: celery -A appname worker -l info --pool solo
After redeploying, scale up one or more workers, e.g. by running
heroku ps:scale worker=1

Two applications using celery scheduled tasks: "Received unregistered task" errors in one of the workers

The scenario:
Two unrelated web apps with celery background tasks running on same server.
One RabbitMQ instance
Each web app has its own virtualenv (including celery). Same celery version in both virtualenvs.
I use the following command lines to start a worker and a beat process for each application.
celery -A firstapp.tasks worker
celery -A firstapp.tasks beat
celery -A secondapp.tasks worker --hostname foobar
celery -A secondapp.tasks beat
Now everything seems to work OK, but in the worker process of secondapp I get the following error:
Received unregistered task of type 'firstapp.tasks.do_something'
Is there a way to isolate the two celery's from each other?
I'm using Celery version 3.1.16, BTW.

I believe I fixed the problem by creating a RabbitMQ vhost and configuring the second app to use that one.
Create vhost (and set permissions):
sudo rabbitmqctl add_vhost /secondapp
sudo rabbitmqctl set_permissions -p /secondapp guest ".*" ".*" ".*"
And then change the command lines for the second app:
celery -A secondapp.tasks -b amqp://localhost//secondapp worker
celery -A secondapp.tasks -b amqp://localhost//secondapp beat

Have to manually start worker on heroku?

In my Procfile I have the following:
worker: cd appname && celery -A appname worker -l info --app=appname.celery_setup:app
However, when my app submits a task it never happens, but I think the celery worker is at least sort of working, because
heroku logs --app appname
every so often gives me one of these:
2016-07-22T07:53:21+00:00 app[heroku-redis]: source=REDIS sample#active-connections=14 sample#load-avg-1m=0.03 sample#load-avg-5m=0.09 sample#load-avg-15m=0.085 sample#read-iops=0 sample#write-iops=0 sample#memory-total=15664884.0kB sample#memory-free=13458244.0kB sample#memory-cached=187136kB sample#memory-redis=566800bytes sample#hit-rate=0.17778 sample#evicted-keys=0
Also, when I open up bash by running
heroku run bash --app appname
and then type in
cd appname && celery -A appname worker -l info --app=appname.celery_setup:app
It immediately tells me the task has been received and then executes it. I would like to have this happen without me having to manually log in and execute the command - is that possible? Do I need a paid account on heroku to do that?

I figured it out. Turns out you also have to do
heroku ps:scale worker=1 --app appname
Or else you won't actually be running a worker.

Django celery tasks working very long

I use celery, supervisor and flower
supervisor conf:
command=/web/venv/bin/python manage.py celeryd -E -l info -c 2 --maxtasksperchild=1 -Ofair
directory=/web/
stdout_logfile=/web/log/
redirect_stderr=true
user=web
killasgroup=true
stopasgroup=true
Problem when start from python shell, task works very fast. If from celery, works very very long. I don't know what the problem. Maybe someone can help ?

you can achieve the same by adding celery from admin, that might solution
http://www.yourhost.com/admin/djcelery/

What is the correct way to leave gunicorn running?

I want to make a Flask+Nginx+Gunicorn deployment. I have Nginx setup and running and I run gunicorn as described in the docs:
gunicorn app:app
But when I logout of the server the gunicorn process exits? What is the correct way to make sure it stay running for Nginx to connect to, and restarts if it crashes?

Use --daemon option while running gunicorn.
Example:
gunicorn grand56.wsgi:application --name grand56 --workers 3 --user=root --group=root --bind=127.0.0.1:1001 --daemon

use --daemon to the binding command of gunicorn.
ex:
gunicorn --bind 0.0.0.0:8001 your_project.wsgi --daemon

I'd look into something like Supervisor.
Very useful tutorial can be found here https://www.codingforentrepreneurs.com/blog/hello-linux-setup-gunicorn-and-supervisor/

Try this:
nohup gunicorn app:app &

The key thing to note is that when you start the process from the command line it is a child of your terminal process (i. e. a child of bash). When you log out of the server your bash process is terminated - as are all its children.
You'll want to use whatever system you have in place to manage nginx also manage gunicorn (anything from init.d or Upstart scripts to specialized application process monitors like Monit, Supervisor, Bluepill, Foreman, etc.)

Pay attention to Sean.
However you can run it on the fly like this:
nohup gunicorn -c config.py </dev/null >/dev/null 2>&1 and it will no longer be dependent on the terminal connection. You could replace >/dev/null with something like >somelogfile if you want to save any output.
But for production use it is best to get it integrated into whatever tool you use for managing processes.

Supervisor is a great cross-platform solution for process management. It is very feature rich and (in my opinion) requires a lot more configuration than some of the vanilla Linux alternatives (upstart, sysv, systemd). You should definitely use something like this to start, monitor and (if need be) restart your process.
No matter what process manager you end up using, you can still very easily leave gunicorn "running improperly" (ie as root user). I think some of the important details left out by other answers are that you should probably have one (non-root) user own the gunicorn process which binds to a unix socket that is owned by that user and the nginx group and has permissions 770. With gunicorn, you specify a mask instead, so invert 770 into 007 and use the -m flag. This way, only gunicorn and nginx can read/write/execute to the socket and no port is needed. You can specify the user and group of your gunicorn process with the -u and -g flags, and it will create the socket with those owners. Whatever you end up using for process mgmt, for nginx/gunicorn, you probably want something like this in your startup script:
exec gunicorn wsgi:app -u gunicorn -g nginx -m 007 -b gunicorn.sock >> /var/log/$<service_name>.sys.log 2>&1
Make sure the gunicorn user has write permission on the log file. Then, in nginx, where you formerly had the ip/port (ie 0.0.0.0:5000), you put the path to the socket (ie /usr/share/nginx/html/gunicorn.sock). Notice I did not use the --daemon flag here, but I used exec, this assumes a process manager, which will run gunicorn as a child process with exec.
You can find all the different flags here.

I tried the systemd option and it worked fine, the link below has my full answer and has all the steps , to invoke your app as a gunicorn service.
https://askubuntu.com/questions/930589/running-upstart-script-on-17-04/1010398#1010398

Running hug api like this.
--daemon is to keep the process in background.
--access-logfile to keep request log
--bind=< ip>:< port> Giving IP will allow to access from other systems(If proxy is not needed).
gunicorn <pyscirpt_name>:__hug_wsgi__ --name caassist -w 4 --access-logfile /var/logs/gunicorn/gunicorn_access.log --daemon --bind=<ip>:<port>

Categories

Resources