We're using Supervisord to run workers started by our Gearman job server. To remove a job from the queue, we have to run :
$ sudo killall supervisord
as to kill all Supervisord subprocesses so the job doesn't spawn when removed, then
$ gearman -n -w -f FUNCTION_NAME > /dev/null
to remove the job completley from the server.
Is there a way to kill only one Supervisord subprocess instead of using killall? For instance, if we have multiple jobs running and a single job is running longer than it should, or starts throwing errors, how can we kill the subprocess and remove the job from the server without killing all subprocesses?
Yes: Use supervisorctl to interact with supervisord. If you need to do so programmatically, there's a web service interface.
Related
Using Python 3.6.1. I am emulating launching an airflow webserver from the command as a process using subprocess.Popen.
After doing some things, I later move to kill (or terminate) it.
webserver_process = subprocess.Popen(["airflow", "webserver"])
webserver_process.kill()
My understanding is that this will send a SIGKILL to the webserver, whose underlying gunicorn should shutdown immediately.
However, when I navigate to http://localhost:8080 I see that the webserver is still running. Similarly when I then run sudo netstat -nlp|grep 8080 (I am using UNIX, and airflow webserver launches on port 8080), I discover:
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN
It's only when I kill the process manually using sudo fuser -k 8080/tcp that it finally dies.
What's going on here?
The python process returned by airflow webserver command actually calls subprocess.Popen to start gunicorn in a subprocess.
You can test this by calling webserver_process.pid, you'll notice that it's a different pid from the gunicorn master process pid.
After I deploy my django project, all I need is touch uwsgi_touch file. And uwsgi will gracefully restart its workers. But what about celery? Now I just restart celery manually when code base of celery tasks is changed. But even if I do it manually I still can't be sure that I will not kill celery task.
Any solutions?
A better way to manage celery workers is to use supervisor
$ pip install supervisor
$ cd /path/to/your/project
$ echo_supervisord_conf > supervisord.conf
Add these to your supervisord.conf file
[program:celeryworker]
command=/path/to/celery worker -A yourapp -l info
stdout_logfile=/path/to/your/logs/celeryd.log
stderr_logfile=/path/to/your/logs/celeryd.log
Now start supervisor with supervisord command in your terminal & use supervisorctl to manage process.
To restart you can do
$ supervisorctl restart celeryworker
I've found answer in celery FAQ
http://docs.celeryproject.org/en/2.2/faq.html#how-do-i-shut-down-celeryd-safely
Use the TERM signal, and the worker will finish all currently
executing jobs and shut down as soon as possible. No tasks should be
lost.
You should never stop celeryd with the KILL signal (-9), unless you’ve
tried TERM a few times and waited a few minutes to let it get a chance
to shut down. As if you do tasks may be terminated mid-execution, and
they will not be re-run unless you have the acks_late option set
(Task.acks_late / CELERY_ACKS_LATE).
I am using celery in an uncommon way - I am creating custom process when celery is started, this process should be running all the time when celery is running.
Celery workers use this process for their tasks (details not needed).
I run celery from command line and everything is ok:
celery -A celery_jobs.tasks.app worker -B --loglevel=warning
But when I use celeryd to daemonize celery, there is no way to stop it.
Command celeryd stop tries to stop celery but never ends.
When I check process trees in both cases, there is the difference - when running from command line, the parent is obviously celery process (main process which has celery workers as childs). Killing (stopping) the parent celery process will stop all celery workers and my custom process.
But when running with celeryd, my custom process has parent /sbin/init - and calling celeryd stop is not working - seems like main celery process is waiting for something, or is unable to stop my custom process as it is not child process of celery.
I don't know much about processes and it is not easy to find information, because I don't know what I should search for, so any tips are appreciated.
I have had the same problem. I needed a quick solution, so I wrote this bash script
#/bin/bash
/etc/init.d/celeryd stop
sleep 10
export PIDS=`ps -ef | grep celery |grep -v 'grep' | awk '{print $2}'`
for PID in $PIDS; do kill -9 $PID; done;
If the process doesn't stop after 10 seconds, it's a long-time-to-stop candidate, so i decided to stop abruptly
I assume your custom process is not a child of any of your pool worker processes and need not be so.
I use supervisord instead of celeryd to daemonize workers. It can be used to daemonize other processes as well. Such as your custom processes.
In your case your supervisord.conf can have multiple sections. One for each celery worker node and one (or more) for your custom process(es).
When you kill the supervisord process (with -TERM) it will take care of terminating all the workers and your custom process as well. If you use -TERM, then you will need to make sure your custom processes handle them.
I create bash to run python script start_queue.sh
content of start_queue.sh
python /tmp/my_python_script.py &
It's work when I run it in terminal. But I want to manage it using supervisord since I have few django website already manage by supervisord.
But I just get the error when start
supervisor: couldn't exec /tmp/start_queue.sh: ENOEXEC
supervisor: child process was not spawned
This is how i config in supervisord
[group:scriptgroup]
programs=script_1
[program:script_1]
command=/tmp/start_queue.sh
directory=/tmp/
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/var/log/x.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=info
It's possible to manage a backgroup process by supervisord, What did i wrong here. Please help.
Add #!/bin/sh at the beginning of script.
You need to execute your shell script spawned. Spawn means when you kill it, it doesn't kill unless a set of cicumstances to be.
notes: supervisor is a python module that control proocessing.
For more see this.
I have written an Upstart job to run celery in my Ubuntu server. Here's my configuration file called celeryd.conf
# celeryd - runs the celery daemon
#
# This task is run on startup to run the celery daemon
description "run celery daemon"
start on startup
expect fork
respawn
exec su - trakklr -c "/app/trakklr/src/trakklr celeryd --events --beat --loglevel=debug --settings=production"
When I execute sudo service celeryd start, the celeryd process starts just fine and all the x number of worker process start fine.
..but when I execute, sudo service celeryd stop, it stops most of the processes but a few processes are left hanging.
Why is this happening? I'm using Celery 2.5.3.
Here's an issue from the Github tracker.
https://github.com/celery/django-celery/issues/142
I still use init.d to run celery so this may not apply. With that in mind, stopping the celery service sends the TERM signal to celery. This tells the workers not to accept new tasks but it does not terminate existing tasks. Therefore, depending on how long your tasks take to execute you may see tasks for some time after telling celery to stop. Eventually, they will all shut down unless you have some other problem.
I wasn't able to figure this out but it seemed to be an issue with my older celery version. I found this issue mentioned on their issue-tracker and I guess it points to the same issue:
https://github.com/celery/django-celery/issues/142
I upgraded my celery and django-celery to the 3.x.x versions and this issue was gone.