I have a celery task calling a bash command (I can use os.system() or subprocess.Popen()).
When I call :
revoke(task_id, terminate=True)
on my task, the subprocess executing my bash command and created by my task is not killed. Is there a way to do that ?
According to the docs, a SIGTERM signal is sent when terminate=True.
http://docs.celeryproject.org/en/latest/userguide/workers.html#revoke-revoking-tasks
Since SIGTERM can be ignored, maybe try sending a SIGKILL?
Related
We are using celery, rabbitmq and ffmpeg-python to read video streams. In a celery task (shared task), we are calling ffmpeg-python which internally uses subprocess to run ffmpeg. Whenever we revoke tasks in celery, the ffmpeg processes become defunct/zombie. Over a time they start getting accumulated and exhausting our pids. Is there any way to gracefully exit the celery task along with its subprocess?
Does this SO answer help you?
Quote:
from celery import Celery
celery = Celery('vwadaptor', broker='redis://workerdb:6379/0',backend='redis://workerdb:6379/0')
celery.control.broadcast('shutdown', destination=[<celery_worker_name>])
[EDIT]
Alternatively, here is a python module that provides warm and cold shutdown behaviour on Celery. Disclaimer: I haven't used it.
I'm developing a ADB client for python, i am planning to invoke adb binary with sub process to get the information.
Here is how i tried to invoke it, to start the adb server.
check_output([ 'adb.exe','start-server'],stderr=STDOUT)
I do see the adb running, but the program is getting stuck after that.
I have tried with shell=True, but that didn't affect it.
When i kill adb from task manager, the program does exit, and prints the right ouput.
How can i fix this, I assume that the command doesn't exit since the daemon is running ?
I was able to overcome this by starting the command in a separate thread, and using the current thread with other adb commands, as they return immediately.
Is there a more elegant solution ?
You can do that with subprocess.Popen.
import subprocess
adb = subprocess.Popen(['adb.exe', 'start-server'])
# Do some other stuff while adb is running...
adb.terminate() # Kill the process once you're done
This also has some advantages, like the possibility of giving input to the process through stdin, by using Popen.communicate()
I start Django server with python manage.py runserver and then quit with CONTROL-C, but I can still access urls in ROOT_URLCONF, why?
Probably you left another process running somewhere else.
Here is how you can list all processes whose command contains manage.py:
ps ax | grep manage.py
Here is how you can kill them:
pkill -f manage.py
Without seeing your script, I would have to say that you have blocking calls, such as socket.recv() or os.system(executable) running at the time of the CTRL+C.
Your script is stuck after the CTRL+C because python executes the KeyboardInterrupt AFTER the the current command is completed, but before the next one. If there is a blocking function waiting for a response, such as an exit code, packet, or URL, until it times out, you're stuck unless you abort it with task manager or by closing the console.
In the case of threading, it kills all threads after it completes its current command. Again, if you have a blocking call, the thread will not exit until it receives its response.
just type exit(), that is what I did and it worked
I am using celery in an uncommon way - I am creating custom process when celery is started, this process should be running all the time when celery is running.
Celery workers use this process for their tasks (details not needed).
I run celery from command line and everything is ok:
celery -A celery_jobs.tasks.app worker -B --loglevel=warning
But when I use celeryd to daemonize celery, there is no way to stop it.
Command celeryd stop tries to stop celery but never ends.
When I check process trees in both cases, there is the difference - when running from command line, the parent is obviously celery process (main process which has celery workers as childs). Killing (stopping) the parent celery process will stop all celery workers and my custom process.
But when running with celeryd, my custom process has parent /sbin/init - and calling celeryd stop is not working - seems like main celery process is waiting for something, or is unable to stop my custom process as it is not child process of celery.
I don't know much about processes and it is not easy to find information, because I don't know what I should search for, so any tips are appreciated.
I have had the same problem. I needed a quick solution, so I wrote this bash script
#/bin/bash
/etc/init.d/celeryd stop
sleep 10
export PIDS=`ps -ef | grep celery |grep -v 'grep' | awk '{print $2}'`
for PID in $PIDS; do kill -9 $PID; done;
If the process doesn't stop after 10 seconds, it's a long-time-to-stop candidate, so i decided to stop abruptly
I assume your custom process is not a child of any of your pool worker processes and need not be so.
I use supervisord instead of celeryd to daemonize workers. It can be used to daemonize other processes as well. Such as your custom processes.
In your case your supervisord.conf can have multiple sections. One for each celery worker node and one (or more) for your custom process(es).
When you kill the supervisord process (with -TERM) it will take care of terminating all the workers and your custom process as well. If you use -TERM, then you will need to make sure your custom processes handle them.
I have a python script that does some jobs. I use multiprocessing.Pool to have a few workers do some commands for me.
My problem is when I try to terminate the script. When I press Ctrl-C, I would like, that every worker immediately cleans up its experiment (which is some custom code, or actually even a subprocess command, not just releasing locks or memory) and stops.
I know that I can catch Ctrl-C with the signal handler. How can I make all current running workers of a multiprocessing.Pool to terminate, still doing their cleanup command?
Pool.terminate() will not be useful, because the processes will be terminated without cleaning up.
How about trying the atexit standard module?
It allows you to register a function that will be executed upon termination.
Are you working with Unix? If yes, why not catch SIGTERM in the subprocesses? In fact, the documentation of Process.terminate() reads:
Terminate the process. On Unix this is done using the SIGTERM signal
(I have not tested this.)