I started to use celery flower for tasks monitoring and it is working like a charm. I have one concern though, how can i "reload" info about monitored tasks after flower restart ?
I use redis as a broker, and i need to have option to check on tasks even in case of unexpected restart of service (or server).
Thanks in advance
I found i out.
It is the matter of setting the persistant flag in command running celery flower.
Related
i have celery running on few computers and using flower for monitoring.
the computers is used by different people.
celery beat is generating jobs for all the workers from one of the computer.
every time new coded task is ready, all the workers less the beat-computer will have task not registered exception.
what is the recommended direction to sync all the code to all other computers in the network, is there a prehook kind of mechanism in celery to check for new code?
Unfortunately, you need to update the code on all the workers (nodes) and after that you need to restart all of them. This is by (good) design.
A clever systemd service could in theory be able to
send the graceful shutdown signal
run pip install -U your-project
start the Celery service
I would like to run APScheduler which is a part of WSGI (via Apache's modwsgi with 3 workers) webapp. I am new in WSGI world thus I would appreciate if you could resolve my doubts:
If APScheduler is a part of webapp - it becomes alive just after first request (first after start/reset Apache) which is run at least by one worker? Starting/resetting Apache won't start it - at least one request is needed.
What about concurrent requests - would every worker run same set of APScheduler's tasks or there will be only one set shared between all workers?
Would once running process (webapp run via worker) keep alive (so APScheduler's tasks will execute) or it could terminate after some idle time (as a consequence - APScheduler's tasks won't execute)?
Thank you!
You're right -- the scheduler won't start until the first request comes in.
Therefore running a scheduler in a WSGI worker is not a good idea. A better idea would be to run the scheduler in a separate process and connect to the scheduler when necessary via some RPC mechanism like RPyC or Execnet.
I have a unit-tests for my django project.
Some of views in my django project run celery tasks and I want to check database after these tasks.
I have a separated tests for the celery tasks, where I call them without .delay() method.
The main problem, what is the best and cleanest way to have a celery worker during the jenkins job?
Currently I just run nohup celery -A myqpp worker & before test and kill all running celery at the end of the job.
The best and cleanest way is not to have any celery workers during the Jenkins job, neither any queue/result backend. Utilize CELERY_ALWAYS_EAGER setting to execute your tasks in unit tests locally by blocking until the task returns.
Check out more in Celery documentation: CELERY_ALWAYS_EAGER docs
Just to extend answer about always eager mode, you can see my answer on other question, how you can run celery worker from test setUp https://stackoverflow.com/a/42107423/590233
But few tings need to be done there:
Connect celery worker to test db
Somehow run message broker instance ... (i think that you run it already before test, but cleanest way is to spawn broker instance from setUp as an celery worker)
I start the worker by executing the following in the terminal:
celery -A cel_test worker --loglevel=INFO --concurrency=10 -n worker1.%h
Then I get a long looping error message stating that celery has received an unregistered task and has triggered:
KeyError: 'cel_test.grp_all_w_codes.mk_dct' #this is the name of the task
The problem with this is that cel_test.grp_all_w_codes.mk_dct doesn't exist. In fact there isn't even a module cel_test.grp_all_w_codes let alone the task mk_dct. There was once a few days ago but I've since deleted it. I thought maybe there was a .pyc file floating around but there isn't. I also can't find a single reference in my code to the task that's throwing the error. I shut down my computer and restarted the rabbitmq server thinking maybe a reference to something was just stuck in memory but it did not help.
Does anyone have any idea what could be the problem here or what I'm missing?
Well, without knowing your conf files, I can see two reasons that would provoke this:
the mk_dct task wasn't completed when you stopped the worker and delete the module. If you're running with CELERY_ACKS_LATE, it will try to relaunch the task everytime you re run the worker. Try remove this setting, or launch the worker with the purge option.
celery -A cel_test worker --loglevel=INFO --concurrency=10 -n worker1.%h --purge
the mk_dct task is launched by your celery beat. If so, try relaunching celery beat and clearing it's database backend if you had a custom one.
If it does not solve the problem, please post your celery conf, and make sure you have cleaned all the .pyc of your project and restarted everything.
I'm running celery and celery flower with redis as a broker. Everything boots up correctly, the worker can find jobs from redis, and the celery worker completes the jobs successfully.
The issue I'm having is the Broker tab in the celery flower web UI doesn't show any of the information from Redis. I know the Redis url is correct, because it's the same URL that celeryd is using. I also know that the celery queue has information in it, because I can manually confirm that via redis-cli.
I'm wondering if celery flower is trying to monitor a different queue in the Broker tab? I don't see any settings in the flower documentation to override or confirm. I'm happy to provide additional information upon request, but I'm not certain what is relevant.
Turns out I needed to start Celery Flower with both the broker and broker_api command line arguments:
celery flower --broker=redis://localhost:6379/0 --broker_api=redis://localhost:6379/0
Hope this helps someone else.
For AMQP this is an example.
/usr/bin/celery -A app_name --broker=amqp://user:pw#host//vhost --broker_api=http://user:pw#host:host_port/api flower
The broker_api is the rabbitmq web ui endpoint with /api
rabbitmq-plugins enable rabbitmq_management
that was help me from http://flower.readthedocs.org/en/latest/config.html?highlight=broker_api#broker-api
Faced the same issue with RabbitMQ. Here is how I have it works:
rabbitmq:
image: rabbitmq:3-management
flower:
image: mher/flower
ports:
- 5555:5555
command:
- "celery"
- "--broker=amqp://guest#rabbitmq:5672//"
- "flower"
- "--broker_api=http://guest:guest#rabbitmq:15672/api//"
depends_on:
- rabbitmq
Brokers & other tabs will show up.