How do I get data(all I really need is the state of the task) from a Celery backend? I am using Redis.
Assuming that you configured the CELERY_RESULT_BACKEND to use redis ( see here ), then you can monitor your application using a variety of methods.
I believe that celeryctl should suffice..
Related
I have a Django application that uses Celery with Redis broker for asynchronous task execution. Currently, the app has 3 queues (& 3 workers) that connect to a single Redis instance for communication. Here, the first two workers are prefork-based workers and the third one is a gevent-based worker.
The Celery setting variables regarding the broker and backend look like this:
CELERY_BROKER_URL="redis://localhost:6379/0"
CELERY_RESULT_BACKEND="redis://localhost:6379/1"
Since Celery uses rpush-blpop to implement the FIFO queue, I was wondering if it'd be correct or even possible to use different Redis databases for different queues like — q1 uses database .../1 and q2 uses database .../2 for messaging? This way each worker will only listen to the dedicated database for that and pick up the task from the queue with less competition.
Does this even make any sense?
If so, how do you implement something like this in Celery?
First, if you are worried about the load, please specify your expected numbers/rates.
In my opinion, you shouldn't be concerned about the Redis capability to handle your load.
Redis has its own scale-out / scale-in capabilities whenever you'll need them.
You can use RabbitMQ as your broker (using rabbitMQ docker is dead-simple as well, you can see example) which again, has its own scale-out capabilities to support a high load, so I don't think you should be worried about this point.
As far as I know, there's no way to use different DBs for Redis broker. You can create different Celery applications with different DBs but then you cannot set dependencies between tasks (canvas: group, chain, etc). I wouldn't recommend such an option.
I have a python backend using celery and Redis to serve long-running tasks. The app has a front end build in Vuejs. For these long-running tasks, I need to provide a real-time update to the user about the status of that task. The approach I think of is to poll the status endpoint to using setTimeOut function in my js code. Is there any better approach to handle this kind of use case?
I currently have setup postgres with flask-sqlalchemy extension on AWS using ElasticBeanstalk (EB). Postgres is running using RDS. Now i want to setup some background tasks. I read about Celery, seems to fit the use case reasonably well.
I want to understand how I can set that up on AWS so that it talks to the same Database. For the actual queues I want to use Redis. The business logic for background process and what I have in flask-webserver is very intertwined. How would the deployment process look like (with or without EB). I am okay with setup a new instance if needed for celery and redis as long as I don't have to separate the business logic a lot.
Another hacky solution i have been thinking is to setup crons on a node that hit certain URLs in Flask application to execute background tasks. But I would rather have a more scalable solution.
I'm using Flask with a similar setup and I followed this answer:
How do you run a worker with AWS Elastic Beanstalk?
I also setup redis using this .config file:
https://gist.github.com/yustam/9086610
However, for my setup, I changed the command to be this:
command=/opt/python/run/venv/bin/python2.7 manage.py celery
My manage.py has:
#manager.command
def celery():
"""
Start the celery worker.
"""
with app.app_context():
return celery_main(['celery', 'worker'])
I'm interested in using the django-celery models to create and monitor recurring tasks. In particular, I am looking at creating recurring cron-like actions and starting/stopping them from the admin.
As I understand it, it is possible to use this only if I am also using Django's default DB as the celery broker. Is it ever going to be possible to use those models with a non-DB broker?
EDIT: To clarify, I am already using RabbitMQ as the broker. My question is: can I, while using RabbigMQ, still somehow use django-celery's models to dynamically create and manage recurring/scheduled tasks?
If you have AMQP installed you can just set in celeryconfig:
BROKER_URL = 'amqp://127.0.0.1//'
Or replace the ip above with the ip where the RabbitMQ server is running.
I have a bunch of Django requests which executes some mathematical computations ( written in C and executed via a Cython module ) which may take an indeterminate amount ( on the order of 1 second ) of time to execute. Also the requests don't need to access the database and are all independent of each other and Django.
Right now everything is synchronous ( using Gunicorn with sync worker types ) but I'd like to make this asynchronous and nonblocking. In short I'd like to do something:
Receive the AJAX request
Allocate task to an available worker ( without blocking the main Django web application )
Worker executes task in some unknown amount of time
Django returns the result of the computation (a list of strings) as JSON whenever the task completes
I am very new to asynchronous Django, and so my question is what is the best stack for doing this.
Is this sort of process something a task queue is well suited for? Would anyone recommend Tornado + Celery + RabbitMQ, or perhaps something else?
Thanks in advance!
Celery would be perfect for this.
Since what you're doing is relatively simple (read: you don't need complex rules about how tasks should be routed), you could probably get away with using the Redis backend, which means you don't need to setup/configure RabbitMQ (which, in my experience, is more difficult).
I use Redis with the most a dev build of Celery, and here are the relevant bits of my config:
# Use redis as a queue
BROKER_BACKEND = "kombu.transport.pyredis.Transport"
BROKER_HOST = "localhost"
BROKER_PORT = 6379
BROKER_VHOST = "0"
# Store results in redis
CELERY_RESULT_BACKEND = "redis"
REDIS_HOST = "localhost"
REDIS_PORT = 6379
REDIS_DB = "0"
I'm also using django-celery, which makes the integration with Django happy.
Comment if you need any more specific advice.
Since you are planning to make it async (presumably using something like gevent), you could also consider making a threaded/forked backend web service for the computational work.
The async frontend server could handle all the light work, get data from databases that are suitable for async (redis or mysql with a special driver), etc. When a computation has to be done, the frontend server can post all input data to the backend server and retrieve the result when the backend server is done computing it.
Since the frontend server is async, it will not block while waiting for the results. The advantage of this as opposed to using celery, is that you can return the result to the client as soon as it becomes available.
client browser <> async frontend server <> backend server for computations