I am implementing an online server using:
Flask
NGINX
Celery
Celery uses:
RabbitMQ as a broker
Redis as a Result backend.
I would like to know if it is possible to use Redis as a cache to avoid doing big calculations if I receive the same request. For example, I want to answer a cache result if I receive a POST containing the same body.
If it is possible, do I have to configure it in Celery or in Redis? And how should I do it?
There are many existing extensions in the flask eco-system that let you do this easily. Including Flask-Redis
Related
I have a FastAPI api code that is executed using uvicorn. Now I want to add a queu system, and I think Celery and Flower can be great tools for me since my api has some endpoints that uses a lot CPU and take some seconds in answering. However, I have a couple of questions about the addition of Celery:
Does Celery substitute Uvicorn? Do I need it any more? I cannot see any example on the website where they consider uvicorn too, and when you execute the Celery seems to do not need it...
I have read a lot about using Celery for creating a queu for FastAPI. However, you can manage a queue in FastAPI without using Celery. What's better? and why?
Does Celery substitute Uvicorn?
No. Celery is not a replacement for Uvicorn. Uvicorn is meant to run your FastAPI application, Celery will not do that for you.
I have read a lot about using Celery for creating a queu for FastAPI. However, you can manage a queue in FastAPI without using Celery. What's better? and why?
I guess you mean the BackgroundTasks here, but that is not a replacement for Celery. FastAPI BackgroundTasks are meant to execute simple tasks (and not CPU bound related tasks).
Answering the question, ideally, you'd have to start both services: Uvicorn, and Celery. You can see an example on how to do it here.
Not that it matters much here, but I'm one of the Uvicorn maintainers.
In my Django project I implemented Celery, which is running using RabbitMQ backend.
What celery does:
Celery puts my tasks into queue, and then under certain conditions runs them. That being said, I basically interact with RabbitMQ message queue exclusively using Celery python interface.
Problem:
I want just to push simple string message into RabbitMQ queue, which should be consumed by 3rd party application.
What I tried:
There is a way how to directly connect to RabbitMQ using Pika library. However I would find it a little clunky - If I have already Celery connected to RabbitMQ, why not use it (if possible) to send simple messages to a specific queue, instead opening another connection using mentioned Pika library.
Any insights appreciated.
You cannot use Celery to send arbitrary messages to your RabbitMQ server.
However, considering that you already use RabbitMQ as a broker, which means you already have all the necesary RabbitMQ support (py-amqp supports it either directly, or via librabbitmq), you can easily send messages to the MQ server from your Celery tasks. If you for whatever reason do not like py-amqp, you may use Pika as you mentioned already.
I want to use GCP cloud run as a technology to run my python flask app, so I have to dockerize it. Most of examples I've seen are either using built in flask server or gunicorn server as an ENTRYPOINT, which gives a warning on a console, that it shouldn't be used on production.
My question is: does it matter with a platform like GCP cloud run which server do I use to run that code? What would be the performance impact of that choice?
You want gunicorn, and you'll need to configure it correctly.
Typically in these setups there will be an external HTTP server proxying requests to your server. So it matters rather less which webserver you're using on the backend, because it's not directly exposed.
That being said, the built-in Flask webserver isn't ideal, so gunicorn would probably be better. You will need to tweak Gunicorn's settings slightly to work correctly in container: logging, heartbeat setting, and parallelism.
See https://pythonspeed.com/articles/gunicorn-in-docker/ for details.
I have a follow-on / clarification question related to an older question
I have 2 servers (for now). 1 server runs a django web application. The other server runs pure python scripts that are CRON-scheduled data acquisition & processing jobs for the web app.
There is a use case where user activity in the web application (updating a certain field) should trigger a series of actions by the backend server. I could stick with CRON but as we scale up, I can imagine running into trouble. Celery seems like a good solution except I'm unclear how to implement it. (Yes, I did read the getting started guide).
I want the web application to send tasks to a specific queue but the backend server to actually execute the work.
Assuming that both servers are using the same broker URL,
Do I need to define stub tasks in Djando or can I just use the celery.send_task method?
Should I still be using django-celery?
Meanwhile the backend server will be running Celery with the full implementation of the tasks and workers?
I decided to try it and work through any issues that came up.
On my django server, I did not use django-celery. I installed celery and redis (via pip) and followed most of the instructions in the First Steps with Django:
updated proj/proj/settings.py file to include the bare minimum of
configuration for Celery such as the BROKER_URL
created the proj/proj/celery.py file but without the task defined
at the bottom
updated the proj/proj/__init__.py file as documented
Since the server running django wasn't actually going to execute any
Celery tasks, in the view that would trigger a task, I added the
following:
from proj.celery import app as celery_app
try:
# send it to celery for backend processing
celery_app.send_task('tasks.mytask', kwargs={'some_id':obj.id,'another_att':obj.att}, queue='my-queue')
except Exception as err:
print('Issue sending task to Celery')
print err
The other server had the following installed: celery and redis (I used an AWS Elasticache redis instance for this testing).
This server had the following files:
celeryconfig.py will all of my Celery configuration and queues
defined, pointing to the same BROKER_URL as the django server
tasks.py with the actual code for all of my tasks
The celery workers were then started on this server, using the standard command: celery -A tasks worker -Q my-queue1,my-queue2
For testing, the above worked. Now I just need to make celery run in the background and optimize the number of workers/queue.
If anyone has additional comments or improvements, I'd love to hear them!
I am working on a web application that uses a permanent object MyService. Using a web interface I am dynamically updating its state and monitor its behavior. Now I would like to periodically call one of its methods. I was thinking of using celery PeriodicTask but run into some scope issues. It seems I need to execute three different processes:
python manage.py runserver
python manage.py celery worker
python manage.py celerybeat
The problem is that even if I ensure that MyService is a singleton that can be safely used by more than one thread, celery creates its own fresh copy of the object. Is there a way I could share this object between both django server and celery main process? I tried to find a way to start celery from within django script but until now with no success. Would appreciate any help.
If you need to share something between multiple processes or maybe even multiple machines (eg. your workers could run on a seperate machine) the best (and probably easiest) practice to share information would be using an external service.
In the simplest case you could use Django's DB, but if you encounter that this is not suitable for you, for example if you have a heavy write load you can use something like Redis or Memcache (which you can also talk to via Django's caching API). These will enable you to be able to handle a big write load and besides you can use eg. Redis as a queue for celery as well.