I have API code which adds tasks to queue and then celery workers consuming those tasks.
Currently I have both code base same. But i want celery workers just to have simple plain Python tasks and no django code as workers will only be processing tasks and does not need django for that. Is it possible.
In order to start celery worker i need to use this line
celery -A django_project worker --queue high
What should i write instead of django_project there
Yes you can. Celery is a generic asynchronous task queue. In place of "django_project" you would point to your module. See http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#application for an example.
Here is an example project layout using celery:
project-dir/
mymodule/
__init__.py
celery.py
tasks.py
tests/
setup.py
etc, etc (e.g. tox.ini, requirements.txt, project management files)
In mymodule/celery.py:
# -*- coding : utf-8 -*-
from __future__ import absolute_import
from celery import Celery
app = Celery('mymodule',
broker='amqp://',
backend='amqp://',
include=['mymodule.tasks'])
if __name__ == '__main__':
app.start()
In mymodule/tasks.py:
from __future__ import absolute_import
from mymodule.celery import app
#app.task
def add(x, y):
return x + y
You can definitely use Celery without using any web framework like Django or Flask. Just create the Celery object and your tasks accordingly and run the following command
celery -A filename.celery_object_name worker --loglevel=info
Later, just run the Python file. You don't need to set anything. It works exactly with or without any Web Framework.
Related
My original problem is that when running celery worker with --detach flag, or using celery multi, my application tasks are not registered with the workers ( although the workers do startup and are reachable, same is this question ). To help debug this problem I have made a plain celery app, which has a different but maybe related issue.
Source structure
setup.py
example
| tasks.py
| celery.py
| __init__.py
| __main__.py
tasks.py:
from example.celery import app
#app.task
def add(x, y):
return x + y
celery.py
from celery import Celery
app = Celery('tasks', broker='redis://localhost:6379', include=["example.tasks"])
init.py
__version__ = "0.0.1"
__package__ = "example"
from example.celery import app
setup.py
from setuptools import setup
import example
setup(
name=example.__package__,
version=example.__version__,
include_package_data=True,
python_requires=">=3.7",
)
Install the package using
$ pip install -e .
From anywhere in the system I can now run
$ celery -A example worker
And I will have a worker with the add task. Adding --detach flag like this:
$ celery -A example.celery.app worker --detach --logfile=$HOME/celery.log
Will give an error trying to connect to AMQPLAIN into the celery.log file, when the app is configured to use Redis:
...
amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
So my understanding is that something is going off when the worker tries to load the celery app in detach mode, but I can't figure out what. Any assistance is greatly appreciated.
Using celery==4.4.7
Update
This works with celery 5.0.2, will be opening a bug ticket.
This is a known bug which I missed.
https://github.com/celery/celery/issues/6370
Why is this happening?
My celery.py:
import os
from celery import Celery
from django.conf import settings
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myshop.settings')
app = Celery('myshop')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
my init.py
# import celery
from .celery import app as celery_app
I even tried renaming celery.py to something else and the error still persisted. Could it be because of my python version?
I`ll post answer in order to move it from comments.
First of all in your __ init__.py file add this line
from __future__ import absolute_import, unicode_literals
Second of all you need to add to your settings, information about brooker.
This is an example configuration file to get you started. It should contain all you need to run a basic Celery set-up.
Broker settings.
broker_url = 'amqp://guest:guest#localhost:5672//'
The next thing is running your celery worker. So if you celery app is named myshop you have to run celery worker (using your environment), by typing this simple command:
celery -A myshop worker -l info
Then try to run your task, and everything should be fine.
its becuase of version
How u have installed celery
pip install celery==3.0.19
if this
then run python by
python manage.py runserver
or
pip3 install celery==3.0.19
if this
then run python by
python3 manage.py runserver
I am working on Django based web app. During unittest, I need to write a test which needs "Celery worker" running in the background.
I have already used:
CELERY_EAGER_PROPAGATES_EXCEPTIONS=True
CELERY_ALWAYS_EAGER=True
BROKER_BACKEND='memory
In over_ride settings, but these are not running celery worker for me in background when needed.
Any help would much appreciated.
Celery won't get run by Django automatically.
You can start a worker process by running from your project root:
$ celery -A my_proj worker
my_proj should be the application name you configured with app = Celery('my_proj')
Here's my celery app config:
from __future__ import absolute_import
from celery import Celery
import os
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tshirtmafia.settings')
app = Celery('tshirtmafia')
app.conf.update(
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
)
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
settings.py:
INSTALLED_APPS:
'kombu.transport.django',
'djcelery',
also:
BROKER_URL = 'django://'
Here's my task:
#shared_task
def test():
send_mail('nesamone bus', 'Files have been successfully generated.', 'marijus.merkevicius#gmail.com',
['marijus.merkevicius#gmail.com'], fail_silently=False)
Now when I run locally python manage.py celeryd locally and then run test.delay() from shell locally it works.
Now I'm trying to deploy my app. When with the exact same configuration I try to open python manage.py celeryd and in other window I open shell and run test task, it doesn't work.
I've also tried to setup background daemon like this:
/etc/default/celeryd configuration:
# Name of nodes to start, here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"
# Where to chdir at start. (CATMAID Django project dir.)
CELERYD_CHDIR="/home/tshirtnation/"
# Python interpreter from environment. (in CATMAID Django dir)
ENV_PYTHON="/usr/bin/python"
# How to call "manage.py celeryd_multi"
CELERYD_MULTI="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryd_multi"
# How to call "manage.py celeryctl"
CELERYCTL="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryctl"
# Extra arguments to celeryd
CELERYD_OPTS="--time-limit=300 --concurrency=1"
# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"
# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.
CELERYD_USER="celery"
CELERYD_GROUP="celery"
# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="settings"
And I use default celery /etc/init.d/celeryd script.
So basically it seems like celeryd starts but doesn't work. No idea how to debug this and what might be wrong.
Let me know if you need anything else
Celery turned to be a very capricious child in Django robust system as for me.
There are too little initial data for understanding the reason of your problems.
The most usual reason of Celery daemon fail is file system permissions.
But to clarify the reason I'd try:
Start celery from a command line by the user-owner of django project:
celery -A proj worker -l info
If it works OK, go further
Start celery in a verbal mode as a root user just like daemon to be:
sudo sh -x /etc/init.d/celeryd start
This will show most of the problems with the daemon script - celery user and group used, but not all, unfortunately: permission fails are not visible.
My little remark.
Usually Celery is started by own celery user, and the django project by another one. After long fighting celery and system, I refused from celery user, and owned celery process by the django project user.
And .. do not forget to start once
update-rc.d celerybeat defaults
update-rc.d celeryd defaults
this is for Ubuntu daemon start, sure.
Good luck
I am new to redis and celery. I have gone through the basic tutorial of both, but I am not getting how to implement then in task scheduling job
I am unable to start with the scripting part. I am not getting how to write a script to make a queue, run the workers etc. I would need a practical example
So here's a cannonical example of how can celery run with Redis (let the script filename be mytasks.py):
from celery import Celery
celery = Celery('tasks', broker='redis://localhost:6379/0')
#celery.task
def add(x, y):
return x + y
As you see, broker argument was set to use Redis installed on your local machine. The next thing is to start celery server:
$ celery -A mytasks worker --loglevel=info
As your tasks celery server has been started, you can now use it to run your task just by importing mytasks script, e.g from Python interpreter interactive mode:
>>> from mytasks import add
>>> add.delay(1, 1)
2
After some time '2' will appear in console.
That's a basic example of how you can setup your tasks execution environment.