file structure
proj/proj/
celery.py
(and other files)
/sitesettings/
tasks.py
(and other files)
celery.py
app = Celery('mooncake',broker_url = 'amqp://')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
sitesettings/tasks.py
from __future__ import absolute_import, unicode_literals
from comma.models import Post
from mooncake.celery import app
app.conf.beat_schedule = {
'every-5-seconds': {
'task': 'sitesettings.tasks.statisticsTag',
'schedule': 5.0,
'args': ()
},
}
#app.task
def statisticsTag():
print(Post.objects.all()[0])
and run it with
celery -A proj beat -l info
it out put with
[2019-02-22 18:21:08,346: INFO/MainProcess] Scheduler: Sending due task every-5-seconds (sitesettings.tasks.statisticsTag)
but no further output.
I used to try write it in proj/celery.py, but it cannot run cuz I have to import from another app, it exit with "app not loaded" error. So what should I do?
The command you are calling to start celery celery -A proj beat -l info is starting a beat scheduler instance of celery which sends due tasks to a worker instance.
You will also need to start a worker server that will execute those due tasks. You can start a celery worker with the command celery -A proj worker -l info. This will need to be running at the same time as your scheduler is running.
Alternatively you can run a worker with embedded beat scheduler celery -A proj worker -B -l info but that is not recommended for production use.
Related
When I run celery -A app.celery worker --loglevel=INFO --pidfile='' I get back the following:
Usage: celery [OPTIONS] COMMAND [ARGS]...
Error: Invalid value for '-A' / '--app':
Unable to load celery application.
'nonetype' object has no attribute '_instantiate_plugins'
To the best of my understanding, in celery -A [name].celery... [name] should be the file where the Celery instance is created and held, which in my case is app.py.
This is my first time working with Celery, so would love help here!
My file structure is as follows:
--app
-- app.py
-- celery_config
-- __init__.py
-- flask_celery.py
app.py
from flask import Flask
from celery_config.flask_celery import make_celery
# Create app
app = Flask(__name__)
app.config.from_envvar("APP_SETTINGS")
...
# Setup Celery
celery = make_celery(app)
flask_celery.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
def make_celery(app):
celery = Celery(app.import_name)
celery.conf.update(
result_backend=app.config["CELERY_RESULT_BACKEND"],
broker_url=app.config["CELERY_BROKER_URL"],
timezone="UTC",
task_serializer="json",
accept_content=["json"],
result_serializer="json"
)
TaskBase = celery.Task
class ContextTask(TaskBase):
abstract = True
def __call__(self, *args, **kwargs):
with app.app_context():
return TaskBase.__call__(self, *args, **kwargs)
celery.Task = ContextTask
return celery
I figured it out.
Turns out since I was running the command from inside the app directory I needed to run celery -A celery worker --loglevel=INFO --pidfile='' rather than celery -A app.celery worker --loglevel=INFO --pidfile='' -- -A app searches for celery within the directory app, but I was already in that directory. I only realized this after finding this GitHub comment.
UPDATE: This was also not the answer, the issue was that I was expecting running the Celery worker to pick up my .env variables, which it doesn't do because it's not a Flask-specific package. I had to export my .env variables into my local environment because it was trying to instantiate the app's database without the DATABASE_URL variable. celery -A app.celery worker --loglevel=INFO --pidfile='' was the right command.
I have a Django application where I defined a few #task functions under task.py to execute at given periodic task. I'm 100% sure that the issue is not caused by task.py or any code related but due to some configuration may be in settings.py or my celery worker.
Task does execute at periodic task but at multiple times.
Here are the celery worker logs:
celery -A cimexmonitor worker --loglevel=info -B -c 4
[2019-09-19 21:22:16,360: INFO/ForkPoolWorker-5] Project Monitor Started : APPProject1
[2019-09-19 21:22:16,361: INFO/ForkPoolWorker-4] Project Monitor Started : APPProject1
[2019-09-19 21:25:22,108: INFO/ForkPoolWorker-4] Project Monitor DONE : APPProject1
[2019-09-19 21:25:45,255: INFO/ForkPoolWorker-5] Project Monitor DONE : APPProject1
[2019-09-20 00:22:16,395: INFO/ForkPoolWorker-4] Project Monitor Started : APPProject2
[2019-09-20 00:22:16,398: INFO/ForkPoolWorker-5] Project Monitor Started : APPProject2
[2019-09-20 01:22:11,554: INFO/ForkPoolWorker-5] Project Monitor DONE : APPProject2
[2019-09-20 01:22:12,047: INFO/ForkPoolWorker-4] Project Monitor DONE : APPProject2
If you check above time interval, tasks.py executes one task but 2 workers of celery takes the task & executes the same task at the same interval. I'm not sure why 2 workers took for one task?
settings.py
..
..
# Internationalization
# https://docs.djangoproject.com/en/2.1/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'Asia/Kolkata'
USE_I18N = True
USE_L10N = True
USE_TZ = True
..
..
..
######## CELERY : CONFIG
CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ENABLE_UTC = True
CELERYBEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'
celery.py
from __future__ import absolute_import, unicode_literals
from celery import Celery
import os
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE','cimexmonitor.settings')
## set the default Django settings module for the 'celery' program.
# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app = Celery('cimexmonitor')
#app.config_from_object('django.conf:settings', namespace='CELERY')
app.config_from_object('django.conf:settings')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks(settings.INSTALLED_APPS)
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
Other information:
→ celery --version
4.3.0 (rhubarb)
→ redis-server --version
Redis server v=3.0.6 sha=00000000:0 malloc=jemalloc-3.6.0 bits=64 build=7785291a3d2152db
django-admin-interface==0.9.2
django-celery-beat==1.5.0
Please help me the ways to debug the problem:
Thanks
Both the worker and beat services need to be running at the same time to execute periodically task as per https://github.com/celery/django-celery-beat
WORKER:
$ celery -A [project-name] worker --loglevel=info -B -c 5
Django scheduler:
celery -A [project-name] beat -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler
I was running both worker,database scheduler at same time, which was said as per the documentation, Which was causing the issues to be executed at the same time, I'm really not sure how celery worker started working as a DB scheduler at the same time.
just running celery worker solved my problem.
From the official documentation: Ensuring a task is only executed one at a time.
Also, I hope you are not running multiple workers the same way (celery -A cimexmonitor worker --loglevel=info -B -c 4) as that would mean you have multiple celery beats scheduling tasks to run... In short - make sure you only have one Celery beat running!
I have django 1.11.4 app and use celery 4.1.0 for background periodic tasks. Celery was daemonized according to documentation and was working fine until... I don't know what happened, basically. It suddenly got broken.
When I execute /etc/init.d/celerybeat start it writes following exception to /var/log/celery/beat.log and halts:
[2017-09-04 18:33:38,485: INFO/MainProcess] beat: Starting...
[2017-09-04 18:33:38,485: INFO/MainProcess] Writing entries...
[2017-09-04 18:33:38,486: CRITICAL/MainProcess] beat raised exception <class 'django.db.utils.InterfaceError'>: InterfaceError
("(0, '')",)
Traceback (most recent call last):
File "/home/hikesadmin/.local/lib/python3.4/site-packages/kombu/utils/objects.py", line 42, in __get__
return obj.__dict__[self.__name__]
KeyError: 'scheduler'
Here is the full log: https://pastebin.com/92iraMCL
I've remove all my tasks and retained simple celery.py tasks file:
import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myapp.settings')
import django
import pymysql
pymysql.install_as_MySQLdb()
django.setup()
from celery import Celery
app = Celery('myapp')
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
#app.task
def gogo():
print("GOGO")
print(123123123)
But celerybeat still does not work. It prints "123123123" and after that halts with same exception.
I've researched deeper and figured out that the problem is in the --detach modifier. When I launch it without it, it works:
/usr/local/bin/celery beat --app=hike_engine -S django -f /var/log/celery/beat.g -l INFO --workdir=/home/hikesadmin/engine --pidfile=/var/run/celery/beat.pid
When I add --detach, celery goes broken.
Please help me to trace and fix the problem. Thanks!
I don't think you can start a detached beat instance via --detach IIRC. We are using supervisor to start/stop/restart our beat instance:
Add /etc/supervisor/conf.d/celery-beat.conf
[program:celery-beat]
command=/path/to/env/bin/celery beat -A your.settings.module --loglevel=INFO --workdir /path/to/your/django/project --pidfile=/var/tmp/celery-beat.pid
directory=/path/to/your/django/project
killasgroup=true
stopasgroup=true
user=YOURUSERID
group=YOURUSERGROUP
numprocs=1
stdout_logfile=/var/log/celery/beat.log
stderr_logfile=/var/log/celery/beat.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600
priority=998
then update the configuration
supervisorctl reread
and finally start the supervised celery beat instance
supervisorctl start celery-beat
init.d is also documented in the celery docs:
http://docs.celeryproject.org/en/latest/userguide/daemonizing.html#init-script-celerybeat
Using celery 3.1.25 with Django 1.10
I can get celery to run tasks by manually getting into the shell and manually launching the tasks. However when I set a task from the django/admin/PeriodicTasks (run every minute), these tasks are not picked up by celery.
I'm using flower to check the status but I don't see any failing task.
The broker node is called celery#USER-vm instead of default, so I don't know if that is affecting this.
my command to run celery is
python manage.py celery -A proj worker --loglevel=INFO -B
Any insight of where to look? MY best guess is that djcelery is not connected to rabbitmq, but not sure where to make those changes.
Thanks!
EDIT:
Settings.py
BROKER_URL="amqp://guest:guest#localhost//"
CELERY_BROKER_URL="amqp://guest:guest#localhost//"
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
CELERY_SEND_TASK_ERROR_EMAILS=True
CELERYD_CONCURRENCY=8
CELERY_TASK_RESULT_EXPIRES=None
CELERY_ACCEPT_CONTENT = ['json', 'application','msgpack', 'yaml']
CELERY_DEFAULT_QUEUE='default'
CELERY_DEFAULT_EXCHANGE_TYPE='direct'
CELERY_DEFAULT_ROUTING_KEY='default'
CELERY_ENABLE_UTC=True
From celery.py
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE','proj.settings')
app = Celery('proj')
app.conf.update(
CELERY_TASK_RESULT_EXPIRES=3600,
)
app.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')
I'm working in a periodic task using django + celery. My project has django-celery==3.1.17, the broker used is Redis and it is working.
In my settings file:
CELERYBEAT_SCHEDULE={
'delivery_send': {
'task': 'delivery.tasks.DeliverySendTask',
'schedule': timedelta(minutes=1),
'args': [],
},
}
CELERY_ROUTES = {
'delivery.tasks.DeliverySendTask': {
'queue': 'periodic_tasks',
'routing_key': 'periodic_tasks'
},
}
There are more celery settings, but I guess these two are the most relevant for this question.
The task:
class DeliverySendTask(Task):
def run(self, *args, **kwargs):
logger.info('executing task!')
from .models import Dispatch
Dispatch.objects.all().delete()
Then I run python manage.py celery beat and see in the console
[2016-06-23 14:32:01,230: INFO/MainProcess] Scheduler: Sending due task delivery_send (delivery.tasks.DeliverySendTask)
But the delete is never actually executed. What am I missing?
Thanks for any help.
celery beat does not execute tasks. Only schedule tasks into queue. celery worker executes tasks.
To execute tasks periodically you have to start both celery beat and celery worker.
python manage.py celery beat
python manage.py celery worker
Install celery too. Then dont run it with manage.py. Run it like this:
celery -A your_app_name worker -l info -B -E
so without python manage.py