In my celery tasks, I have a task which uses a python module(theano) that runs on GPU, this module can only be imported by one thread at the same time. But to start the website, I have to run:
python manage.py runserver
celery -A celery_try worker -l info
So the module will be imported by both celery and django website, which is conflict. Is there a way to decouple Django and Celery so that the module is only imported once?
For testing purposes you can run django development server in single-threaded mode: python manage.py runserver --nothreading.
You want to import theano only in celery worker process, not in django web server process, right? Ok, let's make import conditional, so that it is imported in celery and not in django.
import os
try:
# next line will raise exception in django, but will work fine in celery
is_worker = os.environ['celery_worker']
import theano # celery will import theano, django won't
except Exception as exc:
# django code will catch exception that celery_worker doesn't exist and print it here
print exc
And start your celery worker with celery_worker environment variable set:
celery_worker=yes celery -A celery_try worker -l info
To discriminate between celery worker and django, let's set a bash environment variable in celery process, but not in django process. I called that variable celery_worker. In order to set it, I prepended celery -A celery_try worker -l info with per-command env variable assignment: celery_worker=yes. Now, in python code, I check if the environment variable is present. If it is, we are in celery worker and need to import theano.
If we're in django, os.environ['celery_worker'] shouldn't be defined and should raise an exception.
Related
Using celery 3.1.25 with Django 1.10
I can get celery to run tasks by manually getting into the shell and manually launching the tasks. However when I set a task from the django/admin/PeriodicTasks (run every minute), these tasks are not picked up by celery.
I'm using flower to check the status but I don't see any failing task.
The broker node is called celery#USER-vm instead of default, so I don't know if that is affecting this.
my command to run celery is
python manage.py celery -A proj worker --loglevel=INFO -B
Any insight of where to look? MY best guess is that djcelery is not connected to rabbitmq, but not sure where to make those changes.
Thanks!
EDIT:
Settings.py
BROKER_URL="amqp://guest:guest#localhost//"
CELERY_BROKER_URL="amqp://guest:guest#localhost//"
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
CELERY_SEND_TASK_ERROR_EMAILS=True
CELERYD_CONCURRENCY=8
CELERY_TASK_RESULT_EXPIRES=None
CELERY_ACCEPT_CONTENT = ['json', 'application','msgpack', 'yaml']
CELERY_DEFAULT_QUEUE='default'
CELERY_DEFAULT_EXCHANGE_TYPE='direct'
CELERY_DEFAULT_ROUTING_KEY='default'
CELERY_ENABLE_UTC=True
From celery.py
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE','proj.settings')
app = Celery('proj')
app.conf.update(
CELERY_TASK_RESULT_EXPIRES=3600,
)
app.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')
I am working on Django based web app. During unittest, I need to write a test which needs "Celery worker" running in the background.
I have already used:
CELERY_EAGER_PROPAGATES_EXCEPTIONS=True
CELERY_ALWAYS_EAGER=True
BROKER_BACKEND='memory
In over_ride settings, but these are not running celery worker for me in background when needed.
Any help would much appreciated.
Celery won't get run by Django automatically.
You can start a worker process by running from your project root:
$ celery -A my_proj worker
my_proj should be the application name you configured with app = Celery('my_proj')
Here's my celery app config:
from __future__ import absolute_import
from celery import Celery
import os
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tshirtmafia.settings')
app = Celery('tshirtmafia')
app.conf.update(
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
)
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
settings.py:
INSTALLED_APPS:
'kombu.transport.django',
'djcelery',
also:
BROKER_URL = 'django://'
Here's my task:
#shared_task
def test():
send_mail('nesamone bus', 'Files have been successfully generated.', 'marijus.merkevicius#gmail.com',
['marijus.merkevicius#gmail.com'], fail_silently=False)
Now when I run locally python manage.py celeryd locally and then run test.delay() from shell locally it works.
Now I'm trying to deploy my app. When with the exact same configuration I try to open python manage.py celeryd and in other window I open shell and run test task, it doesn't work.
I've also tried to setup background daemon like this:
/etc/default/celeryd configuration:
# Name of nodes to start, here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"
# Where to chdir at start. (CATMAID Django project dir.)
CELERYD_CHDIR="/home/tshirtnation/"
# Python interpreter from environment. (in CATMAID Django dir)
ENV_PYTHON="/usr/bin/python"
# How to call "manage.py celeryd_multi"
CELERYD_MULTI="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryd_multi"
# How to call "manage.py celeryctl"
CELERYCTL="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryctl"
# Extra arguments to celeryd
CELERYD_OPTS="--time-limit=300 --concurrency=1"
# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"
# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.
CELERYD_USER="celery"
CELERYD_GROUP="celery"
# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="settings"
And I use default celery /etc/init.d/celeryd script.
So basically it seems like celeryd starts but doesn't work. No idea how to debug this and what might be wrong.
Let me know if you need anything else
Celery turned to be a very capricious child in Django robust system as for me.
There are too little initial data for understanding the reason of your problems.
The most usual reason of Celery daemon fail is file system permissions.
But to clarify the reason I'd try:
Start celery from a command line by the user-owner of django project:
celery -A proj worker -l info
If it works OK, go further
Start celery in a verbal mode as a root user just like daemon to be:
sudo sh -x /etc/init.d/celeryd start
This will show most of the problems with the daemon script - celery user and group used, but not all, unfortunately: permission fails are not visible.
My little remark.
Usually Celery is started by own celery user, and the django project by another one. After long fighting celery and system, I refused from celery user, and owned celery process by the django project user.
And .. do not forget to start once
update-rc.d celerybeat defaults
update-rc.d celeryd defaults
this is for Ubuntu daemon start, sure.
Good luck
This does not work for me
$> cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.2 LTS"
django 1.7rc3
celery 3.1.13
python 2.7
I attempt to run
celery worker -A <project_name>
and I get
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
The runserver command works fine so I don't think it has to do with my settings?
python manage.py runserver 0.0.0.0:8080
I've double checked celery.py and confirm it has the correct values for the following lines:
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
app = Celery('proj')
# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
Is there something else i should be doing?
Django 1.7 now requires a different initialization for standalone scripts. When running outside the manage.py context, you now need to include:
import django
django.setup()
Try adding it before the app = Celery('proj') in your script.
I found out that the reason this was failing is because i had this in one of my tasks.py
CURRENT_DOMAIN = Site.objects.get_current().domain
I got side steped this for now with
CURRENT_DOMAIN = lambda: Site.objects.get_current().domain
currently waiting to see if anyone on github cares to offer a better recommendation.
https://github.com/celery/celery/issues/2227
will update if i get one. If not will probably just do a helper function that lazy returns the value i want.
update
at the suggestion of celery author, I've refactored my code so I don't make that call at the module level.
He's also addressed the issue by making sure django.setup() is called before importing task modules
https://github.com/celery/celery/issues/2227
I'm seeing the same issue however only with:
CELERYBEAT_SCHEDULER='djcelery.schedulers.DatabaseScheduler'
Do you have this enabled? If I run it with the default celery scheduler it loads fine. But can't get it to load with the Django scheduler.
I am new to redis and celery. I have gone through the basic tutorial of both, but I am not getting how to implement then in task scheduling job
I am unable to start with the scripting part. I am not getting how to write a script to make a queue, run the workers etc. I would need a practical example
So here's a cannonical example of how can celery run with Redis (let the script filename be mytasks.py):
from celery import Celery
celery = Celery('tasks', broker='redis://localhost:6379/0')
#celery.task
def add(x, y):
return x + y
As you see, broker argument was set to use Redis installed on your local machine. The next thing is to start celery server:
$ celery -A mytasks worker --loglevel=info
As your tasks celery server has been started, you can now use it to run your task just by importing mytasks script, e.g from Python interpreter interactive mode:
>>> from mytasks import add
>>> add.delay(1, 1)
2
After some time '2' will appear in console.
That's a basic example of how you can setup your tasks execution environment.