I tested my project in my local machine, and it worked fine. But after uploading to a remote server(CentOS), I cannot execute celerybeat.
Here is my command.
python manage.py celeryd --events --loglevel=INFO -c 5 --settings=[settings-directory].production
This command works in the local machine(with --settings=[settings-directory].local), but in the remote server, ImportError: cannot import name celeryd occured.
Setting about celery is in base.py. local.py and production.py import the file. In production.py, there are just DEBUG, static, database settings.
I can import djcelery and celery in shell of the remote machine.
How could I solve this?
--
I think this is a version problem.. I'm reading about celery3.1
It turned out I used different version of Django in my remote server.
In Celery 3.1, there is no command named celeryd.
Related
I'm trying to figure out the woeful instructions here
Under the section "Configuring a Celery app" I'm not sure where i put the code:
import os
app.conf.update(BROKER_URL=os.environ['REDIS_URL'],
CELERY_RESULT_BACKEND=os.environ['REDIS_URL'])
Any clarification of these instructions is greatly appreciated.
The instructions are indicating you should put that code in your tasks.py module. However, that's not exactly extensible for multiple packages, each with their own tasks.py module. What I'd recommend is creating a celery.py file in the same directory as your settings.py file.
# tasks.py
import celery
app = celery.Celery('example')
app.conf.update(BROKER_URL=os.environ['REDIS_URL'],
CELERY_RESULT_BACKEND=os.environ['REDIS_URL'])
Or you can specify your settings in settings.py and configure celery as such:
# settings.py
broker_url = os.environ['REDIS_URL']
result_backend = os.environ['REDIS_URL']
# celery.py
from celery import Celery
from celery.utils.collections import DictAttribute
from celery.loaders.base import BaseLoader
from django.conf import settings
from django.apps import apps
class ProjectLoader(BaseLoader):
def read_configuration(self):
"""Load configuration from Django settings.
This may not be needed to be honest. It's what I use in my project.
"""
return DictAttribute(settings)
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
# CELERY_LOADER must be set in the environment. Setting the ``loader``
# kwarg for the app instance does _not_ do what we need it to.
os.environ.setdefault("CELERY_LOADER", "project.celery:ProjectLoader")
app = Celery("project")
app.config_from_object("django.conf:settings")
app.autodiscover_tasks(lambda: [n.name for n in apps.get_app_configs()])
# Procfile
worker: celery worker --app=project.celery
Disclaimer, some of these configs will require adjustment for your project.
Following is the steps i took to make a minimal heroku/django/celery/redis project in conjunction with the instructions here along with other sources I found on the web. Hopefully someone will find this useful.
In your terminal, use the "heroku login" command to log in to the Heroku CLI.
"git clone https://github.com/heroku/python-getting-started.git" to copy a basic django skeleton project to your local.
rename python-getting-started to whatever.
cd into this directory.
run the following command: "pip install -r requirements.txt"
Note: Postgres must be properly installed in order for this step to work properly.
run the following command: "python manage.py collectstatic"
Install redis on Mac: "brew install redis"
Start redis server: "redis-server&" (The & at the end is to run it as a background process)
Test if Redis server is running: "redis-cli ping". If it replies “PONG”, then it’s good to go!
Install celery: "pip install celery"
Make a tasks.py file in your application directory with the following code:
from celery import Celery
app = Celery('tasks', broker='redis://localhost:6379/0')
#app.task
def add(x, y):
return x + y
"cd .." back into root directory.
Run celery: "celery worker -A=location of tasks&"
run: "python manage.py shell" in your root directory.
As your tasks celery server has been started, you can now use it to run your task just by importing tasks.py script, e.g from Python interpreter interactive mode:
import hello.tasks
hello.tasks.add.delay(1, 1)
This should return an Async message.
Push your local to heroku master.
** Note: If you run "celery worker -A=location of tasks.py&" and it gives the message:
consumer: Cannot connect to amqp://guest:**#127.0.0.1:5672//: [Errno 61]
Connection refused.
Try restarting the redis server with the command: "brew services restart redis"
There you have it. A minimal heroku/django/celery/redis project! You can download it here.Instructions on how to deploy this to heroku.
** Note: In the working project the "celery worker" command is already included in the Procfile.
Why is this happening?
My celery.py:
import os
from celery import Celery
from django.conf import settings
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myshop.settings')
app = Celery('myshop')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
my init.py
# import celery
from .celery import app as celery_app
I even tried renaming celery.py to something else and the error still persisted. Could it be because of my python version?
I`ll post answer in order to move it from comments.
First of all in your __ init__.py file add this line
from __future__ import absolute_import, unicode_literals
Second of all you need to add to your settings, information about brooker.
This is an example configuration file to get you started. It should contain all you need to run a basic Celery set-up.
Broker settings.
broker_url = 'amqp://guest:guest#localhost:5672//'
The next thing is running your celery worker. So if you celery app is named myshop you have to run celery worker (using your environment), by typing this simple command:
celery -A myshop worker -l info
Then try to run your task, and everything should be fine.
its becuase of version
How u have installed celery
pip install celery==3.0.19
if this
then run python by
python manage.py runserver
or
pip3 install celery==3.0.19
if this
then run python by
python3 manage.py runserver
I've built a Django project that works, even after I freeze it using Cx_Freeze and Py2exe.
Now I'm trying to set up the project for distribution, which requires a real webserver. I'm going for Gunicorn (will add Nginx once it works). I managed to run the Gunicorn server properly through the command line using :
gunicorn wsgi:application
However, I need to be able to run the server from my Python script, as the server is ment to be localhost. Gunicorn used to be shipped with a command 'run_gunicorn' designed for Django, but this command is now deprecated.
I tried understanding the following method :
How to use Flask-Script and Gunicorn
But I can't figure out how to make it work with Django.
The following doesn't work:
from django.core.wsgi import get_wsgi_application
from gunicorn.app.base import Application
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
application = get_wsgi_application()
Application().run(application)
Could someone tell me how to start the gunicorn server from my Python script ?
Here's my celery app config:
from __future__ import absolute_import
from celery import Celery
import os
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tshirtmafia.settings')
app = Celery('tshirtmafia')
app.conf.update(
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
)
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
settings.py:
INSTALLED_APPS:
'kombu.transport.django',
'djcelery',
also:
BROKER_URL = 'django://'
Here's my task:
#shared_task
def test():
send_mail('nesamone bus', 'Files have been successfully generated.', 'marijus.merkevicius#gmail.com',
['marijus.merkevicius#gmail.com'], fail_silently=False)
Now when I run locally python manage.py celeryd locally and then run test.delay() from shell locally it works.
Now I'm trying to deploy my app. When with the exact same configuration I try to open python manage.py celeryd and in other window I open shell and run test task, it doesn't work.
I've also tried to setup background daemon like this:
/etc/default/celeryd configuration:
# Name of nodes to start, here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"
# Where to chdir at start. (CATMAID Django project dir.)
CELERYD_CHDIR="/home/tshirtnation/"
# Python interpreter from environment. (in CATMAID Django dir)
ENV_PYTHON="/usr/bin/python"
# How to call "manage.py celeryd_multi"
CELERYD_MULTI="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryd_multi"
# How to call "manage.py celeryctl"
CELERYCTL="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryctl"
# Extra arguments to celeryd
CELERYD_OPTS="--time-limit=300 --concurrency=1"
# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"
# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"
# Workers should run as an unprivileged user.
CELERYD_USER="celery"
CELERYD_GROUP="celery"
# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="settings"
And I use default celery /etc/init.d/celeryd script.
So basically it seems like celeryd starts but doesn't work. No idea how to debug this and what might be wrong.
Let me know if you need anything else
Celery turned to be a very capricious child in Django robust system as for me.
There are too little initial data for understanding the reason of your problems.
The most usual reason of Celery daemon fail is file system permissions.
But to clarify the reason I'd try:
Start celery from a command line by the user-owner of django project:
celery -A proj worker -l info
If it works OK, go further
Start celery in a verbal mode as a root user just like daemon to be:
sudo sh -x /etc/init.d/celeryd start
This will show most of the problems with the daemon script - celery user and group used, but not all, unfortunately: permission fails are not visible.
My little remark.
Usually Celery is started by own celery user, and the django project by another one. After long fighting celery and system, I refused from celery user, and owned celery process by the django project user.
And .. do not forget to start once
update-rc.d celerybeat defaults
update-rc.d celeryd defaults
this is for Ubuntu daemon start, sure.
Good luck
I found a MediaCrush open source from here
https://github.com/MediaCrush/MediaCrush
But stuck in last steps.
I started the Redis server, use command
$redis-cli
that received the "PONG" response.
Then used the command
$celery worker -A mediacrush -Q celery,priority
and after
python app.py
But it seem that nothing works. I just installed nginx, run it on my IP ok, but after edit the nginx.conf like a Mediacrush script, then accessing my IP, nothing happens.
So what am I missing here? and how to config nginx server and start redis server to run this script on CentOS (i can change it to Arch like them if required)
Thanks!
I just wanted to run it for fun.. so this may be wrong but what I did after running the celery daemon was edit the app.py script and manually set the host, port, and set debug to false. Then I just executed it like any other python script.
EDIT: This may work
PORT=8000 gunicorn -w 4 app:app
it switches your port to 8000 and runs the gunicron daemon with 4 workers. both approaches worked for me.
Edit File: ./app.py
from mediacrush.app import app
from mediacrush.config import _cfg, _cfgi
import os
app.static_folder = os.path.join(os.getcwd(), "static")
if __name__ == '__main__':
# CHANGE THIS LINE TO REFLECT YOUR DATA
app.run(host=_cfg("debug-host"), port=_cfgi('debug-port'), debug=True)
# FOR EXAMPLE I CHANGED IT TO THIS
# app.run(host="92.222.25.245", port=8000, debug=0)
Also to start redis i believe you should do redis-server& I use the cli to manually tinker with it.
Btw I did this on linux mint / ubuntu 14.04 / debian