Develop Reference

Celery keeps trying to connect to localhost instead of Amazon SQS

So I'm trying to setup Celery in my Django project, and using Amazon SQS as my broker. However, Celery keeps trying to find SQS on localhost for some reason.
This is my settings.py:
CELERY_BROKER_TRANSPORT = "sqs"
CELERY_BROKER_USER = env.str("DJANGO_AWS_ACCESS_KEY_ID")
CELERY_BROKER_PASSWORD = env.str("DJANGO_AWS_SECRET_ACCESS_KEY")
CELERY_BROKER_TRANSPORT_OPTIONS = {
"region": env.str("DJANGO_AWS_SQS_REGION_NAME", default="us-east-2"),
"polling_interval": 10,
}
CELERY_DEFAULT_QUEUE = "default"
CELERY_ACCEPT_CONTENT = ["application/json"]
CELERY_TASK_SERIALIZER = "json"
CELERY_RESULT_SERIALIZER = "json"
CELERY_CONTENT_ENCODING = "utf-8"
CELERY_ENABLE_REMOTE_CONTROL = False
CELERY_SEND_EVENTS = False
CELERY_SQS_QUEUE_NAME = "default"
This is my celery.py :
import os
from celery import Celery
# set the default django settings module
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.production')
app = Celery('consumers') # type: Celery
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
When I start the worker using celery -A src.consumers worker --loglevel=debug, the worker tries to start with the following output and then immediately stops:
-------------- celery#aditya-PC v5.2.7 (dawn-chorus)
--- ***** -----
-- ******* ---- Linux-5.15.0-52-generic-x86_64-with-glibc2.35 2022-10-27 13:56:01
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: consumers:0x7fd77051de40
- ** ---------- .> transport: sqs://AHJJHHFYTA3GHVJHB8:**#localhost:6379//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 12 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. src.consumers.tasks.app1_test
How can I make celery not try to connect to localhost, and connect to SQS instead?

Hey this is kind of a non issue. The way it is mounted makes it look like localhost but it actually writes to the queue.
[2022-10-27 18:46:53,847: INFO/MainProcess] Connected to sqs://localhost//
This is a log from our prod env and everything works
create a message in sqs and you will see it gets processed

Simple periodic task in celery doesn't work but no errors

I'm new in Celery. I'm trying to properly configure Celery with my Django project. To test whether the celery works, I've created a periodic task which should print "periodic_task" each 2 seconds. Unfortunately it doesn't work but no error.
1 Installed rabbitmq
2 Project/project/celery.py
from __future__ import absolute_import
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
from django.conf import settings # noqa
app = Celery('project')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
#app.task(bind=True)
def myfunc():
print 'periodic_task'
#app.task(bind=True)
def debudeg_task(self):
print('Request: {0!r}'.format(self.request))
3 Project/project/__init__.py
from __future__ import absolute_import
from .celery import app as celery_app
4 Settings.py
INSTALLED_APPS = [
'djcelery',
...]
...
...
CELERYBEAT_SCHEDULE = {
'schedule-name': {
'task': 'project.celery.myfunc', # We are going to create a email_sending_method later in this post.
'schedule': timedelta(seconds=2),
},
}
And before python manage.py, I run celery -A project worker -l info
Still can't see any "periodic_task" printed in console every 2 seconds... Do you know what to do?
EDIT CELERY CONSOLE:
-------------- celery#Milwou_NB v3.1.23 (Cipater)
---- **** -----
--- * *** * -- Windows-8-6.2.9200
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: dolava:0x33d1350
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
[tasks]
. project.celery.debudeg_task
. project.celery.myfunc
EDIT:
After changing worker to beat, it seems to work. Something is happening each 2 seconds (changed to 5 seconds) but I can't see the results of the task. (I can put anything into the CELERYBEAT_SCHEDULE, even wrong path and it doesn't raises any error..)
I changed myfunc code to:
#app.task(bind=True)
def myfunc():
# notifications.send_message_to_admin('sdaa','dsadasdsa')
with open('text.txt','a') as f:
f.write('sa')
But I can't see text.txt anywhere.
> celery -A dolava beat -l info
celery beat v3.1.23 (Cipater) is starting.
__ - ... __ - _
Configuration ->
. broker -> amqp://guest:**#localhost:5672//
. loader -> celery.loaders.app.AppLoader
. scheduler -> djcelery.schedulers.DatabaseScheduler
. logfile -> [stderr]#%INFO
. maxinterval -> now (0s)
[2016-10-26 17:46:50,135: INFO/MainProcess] beat: Starting...
[2016-10-26 17:46:50,138: INFO/MainProcess] Writing entries...
[2016-10-26 17:46:51,433: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2016-10-26 17:46:51,433: INFO/MainProcess] Writing entries...
[2016-10-26 17:46:51,812: INFO/MainProcess] Scheduler: Sending due task schedule-name (dolava_app.tasks.myfunc)
[2016-10-26 17:46:51,864: INFO/MainProcess] Writing entries...
[2016-10-26 17:46:57,138: INFO/MainProcess] Scheduler: Sending due task schedule-name (dolava_app.tasks.myfunc)
[2016-10-26 17:47:02,230: INFO/MainProcess] Scheduler: Sending due task schedule-name (dolava_app.tasks.myfunc)

Try to run
$ celery -A project beat -l info

Celery send_task doesn't send tasks

I have a server running Celery with RabbitMQ. But when I try to send tasks using send_task, it just returns with an AsyncResult object.
But the actual task is not running (even though the workers and the queues are empty)
c = Celery("tasks", broker="amqp://guest#127.0.0.1//")
c.send_task("tasks.printing.test_print", (100), queue="print_queue", routing_key="printing.test_print")
My celery configuration is:
CELERY_QUEUES = (
Queue('default', routing_key='task.#'),
Queue('print_queue', routing_key='printing.#'),
)
CELERY_DEFAULT_EXCHANGE = 'tasks'
CELERY_ROUTES = {
'tasks.printing.test_print': {
'queue': 'print_queue',
'routing_key': 'printing.test_print',
}}
BROKER_URL = 'amqp://'
I execute only one worker:
celery -A celerymain worker --loglevel=debug
This is it's initial log:
- ** ---------- [config]
- ** ---------- .> app: __main__:0x7eff96903b50
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: amqp://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues] -------------- .> default exchange=tasks(topic) key=task.#
.> print_queue exchange=tasks(topic) key=printing.#
[tasks] . test_print
This is the task:
class test_print(Task):
name = "test_print"
def run(self,a):
log.info("running")
print a
The rabbitMQ queue 'print_queue' stays empty and there is nothing new in the rabbitMQ logs.
I have 4 GB free space so it's not a disk space problem.
What can be the problem here?

I solved the problem by removing the routing_key parameter from send_task.
I don't really know why this was a problem but at least it works

#app.task(name="test_print")
class test_print(Task):
name = "test_print"
def run(self,a):
log.info("running")
print a

Pydoop call not working when inside Celery task

I've set up two files for a project using Celery and Pydoop, tasks.py and HDFSStorage.py
# tasks.py
from celery import Celery
from celery import shared_task
from celery.utils.log import get_task_logger
from HDFSStorage import HDFSStorage
app = Celery('tasks', broker='amqp://guest#localhost//')
logger = get_task_logger(__name__)
fs = HDFSStorage()
print fs.exists("/myfile.txt")
#shared_task
def add(x,y):
logger.info('Adding {0} + {1}'.format(x, y))
logger.info('Checking if file exists')
fs.exists("/myfile.txt")
logger.info('Done checking if file exists')
return x+y
# HDFSStorage.py
import pydoop
from pydoop.hdfs import hdfs
class HDFSStorage():
def __init__(self):
self.client = hdfs(host="master", port=54310, user="oskar")
def exists(self, name):
return self.client.exists(name)
Running Celery starts with the fs.exists() call outside the task and outputs True as expected.
$ celery -A tasks worker -l info
True
[2016-06-08 15:54:15,298: WARNING/MainProcess] /usr/local/lib/python2.7/dist-packages/ce
lery/apps/worker.py:161: CDeprecationWarning:
Starting from version 3.2 Celery will refuse to accept pickle by default.
The pickle serializer is a security concern as it may give attackers
the ability to execute any command. It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.
If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::
CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']
You must only enable the serializers that you will actually use.
warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))
-------------- celery#master v3.1.23 (Cipater)
---- **** -----
--- * *** * -- Linux-3.19.0-32-generic-x86_64-with-LinuxMint-17.3-rosa
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: tasks:0x7f510d3162d0
- ** ---------- .> transport: amqp://guest:**#localhost:5672//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
[tasks]
. tasks.add
[2016-06-08 15:54:15,371: INFO/MainProcess] Connected to amqp://guest:**#127.0.0.1:5672/
/
[2016-06-08 15:54:15,382: INFO/MainProcess] mingle: searching for neighbors
[2016-06-08 15:54:16,395: INFO/MainProcess] mingle: all alone
[2016-06-08 15:54:16,412: WARNING/MainProcess] celery#master ready.
[2016-06-08 15:54:19,736: INFO/MainProcess] Events of group {task} enabled by remote.
However, running the task which has the same fs.exists() call gets stuck for some unknown reason.
$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from tasks import add
True
>>> print add.delay(5,4).get()
[2016-06-08 15:54:32,833: INFO/MainProcess] Received task: tasks.add[a50409a8-f82d-4376-
ace2-442a09c9ed4f]
[2016-06-08 15:54:32,834: INFO/Worker-2] tasks.add[a50409a8-f82d-4376-ace2-442a09c9ed4f]
: Adding 5 + 3
[2016-06-08 15:54:32,834: INFO/Worker-2] tasks.add[a50409a8-f82d-4376-ace2-442a09c9ed4f]
: Checking if file exists
Removing the fs.exists() call in the task makes the task finish correctly.
What am I doing wrong? What makes Celery to not work with Pydoop?

The HDFSStorage instance must be created inside the task
# tasks.py
from celery import Celery
from celery import shared_task
from celery.utils.log import get_task_logger
from HDFSStorage import HDFSStorage
app = Celery('tasks', broker='amqp://guest#localhost//')
logger = get_task_logger(__name__)
#shared_task
def add(x,y):
fs = HDFSStorage()
logger.info('Adding {0} + {1}'.format(x, y))
logger.info('Checking if file exists')
fs.exists("/myfile.txt")
logger.info('Done checking if file exists')
return x+y

Celery not running chord callback

After looking at a lot of articles about chord callbacks not executing and trying their solutions, I am still unable to get it to work. In fact, the chord_unlock method is also not getting executed for some reason.
celery.py
from __future__ import absolute_import
from celery import Celery
app = Celery('sophie',
broker='redis://localhost:6379/2',
backend='redis://localhost:6379/2',
include=['sophie.lib.chord_test'])
app.conf.update(
CELERY_ACCEPT_CONTENT=["json"],
CELERY_TASK_SERIALIZER="json",
CELERY_TRACK_STARTED=True,
CELERYD_PREFETCH_MULTIPLIER=1, # NO PREFETCHING OF TASKS
BROKER_TRANSPORT_OPTIONS = {
'priority_steps': [0, 1] # ALLOW ONLY 2 TASK PRIORITIES
}
)
if __name__ == '__main__':
app.start()
chord_test.py
from __future__ import absolute_import
from sophie.celery import app
from celery import chord
#app.task(name='sophie.lib.add')
def add(x, y):
return x + y
#app.task(name='sophie.lib.tsum')
def tsum(numbers):
return sum(numbers)
if __name__ == '__main__':
tasks = [add.s(100, 100), add.s(200, 200)]
chord(tasks, tsum.s()).apply_async()
The output of my worker logfile is as follows
$ celery worker -l info --app=sophie.celery -n worker1.%h
-------------- celery#worker1.vagrant-ubuntu-11 v3.1.6 (Cipater)
---- **** -----
--- * *** * -- Linux-3.0.0-12-server-x86_64-with-Ubuntu-11.10-oneiric
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> broker: redis://localhost:6379/2
- ** ---------- .> app: sophie:0x3554250
- ** ---------- .> concurrency: 1 (prefork)
- *** --- * --- .> events: OFF (enable -E to monitor this worker)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
[tasks]
. sophie.lib.add
. sophie.lib.tsum
[2013-12-12 19:37:26,499: INFO/MainProcess] Connected to redis://localhost:6379/2
[2013-12-12 19:37:26,506: INFO/MainProcess] mingle: searching for neighbors
[2013-12-12 19:37:27,512: INFO/MainProcess] mingle: all alone
[2013-12-12 19:37:27,527: WARNING/MainProcess] celery#worker1.vagrant-ubuntu-11 ready.
[2013-12-12 19:37:29,723: INFO/MainProcess] Received task: sophie.lib.add[b7d504c1-217f-43a9-b57e-86f0fcbdbe22]
[2013-12-12 19:37:29,734: INFO/MainProcess] Task sophie.lib.add[b7d504c1-217f-43a9-b57e-86f0fcbdbe22] succeeded in 0.009769904
00522s: 200
[2013-12-12 19:37:29,735: INFO/MainProcess] Received task: sophie.lib.add[eb01a73e-f6c8-401d-8049-6cdbc5f0bd90]
[2013-12-12 19:37:29,737: INFO/MainProcess] Task sophie.lib.add[eb01a73e-f6c8-401d-8049-6cdbc5f0bd90] succeeded in 0.001446505
00442s: 400
There is no chord_unlock being called at all. Some more output to give further context:
$ sudo pip freeze | egrep 'celery|kombu|billiard'
billiard==3.3.0.12
celery==3.1.6
kombu==3.0.7
$ uname -a
Linux vagrant-ubuntu-11 3.0.0-12-server #20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
$ redis-server --version
Redis server version 2.2.11 (00000000:0)

The chords documentation uses the following syntax for chord:
result = chord(header)(callback)
So you could do:
chord(tasks)(tsum.s())

There is an Important Notes section under the chords documentation that says:
Tasks used within a chord must not ignore their results. In practice this means that you must enable a CELERY_RESULT_BACKEND in order to use chords. Additionally, if CELERY_IGNORE_RESULT is set to True in your configuration, be sure that the individual tasks to be used within the chord are defined with ignore_result=False. This applies to both Task subclasses and decorated tasks.
Even when I followed the recommendations the chord callback is not executed, but might help others.
EDIT:
Found what it was in my case! The chord doesn't fully support specifying queue name in all cases:
https://github.com/celery/celery/issues/2085

Happened to me when my task signatures did not match the argments provided by the flow, accepting a *args might help debug such an issue.