Develop Reference

Celery keeps trying to connect to localhost instead of Amazon SQS

So I'm trying to setup Celery in my Django project, and using Amazon SQS as my broker. However, Celery keeps trying to find SQS on localhost for some reason.
This is my settings.py:
CELERY_BROKER_TRANSPORT = "sqs"
CELERY_BROKER_USER = env.str("DJANGO_AWS_ACCESS_KEY_ID")
CELERY_BROKER_PASSWORD = env.str("DJANGO_AWS_SECRET_ACCESS_KEY")
CELERY_BROKER_TRANSPORT_OPTIONS = {
"region": env.str("DJANGO_AWS_SQS_REGION_NAME", default="us-east-2"),
"polling_interval": 10,
}
CELERY_DEFAULT_QUEUE = "default"
CELERY_ACCEPT_CONTENT = ["application/json"]
CELERY_TASK_SERIALIZER = "json"
CELERY_RESULT_SERIALIZER = "json"
CELERY_CONTENT_ENCODING = "utf-8"
CELERY_ENABLE_REMOTE_CONTROL = False
CELERY_SEND_EVENTS = False
CELERY_SQS_QUEUE_NAME = "default"
This is my celery.py :
import os
from celery import Celery
# set the default django settings module
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings.production')
app = Celery('consumers') # type: Celery
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
When I start the worker using celery -A src.consumers worker --loglevel=debug, the worker tries to start with the following output and then immediately stops:
-------------- celery#aditya-PC v5.2.7 (dawn-chorus)
--- ***** -----
-- ******* ---- Linux-5.15.0-52-generic-x86_64-with-glibc2.35 2022-10-27 13:56:01
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: consumers:0x7fd77051de40
- ** ---------- .> transport: sqs://AHJJHHFYTA3GHVJHB8:**#localhost:6379//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 12 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. src.consumers.tasks.app1_test
How can I make celery not try to connect to localhost, and connect to SQS instead?

Hey this is kind of a non issue. The way it is mounted makes it look like localhost but it actually writes to the queue.
[2022-10-27 18:46:53,847: INFO/MainProcess] Connected to sqs://localhost//
This is a log from our prod env and everything works
create a message in sqs and you will see it gets processed

Celery - worker only sometimes picks up tasks

I am building a lead generation portal that can be accessed online. Please don't mind the verbosity of the code, I'm doing a lot of debugging right now.
My Celery worker inconsistently picks up tasks assigned to it, and I'm not sure why.
The weird thing about this, is that sometimes it works 100% perfect: there never are any explicit errors in the terminal.
I am currently in DEBUG = TRUE and REDIS as a broker!
celery start worker terminal command and response
celery -A mysite worker -l info --pool=solo
-------------- celery#DESKTOP-OG8ENRQ v5.0.2 (singularity)
--- ***** -----
-- ******* ---- Windows-10-10.0.19041-SP0 2020-11-09 00:36:13
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: mysite:0x41ba490
- ** ---------- .> transport: redis://localhost:6379//
- ** ---------- .> results: redis://localhost:6379/
- *** --- * --- .> concurrency: 12 (solo)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. mysite.celery.debug_task
. submit
[2020-11-09 00:36:13,899: INFO/MainProcess] Connected to redis://localhost:6379//
[2020-11-09 00:36:14,939: WARNING/MainProcess] c:\users\coole\pycharmprojects\lead_django_retry\venv\lib\site-packages\celery\app\control.py:48: DuplicateNodenameWarning: Received multiple replies from node name: celery#DESKTOP-OG8ENRQ.
Please make sure you give each node a unique nodename using
the celery worker `-n` option.
warnings.warn(DuplicateNodenameWarning(
[2020-11-09 00:36:14,939: INFO/MainProcess] mingle: all alone
[2020-11-09 00:36:14,947: INFO/MainProcess] celery#DESKTOP-OG8ENRQ ready.
views.py
class LeadInputView(FormView):
template_name = 'lead_main.html'
form_class = LeadInput
def form_valid(self, form):
print("I'm at views")
form.submit()
print(form.submit)
return HttpResponseRedirect('./success/')
tasks.py
#task(name="submit")
def start_task(city, category, email):
print("I'm at tasks!")
print(city, category, email)
"""sends an email when feedback form is filled successfully"""
logger.info("Submitted")
return start(city, category, email)
forms.py
class LeadInput(forms.Form):
city = forms.CharField(max_length=50)
category = forms.CharField(max_length=50)
email = forms.EmailField()
def submit(self):
print("I'm at forms!")
x = (start_task.delay(self.cleaned_data['city'], self.cleaned_data['category'], self.cleaned_data['email']))
return x
celery.py
from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')
app = Celery('mysite')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
settings.py
BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'
The runserver terminal will look something like this:
I'm at views
I'm at forms!
<bound method LeadInput.submit of <LeadInput bound=True, valid=True, fields=(city;category;email)>>
But the worker doesn't say that it picked up anything, just that "celery#DESKTOP-OG8ENRQ ready." Except, when it does work... for some reason? I'm at a loss!

Hello to whoever sees this. It turns out, that this is a bug with celery (or maybe redis?)... apparently many windows users run into this. https://github.com/celery/celery/issues/3759
Turns out, the answer is to make -P solo when starting worker. I'm not sure why this is the case... but that solved it!
Thank you Naqib for your help! You put me down the right rabbit hole to a solution.

By default, celery will use the hostname as worker name if your willing to use multiple workers in the same host then specify -n option.
celery -A mysite worker -l info --pool=solo -n worker2#%h
Your code works fine but the task is passed to the first worker, see
DuplicateNodenameWarning with no obvious reason #2938

Celery receives periodic tasks but doesn't execute them

I use celery to run periodic tasks in my Django DRF app. Unfortunately, registered tasks are not executed.
Project structure:
project_name
___ cron_tasks
______ __init__.py
______ celery.py
celery.py:
app = Celery('cron_tasks', include=['cron_tasks.celery'])
app.conf.broker_url = settings.RABBITMQ_URL
app.autodiscover_tasks()
app.conf.redbeat_redis_url = settings.REDBEAT_REDIS_URL
app.conf.broker_pool_limit = 1
app.conf.broker_heartbeat = None
app.conf.broker_connection_timeout = 30
app.conf.worker_prefetch_multiplier = 1
app.conf.beat_schedule = {
'first_warning_overdue': {
'task': 'cron_tasks.celery.test_task',
'schedule': 60.0, # seconds
'options': {'queue': 'default', 'expires': 43100.0}
}
}
#shared_task
def test_task():
app.send_task('cron_tasks.celery.test_action')
def test_action():
print('action!') # print is not executed
# I also tried to change the data, but it never happens too.
from django.contrib.auth import get_user_model
u = get_user_model().objects.get(id=1)
u.first_name = "testttt"
u.save()
setting.py:
RABBITMQ_URL = os.environ.get('RABBITMQ_URL')
REDBEAT_REDIS_URL = os.environ.get('REDBEAT_REDIS_URL')
CELERY_BROKER_URL = os.environ.get('RABBITMQ_URL')
CELERYD_TASK_SOFT_TIME_LIMIT = 60
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_RESULT_BACKEND = os.environ.get('REDBEAT_REDIS_URL')
CELERY_IMPORTS = ("cron_tasks.celery", )
from kombu import Queue
CELERY_DEFAULT_QUEUE = 'default'
CELERY_QUEUES = (
Queue('default'),
)
CELERY_CREATE_MISSING_QUEUES = True
redbeat_redis_url = REDBEAT_REDIS_URL
Rabbitmq is running properly. I can see it's there in the celery worker terminal output:
- ** ---------- .> transport: amqp://admin:**#localhost:5672/my_vhost
Redis is pinging well. I use redis to send beats.
I run:
celery beat -S redbeat.RedBeatScheduler -A cron_tasks.celery:app --loglevel=debug
It shows:
[2019-02-15 09:32:44,477: DEBUG/MainProcess] beat: Waking up in 10.00 seconds.
[2019-02-15 09:32:54,480: DEBUG/MainProcess] beat: Extending lock...
[2019-02-15 09:32:54,481: DEBUG/MainProcess] Selecting tasks
[2019-02-15 09:32:54,482: INFO/MainProcess] Loading 1 tasks
[2019-02-15 09:32:54,483: INFO/MainProcess] Scheduler: Sending due task first_warning_overdue (cron_tasks.celery.test_task)
[2019-02-15 09:32:54,484: DEBUG/MainProcess] cron_tasks.celery.test_task sent. id->f89083aa-11dc-41fc-9ebe-541840951f8f
Celery worker is run this way:
celery worker -Q default -A cron_tasks.celery:app -n .%%h --without-gossip --without-mingle --without-heartbeat --loglevel=info --max-memory-per-child=512000
It says:
-------------- celery#.%me.local v4.2.1 (windowlicker)
---- **** -----
--- * *** * -- Darwin-16.7.0-x86_64-i386-64bit 2019-02-15 09:31:50
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: cron_tasks:0x10e2a5ac8
- ** ---------- .> transport: amqp://admin:**#localhost:5672/my_vhost
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
[tasks]
. cron_tasks.celery.test_task
[2019-02-15 09:31:50,833: INFO/MainProcess] Connected to amqp://admin:**#127.0.0.1:5672/my_vhost
[2019-02-15 09:31:50,867: INFO/MainProcess] celery#.%me.local ready.
[2019-02-15 09:41:46,218: INFO/MainProcess] Received task: cron_tasks.celery.test_task[3c121f04-af3b-4cbe-826b-a32da6cc156e] expires:[2019-02-15 21:40:05.779231+00:00]
[2019-02-15 09:41:46,220: INFO/ForkPoolWorker-2] Task cron_tasks.celery.test_task[3c121f04-af3b-4cbe-826b-a32da6cc156e] succeeded in 0.001324941000000024s: None
Expected behavior:
This should run my test_action().
But, even though the celery worker output says succeeded in 0.001324941000000024s, the function never executes.

Celery not queuing to a remote broker, adding tasks to a localhost instead

My question is same like this Celery not queuing tasks to broker on remote server, adds tasks to localhost instead, but the answer is not working to me.
My celery.py
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
app = Celery('project', broker='amqp://<user>:<user_pass>#remoteserver:5672/<vhost>', backend='amqp')
# app = Celery('project')
# Using a string here means the worker don't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
When I run:
$ celery -A worker -l info
I receive the following output:
-------------- celery#paulo-Inspiron-3420 v4.2.1 (windowlicker)
---- **** -----
--- * *** * -- Linux-4.15.0-36-generic-x86_64-with-Ubuntu-18.04-bionic 2018-10-30 13:44:07
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: mycelery:0x7ff88ca043c8
- ** ---------- .> transport: amqp://<user>:**#<remote_ip>:5672/<vhost>
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
I tried stop rabbitmq server and uninstalled it too, but celery keeps queuing to localhost.
Someone can help?

You need to add something like this to your __init__.py file in the same directory as the celery.py file:
from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app
__all__ = ('celery_app',)
Also, make sure you're starting the worker process from inside your project's virtualenv.

Unsure how to run Celery under windows for periodic tasks

I am having difficulty understanding how to run Celery after setting up some scheduled tasks.
Firstly, my project directory is structured as follows:
blogpodapi\api\__init__.py contains
from tasks import app
import celeryconfig
blogpodapi\api\celeryconfig.py contains
from datetime import timedelta
# Celery settings
CELERY_BROKER_URL = 'redis://localhost:6379/0'
BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/1'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'
CELERY_IMPORTS = ("api.tasks",)
CELERYBEAT_SCHEDULE = {
'write-test': {
'task': 'api.tasks.addrandom',
'schedule': timedelta(seconds=2),
'args': (16000, 42)
},
}
blogpodapi\api\tasks.py contains
from __future__ import absolute_import
import random
from celery import Celery
app = Celery('blogpodapi')
#app.task
def add(x, y):
r = x + y
print "task arguments: {x}, {y}".format(x=x, y=y)
print "task result: {r}".format(r=r)
return r
#app.task
def addrandom(x, *args): # *args are not used, just there to be interchangable with add(x, y)
y = random.randint(1,100)
print "passing to add(x, y)"
return add(x, y)
blogpodapi\blogpodapi\__init__.py contains
from __future__ import absolute_import
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app # noqa
blogpodapi\blogpodapi\settings.py contains
...
# Celery settings
CELERY_BROKER_URL = 'redis://localhost:6379/0'
BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/1'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'
CELERY_IMPORTS = ("api.tasks",)
...
I run celery -A blogpodapi worker --loglevel=info in command prompt and get the following:
D:\blogpodapi>celery -A blogpodapi worker --loglevel=info
-------------- celery#JM v3.1.23 (Cipater)
---- **** -----
--- * *** * -- Windows-8-6.2.9200
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: blogpodapi:0x348a940
- ** ---------- .> transport: redis://localhost:6379/0
- ** ---------- .> results: redis://localhost:6379/1
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
[tasks]
. api.tasks.add
. api.tasks.addrandom
. blogpodapi.celery.debug_task
[2016-08-13 13:01:51,108: INFO/MainProcess] Connected to redis://localhost:6379/0
[2016-08-13 13:01:52,122: INFO/MainProcess] mingle: searching for neighbors
[2016-08-13 13:01:55,138: INFO/MainProcess] mingle: all alone
c:\python27\lib\site-packages\celery\fixups\django.py:265: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2016-08-13 13:02:00,157: WARNING/MainProcess] c:\python27\lib\site-packages\celery\fixups\django.py:265: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2016-08-13 13:02:27,790: WARNING/MainProcess] celery#JM ready.
I then run celery -A blogpodapi beat in command prompt and get the following:
D:\blogpodapi>celery -A blogpodapi beat
celery beat v3.1.23 (Cipater) is starting.
__ - ... __ - _
Configuration ->
. broker -> redis://localhost:6379/0
. loader -> celery.loaders.app.AppLoader
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule
. logfile -> [stderr]#%INFO
. maxinterval -> now (0s)
[2016-08-13 13:02:51,937: INFO/MainProcess] beat: Starting...
For some reason, I can't seem to view my periodic tasks being logged. Is there anything I am doing wrong?
UPDATE: here is my celery.py...
from __future__ import absolute_import
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'blogpodapi.settings')
from django.conf import settings # noqa
app = Celery('blogpodapi')
# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))

You need to run celery beat with the celery settings file
celery -A blogpodapi.celery beat --loglevel=INFO