Develop Reference

Celery Beat: Synchronizing schedule is the last debug message I see

I am running superset and celery on AWS ECS. Celery worker, Celery beat and Superset are running in separate containers of the same task. I have turned on debug logs in celery so that I can see each step celery is taking. Celery is starting up and running. Celery worker goes until the log message DEBUG/MainProcess] | Consumer: Starting Connection, celery beat goes until the first time it wakes up, then it displays the log message DEBUG/MainProcess] beat: Synchronizing schedule... and doesn't wake up again.
The command I am using to start the celery worker is:
celery --app=superset.tasks.celery_app:app worker -E --pool=gevent -c 500 -l DEBUG
The command I am using to start the celery beat is:
celery --app=superset.tasks.celery_app:app beat --pidfile /tmp/celerybeat.pid --schedule /tmp/celerybeat-schedule -l DEBUG
From my superset_config.py, the relevant lines of code are:
class CeleryConfig:
broker_url = "redis://%s:%s/0" % (REDIS_HOST, REDIS_PORT)
imports = (
"superset.sql_lab",
"superset.tasks",
"superset.tasks.thumbnails",
)
result_backend = "redis://%s:%s/0" % (REDIS_HOST, REDIS_PORT)
worker_log_level = "DEBUG"
worker_prefetch_multiplier = 10
task_acks_late = True
task_annotations = {
"sql_lab.get_sql_results": {
"rate_limit": "100/s",
},
"email_reports.send": {
"rate_limit": "1/s",
"time_limit": 600,
"soft_time_limit": 600,
"ignore_result": True,
},
}
beat_schedule = {
"alerts.schedule_check": {
"task": "alerts.schedule_check",
"schedule": crontab(minute="*", hour="*"),
},
"reports.scheduler": {
"task": "reports.scheduler",
"schedule": crontab(minute="*", hour="*"),
},
"reports.prune_log": {
"task": "reports.prune_log",
"schedule": crontab(minute=0, hour=0),
},
}
CELERY_CONFIG = CeleryConfig
WEBDRIVER_BASEURL = "http://0.0.0.0:8088"
Things I have tried:
Configuring a postgres database for Celery
putting celery into the same container as superset
Changed db numer in superset config
Tried variations on WEBDRIVER_BASEURL (localhost:8088, www.actual-url.com)
Changing the dependencies in the ECS task definition
reconfiguring security groups for redis
Various commands for starting celery worker and beat
Ensuring the security group for the container allows ingress on port 6379
I setup flower to run on the ECS instance; flower shows no workers, tasks or monitors
Things I know:
Celery is connecting to Redis. (At one point it wasn't, and that threw very specific errors.)
Celery is reading the schedule, I can see in the logs where it is displaying the three things scheduled for the beat_schedule.
Schedule reports are not firing; there are no logs at the time of the report, nor is there evidence of a report being generated.
I get an error cron_descriptor.GetText:Failed to find locale en_US, when I access the reports page of Superset (although, TBH, I feel like this is unrelated).

So it turns out I was using MemoryDb instead of Elasticache, and the two things are not interchangeable.

Celery+SQS don't receive messages

I'm trying to make a queue system using celery+sqs.
Still in my local environment with localstack I'm not able to receive messages in worker. It just doesn't show anything. There is a question some time ago, but I'm ok in their config.
I'm using all other SQS/SNS activities from other function, but isn't working from celery.
My current setup is like this:
Docker config:
services:
localstack:
image: localstack/localstack:latest
environment:
- SERVICES=sqs,sns
- HOSTNAME=localstack
- HOSTNAME_EXTERNAL=localstack
ports:
- '4566:4566'
networks:
- platform_default
volumes:
- "${TMPDIR:-/tmp}/localstack:/tmp/localstack"
- "/var/run/docker.sock:/var/run/docker.sock"
And the celery instantiation is down, used a get_queue directly to be sure of its link.
return Celery(
"server",
task_default_queue=config.sqs_celery.queue_name,
broker=f"sqs://",
broker_url=f"sqs://{config.sqs_celery.aws_access_key_id}:{config.sqs_celery.aws_secret_access_key}#{config.sqs_celery.broker_site_port}",
# in my case: sqs://localstack:localstack#localhost:4566
broker_transport_options={
'region': config.sqs_celery.region,
"predefined_queues": {
config.sqs_celery.queue_name: {
"url": get_queue_url(config.sqs_celery.queue_name),
# in my case: http://localstack:4566/000000000000/tasks
'region': config.sqs_celery.region,
}
}
}
)
Please maybe you have some ideas to start, because I lost half of the day trying to figure out what is wrong.

SOLVED:
The queue url can't be solved, you have to check everything. I set it to "localhost" and it worked.

Celery Worker not picking task when run inside docker containers

I am facing this issue, when I am running my celery worker inside a docker container it's not picking tasks.
I am using Flask and celery.
Here are my logs when I run it without docker
celery#MacBook-Pro.local v4.4.2 (cliffs)
Darwin-18.2.0-x86_64-i386-64bit 2020-05-26 22:16:40
[config]
.> app: __main__:0x111343470
.> transport: redis://localhost:6379//
.> results: redis://localhost:6379/
.> concurrency: 8 (prefork)
.> task events: ON
[queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. load_data.scraping.tasks.scrape_the_data_daily
. scrape the data daily
You can clearly see that my worker is finding the task but it's not running the periodic task.
When I run the same command in docker here is what I am getting:
celery-worker_1 | /usr/local/lib/python3.6/site-packages/celery/platforms.py:801: RuntimeWarning: You're running the worker with superuser privileges: this is
celery-worker_1 | absolutely not recommended!
celery-worker_1 |
celery-worker_1 | Please specify a different user using the --uid option.
celery-worker_1 |
celery-worker_1 | User information: uid=0 euid=0 gid=0 egid=0
celery-worker_1 |
celery-worker_1 | uid=uid, euid=euid, gid=gid, egid=egid,
celery-worker_1 | [2020-05-26 18:54:02,088: DEBUG/MainProcess] | Worker: Preparing bootsteps.
celery-worker_1 | [2020-05-26 18:54:02,090: DEBUG/MainProcess] | Worker: Building graph...
celery-worker_1 | [2020-05-26 18:54:02,092: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Pool, Autoscaler, StateDB, Beat, Consumer}
So it' looks like it's not finding the app and the tasks.
But if I execute the command from the docker container, I can see that my tasks are found.
Here is how I set up my docker-compose
web:
image: apis
build: .
command: uwsgi --http 0.0.0.0:5000 --module apis.wsgi:app
env_file:
- ./.env
environment:
- POSTGRES_HOST=db
- CELERY_BROKER_URL=redis://redis:6379
- CELERY_RESULT_BACKEND_URL=redis://redis:6379
volumes:
- ./apis:/code/apis
- ./tests:/code/tests
- ./load_data:/code/load_data
- ./db/:/db/
ports:
- "5000:5000"
links:
- redis
redis:
image: redis
celery-beat:
image: apis
command: "celery -A apis.celery_app:app beat -S celerybeatredis.schedulers.RedisScheduler --loglevel=info"
env_file:
- ./.env
depends_on:
- redis
links:
- redis
environment:
- CELERY_BROKER_URL=redis://redis:6379
- CELERY_RESULT_BACKEND_URL=redis://redis:6379
- CELERY_REDIS_SCHEDULER_URL=redis://redis:6379
- C_FORCE_ROOT=true
volumes:
- ./apis:/code/apis
- ./tests:/code/tests
- ./load_data:/code/load_data
- ./db/:/db/
shm_size: '64m'
celery-worker:
image: apis
command: "celery worker -A apis.celery_app:app --loglevel=debug -E"
env_file:
- ./.env
depends_on:
- redis
- celery-beat
links:
- redis
environment:
- CELERY_BROKER_URL=redis://redis:6379
- CELERY_RESULT_BACKEND_URL=redis://redis:6379
- CELERY_REDIS_SCHEDULER_URL=redis://redis:6379
- C_FORCE_ROOT=true
volumes:
- ./apis:/code/apis
- ./tests:/code/tests
- ./load_data:/code/load_data
- ./db/:/db/
shm_size: '64m'
and the celery setup is like this...
from apis.app import init_celery
from celery.schedules import crontab
from apis.config import CELERY_REDIS_SCHEDULER_KEY_PREFIX, CELERY_REDIS_SCHEDULER_URL
from celery.task.control import inspect
app = init_celery()
app.conf.imports = app.conf.imports + ("load_data.scraping.tasks",)
app.conf.imports = app.conf.imports + ("apis.models.address", )
app.conf.beat_schedule = {
'get-data-every-day': {
'task': 'load_data.scraping.tasks.scrape_the_data_daily',
'schedule': crontab(minute='*/5'),
},
}
app.conf.timezone = 'UTC'
app.conf.CELERY_REDIS_SCHEDULER_URL = CELERY_REDIS_SCHEDULER_URL
app.conf.CELERY_REDIS_SCHEDULER_KEY_PREFIX = CELERY_REDIS_SCHEDULER_KEY_PREFIX
i = inspect()
print(10*"===", i.registered_tasks())
And celery is being initialized like this
def init_celery(app=None):
app = app or create_app()
celery.conf.broker_url = app.config["CELERY_BROKER_URL"]
celery.conf.result_backend = app.config["CELERY_RESULT_BACKEND"]
celery.conf.update(app.config)
class ContextTask(celery.Task):
"""Make celery tasks work with Flask app context"""
def __call__(self, *args, **kwargs):
with app.app_context():
return self.run(*args, **kwargs)
celery.Task = ContextTask
return celery
Basically I have 2 questions.
1rst one is why I am not getting the task when running inside the docker container?
2nd Why my tasks are not running?
Any ideas are welcomed.

Okay,
I don't know why the worker logs are not displaying the task on docker and till now.
But the problem was the scheduler beat I was using, for some weird reason, it was not sending schedule for the task.
I just change the scheduler and I found this package, very well documented and it help me to achieve what I wanted.
celery according to the documentation:
from apis.app import init_celery
from celery.schedules import crontab
from apis.config import CELERY_REDIS_SCHEDULER_URL
app = init_celery()
app.conf.imports = app.conf.imports + ("load_data.scraping.tasks",)
app.conf.imports = app.conf.imports + ("apis.models.address", )
app.conf.beat_schedule = {
'get-data-every-day': {
'task': 'load_data.scraping.tasks.scrape_the_data_daily',
'schedule': crontab(minute='*/60'),
},
}
app.conf.timezone = 'UTC'
app.conf.redbeat_redis_url = my redis url
And I updated the script that run the beat with this:
celery -A apis.celery_app:app beat -S redbeat.RedBeatScheduler --loglevel=info

I cannot comment as I don't have 50 karma. I'm willing to bet there is a networking issue present. Ensure all your containers are listening to the correct interface.
What makes me think this is that your redis service in docker-compose isn't declaring any networking parameters so the default will be used (which is localhost). This would mean that the redis container isn't accessible from outside the container.
After you docker-compose up run docker ps -a to see what interface redis is listening on.

How to get public_dns_name according to ec2 resource instance?

Now I want to get the public_dns_name, and then log on to an instance of aws to get the CPU utilization. I know that I can check it with cloudwatch, but now I don't want to rely on the function provided by amazon to check it directly based on Linux. How can I get it?
ec2 = boto3.resource('ec2',
aws_access_key_id=settings.AWS.get("aws_access_key_id"),
aws_secret_access_key=settings.AWS.get("aws_secret_access_key"),
region_name=settings.AWS.get("region_name"))
user_data_script = """#cloud-config
runcmd:
- sudo systemctl start docker
- cd /srv/app/kfquantum
- sudo docker run --env broker=%s \
--env result=redis://10.0.0.245:6379/0 \
--env host=kf-db.cluster-crietao0wrdu.rds.cn-northwest-1.amazonaws.com.cn \
--env user=db \
--env pw=y6y83aGagY8hgXZ \
-w /kfquantum -d --restart unless-stopped --name kf-celery kf-celery \
celery -A kfquantum worker --pool=solo -l info
""" % settings.BROKER_URL
instance_type = "c5.large"
instance = ec2.create_instances(
BlockDeviceMappings=[
{
"DeviceName": "/dev/xvda",
'Ebs': {
'DeleteOnTermination': True,
'VolumeSize': 8
},
},
],
ImageId="ami-09feea664bfc54194",
MinCount=1,
MaxCount=1,
Placement={'AvailabilityZone': 'cn-northwest-1a'},
InstanceType=instance_type,
UserData=user_data_script,
NetworkInterfaces=[
{
'DeviceIndex': 0,
'SubnetId': "subnet-093161e8b00deb367",
'AssociatePublicIpAddress': True
},
]
)
instance = instance[0]
print("1",instance.private_ip_address)
print("2",instance.public_dns_name)
Console
1,10.0.0.73
2

Try this...
instance= instance[0]
print('***Success!! instance:', instance.id, 'created, and instance-state-name:', instance.state['Name'])
instance.wait_until_running()
instance.reload()
print('***instance:', instance.id, 'is now up, and instance-state-name:', instance.state['Name'])
print('***private ipa:', instance.private_ip_address)
print('***public ipa:', instance.public_dns_name)
Hope it helps...
r0ck

The value for public_dns_name is not assigned yet.
This could be due to
The instance is not running yet. You are trying to fetch the DNS name as soon as the create_instances method is invoked. It takes a few seconds for the instance to reach running state. The public_dns_name is not available until the instance reaches running state. Check for the instance state and wait for it to be running before trying to get the Public DNS Name.
An easy fix without disturbing the rest of your code would be,
Update:
The instance object must be reloaded after the waiter for the updated instance attributes to be detected.
instance = instance[0]
instance.wait_until_running()
instance.reload()
print("1",instance.private_ip_address)
print("2",instance.public_dns_name)
DNS hostnames are not enabled in your VPC.

Django/Celery multiple queues on localhost - routing not working

I followed celery docs to define 2 queues on my dev machine.
My celery settings:
CELERY_ALWAYS_EAGER = True
CELERY_TASK_RESULT_EXPIRES = 60 # 1 mins
CELERYD_CONCURRENCY = 2
CELERYD_MAX_TASKS_PER_CHILD = 4
CELERYD_PREFETCH_MULTIPLIER = 1
CELERY_CREATE_MISSING_QUEUES = True
CELERY_QUEUES = (
Queue('default', Exchange('default'), routing_key='default'),
Queue('feeds', Exchange('feeds'), routing_key='arena.social.tasks.#'),
)
CELERY_ROUTES = {
'arena.social.tasks.Update': {
'queue': 'fs_feeds',
},
}
i opened two terminal windows, in virtualenv of my project, and ran following commands:
terminal_1$ celery -A arena worker -Q default -B -l debug --purge -n deafult_worker
terminal_2$ celery -A arena worker -Q feeds -B -l debug --purge -n feeds_worker
what i get is that all tasks are being processed by both queues.
My goal is to have one queue to process only the one task defined in CELERY_ROUTES and default queue to process all other tasks.
I also followed this SO question, rabbitmqctl list_queues returns celery 0, and running rabbitmqctl list_bindings returns exchange celery queue celery [] twice. Restarting rabbit server didn't change anything.

Ok, so i figured it out. Following is my whole setup, settings and how to run celery, for those who might be wondering about same thing as my question did.
Settings
CELERY_TIMEZONE = TIME_ZONE
CELERY_ACCEPT_CONTENT = ['json', 'pickle']
CELERYD_CONCURRENCY = 2
CELERYD_MAX_TASKS_PER_CHILD = 4
CELERYD_PREFETCH_MULTIPLIER = 1
# celery queues setup
CELERY_DEFAULT_QUEUE = 'default'
CELERY_DEFAULT_EXCHANGE_TYPE = 'topic'
CELERY_DEFAULT_ROUTING_KEY = 'default'
CELERY_QUEUES = (
Queue('default', Exchange('default'), routing_key='default'),
Queue('feeds', Exchange('feeds'), routing_key='long_tasks'),
)
CELERY_ROUTES = {
'arena.social.tasks.Update': {
'queue': 'feeds',
'routing_key': 'long_tasks',
},
}
How to run celery?
terminal - tab 1:
celery -A proj worker -Q default -l debug -n default_worker
this will start first worker that consumes tasks from default queue. NOTE! -n default_worker is not a must for the first worker, but is a must if you have any other celery instances up and running. Setting -n worker_name is the same as --hostname=default#%h.
terminal - tab 2:
celery -A proj worker -Q feeds -l debug -n feeds_worker
this will start second worker that consumers tasks from feeds queue. Notice -n feeds_worker, if you are running with -l debug (log level = debug), you will see that both workers are syncing between them.
terminal - tab 3:
celery -A proj beat -l debug
this will start the beat, executing tasks according to the schedule in your CELERYBEAT_SCHEDULE.
I didn't have to change the task, or the CELERYBEAT_SCHEDULE.
For example, this is how looks my CELERYBEAT_SCHEDULE for the task that should go to feeds queue:
CELERYBEAT_SCHEDULE = {
...
'update_feeds': {
'task': 'arena.social.tasks.Update',
'schedule': crontab(minute='*/6'),
},
...
}
As you can see, no need for adding 'options': {'routing_key': 'long_tasks'} or specifying to what queue it should go. Also, if you were wondering why Update is upper cased, its because its a custom task, which are defined as sub classes of celery.Task.
Update Celery 5.0+
Celery made a couple changes since version 5, here is an updated setup for routing of tasks.
How to create the queues?
Celery can create the queues automatically. It works perfectly for simple cases, where celery default values for routing are ok.
task_create_missing_queues=True or, if you're using django settings and you're namespacing all celery configs under CELERY_ key, CELERY_TASK_CREATE_MISSING_QUEUES=True. Note, that it is on by default.
Automatic scheduled task routing
After configuring celery app:
celery_app.conf.beat_schedule = {
"some_scheduled_task": {
"task": "module.path.some_task",
"schedule": crontab(minute="*/10"),
"options": {"queue": "queue1"}
}
}
Automatic task routing
Celery app still has to be configured first and then:
app.conf.task_routes = {
"module.path.task2": {"queue": "queue2"},
}
Manual routing of tasks
In case and you want to route the tasks dynamically, then when sending the task specify the queue:
from module import task
def do_work():
# do some work and launch the task
task.apply_async(args=(arg1, arg2), queue="queue3")
More details re routing can be found here:
https://docs.celeryproject.org/en/stable/userguide/routing.html
And regarding calling tasks here:
https://docs.celeryproject.org/en/stable/userguide/calling.html

In addition to accepted answer, if anyone comes here and still wonders why his settings aren't working (as I did just moments ago), here's why: celery documentation isn't listing settings names properly.
For celery 5.0.5 settings CELERY_DEFAULT_QUEUE, CELERY_QUEUES, CELERY_ROUTES should be named CELERY_TASK_DEFAULT_QUEUE, CELERY_TASK_QUEUESand CELERY_TASK_ROUTES instead. These are settings that I've tested, but my guess is the same rule applies for exchange and routing key aswell.