Develop Reference

Run a custom task asynchronously in airflow using existing celery

I have a running airflow with celery and redis.
This by default sends dag's task to celery worker.
I want to run a custom task from one of DAG's task from python code.
In tasks.py I have following code.
from airflow.configuration import conf
from airflow.config_templates.default_celery import DEFAULT_CELERY_CONFIG
from celery import Celery
from celery import shared_task
if conf.has_option('celery', 'celery_config_options'):
celery_configuration = conf.getimport('celery', 'celery_config_options')
else:
celery_configuration = DEFAULT_CELERY_CONFIG
app = Celery(conf.get('celery', 'CELERY_APP_NAME'), config_source=celery_configuration,include=["dags.tasks"])
app.autodiscover_tasks(force=True)
print("here")
print(conf.get('celery', 'CELERY_APP_NAME'))
print(celery_configuration)
print(app)
#app.task(name='maximum')
def maximum(x=10, y=11):
#print("here")
print(x)
if x > y:
return x
else:
return y
tasks = app.tasks.keys()
print(tasks)
I am calling this from one of the DAG's task.
max=maximum.apply_async(kwargs={'x':5, 'y':4})
print(max)
print(max.get(timeout=5))
I am geting
File "/home/airflow/.local/lib/python3.7/site-packages/celery/result.py", line 336, in maybe_throw
self.throw(value, self._to_remote_traceback(tb))
File "/home/airflow/.local/lib/python3.7/site-packages/celery/result.py", line 329, in throw
self.on_ready.throw(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/vine/promises.py", line 234, in throw
reraise(type(exc), exc, tb)
File "/home/airflow/.local/lib/python3.7/site-packages/vine/utils.py", line 30, in reraise
raise value
celery.exceptions.NotRegistered: 'maximum'
In the registered tasks from above I am getting :
tasks = app.tasks.keys()
print(tasks)
output
dict_keys(['celery.chunks', 'airflow.executors.celery_executor.execute_command', 'maximum', 'celery.backend_cleanup', 'celery.chord_unlock', 'celery.group', 'celery.map', 'celery.accumulate', 'celery.chain', 'celery.starmap', 'celery.chord'])
Maximum is there in registered tasks.
The airflow worker is run from docker as follows(snip from docker-compose.yaml):
airflow-worker:
<<: *airflow-common
command: celery worker
healthcheck:
test:
- "CMD-SHELL"
- 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery#$${HOSTNAME}"'
interval: 10s
timeout: 10s
retries: 5
restart: always
Full docker-compose.yaml
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-tanesca-airflow:2.1.0}
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow#postgres/airflow
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow#postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow#postgres/airflow
AIRFLOW__CELERY__BROKER_URL: redis://:#redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth'
# _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-pandas kiteconnect}
# _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-pandas}
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
# user: "${AIRFLOW_UID:-50000}:0"
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
services:
postgres:
image: postgres:13
environment:
POSTGRES_USER: ****
POSTGRES_PASSWORD: ***
POSTGRES_DB: ***
volumes:
- postgres-db-volume:/var/lib/postgresql/data
ports:
- 5432:5432
healthcheck:
test: ["CMD", "pg_isready", "-U", "airflow"]
interval: 5s
retries: 5
restart: always
redis:
image: redis:latest
ports:
- 6379:6379
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 30s
retries: 50
restart: always
airflow-webserver:
<<: *airflow-common
command: webserver
ports:
- 8080:8080
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-scheduler:
<<: *airflow-common
command: scheduler
healthcheck:
test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"']
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-worker:
<<: *airflow-common
command: celery worker
healthcheck:
test:
- "CMD-SHELL"
- 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery#$${HOSTNAME}"'
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-init:
<<: *airflow-common
command: version
environment:
<<: *airflow-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
flower:
<<: *airflow-common
command: celery flower
ports:
- 5555:5555
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
interval: 10s
timeout: 10s
retries: 5
restart: always
volumes:
postgres-db-volume:
airflow worker logs
-------------- celery#eecdca8a08ff v5.2.7 (dawn-chorus)
--- ***** -----
-- ******* ---- Linux-5.15.0-1019-aws-x86_64-with-debian-11.4 2022-09-02 12:35:42
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: airflow.executors.celery_executor:0x7fa27b38b0d0
- ** ---------- .> transport: redis://redis:6379/0
- ** ---------- .> results: postgresql://airflow:**#postgres/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> default exchange=default(direct) key=default
[tasks]
. airflow.executors.celery_executor.execute_command
[2022-09-02 12:35:50,295: INFO/MainProcess] Connected to redis://redis:6379/0
[2022-09-02 12:35:50,310: INFO/MainProcess] mingle: searching for neighbors

I assume that you simply want to run custom Python code within your task. Not sure why you are using Celery decorator, maybe I missed something.
Anyway, I would recommend using PythonOperator for that. You need to implement your own logic and it will run in celery worker.
Based on your code above, I've created a short example:
import logging
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
def maximum(**kwargs):
logging.warning(f"got this args: {kwargs}")
x = kwargs.get("x")
y = kwargs.get("y")
if x > y:
return x
else:
return y
def minimum(**kwargs):
logging.warning(f"got this args: {kwargs}")
x = kwargs.get("x")
y = kwargs.get("y")
if x > y:
return x
else:
return y
with DAG(
'tutorial',
default_args={
},
description='A simple tutorial DAG',
schedule_interval=timedelta(days=1),
start_date=datetime(2021, 1, 1),
catchup=False,
tags=['example'],
) as dag:
op_kwargs = {
"x": 10,
"y": 11,
}
t1 = PythonOperator(
task_id="my_max_python_task",
python_callable=maximum,
dag=dag,
op_kwargs=op_kwargs
)
t2 = PythonOperator(
task_id="my_min_python_task",
python_callable=minimum,
dag=dag,
op_kwargs=op_kwargs
)
t1 >> t2
You can see that it ran:
And return the result (if you have downstream task to consume that):

Celery task is always PENDING inside Docker container (Flask + Celery + RabbitMQ + Docker)

I'm creating a basic project to test Flask + Celery + RabbitMQ + Docker.
For some reason, that I do not know, when I call the celery, the task seems to call RabbitMQ, but it stays at the PENDING state always, it never changes to another state. I try to use task.get(), but the code freezes. Example:
The celery worker (e.g. worker_a.py) is something like this:
from celery import Celery
# Initialize Celery
celery = Celery('worker_a',
broker='amqp://guest:guest#tfcd_rabbit:5672//',
backend='rpc://')
[...]
#celery.task()
def add_nums(a, b):
return a + b
While docker-compose.yml is something like this:
[...]
tfcd_rabbit:
container_name: tfcd_rabbit
hostname: tfcd_rabbit
image: rabbitmq:3.8.11-management
environment:
- RABBITMQ_ERLANG_COOKIE=test
- RABBITMQ_DEFAULT_USER=guest
- RABBITMQ_DEFAULT_PASS=guest
ports:
- 5672:5672
- 15672:15672
networks:
- tfcd
tfcd_worker_a:
container_name: tfcd_worker_a
hostname: tfcd_worker_1
image: test_flask_celery_docker
entrypoint: celery
command: -A worker_a worker -l INFO -Q worker_a
volumes:
- .:/app
links:
- tfcd_rabbit
depends_on:
- tfcd_rabbit
networks:
- tfcd
[...]
The repository with all the files and instructions to run it can be found here.
Would anyone know what might be going on?
Thank you in advance.

After a while, a friend of mine discovered the problem:
The correct queue name was missing when creating the task, because Celery was using the default name "celery" instead of the correct queue name.
The final code is this:
[...]
#celery.task(queue='worker_a')
def add_nums(a, b):
return a + b

Job is visible only from redis cli but not showing in rq dashboard and not executed

I want to build a pipeline using Redis and RQ. I created a worker, server and a job, the worker is running and listening to queue, the server is dispatching a job to a queue, the job is dispatched and I print the job ID, in console, I can see the worker logs sth that receive a job in a queue. The job is never executing and never shows in rq dashboard, but I can see it in Redis CLI.
Verions I am using:
rq==1.7.0
redis==3.5.0
Here is my code:
Worker in run.py
import os
import redis
from rq import Worker, Queue, Connection
listen = ['stance_queue','default']
redis_url = os.getenv('REDIS_URL', 'redis://redis:6379')
conn = redis.from_url(redis_url)
# conn = redis.Redis(host='redis', port=6379)
if __name__ == '__main__':
with Connection(conn):
print("Createing worker")
worker = Worker(map(Queue, listen))
# worker = Worker([Queue()])
worker.work()
And here were I dispatch a job
from workers.stance.run import conn
q = Queue('default', connection=conn)
#server.route("/task")
def home():
if request.args.get("n"):
print('create a job in default queue')
job = q.enqueue( background_task, args=(20,))
return f"Task ({job.id}) added to queue at {job.enqueued_at}"
return "No value for count provided"
And here is the background job
def background_task(n):
""" Function that returns len(n) and simulates a delay """
delay = 2
print("Task running", flush=True)
print(f"Simulating a {delay} second delay", flush=True)
time.sleep(delay)
print(len(n))
print("Task complete")
return len(n)
Here is a screenshot for rq-dashboard
And here is the logs in the worker
Attaching to annotators_server_stance_worker_1
stance_worker_1 | Createing worker
stance_worker_1 | 08:33:44 Worker rq:worker:cae161cf792b4c998376cde2c0848291: started, version 1.7.0
stance_worker_1 | 08:33:44 Subscribing to channel rq:pubsub:cae161cf792b4c998376cde2c0848291
stance_worker_1 | 08:33:44 *** Listening on stance_queue, default...
stance_worker_1 | 08:33:44 Cleaning registries for queue: stance_queue
stance_worker_1 | 08:33:44 Cleaning registries for queue: default
stance_worker_1 | 08:33:49 default: home.annotator_server.background_task(20) (9f1f31e0-f465-4019-9dc6-85bc349feab9)
and here is the logs from redis-cli
mpose exec redis redis-cli
127.0.0.1:6379> keys *
1) "rq:workers"
2) "rq:failed:default"
3) "rq:clean_registries:default"
4) "rq:queues"
5) "rq:job:9f1f31e0-f465-4019-9dc6-85bc349feab9"
6) "rq:worker:cae161cf792b4c998376cde2c0848291"
7) "rq:workers:default"
8) "rq:clean_registries:stance_queue"
9) "rq:workers:stance_queue"
And here is my compose
version: '3'
services:
annotators_server:
build:
context: .
dockerfile: Dockerfile
ports:
- "5000:5000"
volumes:
- ./app:/home
depends_on:
- redis
redis:
image: "redis:alpine"
dashboard:
image: "godber/rq-dashboard"
ports:
- 9181:9181
command: rq-dashboard -H redis
depends_on:
- redis
stance_worker:
build:
context: ./app/workers/stance
dockerfile: Dockerfile
environment:
- REDIS_URL=redis://redis:6379
depends_on:
- redis
I never see a logs for the job excution, I tried to add TTL and TIMEOUT but still facing the samething.

Pass the redis database to the connection string, when starting dashboard and worker.
Redis url = redis://redis-host:6379/0 ( this refers to db 0 ).

Celery - worker only sometimes picks up tasks

I am building a lead generation portal that can be accessed online. Please don't mind the verbosity of the code, I'm doing a lot of debugging right now.
My Celery worker inconsistently picks up tasks assigned to it, and I'm not sure why.
The weird thing about this, is that sometimes it works 100% perfect: there never are any explicit errors in the terminal.
I am currently in DEBUG = TRUE and REDIS as a broker!
celery start worker terminal command and response
celery -A mysite worker -l info --pool=solo
-------------- celery#DESKTOP-OG8ENRQ v5.0.2 (singularity)
--- ***** -----
-- ******* ---- Windows-10-10.0.19041-SP0 2020-11-09 00:36:13
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: mysite:0x41ba490
- ** ---------- .> transport: redis://localhost:6379//
- ** ---------- .> results: redis://localhost:6379/
- *** --- * --- .> concurrency: 12 (solo)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. mysite.celery.debug_task
. submit
[2020-11-09 00:36:13,899: INFO/MainProcess] Connected to redis://localhost:6379//
[2020-11-09 00:36:14,939: WARNING/MainProcess] c:\users\coole\pycharmprojects\lead_django_retry\venv\lib\site-packages\celery\app\control.py:48: DuplicateNodenameWarning: Received multiple replies from node name: celery#DESKTOP-OG8ENRQ.
Please make sure you give each node a unique nodename using
the celery worker `-n` option.
warnings.warn(DuplicateNodenameWarning(
[2020-11-09 00:36:14,939: INFO/MainProcess] mingle: all alone
[2020-11-09 00:36:14,947: INFO/MainProcess] celery#DESKTOP-OG8ENRQ ready.
views.py
class LeadInputView(FormView):
template_name = 'lead_main.html'
form_class = LeadInput
def form_valid(self, form):
print("I'm at views")
form.submit()
print(form.submit)
return HttpResponseRedirect('./success/')
tasks.py
#task(name="submit")
def start_task(city, category, email):
print("I'm at tasks!")
print(city, category, email)
"""sends an email when feedback form is filled successfully"""
logger.info("Submitted")
return start(city, category, email)
forms.py
class LeadInput(forms.Form):
city = forms.CharField(max_length=50)
category = forms.CharField(max_length=50)
email = forms.EmailField()
def submit(self):
print("I'm at forms!")
x = (start_task.delay(self.cleaned_data['city'], self.cleaned_data['category'], self.cleaned_data['email']))
return x
celery.py
from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')
app = Celery('mysite')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
settings.py
BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'
The runserver terminal will look something like this:
I'm at views
I'm at forms!
<bound method LeadInput.submit of <LeadInput bound=True, valid=True, fields=(city;category;email)>>
But the worker doesn't say that it picked up anything, just that "celery#DESKTOP-OG8ENRQ ready." Except, when it does work... for some reason? I'm at a loss!

Hello to whoever sees this. It turns out, that this is a bug with celery (or maybe redis?)... apparently many windows users run into this. https://github.com/celery/celery/issues/3759
Turns out, the answer is to make -P solo when starting worker. I'm not sure why this is the case... but that solved it!
Thank you Naqib for your help! You put me down the right rabbit hole to a solution.

By default, celery will use the hostname as worker name if your willing to use multiple workers in the same host then specify -n option.
celery -A mysite worker -l info --pool=solo -n worker2#%h
Your code works fine but the task is passed to the first worker, see
DuplicateNodenameWarning with no obvious reason #2938

Celery sees my task but does not execute it

I'm trying to run a celery chain in local, with a Redis broker.
The problem: As you can see below the worker command, tasks are not ran. Nothing happens.
Find below all the informations needed.
The redis-server is running
redis-server
[15048] 05 Feb 11:36:30 * Server started, Redis version 2.4.5
[15048] 05 Feb 11:36:30 * DB loaded from disk: 0 seconds
[15048] 05 Feb 11:36:30 * The server is now ready to accept connections on port 6379
...
The celery beat is also running
celery beat -b redis://localhost:6379/0
celery beat v4.2.1 (windowlicker) is starting.
__ - ... __ - _
LocalTime -> 2020-02-05 11:37:30
Configuration ->
. broker -> redis://localhost:6379/0
. loader -> celery.loaders.default.Loader
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule
. logfile -> [stderr]#%WARNING
. maxinterval -> 5.00 minutes (300s)
But when I run the worker, it can see the tasks, but does not execute them:
celery -A module.tasks worker --loglevel=info -Q training_queue -P eventlet
[tasks]
. module.tasks.generate_the_chain
. module.tasks.post_pipeline_message
. module.tasks.train_by_site_post
[2020-02-05 11:39:14,700: INFO/MainProcess] Connected to redis://localhost:6379/0
[2020-02-05 11:39:16,746: INFO/MainProcess] mingle: searching for neighbors
[2020-02-05 11:39:20,804: INFO/MainProcess] mingle: all alone
[2020-02-05 11:39:21,836: INFO/MainProcess] pidbox: Connected to redis://localhost:6379/0.
[2020-02-05 11:39:22,850: INFO/MainProcess] celery#HOST ready.
Nothing happens after this line.
BELOW is the code:
init.py
import os
from celery import Celery
CELERY_APP_NAME = 'app_name'
CELERY_BROKER = 'redis://localhost:6379/0'
CELERY_BACKEND = CELERY_BROKER
CELERY = Celery(CELERY_APP_NAME,
backend=CELERY_BACKEND,
broker=CELERY_BROKER,
include=['module.tasks']
)
And tasks.py
import os
import logging
import time
from celery import Celery, chain, signature
from celery.schedules import crontab
from module import CELERY
#CELERY.on_after_finalize.connect
def setup_periodic_tasks(sender, **kwargs):
site_id = "f1ae"
sign = signature(
'module.tasks.generate_the_chain',
args=(site_id),
queue='training_queue'
)
sender.add_periodic_task(5.0, sign, name='training')
#CELERY.task
def generate_the_chain(site_id):
print("Hello World")
"""
This function run the chain of functions and is then
scheduled with Celery beat
"""
chain = signature(
'module.tasks.site_post',
args=(site_id),
queue='training_queue'
)
chain |= signature(
'module.tasks.post_pipeline_message',
args=(),
queue='training_queue'
)
chain.apply_async()
#CELERY.task
def site_post(site_id):
print(f"Hello {site_id}")
return True
#CELERY.task
def post_pipeline_message(success):
if success is True:
LOGGER.info("Pipeline build with success")