Develop Reference

ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile

I'm getting this error when I try to perform celery -A draft1 beat in my terminal.
Traceback (most recent call last):
File "/home/james/postr/env/lib/python3.5/site-packages/celery/apps/beat.py", line 107, in start_scheduler
service.start()
File "/home/james/postr/env/lib/python3.5/site-packages/celery/beat.py", line 558, in start
interval = self.scheduler.tick()
File "/home/james/postr/env/lib/python3.5/site-packages/celery/beat.py", line 279, in tick
self.apply_entry(entry, producer=self.producer)
File "/home/james/postr/env/lib/python3.5/site-packages/kombu/utils/objects.py", line 44, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/home/james/postr/env/lib/python3.5/site-packages/celery/beat.py", line 411, in producer
return self.Producer(self._ensure_connected(), auto_declare=False)
File "/home/james/postr/env/lib/python3.5/site-packages/celery/beat.py", line 395, in _ensure_connected
_error_handler, self.app.conf.broker_connection_max_retries
File "/home/james/postr/env/lib/python3.5/site-packages/kombu/connection.py", line 405, in ensure_connection
callback)
File "/home/james/postr/env/lib/python3.5/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
return fun(*args, **kwargs)
File "/home/james/postr/env/lib/python3.5/site-packages/kombu/connection.py", line 261, in connect
return self.connection
File "/home/james/postr/env/lib/python3.5/site-packages/kombu/connection.py", line 802, in connection
self._connection = self._establish_connection()
File "/home/james/postr/env/lib/python3.5/site-packages/kombu/connection.py", line 757, in _establish_connection
conn = self.transport.establish_connection()
File "/home/james/postr/env/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/home/james/postr/env/lib/python3.5/site-packages/amqp/connection.py", line 288, in connect
self.drain_events(timeout=self.connect_timeout)
File "/home/james/postr/env/lib/python3.5/site-packages/amqp/connection.py", line 471, in drain_events
while not self.blocking_read(timeout):
File "/home/james/postr/env/lib/python3.5/site-packages/amqp/connection.py", line 477, in blocking_read
return self.on_inbound_frame(frame)
File "/home/james/postr/env/lib/python3.5/site-packages/amqp/method_framing.py", line 55, in on_frame
callback(channel, method_sig, buf, None)
File "/home/james/postr/env/lib/python3.5/site-packages/amqp/connection.py", line 481, in on_inbound_method
method_sig, payload, content,
File "/home/james/postr/env/lib/python3.5/site-packages/amqp/abstract_channel.py", line 128, in dispatch_method
listener(*args)
File "/home/james/postr/env/lib/python3.5/site-packages/amqp/connection.py", line 603, in _on_close
(class_id, method_id), ConnectionError)
amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
I'm running celery on my remote Ubuntu django server.
Any idea what the problem is?
Here's my code:
settings
CELERYBEAT_SCHEDULE = {
'post_jobs': {
'task': 'post.tasks.post_jobs', # the same goes in the task name
'schedule': crontab(minute=40),
},
'test_post': {
'task': 'post.tasks.test_post',
'schedule': crontab(minute=40),
}
}
draft1/celery.py
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'draft1.settings')
app = Celery("draft1", broker=CELERY_BROKER_URL)
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
post/celery.py
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'draft1.settings')
app = Celery(broker=CELERY_BROKER_URL)
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
post/tasks.py
#app.task
def test_post():
from .models import Post
for i in Post.objects.all():
if i.entered_category == "test":
i.entered_category = "not_test"
i.save()
return HttpResponseRedirect('/')
postr-celery.conf
[program:postr-celery]
command=/home/james/postr/env/bin/celery -A post worker --loglevel=INFO
directory=/home/james/postr
user=james
numprocs=1
stdout_logfile=/var/log/supervisor/celery.log
stderr_logfile=/var/log/supervisor/celery.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
stopasgroup=true
; Set Celery priority higher than default (999)
; so, if rabbitmq is supervised, it will start first.
priority=1000
I'm not sure what sparked this error, my celery was working recently. Any idea what the problem is?

There is a good chance that a software update of the SSL features caused the problem. I found my issue in the /var/log/rabbitmq/rabbit#host.log file.
In my case, I found the following, and to resolve I had to re-install my rabbitmq / celery.
/lib/erlang/lib/crypto-4.2/priv/lib/crypto: 'libcrypto.so.1.0.0: cannot open shared object file: No such file
a test of rabbitmqctl list_users and add_users can help determine if your rabbitmqctl is installed correctly when you monitor the logs.

Celery tries to import the broker URL as a module?

I recently added Celery to my back end but i got this weird error below
[2017-10-25 21:41:37,142: CRITICAL/MainProcess] Unrecoverable error: ImportError('No module named myredisserverip.com',)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/usr/local/lib/python2.7/dist-packages/celery/bootsteps.py", line 115, in start
self.on_start()
File "/usr/local/lib/python2.7/dist-packages/celery/apps/worker.py", line 143, in on_start
self.emit_banner()
File "/usr/local/lib/python2.7/dist-packages/celery/apps/worker.py", line 158, in emit_banner
' \n', self.startup_info(artlines=not use_image))),
File "/usr/local/lib/python2.7/dist-packages/celery/apps/worker.py", line 221, in startup_info
results=self.app.backend.as_uri(),
File "/usr/local/lib/python2.7/dist-packages/kombu/utils/objects.py", line 44, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/usr/local/lib/python2.7/dist-packages/celery/app/base.py", line 1183, in backend
return self._get_backend()
File "/usr/local/lib/python2.7/dist-packages/celery/app/base.py", line 901, in _get_backend
self.loader)
File "/usr/local/lib/python2.7/dist-packages/celery/app/backends.py", line 66, in by_url
return by_name(backend, loader), url
File "/usr/local/lib/python2.7/dist-packages/celery/app/backends.py", line 46, in by_name
cls = symbol_by_name(backend, aliases)
File "/usr/local/lib/python2.7/dist-packages/kombu/utils/imports.py", line 56, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named myredisserverip.com
My config.py looks like this
class BaseConfig(object):
""" A base configuration of the app """
DEBUG = False
SERVER_NAME = "my-production-ip"
SECRET_KEY = os.environ['SECRET']
BASE_DIR = os.path.abspath(os.path.dirname(__file__))
SQLALCHEMY_DATABASE_URI = os.environ['SQL_PRODUCTION']
SQLALCHEMY_TRACK_MODIFICATIONS = True
CELERY_BROKER_URL = os.environ['CELERY_BROKER_PROD']
CELERY_RESULT_BACKEND = os.environ['CELERY_RESULT_BACKEND_PROD']
DATABASE_CONNECT_OPTIONS = {}
THREADS_PER_PAGE = 2
CSRF_ENABLED = True
CSRF_SESSION_KEY = "secret"
MAIL_SERVER = "smtp.gmail.com"
MAIL_PORT = 465
MAIL_USE_SSL = True
MAIL_USE_TSL = False
MAIL_USERNAME = "blabla"
MAIL_PASSWORD = "pwd"
CELERY_BROKER_PROD and CELERY_RESULT_BACKEND are both the same and they contain the URL of the redis instance im running on amazon AWS. When i try to run
celery worker -A app.celery
from within my project directory i get this error, what is happening?
The way i setup celery is this
celery = Celery(app.name, broker=app.config['CELERY_BROKER_URL'])
celery.conf.update(app.config)

can you confirm what the values of CELERY_BROKER_PROD and CELERY_RESULTS_BACKEND are?
It seems that you have defined this as myredisserverip.com, however according to the celery docs, redis server should be defined as:
CELERY_RESULT_BACKEND = 'redis://:password#host:port/db'
as per the documentation here: http://docs.celeryproject.org/en/3.1/configuration.html#redis-backend-settings
For the broker_url you need to define the transport, documentation on this can be found here:
http://docs.celeryproject.org/en/3.1/configuration.html#broker-url
Is the issue that you are missing the transport, ie the redis:// prefix in your environment variables?

Error 111 connection refused (Python, celery, redis)

I tried to get all the active/scheduled/reserved tasks in redis:
from celery.task.control import inspect
inspect_obj = inspect()
inspect_obj.active()
inspect_obj.scheduled()
inspect_obj.reserved()
But was greeted with a list of errors as follows:
My virtual environment ==> HubblerAPI.
Iam using this from the ec2 console
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/celery/app/control.py", line 81, in active
return self._request('dump_active', safe=safe)
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/celery/app/control.py", line 71, in _request
timeout=self.timeout, reply=True,
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/celery/app/control.py", line 316, in broadcast
limit, callback, channel=channel,
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/kombu/pidbox.py", line 283, in _broadcast
chan = channel or self.connection.default_channel
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/kombu/connection.py", line 771, in default_channel
self.connection
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/kombu/connection.py", line 756, in connection
self._connection = self._establish_connection()
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/kombu/connection.py", line 711, in _establish_connection
conn = self.transport.establish_connection()
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/kombu/transport/pyamqp.py", line 116, in establish_connection
conn = self.Connection(**opts)
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/amqp/connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/amqp/connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/amqp/transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "/home/ec2-user/HubblerAPI/local/lib/python3.4/site-
packages/amqp/transport.py", line 95, in __init__
raise socket.error(last_err)
**OSError: [Errno 111] Connection refused**
My celery config file is as follows:
BROKER_TRANSPORT = 'redis'
BROKER_TRANSPORT_OPTIONS = {
'queue_name_prefix': 'dev-',
'wait_time_seconds': 10,
'polling_interval': 30,
# The polling interval decides the number of seconds to sleep
between unsuccessful polls
'visibility_timeout': 3600 * 5,
# If a task is not acknowledged within the visibility_timeout, the
task will be redelivered to another worker and executed.
}
CELERY_MESSAGES_DB = 6
BROKER_URL = "redis://%s:%s/%s" % (AWS_REDIS_ENDPOINT, AWS_REDIS_PORT,
CELERY_MESSAGES_DB)
What am i doing wrong here as the error log suggests that its not using the redis broker.

Looks like your python code doesn't recognize your configs since it is attempting to use RabbitMQ's ampq protocol instead of the configured broker.
I suggest the following
https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html
Your configs look similar to Django configs for Celery yet it doesn't seem you are using Celery with Django.
https://docs.celeryq.dev/en/latest/django/first-steps-with-django.html

The issue is using "BROKER_URL" instead of "CELERY_BROKER_URL" in settings.py. Celery wasn't finding the URL and was defaulting to the rabbitmq port instead of the redis port.

How to debug "pika.exceptions.AuthenticationError: EXTERNAL" error when establishing TLS connection to RabbitMQ?

I have a RabbitMQ 3.6.1 server on Ubuntu 14.04 running properly. I tried to configure an SSL listener according to official documentation. No problems during the startup.
However when trying to establish a connection, I get the following error on Python/pika side (full transcript below):
pika.exceptions.AuthenticationError: EXTERNAL
What does EXTERNAL mean here? How to debug / get further details of the error?
Course of actions (to test I used a Vagrant box and a local connection):
RabbitMQ starts SSL Listener on port 5671 (per /var/log/rabbitmq/rabbit#rabbitmq-server.log):
started SSL Listener on [::]:5671
I execute the pika.BlockingConnection on the client side.
On the server side I can see an incoming connection:
=INFO REPORT==== 17-Apr-2016::17:07:15 ===
accepting AMQP connection <0.2788.0> (127.0.0.1:48404 -> 127.0.0.1:5671)
Client fails with:
pika.exceptions.AuthenticationError: EXTERNAL
Server timeouts:
=ERROR REPORT==== 17-Apr-2016::17:07:25 ===
closing AMQP connection <0.2788.0> (127.0.0.1:48404 -> 127.0.0.1:5671):
{handshake_timeout,frame_header}
Full transcript of the client side:
>>> import pika, ssl
>>> from pika.credentials import ExternalCredentials
>>> ssl_options = ({"ca_certs": "/etc/rabbitmq/certs/testca/cacert.pem",
... "certfile": "/etc/rabbitmq/certs/client/cert.pem",
... "keyfile": "/etc/rabbitmq/certs/client/key.pem",
... "cert_reqs": ssl.CERT_REQUIRED,
... "server_side": False})
>>> host = "localhost"
>>> connection = pika.BlockingConnection(
... pika.ConnectionParameters(
... host, 5671, credentials=ExternalCredentials(),
... ssl=True, ssl_options=ssl_options))
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 339, in __init__
self._process_io_for_connection_setup()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 374, in _process_io_for_connection_setup
self._open_error_result.is_ready)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 410, in _flush_output
self._impl.ioloop.poll()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/select_connection.py", line 602, in poll
self._process_fd_events(fd_event_map, write_only)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/select_connection.py", line 443, in _process_fd_events
handler(fileno, events, write_only=write_only)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 364, in _handle_events
self._handle_read()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 415, in _handle_read
self._on_data_available(data)
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1347, in _on_data_available
self._process_frame(frame_value)
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1414, in _process_frame
if self._process_callbacks(frame_value):
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1384, in _process_callbacks
frame_value) # Args
File "/usr/local/lib/python2.7/dist-packages/pika/callback.py", line 60, in wrapper
return function(*tuple(args), **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pika/callback.py", line 92, in wrapper
return function(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pika/callback.py", line 236, in process
callback(*args, **keywords)
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1298, in _on_connection_start
self._send_connection_start_ok(*self._get_credentials(method_frame))
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1077, in _get_credentials
raise exceptions.AuthenticationError(self.params.credentials.TYPE)
pika.exceptions.AuthenticationError: EXTERNAL
>>>

The Python / pika code in the question is correct.
The error:
pika.exceptions.AuthenticationError: EXTERNAL
is reported when client certificate authorisation is not enabled on the RabbitMQ server side. The word EXTERNAL in the error refers to the authentication mechanism as described here.
To enable:
rabbitmq-plugins enable rabbitmq_auth_mechanism_ssl

Error using paho-mqtt in App Engine Python App

I am trying to right a Google Cloud Platform app in python with Flask that makes an MQTT connection. I have included the paho python library by doing pip install paho-mqtt -t libs/. However, when I try to run the app, even if I don't try to connect to MQTT. I get a weird error about IP address checking:
RuntimeError: error('illegal IP address string passed to inet_pton',)
It seems something in the remote_socket lib is causing a problem. Is this a security issue? Is there someway to disable it?
Relevant code:
from flask import Flask
import paho.mqtt.client as mqtt
import logging as logger
app = Flask(__name__)
# Note: We don't need to call run() since our application is embedded within
# the App Engine WSGI application server.
#callback to print out connection status
def on_connect(mosq, obj, rc):
logger.info('on_connect')
if rc == 0:
logger.info("Connected")
mqttc.subscribe('test', 0)
else:
logger.info(rc)
def on_message(mqttc, obj, msg):
logger.info(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))
mqttc = mqtt.Client("mqttpy")
mqttc.on_message = on_message
mqttc.on_connect = on_connect
As well as full stack trace:
ERROR 2014-06-03 15:14:57,285 wsgi.py:262]
Traceback (most recent call last):
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 239, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 298, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 84, in LoadObject
obj = __import__(path[0])
File "/Users/cbarnes/code/ignite/tank-demo/appengine-flask-demo/main.py", line 24, in <module>
mqttc = mqtt.Client("mqtthtpp")
File "/Users/cbarnes/code/ignite/tank-demo/appengine-flask-demo/lib/paho/mqtt/client.py", line 403, in __init__
self._sockpairR, self._sockpairW = _socketpair_compat()
File "/Users/cbarnes/code/ignite/tank-demo/appengine-flask-demo/lib/paho/mqtt/client.py", line 255, in _socketpair_compat
listensock.bind(("localhost", 0))
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/dist27/socket.py", line 222, in meth
return getattr(self._sock,name)(*args)
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 668, in bind
self._SetProtoFromAddr(request.mutable_proxy_external_ip(), address)
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 632, in _SetProtoFromAddr
proto.set_packed_address(self._GetPackedAddr(address))
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 627, in _GetPackedAddr
AI_NUMERICSERV|AI_PASSIVE):
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 338, in getaddrinfo
canonical=(flags & AI_CANONNAME))
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 211, in _Resolve
canon, aliases, addresses = _ResolveName(name, families)
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/remote_socket/_remote_socket.py", line 229, in _ResolveName
apiproxy_stub_map.MakeSyncCall('remote_socket', 'Resolve', request, reply)
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 94, in MakeSyncCall
return stubmap.MakeSyncCall(service, call, request, response)
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 328, in MakeSyncCall
rpc.CheckSuccess()
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_rpc.py", line 156, in _WaitImpl
self.request, self.response)
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 200, in MakeSyncCall
self._MakeRealSyncCall(service, call, request, response)
File "/Users/cbarnes/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 234, in _MakeRealSyncCall
raise pickle.loads(response_pb.exception())
RuntimeError: error('illegal IP address string passed to inet_pton',)
INFO 2014-06-03 15:14:57,291 module.py:639] default: "GET / HTTP/1.1" 500 -
Thanks!

You're trying to connect to host test and port 0 where typically you should use localhost and 1883
See the getting started example here:
https://pypi.python.org/pypi/paho-mqtt/0.9#usage-and-api