Develop Reference

Celery + Azure Service Bus (Broker) = claim is empty or token is invalid

I am trying to use Azure Service Bus as the broker for my celery app.
I have patched the solution by referring to various sources.
The goal is to use Azure Service Bus as the broker and PostgresSQL as the backend.
I created an Azure Service Bus and copied the credentials for the RootManageSharedAccessKey to the celery app.
Following is the task.py
from time import sleep
from celery import Celery
from kombu.utils.url import safequote
SAS_policy = safequote("RootManageSharedAccessKey") #SAS Policy
SAS_key = safequote("1234222zUY28tRUtp+A2YoHmDYcABCD") #Primary key from the previous SS
namespace = safequote("bluenode-dev")
app = Celery('tasks', backend='db+postgresql://afsan.gujarati:admin#localhost/local_dev',
broker=f'azureservicebus://{SAS_policy}:{SAS_key}=#{namespace}')
#app.task
def divide(x, y):
sleep(30)
return x/y
When I try to run the Celery app using the following command:
celery -A tasks worker --loglevel=INFO
I get the following error
[2020-10-09 14:00:32,035: CRITICAL/MainProcess] Unrecoverable error: AzureHttpError('Unauthorized\n<Error><Code>401</Code><Detail>claim is empty or token is invalid. TrackingId:295f7c76-770e-40cc-8489-e0eb56248b09_G5S1, SystemTracker:bluenode-dev.servicebus.windows.net:$Resources/Queues, Timestamp:2020-10-09T20:00:31</Detail></Error>')
Traceback (most recent call last):
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 918, in create_channel
return self._avail_channels.pop()
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/azure/servicebus/control_client/servicebusservice.py", line 1225, in _perform_request
resp = self._filter(request)
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/azure/servicebus/control_client/_http/httpclient.py", line 211, in perform_request
raise HTTPError(status, message, respheaders, respbody)
azure.servicebus.control_client._http.HTTPError: Unauthorized
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/celery/bootsteps.py", line 365, in start
return self.obj.start()
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 311, in start
blueprint.start(self)
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/celery/worker/consumer/connection.py", line 21, in start
c.connection = c.connect()
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 398, in connect
conn = self.connection_for_read(heartbeat=self.amqheartbeat)
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 404, in connection_for_read
return self.ensure_connected(
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 430, in ensure_connected
conn = conn.ensure_connection(
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/kombu/connection.py", line 383, in ensure_connection
self._ensure_connection(*args, **kwargs)
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/kombu/connection.py", line 435, in _ensure_connection
return retry_over_time(
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/kombu/utils/functional.py", line 325, in retry_over_time
return fun(*args, **kwargs)
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/kombu/connection.py", line 866, in _connection_factory
self._connection = self._establish_connection()
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/kombu/connection.py", line 801, in _establish_connection
conn = self.transport.establish_connection()
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 938, in establish_connection
self._avail_channels.append(self.create_channel(self))
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 920, in create_channel
channel = self.Channel(connection)
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/kombu/transport/azureservicebus.py", line 64, in __init__
for queue in self.queue_service.list_queues():
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/azure/servicebus/control_client/servicebusservice.py", line 313, in list_queues
response = self._perform_request(request)
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/azure/servicebus/control_client/servicebusservice.py", line 1227, in _perform_request
return _service_bus_error_handler(ex)
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/azure/servicebus/control_client/_serialization.py", line 569, in _service_bus_error_handler
return _general_error_handler(http_error)
File "/Users/afsan.gujarati/.pyenv/versions/3.8.1/envs/celery-servicebus/lib/python3.8/site-packages/azure/servicebus/control_client/_common_error.py", line 41, in _general_error_handler
raise AzureHttpError(message, http_error.status)
azure.common.AzureHttpError: Unauthorized
<Error><Code>401</Code><Detail>claim is empty or token is invalid. TrackingId:295f7c76-770e-40cc-8489-e0eb56248b09_G5S1, SystemTracker:bluenode-dev.servicebus.windows.net:$Resources/Queues, Timestamp:2020-10-09T20:00:31</Detail></Error>
I don't see a straight solution for this anywhere. What am I missing?
P.S. I did not create the Queue in Azure Service Bus. I am assuming that celery would create the Queue by itself when the celery app is executed.
P.S.S. I also tried to use the exact same credentials in Python's Service Bus Client and it seemed to work. It feels like a Celery issue, but I am not able to figure out exactly what.

If you want to use Azure Service Bus Transport to connect Azure service bus, the URL should be azureservicebus://{SAS policy name}:{SAS key}#{Service Bus Namespace}.
For example
Get Shared access policies RootManageSharedAccessKey
Code
from celery import Celery
from kombu.utils.url import safequote
SAS_policy = "RootManageSharedAccessKey" # SAS Policy
# Primary key from the previous SS
SAS_key = safequote("X/*****qyY=")
namespace = "bowman1012"
app = Celery('tasks', backend='db+postgresql://<>#localhost/<>',
broker=f'azureservicebus://{SAS_policy}:{SAS_key}#{namespace}')
#app.task
def add(x, y):
return x + y

Airflow Scheduler Crashes when setting Postgres celery result_backend

I try to implement Apache Airflow with the CeleryExecutor. For the database I use Postgres, for the celery message queue I use Redis. When using LocalExecutor everything works fine, but when I set the CeleryExecutor in the airflow.cfg and want to set the Postgres database as the result_backend
result_backend = postgresql+psycopg2://airflow_user:*******#localhost/airflow
I get this error when running the Airflow scheduler no matter which DAG I trigger:
[2020-03-18 14:14:13,341] {scheduler_job.py:1382} ERROR - Exception when executing execute_helper
Traceback (most recent call last):
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
return obj.__dict__[self.__name__]
KeyError: 'backend'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1380, in _execute
self._execute_helper()
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1441, in _execute_helper
if not self._validate_and_run_task_instances(simple_dag_bag=simple_dag_bag):
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1503, in _validate_and_run_task_instances
self.executor.heartbeat()
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/airflow/executors/base_executor.py", line 130, in heartbeat
self.trigger_tasks(open_slots)
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/airflow/executors/celery_executor.py", line 205, in trigger_tasks
cached_celery_backend = tasks[0].backend
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/celery/local.py", line 146, in __getattr__
return getattr(self._get_current_object(), name)
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/celery/app/task.py", line 1037, in backend
return self.app.backend
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/kombu/utils/objects.py", line 44, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/celery/app/base.py", line 1227, in backend
return self._get_backend()
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/celery/app/base.py", line 944, in _get_backend
self.loader)
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/celery/app/backends.py", line 74, in by_url
return by_name(backend, loader), url
File "<PATH_TO_VIRTUALENV>/lib/python3.6/site-packages/celery/app/backends.py", line 60, in by_name
backend, 'is a Python module, not a backend class.'))
celery.exceptions.ImproperlyConfigured: Unknown result backend: 'postgresql'. Did you spell that correctly? ('is a Python module, not a backend class.')
The exact same parameter to direct to the database works
sql_alchemy_conn = postgresql+psycopg2://airflow_user:*******#localhost/airflow
Setting Redis as the celery result_backend works, but I read it is not the recommended way.
result_backend = redis://localhost:6379/0
Does anyone see what I am doing wrong?

You need to add the db+ prefix to the database connection string:
f"db+postgresql+psycopg2://{user}:{password}#{host}/{database}"
This is also mentioned in the docs: https://docs.celeryproject.org/en/stable/userguide/configuration.html#database-url-examples

You need to add the db+ prefix to the database connection string:
result_backend = db+postgresql://airflow_user:*******#localhost/airflow

Django server won't run

I just tries to start django project on win7(x64), but i faced with following issue:
$ python manage.py runserver
Performing system checks...
System check identified no issues (0 silenced).
March 24, 2018 - 14:24:08
Django version 1.11.3, using settings 'superlists.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
Unhandled exception in thread started by <function check_errors.<locals>.wrapper
at 0x035BD978>
Traceback (most recent call last):
File "C:\Users\alesya\.virtualenvs\superlists\lib\site-packages\django\utils\a
utoreload.py", line 227, in wrapper
fn(*args, **kwargs)
File "C:\Users\alesya\.virtualenvs\superlists\lib\site-packages\django\core\ma
nagement\commands\runserver.py", line 149, in inner_run
ipv6=self.use_ipv6, threading=threading, server_cls=self.server_cls)
File "C:\Users\alesya\.virtualenvs\superlists\lib\site-packages\django\core\se
rvers\basehttp.py", line 164, in run
httpd = httpd_cls(server_address, WSGIRequestHandler, ipv6=ipv6)
File "C:\Users\alesya\.virtualenvs\superlists\lib\site-packages\django\core\se
rvers\basehttp.py", line 74, in __init__
super(WSGIServer, self).__init__(*args, **kwargs)
File "c:\users\alesya\appdata\local\programs\python\python36-32\Lib\socketserv
er.py", line 453, in __init__
self.server_bind()
File "c:\users\alesya\appdata\local\programs\python\python36-32\Lib\wsgiref\si
mple_server.py", line 50, in server_bind
HTTPServer.server_bind(self)
File "c:\users\alesya\appdata\local\programs\python\python36-32\Lib\http\serve
r.py", line 138, in server_bind
self.server_name = socket.getfqdn(host)
File "c:\users\alesya\appdata\local\programs\python\python36-32\Lib\socket.py"
, line 673, in getfqdn
hostname, aliases, ipaddrs = gethostbyaddr(name)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbb in position 14: invalid
start byte
My computer has an ASCII name, so I even not realized, what happens.
Did all these things on another win7 and everything was ok.
Maybe someone can help with?
UPD. My problem was due to the changed 'hosts' file - there are a lot of disabled addresses.
Thanks all for the answers.

use python3, if you use python2.x many letters like accents or others, they cause abnormal crashes

try this:
a.encode('utf-8').strip()
if "a" is the string with non-ascii character

Password input may be echoed when run Python Fabric command in crontab

I encountered issue when trying to run my program through crontab on Mac OS. My program works fine when run it independently. Normally, I never set env.password for remote system password. For instead, I set env.key_filename. It works fine if i don't have to run sudo command. So ideally, it shouldn't prompt any password typing.
By referring to https://github.com/fabric/fabric/issues/1230, i also tried to set environment variables to pass the password. Yet, i still get the same error. What did i miss? Anyone can help pls?
Thx
Error msg:
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/getpass.py:83: GetPassWarning: Can not control echo on the terminal.
passwd = fallback_getpass(prompt, stream)
Warning: Password input may be echoed.
[xxx.xxx.xxx.xxx] Login password for 'xxx': [xxx.xxx.xxx.xxx] Executing task 'System.Monitor.free_mem'
[+] Start checking system total/free memory in MB:
[xxx.xxx.xxx.xxx] run: free -m
Traceback (most recent call last):
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/main.py", line 757, in main
*args, **kwargs
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/tasks.py", line 386, in execute
multiprocessing
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/tasks.py", line 276, in _execute
return task.run(*args, **kwargs)
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
return self.wrapped(*args, **kwargs)
File "/Users/thomas.pan/Python-ninja/playwith/DevOps/System/Monitor.py", line 69, in free_mem
run("free -m")
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/network.py", line 687, in host_prompting_wrapper
return func(*args, **kwargs)
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/operations.py", line 1090, in run
shell_escape=shell_escape, capture_buffer_size=capture_buffer_size,
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/operations.py", line 930, in _run_command
channel=default_channel(), command=wrapped_command, pty=pty,
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/state.py", line 424, in default_channel
chan = _open_session()
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/state.py", line 416, in _open_session
return connections[env.host_string].get_transport().open_session()
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/network.py", line 159, in __getitem__
self.connect(key)
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/network.py", line 151, in connect
user, host, port, cache=self, seek_gateway=seek_gateway)
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/network.py", line 569, in connect
password = prompt_for_password(text)
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/network.py", line 652, in prompt_for_password
new_password = _password_prompt(password_prompt, stream)
File "/usr/local/var/pyenv/versions/2.7.10/lib/python2.7/site-packages/fabric/network.py", line 624, in _password_prompt
return getpass.getpass(prompt.encode('ascii', 'ignore'), stream)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/getpass.py", line 83, in unix_getpass
passwd = fallback_getpass(prompt, stream)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/getpass.py", line 118, in fallback_getpass
return _raw_input(prompt, stream)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/getpass.py", line 135, in _raw_input
raise EOFError
EOFError

Issue closed by switch to launchd with LaunchControl tool. It's not Fabric issue.

In case it helps anyone - if you're running a command that is logging into an instance, this GetPassWarning/OEFError issue can happen with a cron job, since the shell session does not know how to log into that instance.
To fix this, you may need to give ssh context to crond. ssh-cron can do this, for example, since it looks like there is difficulty getting that all set up in crontab.

Authentication failed to connect to mongodb using pymongo

We have written a piece of code in python script using pymongo that connects to mongodb.
username = 'abc'
password = 'xxxxxx'
server = 'dns name of that server'
port = 27017
In program, the code looks like:
import pymongo
from pymongo import MongoClient
client = MongoClient(url, serverSelectionTimeoutMS=300)
database = client.database_name
data_insert = database.collection_name.insert_one({'id': 1, 'name': xyz})
When I tried to do these operations, it raises an error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1114, in next
if len(self.__data) or self._refresh():
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1036, in _refresh
self.__collation))
File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 873, in __send_message
**kwargs)
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 905, in _send_message_with_response
exhaust)
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 916, in _reset_on_error
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pymongo/server.py", line 99, in send_message_with_response
with self.get_socket(all_credentials, exhaust) as sock_info:
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/local/lib/python2.7/dist-packages/pymongo/server.py", line 168, in get_socket
with self.pool.get_socket(all_credentials, checkout) as sock_info:
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/local/lib/python2.7/dist-packages/pymongo/pool.py", line 792, in get_socket
sock_info.check_auth(all_credentials)
File "/usr/local/lib/python2.7/dist-packages/pymongo/pool.py", line 512, in check_auth
auth.authenticate(credentials, self)
File "/usr/local/lib/python2.7/dist-packages/pymongo/auth.py", line 470, in authenticate
auth_func(credentials, sock_info)
File "/usr/local/lib/python2.7/dist-packages/pymongo/auth.py", line 450, in _authenticate_default
return _authenticate_scram_sha1(credentials, sock_info)
File "/usr/local/lib/python2.7/dist-packages/pymongo/auth.py", line 201, in _authenticate_scram_sha1
res = sock_info.command(source, cmd)
File "/usr/local/lib/python2.7/dist-packages/pymongo/pool.py", line 419, in command
collation=collation)
File "/usr/local/lib/python2.7/dist-packages/pymongo/network.py", line 116, in command
parse_write_concern_error=parse_write_concern_error)
File "/usr/local/lib/python2.7/dist-packages/pymongo/helpers.py", line 210, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: Authentication failed.
In MongoDB, while performing queries we are getting the responses normally, without raising any errors.

Because the other answers to your question didn't work for me, I'm going to copy and paste my answer from a similar question.
If you've tried the above answers and you're still getting an error:
pymongo.errors.OperationFailure: Authentication failed.
There's a good chance you need to add ?authSource=admin to the end of your uri.
Here's a working solution that I'm using with MongoDB server version 4.2.6 and MongoDB shell version v3.6.9.
from pymongo import MongoClient
# Replace these with your server details
MONGO_HOST = "XX.XXX.XXX.XXX"
MONGO_PORT = "27017"
MONGO_DB = "database"
MONGO_USER = "admin"
MONGO_PASS = "pass"
uri = "mongodb://{}:{}#{}:{}/{}?authSource=admin".format(MONGO_USER, MONGO_PASS, MONGO_HOST, MONGO_PORT, MONGO_DB)
client = MongoClient(uri)
Similar fix for command line is adding --authenticationDatabase admin

Well, I have been stuck with the same error for almost 3-4 hours. I came across solution with the following steps:
from your shell connect to MongoDB by typing: mongo
afterwards, create a database: use test_database
Now create a user with the following command with readWrite and dbAdmin privileges.
db.createUser(
{
user: "test_user",
pwd: "testing12345",
roles: [ "readWrite", "dbAdmin" ]
}
);
This will prompt Successfully added user: { "user" : "test_user", "roles" : [ "readWrite", "dbAdmin" ] }
you can check by typing: show users.
It will also show you DB name you created before in the json.
now you should be able to insert data to your database:
client = MongoClient("mongodb://test_user:myuser123#localhost:27017/test_database")
db = client.test_database
data = {"initial_test":"testing"}
db["my_collection"].insert_one(data).inserted_id

I ran into this error, and my problem was with the password.
I had what I believe to be a special character in the Master account. Changing the password to be only alphanumeric fixed it for me.
Code snippet
client = pymongo.MongoClient(
'mongodb://username:alphaNumericPassword#localhost:27017/?ssl=true&ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false'
)
# Specify the database to be used
db = client['prod-db']