Develop Reference

Access CosmosDB Data from Azure App Service by using managed identity (Failed)

A FastAPI-based API written in Python has been deployed as an Azure App Service. The API needs to read and write data from CosmosDB, and I attempted to use Managed Identity for this purpose, but encountered an error, stating Unrecognized credential type
These are the key steps that I took towards that goal
Step One: I used Terraform to configure the managed identity for Azure App Service, and assigned the 'contributor' role to the identity so that it can access and write data to CosmosDB. The role assignment was carried out in the file where the Azure App Service is provisioned.
resource "azurerm_linux_web_app" "this" {
name = var.appname
location = var.location
resource_group_name = var.rg_name
service_plan_id = azurerm_service_plan.this.id
app_settings = {
"PROD" = false
"DOCKER_ENABLE_CI" = true
"DOCKER_REGISTRY_SERVER_URL" = data.azurerm_container_registry.this.login_server
"WEBSITE_HTTPLOGGING_RETENTION_DAYS" = "30"
"WEBSITE_ENABLE_APP_SERVICE_STORAGE" = false
}
lifecycle {
ignore_changes = [
app_settings["WEBSITE_HTTPLOGGING_RETENTION_DAYS"]
]
}
https_only = true
identity {
type = "SystemAssigned"
}
data "azurerm_cosmosdb_account" "this" {
name = var.cosmosdb_account_name
resource_group_name = var.cosmosdb_resource_group_name
}
// built-in role that allow the app-service to read and write to an Azure Cosmos DB
resource "azurerm_role_assignment" "cosmosdbContributor" {
scope = data.azurerm_cosmosdb_account.this.id
principal_id = azurerm_linux_web_app.this.identity.0.principal_id
role_definition_name = "Contributor"
}
Step Two: I used the managed identity library to fetch the necessary credentials in the Python code.
from azure.identity import ManagedIdentityCredential
from azure.cosmos.cosmos_client import CosmosClient
client = CosmosClient(get_endpoint(),credential=ManagedIdentityCredential())
client = self._get_or_create_client()
database = client.get_database_client(DB_NAME)
container = database.get_container_client(CONTAINER_NAME)
container.query_items(query)
I received the following error when running the code locally and from Azure (the error can be viewed from the Log stream of the Azure App Service):
raise TypeError(
TypeError: Unrecognized credential type. Please supply the master key as str, or a dictionary or resource tokens, or a list of permissions.
Any help or discussion is welcome

If you are using the Python SDK, you can directly do this ,check the sample here
aad_credentials = ClientSecretCredential(
tenant_id="<azure-ad-tenant-id>",
client_id="<client-application-id>",
client_secret="<client-application-secret>")
client = CosmosClient("<account-endpoint>", aad_credentials)

Office 365 IMAP authentication via OAuth2 and python MSAL library

I'm trying to upgrade a legacy mail bot to authenticate via Oauth2 instead of Basic authentication, as it's now deprecated two days from now.
The document states applications can retain their original logic, while swapping out only the authentication bit
Application developers who have built apps that send, read, or
otherwise process email using these protocols will be able to keep the
same protocol, but need to implement secure, Modern authentication
experiences for their users. This functionality is built on top of
Microsoft Identity platform v2.0 and supports access to Microsoft 365
email accounts.
Note I've explicitly chosen the client credentials flow, because the documentation states
This type of grant is commonly used for server-to-server interactions
that must run in the background, without immediate interaction with a
user.
So I've got a python script that retrieves an Access Token using the MSAL python library. Now I'm trying to authenticate with the IMAP server, using that Access Token. There's some existing threads out there showing how to connect to Google, I imagine my case is pretty close to this one, except I'm connecting to a Office 365 IMAP server. Here's my script
import imaplib
import msal
import logging
app = msal.ConfidentialClientApplication(
'client-id',
authority='https://login.microsoftonline.com/tenant-id',
client_credential='secret-key'
)
result = app.acquire_token_for_client(scopes=['https://graph.microsoft.com/.default'])
def generate_auth_string(user, token):
return 'user=%s\1auth=Bearer %s\1\1' % (user, token)
# IMAP time!
mailserver = 'outlook.office365.com'
imapport = 993
M = imaplib.IMAP4_SSL(mailserver,imapport)
M.debug = 4
M.authenticate('XOAUTH2', lambda x: generate_auth_string('user#mydomain.com', result['access_token']))
print(result)
The IMAP authentication is failing and despite setting M.debug = 4, the output isn't very helpful
22:56.53 > b'DBDH1 AUTHENTICATE XOAUTH2'
22:56.53 < b'+ '
22:56.53 write literal size 2048
22:57.84 < b'DBDH1 NO AUTHENTICATE failed.'
22:57.84 NO response: b'AUTHENTICATE failed.'
Traceback (most recent call last):
File "/home/ubuntu/mini-oauth.py", line 21, in <module>
M.authenticate("XOAUTH2", lambda x: generate_auth_string('user#mydomain.com', result['access_token']))
File "/usr/lib/python3.10/imaplib.py", line 444, in authenticate
raise self.error(dat[-1].decode('utf-8', 'replace'))
imaplib.IMAP4.error: AUTHENTICATE failed.
Any idea where I might be going wrong, or how to get more robust information from the IMAP server about why the authentication is failing?
Things I've looked at
Note this answer no longer works as the suggested scopes fail to generate an Access Token.
The client credentials flow seems to mandate the https://graph.microsoft.com/.default grant. I'm not sure if that includes the scope required for the IMAP resource
https://outlook.office.com/IMAP.AccessAsUser.All?
Verified the code lifted from the Google thread produces the SASL XOAUTH2 string correctly, per example on the MS docs
import base64
user = 'test#contoso.onmicrosoft.com'
token = 'EwBAAl3BAAUFFpUAo7J3Ve0bjLBWZWCclRC3EoAA'
xoauth = "user=%s\1auth=Bearer %s\1\1" % (user, token)
xoauth = xoauth.encode('ascii')
xoauth = base64.b64encode(xoauth)
xoauth = xoauth.decode('ascii')
xsanity = 'dXNlcj10ZXN0QGNvbnRvc28ub25taWNyb3NvZnQuY29tAWF1dGg9QmVhcmVyIEV3QkFBbDNCQUFVRkZwVUFvN0ozVmUwYmpMQldaV0NjbFJDM0VvQUEBAQ=='
print(xoauth == xsanity) # prints True
This thread seems to suggest multiple tokens need to be fetched, one for graph, then another for the IMAP connection; could that be what I'm missing?

Try the below steps.
For Client Credentials Flow you need to assign “Application permissions” in the app registration, instead of “Delegated permissions”.
Add permission “Office 365 Exchange Online / IMAP.AccessAsApp” (application).
Grant admin consent to you application
Service Principals and Exchange.
Once a service principal is registered with Exchange Online, administrators can run the Add-Mailbox Permission cmdlet to assign receive permissions to the service principal.
Use scope 'https://outlook.office365.com/.default'.
Now you can generate the SALS authentication string by combining this access token and the mailbox username to authenticate with IMAP4.
#Python code
def get_access_token():
tenantID = 'abc'
authority = 'https://login.microsoftonline.com/' + tenantID
clientID = 'abc'
clientSecret = 'abc'
scope = ['https://outlook.office365.com/.default']
app = ConfidentialClientApplication(clientID,
authority=authority,
client_credential = clientSecret)
access_token = app.acquire_token_for_client(scopes=scope)
return access_token
def generate_auth_string(user, token):
auth_string = f"user={user}\1auth=Bearer {token}\1\1"
return auth_string
#IMAP AUTHENTICATE
imap = imaplib.IMAP4_SSL(imap_host, 993)
imap.debug = 4
access_token = get_access_token_to_authenticate_imap()
imap.authenticate("XOAUTH2", lambda x:generate_auth_string(
'useremail',
access_token['access_token']))
imap.select('inbox')

The imaplib.IMAP4.error: AUTHENTICATE failed Error occured because one point in the documentation is not that clear.
When setting up the the Service Principal via Powershell you need to enter the App-ID and an Object-ID. Many people will think, it is the Object-ID you see on the overview page of the registered App, but its not!
At this point you need the Object-ID from "Azure Active Directory -> Enterprise Applications --> Your-App --> Object-ID"
New-ServicePrincipal -AppId <APPLICATION_ID> -ServiceId <OBJECT_ID> [-Organization <ORGANIZATION_ID>]
Microsoft says:
The OBJECT_ID is the Object ID from the Overview page of the
Enterprise Application node (Azure Portal) for the application
registration. It is not the Object ID from the Overview of the App
Registrations node. Using the incorrect Object ID will cause an
authentication failure.
Ofcourse you need to take care for the API-permissions and the other stuff, but this was for me the point.
So lets go trough it again, like it is explained on the documentation page.
Authenticate an IMAP, POP or SMTP connection using OAuth
Register the Application in your Tenant
Setup a Client-Key for the application
Setup the API permissions, select the APIs my organization uses tab and search for "Office 365 Exchange Online" -> Application permissions -> Choose IMAP and IMAP.AccessAsApp
Setup the Service Principal and full access for your Application on the mailbox
Check if IMAP is activated for the mailbox
Thats the code I use to test it:
import imaplib
import msal
import pprint
conf = {
"authority": "https://login.microsoftonline.com/XXXXyourtenantIDXXXXX",
"client_id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX", #AppID
"scope": ['https://outlook.office365.com/.default'],
"secret": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-Value
"secret-id": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-ID
}
def generate_auth_string(user, token):
return f"user={user}\x01auth=Bearer {token}\x01\x01"
if __name__ == "__main__":
app = msal.ConfidentialClientApplication(conf['client_id'], authority=conf['authority'],
client_credential=conf['secret'])
result = app.acquire_token_silent(conf['scope'], account=None)
if not result:
print("No suitable token in cache. Get new one.")
result = app.acquire_token_for_client(scopes=conf['scope'])
if "access_token" in result:
print(result['token_type'])
pprint.pprint(result)
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id"))
imap = imaplib.IMAP4('outlook.office365.com')
imap.starttls()
imap.authenticate("XOAUTH2", lambda x: generate_auth_string("target_mailbox#example.com", result['access_token']).encode("utf-8"))
After setting up the Service Principal and giving the App full access on the mailbox, wait 15 - 30 minutes for the changes to take effect and test it.

Try with this script:
import json
import msal
import requests
client_id = '***'
client_secret = '***'
tenant_id = '***'
authority = f"https://login.microsoftonline.com/{tenant_id}"
app = msal.ConfidentialClientApplication(
client_id=client_id,
client_credential=client_secret,
authority=authority)
scopes = ["https://graph.microsoft.com/.default"]
result = None
result = app.acquire_token_silent(scopes, account=None)
if not result:
print(
"No suitable token exists in cache. Let's get a new one from Azure Active Directory.")
result = app.acquire_token_for_client(scopes=scopes)
# if "access_token" in result:
# print("Access token is " + result["access_token"])
if "access_token" in result:
userId = "***"
endpoint = f'https://graph.microsoft.com/v1.0/users/{userId}/sendMail'
toUserEmail = "***"
email_msg = {'Message': {'Subject': "Test Sending Email from Python",
'Body': {'ContentType': 'Text', 'Content': "This is a test email."},
'ToRecipients': [{'EmailAddress': {'Address': toUserEmail}}]
},
'SaveToSentItems': 'true'}
r = requests.post(endpoint,
headers={'Authorization': 'Bearer ' + result['access_token']}, json=email_msg)
if r.ok:
print('Sent email successfully')
else:
print(r.json())
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id"))
Source: https://kontext.tech/article/795/python-send-email-via-microsoft-graph-api

Error call Google Vertex AI endpoint from a python backend

I am trying to send an http post request to my google vertex ai endpoint for prediction. Though I do set the Bearer Token in the request header, the request still fails with the below error:
{
"error": {
"code": 401,
"message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED",
"details": [
{
"#type": "type.googleapis.com/google.rpc.ErrorInfo",
"reason": "ACCESS_TOKEN_TYPE_UNSUPPORTED",
"metadata": {
"service": "aiplatform.googleapis.com",
"method": "google.cloud.aiplatform.v1.PredictionService.Predict"
}
}
]
}
}
Since I am making this call from a python backend, I'm not sure if OAuth 2 as suggested in the message would be wise and applicable choice.
The model is already deployed and endpointed test on vertex ai and it worked fine. What I am trying to do is send same prediction task via an http post request using postman and this is what failed.
The request url looks like this:
https://[LOCATION]-aiplatform.googleapis.com/v1/projects/[PROJECT ID]/locations/[LOCATION]/endpoints/[ENDPOINT ID]:predict
Where token bearer is set in the potman authorization tab and instance set in request body.

I usually call Vertex AI endpoint this way:
from google.cloud import aiplatform
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="/home/user/1a2b3c4d.json"
aip_endpoint_name = (
f"projects/your-project/locations/us-west1/endpoints/1234567"
)
endpoint = aiplatform.Endpoint(aip_endpoint_name)
Here, you encode the input according to your needs.
def encode_64(input):
message = input
message_bytes = message.encode('ascii')
base64_bytes = base64.b64encode(message_bytes)
base64_message = base64_bytes.decode('ascii')
return base64_message
Check the model signature for the input type:
saved_model_cli show --dir /home/jupyter/model --all
Then call the endpoint
instances_list = [{"input_1": {"b64": encode_64("doctor")}}]
instances = [json_format.ParseDict(s, Value()) for s in instances_list]
results = endpoint.predict(instances=instances)
print(results.predictions)
According to the type of the input you are submitting to Vertex AI endpoint (integer, array, string, image), you may have to change the encoding.

If you don't want to use oAuth 2.0 for authentication when making http post request, you may want to use Application Default Credentials. In this documentation you can follow the step by step on getting service account key to passing credentials via environment variable.

Can you try following:
import subprocess
import base64
import requests
import json
def get_headers():
gcloud_access_token = subprocess.check_output("gcloud auth print-access-token".split(' ')).decode().rstrip('\n')
return {"authorization": "Bearer " + gcloud_access_token}
def send_get_request(uri):
return requests.get(uri, headers=get_headers(), verify=False)
It should work ff you have gcloud configured (you can do this by using gcloud init) on the machine you are sending request from and have permissions to access Vertex Prediction.

401 error code when trying to export powerbi reports

I am able to get a token using the default scope for powerbi ( "scope" :
"https://analysis.windows.net/powerbi/api/.default"] )
With that token, I am able to read the workspaces my user has access to, ( "https://api.powerbi.com/v1.0/myorg/groups") and the reports information inside each of those workspaces (
"https://api.powerbi.com/v1.0/myorg/reports/")
But it does not matter if I reuse the same token or just acquire a brand new, if I try to export a specific report, I got a 401 error code. This is the way I am issuing the requests.get
token_ = <new token or reused from previous get requests>
reports = requests.get( # Use token to call downstream service
config['reports']+report_id+'/Export',
headers={'Authorization': 'Bearer ' + token_ },)
Now, if I go to https://learn.microsoft.com/en-us/rest/api/power-bi/reports/getreportsingroup
and sign in (with the same user I am using on my python script). Get the token from that page and use it on my script. It works, If I use it in postman, it works
I I try to use the token acquired by my script in Postman, I also get a 401 error. So, yes, my script is not getting the correct token for this particular entry point but it is good enough for the groups and reports entry point.
Is there anything I need to add to the request for the token on this particular entry point?
Thank you very much,
Andres
Here is the full script I am using, there is also a params.json that looks like this:
{
"authority": "https://login.microsoftonline.com/1abcdefg-abcd-48b6-9b3c-bd5123456",
"client_id": "5d2545-abcd-4765-8fbb-53555f2fa91",
"username":"myusername#tenant",
"password": "mypass",
"scope" : ["https://analysis.windows.net/powerbi/api/.default"],
"workspaces" : "https://api.powerbi.com/v1.0/myorg/groups",
"reports": "https://api.powerbi.com/v1.0/myorg/reports/"
}
#script based on msal github library sample
import sys # For simplicity, we'll read config file from 1st CLI param sys.argv[1]
import json
import logging
import requests
import msal
def exportReport(report_id,token_):
result = app.acquire_token_by_username_password( config["username"], config["password"], scopes=config["scope"])
token_ = result['access_token']
print(f'Using token: {token_}')
reports = requests.get( # Use token to call downstream service
config['reports']+report_id+'/Export',
headers={'Authorization': 'Bearer ' + token_ },)
print(f'-reports: {reports.status_code}')
def list_reports(workspace_id,ws_id,ws_name,token_):
print(f'reports id for workspace {ws_name}')
for rp in workspace_id['value']:
if rp["id"] == "1d509119-76a1-42ce-8afd-bd3c420dd62d":
exportReport("1d509119-76a1-42ce-8afd-bd0c420dd62d",token_)
def list_workspaces(workspaces_dict):
for ws in workspaces_dict['value']:
yield (ws['id'],ws['name'])
config = json.load(open('params.json'))
app = msal.PublicClientApplication(
config["client_id"], authority=config["authority"],
)
result = None
if not result:
logging.info("No suitable token exists in cache. Let's get a new one from AAD.")
result = app.acquire_token_by_username_password(
config["username"], config["password"], scopes=config["scope"])
if "access_token" in result:
workspaces = requests.get( # Use token to call downstream service
config['workspaces'],
headers={'Authorization': 'Bearer ' + result['access_token']},).json()
ids=list_workspaces(workspaces) #prepare workspace generator
headers = {'Authorization': 'Bearer ' + result['access_token']}
while True:
try:
ws_id,ws_name=next(ids)
reports = requests.get( # Use token to call downstream service
config['workspaces']+'/'+ws_id+'/reports',
headers={'Authorization': 'Bearer ' + result['access_token']},).json()
list_reports(reports,ws_id,ws_name,result['access_token'])
except StopIteration:
exit(0)
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id")) # You may need this when reporting a bug
if 65001 in result.get("error_codes", []):
# AAD requires user consent for U/P flow
print("Visit this to consent:", app.get_authorization_request_url(config["scope"]))

From your description, I suppose you didn't grant the correct permission for your AD App used to login your user account in the code, please follow the steps below.
Navigate to the Azure portal -> Azure Active Directory -> App registrations -> find your AD App used in the code(filter with All applications) -> API permissions -> add the Report.Read.All Delegated permission in Power BI Service API(this permission is just for read action, if you need some further write operation, choose Report.ReadWrite.All) -> click the Grant admin consent for xxx button at last.
Update:
Use the application id of the access token got from Get-PowerBIAccessToken solve the issue.

Getting auth token from keystone in horizon

I want to get the auth token from keystone using horizon and then wants to pass that auth token to my backed code.
i don't know how to get this, please help me out.
I read many articles and blogs blogs but i am not able to find the answer. Please just point me into the right direction.

Easiest is to use a Rest client to login and just take the token from the response. I like the Firefox RESTClient add-on but you can use any client you want.
Post a request to the Openstack Identity URL:
POST keystone_ip:port/version/tokens
(e.g. 127.0.0.1:5000/v2.0/tokens)
with header:
Content-Type: application/json
and body:
{
"auth": {
"tenantName": "enter_your_tenantname",
"passwordCredentials": {
"username": "enter_your_username",
"password": "enter_your_password"
}
}
}
Note: If you're not sure what is the correct identity (keystone) URL you can log in manually to Horizon and look for a list of API endpoints.
The response body will include something like this:
{
"token": {
"issued_at": "2014-02-25T08:34:56.068363",
"expires": "2014-02-26T08:34:55Z",
"id": "529e3a0e1c375j498315c71d08134837"
}
}
Use the returned token id as a header in new rest calls. For example, to get a list of servers use request:
GET compute_endpoint_ip:port/v2/tenant_id/servers
with Headers:
X-Auth-Token: 529e3a0e1c375j498315c71d08134837
Content-Type: application/json

As an example of how to get at it:
import keystoneclient.v2_0.client as ksclient
# authenticate with keystone to get a token
keystone = ksclient.Client(auth_url="http://192.168.10.5:35357/v2.0",
username="admin",
password="admin",
tenant_name="admin")
token = keystone.auth_ref['token']['id']
# use this token for whatever other services you are accessing.
print token

You can use python-keystoneclient. To authenticate the user, use for example
username='admin'
password='1234'
tenant_name='admin'
auth_url='http://127.0.0.1:5000/v2.0'
keystone = client.Client(username=username, password=password, tenant_name=tenant_name, auth_url=auth_url)
Once, the user is authenticated, a token is generated. The auth_ref property on the client ( keystone variable in this example) will give you a dictionary like structure having all the information you need about the token, which will enable you to re-use the token or pass it to the back-end in your case.
token_dict = keystone.auth_ref
Now,the token_dict is the variable that you can pass to your back-end.

Go to the node where you have installed Keystone services. Open vi /etc/keystone/keystone.conf
Check for the third line starting admin_token. It should be a long random string:
admin_token = 05131394ad6b49c56f217
That is your keystone token. Using python:
>>> from keystoneclient.v2_0.client as ksclient
>>> keystone = ksclient.Client(auth_url="http://service-stack.linxsol.com:35357/v2.0", username="admin", password="123456", tenant_name="admin")
Ofcourse, you will change auth_url, *username, password* and tenant_name to your choice. Now you can use keystone to execute all the api tasks:
keystone.tenants.list()
keystone.users.list()
keystone.roles.list()
Or use dir(keystone) to list all the available options.
You can reuse the token as follows:
auth_ref = keystone.auth_ref or token = ksclient.get_raw_token_from_identity_service(auth_url="http://service-stack.linxsol.com:35357/v2.0", username="admin", password="123456", tenant_name="admin")
But remember it returns a dictionary and a raw token not in a form of a token as you can see above.
For further information please check the python-keystoneclient.
I hope that helps.

Use the python-keystoneclient package.
Look into the Client.get_raw_token_from_identity_service method.

First you have to install python-keystoneclient.
To generate the token you can use the following code, here I want to mention you can change the authentication url with your server url but port number will be same,
from keystoneclient.v2_0 import client
username='admin'
password='1234'
tenant_name='demo'
auth_url='http://10.0.2.15:5000/v2.0' # Or auth_url='http://192.168.206.133:5000/v2.0'
if your username, password, or tenant_name is wrong then you will get keystoneclient.openstack.common.apiclient.exceptions.Unauthorized: Invalid user / password
keystone = client.Client(username=username, password=password, tenant_name=tenant_name, auth_url=auth_url)
token_dict = keystone.auth_ref
token_dict