Develop Reference

Office 365 IMAP authentication via OAuth2 and python MSAL library

I'm trying to upgrade a legacy mail bot to authenticate via Oauth2 instead of Basic authentication, as it's now deprecated two days from now.
The document states applications can retain their original logic, while swapping out only the authentication bit
Application developers who have built apps that send, read, or
otherwise process email using these protocols will be able to keep the
same protocol, but need to implement secure, Modern authentication
experiences for their users. This functionality is built on top of
Microsoft Identity platform v2.0 and supports access to Microsoft 365
email accounts.
Note I've explicitly chosen the client credentials flow, because the documentation states
This type of grant is commonly used for server-to-server interactions
that must run in the background, without immediate interaction with a
user.
So I've got a python script that retrieves an Access Token using the MSAL python library. Now I'm trying to authenticate with the IMAP server, using that Access Token. There's some existing threads out there showing how to connect to Google, I imagine my case is pretty close to this one, except I'm connecting to a Office 365 IMAP server. Here's my script
import imaplib
import msal
import logging
app = msal.ConfidentialClientApplication(
'client-id',
authority='https://login.microsoftonline.com/tenant-id',
client_credential='secret-key'
)
result = app.acquire_token_for_client(scopes=['https://graph.microsoft.com/.default'])
def generate_auth_string(user, token):
return 'user=%s\1auth=Bearer %s\1\1' % (user, token)
# IMAP time!
mailserver = 'outlook.office365.com'
imapport = 993
M = imaplib.IMAP4_SSL(mailserver,imapport)
M.debug = 4
M.authenticate('XOAUTH2', lambda x: generate_auth_string('user#mydomain.com', result['access_token']))
print(result)
The IMAP authentication is failing and despite setting M.debug = 4, the output isn't very helpful
22:56.53 > b'DBDH1 AUTHENTICATE XOAUTH2'
22:56.53 < b'+ '
22:56.53 write literal size 2048
22:57.84 < b'DBDH1 NO AUTHENTICATE failed.'
22:57.84 NO response: b'AUTHENTICATE failed.'
Traceback (most recent call last):
File "/home/ubuntu/mini-oauth.py", line 21, in <module>
M.authenticate("XOAUTH2", lambda x: generate_auth_string('user#mydomain.com', result['access_token']))
File "/usr/lib/python3.10/imaplib.py", line 444, in authenticate
raise self.error(dat[-1].decode('utf-8', 'replace'))
imaplib.IMAP4.error: AUTHENTICATE failed.
Any idea where I might be going wrong, or how to get more robust information from the IMAP server about why the authentication is failing?
Things I've looked at
Note this answer no longer works as the suggested scopes fail to generate an Access Token.
The client credentials flow seems to mandate the https://graph.microsoft.com/.default grant. I'm not sure if that includes the scope required for the IMAP resource
https://outlook.office.com/IMAP.AccessAsUser.All?
Verified the code lifted from the Google thread produces the SASL XOAUTH2 string correctly, per example on the MS docs
import base64
user = 'test#contoso.onmicrosoft.com'
token = 'EwBAAl3BAAUFFpUAo7J3Ve0bjLBWZWCclRC3EoAA'
xoauth = "user=%s\1auth=Bearer %s\1\1" % (user, token)
xoauth = xoauth.encode('ascii')
xoauth = base64.b64encode(xoauth)
xoauth = xoauth.decode('ascii')
xsanity = 'dXNlcj10ZXN0QGNvbnRvc28ub25taWNyb3NvZnQuY29tAWF1dGg9QmVhcmVyIEV3QkFBbDNCQUFVRkZwVUFvN0ozVmUwYmpMQldaV0NjbFJDM0VvQUEBAQ=='
print(xoauth == xsanity) # prints True
This thread seems to suggest multiple tokens need to be fetched, one for graph, then another for the IMAP connection; could that be what I'm missing?

Try the below steps.
For Client Credentials Flow you need to assign “Application permissions” in the app registration, instead of “Delegated permissions”.
Add permission “Office 365 Exchange Online / IMAP.AccessAsApp” (application).
Grant admin consent to you application
Service Principals and Exchange.
Once a service principal is registered with Exchange Online, administrators can run the Add-Mailbox Permission cmdlet to assign receive permissions to the service principal.
Use scope 'https://outlook.office365.com/.default'.
Now you can generate the SALS authentication string by combining this access token and the mailbox username to authenticate with IMAP4.
#Python code
def get_access_token():
tenantID = 'abc'
authority = 'https://login.microsoftonline.com/' + tenantID
clientID = 'abc'
clientSecret = 'abc'
scope = ['https://outlook.office365.com/.default']
app = ConfidentialClientApplication(clientID,
authority=authority,
client_credential = clientSecret)
access_token = app.acquire_token_for_client(scopes=scope)
return access_token
def generate_auth_string(user, token):
auth_string = f"user={user}\1auth=Bearer {token}\1\1"
return auth_string
#IMAP AUTHENTICATE
imap = imaplib.IMAP4_SSL(imap_host, 993)
imap.debug = 4
access_token = get_access_token_to_authenticate_imap()
imap.authenticate("XOAUTH2", lambda x:generate_auth_string(
'useremail',
access_token['access_token']))
imap.select('inbox')

The imaplib.IMAP4.error: AUTHENTICATE failed Error occured because one point in the documentation is not that clear.
When setting up the the Service Principal via Powershell you need to enter the App-ID and an Object-ID. Many people will think, it is the Object-ID you see on the overview page of the registered App, but its not!
At this point you need the Object-ID from "Azure Active Directory -> Enterprise Applications --> Your-App --> Object-ID"
New-ServicePrincipal -AppId <APPLICATION_ID> -ServiceId <OBJECT_ID> [-Organization <ORGANIZATION_ID>]
Microsoft says:
The OBJECT_ID is the Object ID from the Overview page of the
Enterprise Application node (Azure Portal) for the application
registration. It is not the Object ID from the Overview of the App
Registrations node. Using the incorrect Object ID will cause an
authentication failure.
Ofcourse you need to take care for the API-permissions and the other stuff, but this was for me the point.
So lets go trough it again, like it is explained on the documentation page.
Authenticate an IMAP, POP or SMTP connection using OAuth
Register the Application in your Tenant
Setup a Client-Key for the application
Setup the API permissions, select the APIs my organization uses tab and search for "Office 365 Exchange Online" -> Application permissions -> Choose IMAP and IMAP.AccessAsApp
Setup the Service Principal and full access for your Application on the mailbox
Check if IMAP is activated for the mailbox
Thats the code I use to test it:
import imaplib
import msal
import pprint
conf = {
"authority": "https://login.microsoftonline.com/XXXXyourtenantIDXXXXX",
"client_id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX", #AppID
"scope": ['https://outlook.office365.com/.default'],
"secret": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-Value
"secret-id": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-ID
}
def generate_auth_string(user, token):
return f"user={user}\x01auth=Bearer {token}\x01\x01"
if __name__ == "__main__":
app = msal.ConfidentialClientApplication(conf['client_id'], authority=conf['authority'],
client_credential=conf['secret'])
result = app.acquire_token_silent(conf['scope'], account=None)
if not result:
print("No suitable token in cache. Get new one.")
result = app.acquire_token_for_client(scopes=conf['scope'])
if "access_token" in result:
print(result['token_type'])
pprint.pprint(result)
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id"))
imap = imaplib.IMAP4('outlook.office365.com')
imap.starttls()
imap.authenticate("XOAUTH2", lambda x: generate_auth_string("target_mailbox#example.com", result['access_token']).encode("utf-8"))
After setting up the Service Principal and giving the App full access on the mailbox, wait 15 - 30 minutes for the changes to take effect and test it.

Try with this script:
import json
import msal
import requests
client_id = '***'
client_secret = '***'
tenant_id = '***'
authority = f"https://login.microsoftonline.com/{tenant_id}"
app = msal.ConfidentialClientApplication(
client_id=client_id,
client_credential=client_secret,
authority=authority)
scopes = ["https://graph.microsoft.com/.default"]
result = None
result = app.acquire_token_silent(scopes, account=None)
if not result:
print(
"No suitable token exists in cache. Let's get a new one from Azure Active Directory.")
result = app.acquire_token_for_client(scopes=scopes)
# if "access_token" in result:
# print("Access token is " + result["access_token"])
if "access_token" in result:
userId = "***"
endpoint = f'https://graph.microsoft.com/v1.0/users/{userId}/sendMail'
toUserEmail = "***"
email_msg = {'Message': {'Subject': "Test Sending Email from Python",
'Body': {'ContentType': 'Text', 'Content': "This is a test email."},
'ToRecipients': [{'EmailAddress': {'Address': toUserEmail}}]
},
'SaveToSentItems': 'true'}
r = requests.post(endpoint,
headers={'Authorization': 'Bearer ' + result['access_token']}, json=email_msg)
if r.ok:
print('Sent email successfully')
else:
print(r.json())
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id"))
Source: https://kontext.tech/article/795/python-send-email-via-microsoft-graph-api

Validate signature-message before request-body validation in Connexion

Currently, I have a working API that uses Connexion and receives an OpenAPI specification:
connexion_app.add_api(
"openapi.yaml",
options={"swagger_ui": False},
validate_responses=True,
strict_validation=True, # Changing this also didn't help
)
A response gets validated in the following order:
Check if API-Key is valid
Validate if the request body contains all necessary parameters
Validate message-signature
Handle request and send response
The verification of the API-Key is done via the OpenAPI spec:
securitySchemes:
apiKeyAuth:
type: apiKey
in: header
name: API-Key
x-apikeyInfoFunc: server.security.check_api_key
security:
- apiKeyAuth: []
The validation is also done via the OpenAPI spec.
The signature gets verified in the endpoint:
if not verify_signature(kwargs):
abort(401, "Signature could not be verified")
Where verify_signature is basically this:
def verify_signature(request) -> bool:
"""Calculate the signature using the header and data."""
signature = re.findall(r'"([A-Za-z0-9+/=]+)"', connexion.request.headers.get("Message-Signature", ""))
created = re.findall(r"created=(\d+)", connexion.request.headers.get("Message-Signature", ""))
if len(signature) == 0:
abort(401, "No valid Signature found.")
if len(created) == 0:
abort(401, "No valid created timestamp found.")
signature = signature[0]
created = int(created[0])
method, path, host, api_key, content_type = _get_attributes_from_request()
message = create_signature_message(request["body"], created, method, path, host, api_key, content_type)
recreated_signature = _encode_message(message)
return recreated_signature == str(signature)
For security purposes I would like to swap 2. and 3.:
Check if API-Key is valid
Validate message-signature
Validate if the request body contains all necessary parameters
Handle request and send response
The problem is that Connexion validates the body before I get to my endpoint in which I execute my Python code such as verify_signature.
I tried adding the following to my OpenAPI.yaml:
signatureAuth:
type: http
scheme: basic
x-basicInfoFunc: server.security.verify_signature
security:
- apiKeyAuth: []
signatureAuth: []
But I think this is the wrong approach since I think this is only used as a simple verification method and I get the following error message:
No authorization token provided.
Now to my question:
Is there a way to execute a function which receives the whole request that gets executed before Connexion validates the body?

Yes you can use the Connexion before_request annotation so it runs a function on a new request before validating the body. Here's an example that logs the headers and content:
import connexion
import logging
from flask import request
logger = logging.getLogger(__name__)
conn_app = connexion.FlaskApp(__name__)
#conn_app.app.before_request
def before_request():
for h in request.headers:
logger.debug('header %s', h)
logger.debug('data %s', request.get_data())

Error Getting Managed Identity Access Token from Azure Function

I'm having an issue retrieving an Azure Managed Identity access token from my Function App. The function gets a token then accesses a Mysql database using that token as the password.
I am getting this response from the function:
9103 (HY000): An error occurred while validating the access token. Please acquire a new token and retry.
Code:
import logging
import mysql.connector
import requests
import azure.functions as func
def main(req: func.HttpRequest) -> func.HttpResponse:
def get_access_token():
URL = "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fossrdbms-aad.database.windows.net&client_id=<client_id>"
headers = {"Metadata":"true"}
try:
req = requests.get(URL, headers=headers)
except Exception as e:
print(str(e))
return str(e)
else:
password = req.json()["access_token"]
return password
def get_mysql_connection(password):
"""
Get a Mysql Connection.
"""
try:
con = mysql.connector.connect(
host='<host>.mysql.database.azure.com',
user='<user>#<db>',
password=password,
database = 'materials_db',
auth_plugin='mysql_clear_password'
)
except Exception as e:
print(str(e))
return str(e)
else:
return "Connected to DB!"
password = get_access_token()
return func.HttpResponse(get_mysql_connection(password))
Running a modified version of this code on a VM with my managed identity works. It seems that the Function App is not allowed to get an access token. Any help would be appreciated.
Note: I have previously logged in as AzureAD Manager to the DB and created this user with all privileges to this DB.
Edit: No longer calling endpoint for VMs.
def get_access_token():
identity_endpoint = os.environ["IDENTITY_ENDPOINT"] # Env var provided by Azure. Local to service doing the requesting.
identity_header = os.environ["IDENTITY_HEADER"] # Env var provided by Azure. Local to service doing the requesting.
api_version = "2019-08-01" # "2018-02-01" #"2019-03-01" #"2019-08-01"
CLIENT_ID = "<client_id>"
resource_requested = "https%3A%2F%2Fossrdbms-aad.database.windows.net"
# resource_requested = "https://ossrdbms-aad.database.windows.net"
URL = f"{identity_endpoint}?api-version={api_version}&resource={resource_requested}&client_id={CLIENT_ID}"
headers = {"X-IDENTITY-HEADER":identity_header}
try:
req = requests.get(URL, headers=headers)
except Exception as e:
print(str(e))
return str(e)
else:
try:
password = req.json()["access_token"]
except:
password = str(req.text)
return password
But now I am getting this Error:
{"error":{"code":"UnsupportedApiVersion","message":"The HTTP resource that matches the request URI 'http://localhost:8081/msi/token?api-version=2019-08-01&resource=https%3A%2F%2Fossrdbms-aad.database.windows.net&client_id=<client_idxxxxx>' does not support the API version '2019-08-01'.","innerError":null}}
Upon inspection this seems to be a general error. This error message is propagated even if it's not the underlying issue. Noted several times in Github.
Is my endpoint correct now?

For this problem, it was caused by the wrong endpoint you request for the access token. We can just use the endpoint http://169.254.169.254/metadata/identity..... in azure VM, but if in azure function we can not use it.
In azure function, we need to get the IDENTITY_ENDPOINT from the environment.
identity_endpoint = os.environ["IDENTITY_ENDPOINT"]
The endpoint is like:
http://127.0.0.1:xxxxx/MSI/token/
You can refer to this tutorial about it, you can also find the python code sample in the tutorial.
In my function code, I also add the client id of the managed identity I created in the token_auth_uri but I'm not sure if the client_id is necessary here (In my case, I use user-assigned identity but not system-assigned identity).
token_auth_uri = f"{identity_endpoint}?resource={resource_uri}&api-version=2019-08-01&client_id={client_id}"
Update:
#r "Newtonsoft.Json"
using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;
public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{
string resource="https://ossrdbms-aad.database.windows.net";
string clientId="xxxxxxxx";
log.LogInformation("C# HTTP trigger function processed a request.");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(String.Format("{0}/?resource={1}&api-version=2019-08-01&client_id={2}", Environment.GetEnvironmentVariable("IDENTITY_ENDPOINT"), resource,clientId));
request.Headers["X-IDENTITY-HEADER"] = Environment.GetEnvironmentVariable("IDENTITY_HEADER");
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader streamResponse = new StreamReader(response.GetResponseStream());
string stringResponse = streamResponse.ReadToEnd();
log.LogInformation("test:"+stringResponse);
string name = req.Query["name"];
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
name = name ?? data?.name;
return name != null
? (ActionResult)new OkObjectResult($"Hello, {name}")
: new BadRequestObjectResult("Please pass a name on the query string or in the request body");
}

For your latest issue, where you are seeing UnsupportedApiVersion, it is probably this issue: https://github.com/MicrosoftDocs/azure-docs/issues/53726
Here are a couple of options that worked for me:
I am assuming you are hosting the Function app on Linux. I noticed that ApiVersion 2017-09-01 works, but you need to make additional changes (instead of "X-IDENTITY-HEADER", use "secret" header). And also use a system-assigned managed identity for your function app, and not a user assigned identity.
When I hosted the function app on Windows, I didn't have the same issues. So if you want to use an user-assigned managed identity, you can try this option instead. (with the api-version=2019-08-01, and X-IDENTITY-HEADER.

Call functions that use specific types on Suds, Python

I have the following code:
import logging
logging.basicConfig(level=logging.INFO)
logging.getLogger('suds.client').setLevel(logging.DEBUG)
from suds.client import Client
url = 'https://webpay3gint.transbank.cl/WSWebpayTransaction/cxf/WSWebpayService?wsdl'
client = Client(url)
print client
Which produces the following output:
Service ( WSWebpayServiceImplService ) tns="http://service.wswebpay.webpay.transbank.com/"
Prefixes (1)
ns0 = "http://service.wswebpay.webpay.transbank.com/"
Ports (1):
(WSWebpayServiceImplPort)
Methods (3):
acknowledgeTransaction(xs:string tokenInput, )
getTransactionResult(xs:string tokenInput, )
initTransaction(wsInitTransactionInput wsInitTransactionInput, )
Types (14):
acknowledgeTransaction
acknowledgeTransactionResponse
cardDetail
getTransactionResult
getTransactionResultResponse
initTransaction
initTransactionResponse
transactionResultOutput
wpmDetailInput
wsInitTransactionInput
wsInitTransactionOutput
wsTransactionDetail
wsTransactionDetailOutput
wsTransactionType
As you can see, some methods (initTransaction) use custom types (wsInitTransactionInput), how can I create an element of that custom type so I'm able to call initTransaction()?

The solution was to write:
object = client.factory.create('wsInitTransactionInput')
client.service.initTransaction(object)
And it was in the suds documentation.

Unable utilize a methodname(from webservice)present in the form of a string.I tried utilizing getattr,but then not unable to extend it for suds

from suds.client import Client ##UnresolvedImport
from suds.transport.https import HttpAuthenticated ##UnresolvedImport
import urllib2
class methodinvokeclass():
def methodinvokemethod(self,*args):
method=args[1]
c=args[2]
print c
response=c.service.method("90210")# I know this wont work,coz of method, but even I cant get me way thru with getattr
#response=c.service.LatLonListZipCode("90210")
print response
if __name__=="__main__":
invokemethodname="LatLonListZipCode"#Webservice name which I want to invoke...later !
f=open("C:\wsdllocation.txt",'r')# picks up the WSDL file location from the above file
webservwsdl=f.readline()
f.close()
y=methodinvokeclass()#dummy object
z=methodinvokeclass()#dummy object
t = HttpAuthenticated(username='x', password='x')#helps me getting thru my corporate firewall
t.handler = urllib2.HTTPBasicAuthHandler(t.pm)#helps me getting thru my corporate firewall
t.urlopener = urllib2.build_opener(t.handler)#helps me getting thru my corporate firewall
c = Client(url=webservwsdl,transport=t)#SUDs client !!!!!
x=y.methodinvokemethod(z,invokemethodname,c)# invoking the code above

You can use getattr to retrieve the SOAP method from the service. Use this instead:
impl = getattr(c.service, method)
response = impl('90210')
print response
Here is a working example, using a webservicex test SOAP service:
url = 'http://www.webservicex.net/stockquote.asmx?WSDL'
client = Client(url=url)
name = 'GetQuote'
impl = getattr(client.service, name)
print impl('IBM')
Output:
<StockQuotes>
<Stock>
<Symbol>IBM</Symbol>
<Last>163.81</Last>
<Date>4/7/2011</Date>
<Time>11:47am</Time>
<Change>-0.23</Change>
<Open>164.10</Open>
<High>164.5463</High>
<Low>163.44</Low>
<Volume>1573461</Volume>
<MktCap>199.8B</MktCap>
<PreviousClose>164.04</PreviousClose>
<PercentageChange>-0.14%</PercentageChange>
<AnnRange>116.00 - 167.72</AnnRange>
<Earns>11.52</Earns>
<P-E>14.24</P-E>
<Name>International Bus</Name>
</Stock>
</StockQuotes>