Develop Reference

Office 365 IMAP authentication via OAuth2 and python MSAL library

I'm trying to upgrade a legacy mail bot to authenticate via Oauth2 instead of Basic authentication, as it's now deprecated two days from now.
The document states applications can retain their original logic, while swapping out only the authentication bit
Application developers who have built apps that send, read, or
otherwise process email using these protocols will be able to keep the
same protocol, but need to implement secure, Modern authentication
experiences for their users. This functionality is built on top of
Microsoft Identity platform v2.0 and supports access to Microsoft 365
email accounts.
Note I've explicitly chosen the client credentials flow, because the documentation states
This type of grant is commonly used for server-to-server interactions
that must run in the background, without immediate interaction with a
user.
So I've got a python script that retrieves an Access Token using the MSAL python library. Now I'm trying to authenticate with the IMAP server, using that Access Token. There's some existing threads out there showing how to connect to Google, I imagine my case is pretty close to this one, except I'm connecting to a Office 365 IMAP server. Here's my script
import imaplib
import msal
import logging
app = msal.ConfidentialClientApplication(
'client-id',
authority='https://login.microsoftonline.com/tenant-id',
client_credential='secret-key'
)
result = app.acquire_token_for_client(scopes=['https://graph.microsoft.com/.default'])
def generate_auth_string(user, token):
return 'user=%s\1auth=Bearer %s\1\1' % (user, token)
# IMAP time!
mailserver = 'outlook.office365.com'
imapport = 993
M = imaplib.IMAP4_SSL(mailserver,imapport)
M.debug = 4
M.authenticate('XOAUTH2', lambda x: generate_auth_string('user#mydomain.com', result['access_token']))
print(result)
The IMAP authentication is failing and despite setting M.debug = 4, the output isn't very helpful
22:56.53 > b'DBDH1 AUTHENTICATE XOAUTH2'
22:56.53 < b'+ '
22:56.53 write literal size 2048
22:57.84 < b'DBDH1 NO AUTHENTICATE failed.'
22:57.84 NO response: b'AUTHENTICATE failed.'
Traceback (most recent call last):
File "/home/ubuntu/mini-oauth.py", line 21, in <module>
M.authenticate("XOAUTH2", lambda x: generate_auth_string('user#mydomain.com', result['access_token']))
File "/usr/lib/python3.10/imaplib.py", line 444, in authenticate
raise self.error(dat[-1].decode('utf-8', 'replace'))
imaplib.IMAP4.error: AUTHENTICATE failed.
Any idea where I might be going wrong, or how to get more robust information from the IMAP server about why the authentication is failing?
Things I've looked at
Note this answer no longer works as the suggested scopes fail to generate an Access Token.
The client credentials flow seems to mandate the https://graph.microsoft.com/.default grant. I'm not sure if that includes the scope required for the IMAP resource
https://outlook.office.com/IMAP.AccessAsUser.All?
Verified the code lifted from the Google thread produces the SASL XOAUTH2 string correctly, per example on the MS docs
import base64
user = 'test#contoso.onmicrosoft.com'
token = 'EwBAAl3BAAUFFpUAo7J3Ve0bjLBWZWCclRC3EoAA'
xoauth = "user=%s\1auth=Bearer %s\1\1" % (user, token)
xoauth = xoauth.encode('ascii')
xoauth = base64.b64encode(xoauth)
xoauth = xoauth.decode('ascii')
xsanity = 'dXNlcj10ZXN0QGNvbnRvc28ub25taWNyb3NvZnQuY29tAWF1dGg9QmVhcmVyIEV3QkFBbDNCQUFVRkZwVUFvN0ozVmUwYmpMQldaV0NjbFJDM0VvQUEBAQ=='
print(xoauth == xsanity) # prints True
This thread seems to suggest multiple tokens need to be fetched, one for graph, then another for the IMAP connection; could that be what I'm missing?

Try the below steps.
For Client Credentials Flow you need to assign “Application permissions” in the app registration, instead of “Delegated permissions”.
Add permission “Office 365 Exchange Online / IMAP.AccessAsApp” (application).
Grant admin consent to you application
Service Principals and Exchange.
Once a service principal is registered with Exchange Online, administrators can run the Add-Mailbox Permission cmdlet to assign receive permissions to the service principal.
Use scope 'https://outlook.office365.com/.default'.
Now you can generate the SALS authentication string by combining this access token and the mailbox username to authenticate with IMAP4.
#Python code
def get_access_token():
tenantID = 'abc'
authority = 'https://login.microsoftonline.com/' + tenantID
clientID = 'abc'
clientSecret = 'abc'
scope = ['https://outlook.office365.com/.default']
app = ConfidentialClientApplication(clientID,
authority=authority,
client_credential = clientSecret)
access_token = app.acquire_token_for_client(scopes=scope)
return access_token
def generate_auth_string(user, token):
auth_string = f"user={user}\1auth=Bearer {token}\1\1"
return auth_string
#IMAP AUTHENTICATE
imap = imaplib.IMAP4_SSL(imap_host, 993)
imap.debug = 4
access_token = get_access_token_to_authenticate_imap()
imap.authenticate("XOAUTH2", lambda x:generate_auth_string(
'useremail',
access_token['access_token']))
imap.select('inbox')

The imaplib.IMAP4.error: AUTHENTICATE failed Error occured because one point in the documentation is not that clear.
When setting up the the Service Principal via Powershell you need to enter the App-ID and an Object-ID. Many people will think, it is the Object-ID you see on the overview page of the registered App, but its not!
At this point you need the Object-ID from "Azure Active Directory -> Enterprise Applications --> Your-App --> Object-ID"
New-ServicePrincipal -AppId <APPLICATION_ID> -ServiceId <OBJECT_ID> [-Organization <ORGANIZATION_ID>]
Microsoft says:
The OBJECT_ID is the Object ID from the Overview page of the
Enterprise Application node (Azure Portal) for the application
registration. It is not the Object ID from the Overview of the App
Registrations node. Using the incorrect Object ID will cause an
authentication failure.
Ofcourse you need to take care for the API-permissions and the other stuff, but this was for me the point.
So lets go trough it again, like it is explained on the documentation page.
Authenticate an IMAP, POP or SMTP connection using OAuth
Register the Application in your Tenant
Setup a Client-Key for the application
Setup the API permissions, select the APIs my organization uses tab and search for "Office 365 Exchange Online" -> Application permissions -> Choose IMAP and IMAP.AccessAsApp
Setup the Service Principal and full access for your Application on the mailbox
Check if IMAP is activated for the mailbox
Thats the code I use to test it:
import imaplib
import msal
import pprint
conf = {
"authority": "https://login.microsoftonline.com/XXXXyourtenantIDXXXXX",
"client_id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX", #AppID
"scope": ['https://outlook.office365.com/.default'],
"secret": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-Value
"secret-id": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-ID
}
def generate_auth_string(user, token):
return f"user={user}\x01auth=Bearer {token}\x01\x01"
if __name__ == "__main__":
app = msal.ConfidentialClientApplication(conf['client_id'], authority=conf['authority'],
client_credential=conf['secret'])
result = app.acquire_token_silent(conf['scope'], account=None)
if not result:
print("No suitable token in cache. Get new one.")
result = app.acquire_token_for_client(scopes=conf['scope'])
if "access_token" in result:
print(result['token_type'])
pprint.pprint(result)
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id"))
imap = imaplib.IMAP4('outlook.office365.com')
imap.starttls()
imap.authenticate("XOAUTH2", lambda x: generate_auth_string("target_mailbox#example.com", result['access_token']).encode("utf-8"))
After setting up the Service Principal and giving the App full access on the mailbox, wait 15 - 30 minutes for the changes to take effect and test it.

Try with this script:
import json
import msal
import requests
client_id = '***'
client_secret = '***'
tenant_id = '***'
authority = f"https://login.microsoftonline.com/{tenant_id}"
app = msal.ConfidentialClientApplication(
client_id=client_id,
client_credential=client_secret,
authority=authority)
scopes = ["https://graph.microsoft.com/.default"]
result = None
result = app.acquire_token_silent(scopes, account=None)
if not result:
print(
"No suitable token exists in cache. Let's get a new one from Azure Active Directory.")
result = app.acquire_token_for_client(scopes=scopes)
# if "access_token" in result:
# print("Access token is " + result["access_token"])
if "access_token" in result:
userId = "***"
endpoint = f'https://graph.microsoft.com/v1.0/users/{userId}/sendMail'
toUserEmail = "***"
email_msg = {'Message': {'Subject': "Test Sending Email from Python",
'Body': {'ContentType': 'Text', 'Content': "This is a test email."},
'ToRecipients': [{'EmailAddress': {'Address': toUserEmail}}]
},
'SaveToSentItems': 'true'}
r = requests.post(endpoint,
headers={'Authorization': 'Bearer ' + result['access_token']}, json=email_msg)
if r.ok:
print('Sent email successfully')
else:
print(r.json())
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id"))
Source: https://kontext.tech/article/795/python-send-email-via-microsoft-graph-api

eBay Marketplace Account Deletion/Closure Notifications

I am trying to start an eBay API in Python and I can't find a single answer as to how to get an API key with eBay's new requirements of "Account Deletion/Closure Notifications." Here's the link: https://developer.ebay.com/marketplace-account-deletion
Specifically, I am told that "Your Keyset is currently disabled" because I have not completed whatever process is needed for this marketplace account deletion/closure notification.
The problems?
I have no idea if I need this.
I have no idea how to actually do this.
Re: 1. It looks like this is for anyone who stores user data. I don’t think that’s me intentionally because I really just want to get sold data and current listings, but is it actually me?
Re: 2. I don’t understand how to validate it and send back the proper responses. I’ve gotten quite good at python but I’m lost here.
eBay forums are completely useless and I see no one with an answer to this. Any help is greatly appreciated.

Re: 1. Same. Here's my interpretation: In order to use their APIs, you need to provide (and configure) your own API, so they can communicate with you —programatically— and tell you what users have asked to have their accounts/data deleted.
Re: 2. To handle their GET and POST requests, I guess you'll need to configure a website's URL as an API endpoint. In Django, I might use something like this (untested) code:
import hashlib
import json
from django.http import (
HttpResponse,
JsonResponse,
HttpResponseBadRequest
)
def your_api_endpoint(request):
"""
API Endpoint to handle the verification's challenge code and
receive eBay's Marketplace Account Deletion/Closure Notifications.
"""
# STEP 1: Handle verification's challenge code
challengeCode = request.GET.get('challenge_code')
if challengeCode is not None:
# Token needs to be 32-80 characters long
verificationToken = "your-token-012345678901234567890123456789"
# URL needs to use HTTPS protocol
endpoint_url = "https://your-domain.com/your-endpoint"
# Hash elements need to be ordered as follows
m = hashlib.sha256((challengeCode+verificationToken+endpoint_url).encode('utf-8'))
# JSON field needs to be called challengeResponse
return JsonResponse({"challengeResponse": m.hexdigest()}, status=200)
# STEP 2: Handle account deletion/closure notification
elif request.method == 'POST':
notification_details = json.loads(request.body)
# Verify notification is actually from eBay
# ...
# Delete/close account
# ...
# Acknowledge notification reception
return HttpResponse(status=200)
else:
return HttpResponseBadRequest()
If you find the answer to question number one, please do let me know.

Re: 1. You need to comply with eBay's Marketplace Account Deletion/Closure Notification workflow if you are storing user data into your own database. For example, using eBay's Buy APIs, you may get access to what users are selling on eBay (for ex. an eBay feed of products). If those eBay sellers decide they want to remove all of their personal data from eBay's database, eBay is requesting you remove their data from your database as well. If you are NOT storing any eBay user data into your database, you do not need to comply. Here is where you can find more info: https://partnerhelp.ebay.com/helpcenter/s/article/Complying-with-the-eBay-Marketplace-Account-Deletion-Closure-Notification-workflow?language=en_US
Re: 2. To be honest I've spent days trying to figure this out in Python (Django), but I have a solution now and am happy to share it with whoever else comes across this issue. Here's my solution:
import os
import json
import base64
import hashlib
import requests
import logging
from OpenSSL import crypto
from rest_framework import status
from rest_framework.views import APIView
from django.http import JsonResponse
logger = logging.getLogger(__name__)
class EbayMarketplaceAccountDeletion(APIView):
"""
This is required as per eBay Marketplace Account Deletion Requirements.
See documentation here: https://developer.ebay.com/marketplace-account-deletion
"""
# Ebay Config Values
CHALLENGE_CODE = 'challenge_code'
VERIFICATION_TOKEN = os.environ.get('VERIFICATION_TOKEN')
# ^ NOTE: You can make this value up so long as it is between 32-80 characters.
ENDPOINT = 'https://example.com/ebay_marketplace_account_deletion'
# ^ NOTE: Replace this with your own endpoint
X_EBAY_SIGNATURE = 'X-Ebay-Signature'
EBAY_BASE64_AUTHORIZATION_TOKEN = os.environ.get('EBAY_BASE64_AUTHORIZATION_TOKEN')
# ^ NOTE: Here's how you can get your EBAY_BASE64_AUTHORIZATION_TOKEN:
# import base64
# base64.b64encode(b'{CLIENT_ID}:{CLIENT_SECRET}')
def __init__(self):
super(EbayMarketplaceAccountDeletion, self).__init__()
def get(self, request):
"""
Get challenge code and return challengeResponse: challengeCode + verificationToken + endpoint
:return: Response
"""
challenge_code = request.GET.get(self.CHALLENGE_CODE)
challenge_response = hashlib.sha256(challenge_code.encode('utf-8') +
self.VERIFICATION_TOKEN.encode('utf-8') +
self.ENDPOINT.encode('utf-8'))
response_parameters = {
"challengeResponse": challenge_response.hexdigest()
}
return JsonResponse(response_parameters, status=status.HTTP_200_OK)
def post(self, request):
"""
Return 200 status code and remove from db.
See how to validate the notification here:
https://developer.ebay.com/api-docs/commerce/notification/overview.html#use
"""
# Verify notification is actually from eBay #
# 1. Use a Base64 function to decode the X-EBAY-SIGNATURE header and retrieve the public key ID and signature
x_ebay_signature = request.headers[self.X_EBAY_SIGNATURE]
x_ebay_signature_decoded = json.loads(base64.b64decode(x_ebay_signature).decode('utf-8'))
kid = x_ebay_signature_decoded['kid']
signature = x_ebay_signature_decoded['signature']
# 2. Call the getPublicKey Notification API method, passing in the public key ID ("kid") retrieved from the
# decoded signature header. Documentation on getPublicKey:
# https://developer.ebay.com/api-docs/commerce/notification/resources/public_key/methods/getPublicKey
public_key = None
try:
ebay_verification_url = f'https://api.ebay.com/commerce/notification/v1/public_key/{kid}'
oauth_access_token = self.get_oauth_token()
headers = {
'Authorization': f'Bearer {oauth_access_token}'
}
public_key_request = requests.get(url=ebay_verification_url, headers=headers, data={})
if public_key_request.status_code == 200:
public_key_response = public_key_request.json()
public_key = public_key_response['key']
except Exception as e:
message_title = "Ebay Marketplace Account Deletion: Error calling getPublicKey Notfication API."
logger.error(f"{message_title} Error: {e}")
return JsonResponse({}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
# 3. Initialize the cryptographic library to perform the verification with the public key that is returned from
# the getPublicKey method. If the signature verification fails, an HTTP status of 412 Precondition Failed is returned.
pkey = crypto.load_publickey(crypto.FILETYPE_PEM, self.get_public_key_into_proper_format(public_key))
certification = crypto.X509()
certification.set_pubkey(pkey)
notification_payload = request.body
signature_decoded = base64.b64decode(signature)
try:
crypto.verify(certification, signature_decoded, notification_payload, 'sha1')
except crypto.Error as e:
message_title = f"Ebay Marketplace Account Deletion: Signature Invalid. " \
f"The signature is invalid or there is a problem verifying the signature. "
logger.warning(f"{message_title} Error: {e}")
return JsonResponse({}, status=status.HTTP_412_PRECONDITION_FAILED)
except Exception as e:
message_title = f"Ebay Marketplace Account Deletion: Error performing cryptographic validation."
logger.error(f"{message_title} Error: {e}")
return JsonResponse({}, status=status.HTTP_412_PRECONDITION_FAILED)
# Take appropriate action to delete the user data. Deletion should be done in a manner such that even the
# highest system privilege cannot reverse the deletion #
# TODO: Replace with your own data removal here
# Acknowledge notification reception
return JsonResponse({}, status=status.HTTP_200_OK)
def get_oauth_token(self):
"""
Returns the OAuth Token from eBay which can be used for making other API requests such as getPublicKey
"""
url = 'https://api.ebay.com/identity/v1/oauth2/token'
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'Authorization': f"Basic {self.EBAY_BASE64_AUTHORIZATION_TOKEN}"
}
payload = 'grant_type=client_credentials&scope=https%3A%2F%2Fapi.ebay.com%2Foauth%2Fapi_scope'
request = requests.post(url=url, headers=headers, data=payload)
data = request.json()
return data['access_token']
#staticmethod
def get_public_key_into_proper_format(public_key):
"""
Public key needs to have \n in places to be properly assessed by crypto library.
"""
return public_key[:26] + '\n' + public_key[26:-24] + '\n' + public_key[-24:]

This is how I am dealing with the ebay notification requirement using Python3 cgi. Because bytes are sent, cannot use cgi.FieldStorage()
import os
import sys
import hashlib
import json
from datetime import datetime
from html import escape
import cgi
import cgitb
import io
include_path = '/var/domain_name/www'
sys.path.insert(0, include_path)
cgitb.enable(display=0, logdir=f"""{include_path}/tmp_errors""") # include_path is OUTDIR
dt_now = datetime.now()
current_dt_now = dt_now.strftime("%Y-%m-%d_%H-%M-%S")
def enc_print(string='', encoding='utf8'):
sys.stdout.buffer.write(string.encode(encoding) + b'\n')
html = ''
challengeCode = ''
# GET
myQuery = os.environ.get('QUERY_STRING')
if myQuery.find('=') != -1:
pos = myQuery.find('=')
var_name = myQuery[:pos]
var_val = myQuery[pos+1:]
challengeCode = var_val
# POST
if os.environ.get('CONTENT_LENGTH') != None:
totalBytes=int(os.environ.get('CONTENT_LENGTH'))
reqbytes=io.open(sys.stdin.fileno(),"rb").read(totalBytes)
if challengeCode != '' :
"""
API Endpoint to handle the verification's challenge code and
receive eBay's Marketplace Account Deletion/Closure Notifications.
"""
# STEP 1: Handle verification's challenge code
# Token needs to be 32-80 characters long
verificationToken = "0123456789012345678901234567890123456789" #sample token
# URL needs to use HTTPS protocol
endpoint = "https://domain_name.com/ebay/notification.py" # sample endpoint
# Hash elements need to be ordered as follows
m = hashlib.sha256( (challengeCode+verificationToken+endpoint).encode('utf-8') )
# JSON field needs to be called challengeResponse
enc_print("Content-Type: application/json")
enc_print("Status: 200 OK")
enc_print()
enc_print('{"challengeResponse":"' + m.hexdigest() + '"}')
exit()
else :
#html += 'var length:' + str(totalBytes) + '\n'
html += reqbytes.decode('utf-8') + '\n'
# STEP 2: Handle account deletion/closure notification
# Verify notification is actually from eBay
# ...
# Delete/close account
# ...
# Acknowledge notification reception
with open( f"""./notifications/{current_dt_now}_user_notification.txt""", 'w') as f:
f.write(html)
enc_print("Content-Type: application/json")
enc_print("Status: 200 OK")
enc_print()
exit()

I've been trying #José Matías Arévalo code. It works except "STEP 2" branch - Django returns 403 error. This is because of by default Django uses CSRF middleware (Cross Site Request Forgery protection). To avoid 403 error we need to marks a view as being exempt from the protection as described here https://docs.djangoproject.com/en/dev/ref/csrf/#utilities so add couple strings in code:
from django.views.decorators.csrf import csrf_exempt
#csrf_exempt
def your_api_endpoint(request):
And in my case I use url "https://your-domain.com/your-endpoint/" with slash symbol "/" at the end of url. Without this slash eBay doesn't confirm subscription.

I am using Flask and this is the code I have used:
from flask import Flask, request
import hashlib
# Create a random verification token, it needs to be 32-80 characters long
verification_token = 'a94cbd68e463cb9780e2008b1f61986110a5fd0ff8b99c9cba15f1f802ad65f9'
endpoint_url = 'https://dev.example.com'
app = Flask(__name__)
# There will be errors if you just use '/' as the route as it will redirect eBays request
# eBay will send a request to https://dev.example.com?challenge_code=123
# The request will get redirected by Flask to https://dev.example.com/?challenge_code=123 which eBay will not accept
endpoint = endpoint_url + '/test'
# The Content-Type header will be added automatically by Flask as 'application/json'
#app.route('/test')
def test():
code = request.args.get('challenge_code')
print('Requests argument:', code)
code = code + token + endpoint
code = code.encode('utf-8')
code = hashlib.sha256(code)
code = code.hexdigest()
print('Hexdigest:', code)
final = {"challengeResponse": code}
return final
## To run locally first use this:
# app.run(port=29)

How do I send credentials so I can unit test MSAL for Python?

The flow to this admin app is basically:
User goes to /admin.
#azure/msal-react checks if the user is logged in and if not redirects them to login.
When the accessToken, idToken, and oid have been received on the FE, they are sent to my API (/api) where they are validated.
If the tokens are validated, and verified as belonging to the oid that came from the FE, then a JWT is issued for the user indicating they are authenticated with the API as well.
The JWT is sent back to the FE where having both the MSAL isAuthenticated() and their API JWT isApiAuthenticated() indicates they are fully authenticated and it renders the FE components.
I'm trying to write unit tests for the what I have in the aad.py (bottom).
The issue I'm seeing is that I need to retrieve and send it a legit MSAL access_token, id_token, and oid. That means I actually need to log in a user against our AAD.
I do have a dummy user to use for this purpose, but I'm not seeing in the Python MSAL documentation how to sign the user in without interaction and just submit username and password.
Suggestions for how to write unit tests to test? Do I need to mockup something else, or should I be able to send a dummy user's username and password with Python MSAL?
# Python libraries
import json
# Third-party dependencies
import jwt
import requests
from base64 import b64decode, b64encode
from cryptography import x509
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import serialization
from config import settings
def ms_graph(access_token):
"""
Takes the access_token from the FE and uses it query MS Graph for user
information.
Returns an object.
Takes one argument:
access_token: sent from FE.
"""
graph_response = requests.get( # Use token to call downstream service
settings.GRAPH_URI,
headers={'Authorization': 'Bearer ' + access_token},)
return json.loads(graph_response.text)
def validate_token(token):
"""
Used to decode the token sent from the FE. The result is
a verified signature that contains information about the user and tokens.
Returns an object.
Takes one argument:
token: The idToken sent from the FE that needs to be validated.
"""
# Get a list of the possible public keys from the JWKS_URI endpoint
jwkeys = requests.get(settings.JWKS_URI).json()['keys']
# Extract the 'kid' from the unverified header to get the public key
token_key_id = jwt.get_unverified_header(token)['kid']
# Get the object corresponding to the 'kid' key
jwk = [key for key in jwkeys if key['kid'] == token_key_id][0]
# Encode the 'x5c' to eventually get the public key for decoding
der_cert = b64decode(jwk['x5c'][0])
cert = x509.load_der_x509_certificate(der_cert, default_backend())
public_key = cert.public_key()
pem_key = public_key.public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo
)
# Finally, validate and decode the token
token_claims = jwt.decode(
token,
pem_key,
algorithms=['RS256'],
audience=settings.CLIENT_ID
)
return token_claims
def validate_user(access_token, id_token, client_oid):
"""
Takes the id_token sent from the FE and validates the signature.
The validated signature also contains an oid claim which is used to verify
the oid sent from the FE.
Returns an object containing:
{
"user_validated": bool,
"graph_response": object
}
Takes two arguments:
access_token: access_token sent from the FE.
id_token: the id_token sent from the FE.
client_oid: the oid sent from the FE.
"""
# Decode the token to retrieve claims
id_token_payload = validate_token(id_token)
# Check if the claim oid and matches the oid sent from the FE
if id_token_payload['oid'] == client_oid:
graph_response = ms_graph(access_token)
res = dict()
res['user_validated'] = graph_response['id'] == client_oid
res['graph_response'] = graph_response
return res

If you really really want to obtain tokens for unit tests with username and password then look at acquire_token_by_username_password() and its example in MS docs.
But there are reasons to not let unit tests use the real MS Graph API:
dependency on network connection can make unit tests run slow or fail, even if your code still works fine,
storing credentials in repository is dangerous, keeping them properly is difficult,
the dummy account can be locked,
numerous test calls can cause throttling,
it is kind of unpolite to abuse the infrastructure with non-productive requests.
The suggestion for unit tests is usual: isolate test cases, then mock and patch any external dependencies not related to code being tested.
MSAL uses requests for calling the cloud, and there is a great package for mocking its work: meet responses. Prevent MSAL for sending out any requests and mock OpenID configuration call:
import responses
import time
import unittest
MS_OPENID_CONFIG = {
"authorization_endpoint": "https://login.microsoftonline.com/common/"
"oauth2/v2.0/authorize",
"token_endpoint": "https://login.microsoftonline.com/common/oauth2/"
"v2.0/token",
}
MS_TOKENS = {
'id_token_claims': {
'exp': time.time() + 60,
'oid': '00000000-0000-0000-0000-000000000000',
'upn': 'test_user_1_'
},
'id_token': {},
'access_token': {}
}
class BaseTest(unittest.TestCase):
def afterSetUp(self):
self.responses = responses.RequestsMock()
self.responses.start()
self.responses.add(
responses.GET,
'https://login.microsoftonline.com/common/v2.0/'
'.well-known/openid-configuration',
json=MS_OPENID_CONFIG
)
# add_calback() allows modifying mock token in runtime
self.responses.add_callback(
responses.POST,
'https://login.microsoftonline.com/common/oauth2/v2.0/token',
callback=lambda request: (200, {}, json.dumps(MS_TOKENS)),
content_type='application/json',
)
self.addCleanup(self.responses.stop)
self.addCleanup(self.responses.reset)
For testing validate_token() you can bundle pre-generated test RSA key pair:
openssl genrsa -out test_rsa_private.pem 512
openssl rsa -in test_rsa_private.pem -pubout -outform PEM -out test_rsa_public.pem
Then you can mock response to settings.JWKS_URI in the same way as OpenID config, returning test public key with known kid.
If you'd prefer to save time on signing test tokens, then just mock jwt signature verification to always return True:
from unittest import Mock, patch
def test_validate_token(self)
...
with patch('jwt.algorithms.RSAAlgorithm.verify', new=Mock(return_value=True)):
# jwt.decode(at, key=open('test_rsa_public.pem').read(), algorithms=['RS256'], options={'verify_exp': False, 'verify_aud': False})
validate_token(test_token)
And so on, unit test should only test the code of unit itself, everything around can be mocked as needed. Read the source of MSAL and jwt. Don't mock what looks like implementation details, they might change in future versions.

Passwordless authentication flow using Cognito & API Gateway & Lambda (Python)

I've been trying to implement passwordless authentication using AWS Cognito & API Gateway & Lambda (Python)
I have followed these articles:
https://medium.com/digicred/password-less-authentication-in-cognito-cafa016d4db7
https://medium.com/#pjatocheseminario/passwordless-api-using-cognito-and-serverless-framework-7fa952191352
I have configured Cognito (to accept CUSTOM_AUTH), added the Lambdas, and created the API endpoints:
/sign-up
/initiate-auth (aka initiate login)
/respond-to-auth-challenge (aka (verify login)
When calling initiateAuth I receive the following response:
An error occurred (NotAuthorizedException) when calling the InitiateAuth operation: Incorrect username or password."
I'm using CUSTOM_AUTH which doesn't require password, and the user name is definitely correct because it actually initiates the authentication flow and I receive a code, however because boto3 doesn't respond with a session I can't continue the authentication.
This is how I call Cognito:
res = cognito.initiate_auth(
ClientId=client_id,
AuthFlow="CUSTOM_AUTH",
AuthParameters={
"USERNAME": email,
"PASSWORD": random_password
}
)
It's probably something small I'm missing but I can't figure out what.

Your client code looks OK, mine has ClientId param in it but if your code is not raising an exception then it should be fine. Unless you had Generate client secret option checked when you created your app client.
If that is the case then you have to pass in SECRET_HASH in AuthParameters like the following:
import hmac
import hashlib
import base64
def get_secret_hash(email, client_id, client_secret):
"""
A keyed-hash message authentication code (HMAC) calculated using
the secret key of a user pool client and username plus the client
ID in the message.
"""
message = email + client_id
client_secret = str.encode(client_secret)
dig = hmac.new(client_secret, msg=message.encode('UTF-8'), digestmod=hashlib.sha256).digest()
return base64.b64encode(dig).decode()
client.admin_initiate_auth(
UserPoolId=COGNITO_USER_POOL_ID,
ClientId=CLIENT_ID,
AuthFlow='CUSTOM_AUTH',
AuthParameters={
'USERNAME': email,
'SECRET_HASH': get_secret_hash(email, CLIENT_ID, CLIENT_SECRET) # Omit if secret key option is disabled.
},
)
Next, double check the following:
Under App clients > * > Auth Flows Configuration, is ALLOW_CUSTOM_AUTH option enabled for your client?
Under App integration > App client settings > * > Enabled Identity Providers, is your user pool selected?
If you have Cognito setup correctly and your code still doesn't work then it is probably the lambda code. You probably know this but for password-less custom auth you need to use 3 lambda triggers: Define Auth Challenge, Create Auth Challenge, and Verify Auth Challenge.
Custom auth lambdas events are triggered in the following order:
DefineAuthChallenge_Authentication:
Technically, issueTokens can be set to True here to return tokens without going through the rest of the steps.
def lambda_handler(event, context):
if event['triggerSource'] == 'DefineAuthChallenge_Authentication':
event['response']['challengeName'] = 'CUSTOM_CHALLENGE'
event['response']['issueTokens'] = False
event['response']['failAuthentication'] = False
if event['request']['session']: # Needed for step 4.
# If all of the challenges are answered, issue tokens.
event['response']['issueTokens'] = all(
answered_challenge['challengeResult'] for answered_challenge in event['request']['session'])
return event
CreateAuthChallenge_Authentication:
def lambda_handler(event, context):
if event['triggerSource'] == 'CreateAuthChallenge_Authentication':
if event['request']['challengeName'] == 'CUSTOM_CHALLENGE':
event['response']['privateChallengeParameters'] = {}
event['response']['privateChallengeParameters']['answer'] = 'YOUR CHALLENGE ANSWER HERE'
event['response']['challengeMetadata'] = 'AUTHENTICATE_AS_CHALLENGE'
return event
Then your client must respond to the challenge:
client.respond_to_auth_challenge(
ClientId=CLIENT_ID,
ChallengeName='CUSTOM_CHALLENGE',
Session=session,
ChallengeResponses={
'USERNAME': email,
'ANSWER': 'Extra Protection!',
'SECRET_HASH': get_secret_hash(email, CLIENT_ID, CLIENT_SECRET) # Omit if secret key option is disabled.
}
)
VerifyAuthChallengeResponse_Authentication:
def lambda_handler(event, context):
if event['triggerSource'] == 'VerifyAuthChallengeResponse_Authentication':
if event['request']['challengeAnswer'] == event['request']['privateChallengeParameters']['answer']:
event['response']['answerCorrect'] = True
return event
DefineAuthChallenge_Authentication:
Set event['response']['issueTokens'] to True to return tokens (code shown in step 1), or issue another challenge to keep repeating steps 1-3.
Finally, make sure that if case-insensitivity option is enabled for your user pool too. Also, I can't exactly recall if CUSTOM_AUTH flow worked if the user is in FORCE_CHANGE_PASSWORD status. If the user is in that state, then try settings a permanent password with the sdk to set the status to CONFIRMED.

I was facing the same error, and I think that the error message is misleading.
When you did not respond correctly in Create-Auth-Challenge lambda, you will get this error. So make sure everything is right in your lambda.

How to create a SECRET_HASH for AWS Cognito using boto3?

I want to create/calculate a SECRET_HASH for AWS Cognito using boto3 and python. This will be incorporated in to my fork of warrant.
I configured my cognito app client to use an app client secret. However, this broke the following code.
def renew_access_token(self):
"""
Sets a new access token on the User using the refresh token.
NOTE:
Does not work if "App client secret" is enabled. 'SECRET_HASH' is needed in AuthParameters.
'SECRET_HASH' requires HMAC calculations.
Does not work if "Device Tracking" is turned on.
https://stackoverflow.com/a/40875783/1783439
'DEVICE_KEY' is needed in AuthParameters. See AuthParameters section.
https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_InitiateAuth.html
"""
refresh_response = self.client.initiate_auth(
ClientId=self.client_id,
AuthFlow='REFRESH_TOKEN',
AuthParameters={
'REFRESH_TOKEN': self.refresh_token
# 'SECRET_HASH': How to generate this?
},
)
self._set_attributes(
refresh_response,
{
'access_token': refresh_response['AuthenticationResult']['AccessToken'],
'id_token': refresh_response['AuthenticationResult']['IdToken'],
'token_type': refresh_response['AuthenticationResult']['TokenType']
}
)
When I run this I receive the following exception:
botocore.errorfactory.NotAuthorizedException:
An error occurred (NotAuthorizedException) when calling the InitiateAuth operation:
Unable to verify secret hash for client <client id echoed here>.
This answer informed me that a SECRET_HASH is required to use the cognito client secret.
The aws API reference docs AuthParameters section states the following:
For REFRESH_TOKEN_AUTH/REFRESH_TOKEN: USERNAME (required), SECRET_HASH
(required if the app client is configured with a client secret),
REFRESH_TOKEN (required), DEVICE_KEY
The boto3 docs state that a SECRET_HASH is
A keyed-hash message authentication code (HMAC) calculated using the
secret key of a user pool client and username plus the client ID in
the message.
The docs explain what is needed, but not how to achieve this.

The below get_secret_hash method is a solution that I wrote in Python for a Cognito User Pool implementation, with example usage:
import boto3
import botocore
import hmac
import hashlib
import base64
class Cognito:
client_id = app.config.get('AWS_CLIENT_ID')
user_pool_id = app.config.get('AWS_USER_POOL_ID')
identity_pool_id = app.config.get('AWS_IDENTITY_POOL_ID')
client_secret = app.config.get('AWS_APP_CLIENT_SECRET')
# Public Keys used to verify tokens returned by Cognito:
# http://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-with-identity-providers.html#amazon-cognito-identity-user-pools-using-id-and-access-tokens-in-web-api
id_token_public_key = app.config.get('JWT_ID_TOKEN_PUB_KEY')
access_token_public_key = app.config.get('JWT_ACCESS_TOKEN_PUB_KEY')
def __get_client(self):
return boto3.client('cognito-idp')
def get_secret_hash(self, username):
# A keyed-hash message authentication code (HMAC) calculated using
# the secret key of a user pool client and username plus the client
# ID in the message.
message = username + self.client_id
dig = hmac.new(self.client_secret, msg=message.encode('UTF-8'),
digestmod=hashlib.sha256).digest()
return base64.b64encode(dig).decode()
# REQUIRES that `ADMIN_NO_SRP_AUTH` be enabled on Client App for User Pool
def login_user(self, username_or_alias, password):
try:
return self.__get_client().admin_initiate_auth(
UserPoolId=self.user_pool_id,
ClientId=self.client_id,
AuthFlow='ADMIN_NO_SRP_AUTH',
AuthParameters={
'USERNAME': username_or_alias,
'PASSWORD': password,
'SECRET_HASH': self.get_secret_hash(username_or_alias)
}
)
except botocore.exceptions.ClientError as e:
return e.response

I also got a TypeError when I tried the above solution. Here is the solution that worked for me:
import hmac
import hashlib
import base64
# Function used to calculate SecretHash value for a given client
def calculateSecretHash(client_id, client_secret, username):
key = bytes(client_secret, 'utf-8')
message = bytes(f'{username}{client_id}', 'utf-8')
return base64.b64encode(hmac.new(key, message, digestmod=hashlib.sha256).digest()).decode()
# Usage example
calculateSecretHash(client_id, client_secret, username)