Develop Reference

Office 365 IMAP authentication via OAuth2 and python MSAL library

I'm trying to upgrade a legacy mail bot to authenticate via Oauth2 instead of Basic authentication, as it's now deprecated two days from now.
The document states applications can retain their original logic, while swapping out only the authentication bit
Application developers who have built apps that send, read, or
otherwise process email using these protocols will be able to keep the
same protocol, but need to implement secure, Modern authentication
experiences for their users. This functionality is built on top of
Microsoft Identity platform v2.0 and supports access to Microsoft 365
email accounts.
Note I've explicitly chosen the client credentials flow, because the documentation states
This type of grant is commonly used for server-to-server interactions
that must run in the background, without immediate interaction with a
user.
So I've got a python script that retrieves an Access Token using the MSAL python library. Now I'm trying to authenticate with the IMAP server, using that Access Token. There's some existing threads out there showing how to connect to Google, I imagine my case is pretty close to this one, except I'm connecting to a Office 365 IMAP server. Here's my script
import imaplib
import msal
import logging
app = msal.ConfidentialClientApplication(
'client-id',
authority='https://login.microsoftonline.com/tenant-id',
client_credential='secret-key'
)
result = app.acquire_token_for_client(scopes=['https://graph.microsoft.com/.default'])
def generate_auth_string(user, token):
return 'user=%s\1auth=Bearer %s\1\1' % (user, token)
# IMAP time!
mailserver = 'outlook.office365.com'
imapport = 993
M = imaplib.IMAP4_SSL(mailserver,imapport)
M.debug = 4
M.authenticate('XOAUTH2', lambda x: generate_auth_string('user#mydomain.com', result['access_token']))
print(result)
The IMAP authentication is failing and despite setting M.debug = 4, the output isn't very helpful
22:56.53 > b'DBDH1 AUTHENTICATE XOAUTH2'
22:56.53 < b'+ '
22:56.53 write literal size 2048
22:57.84 < b'DBDH1 NO AUTHENTICATE failed.'
22:57.84 NO response: b'AUTHENTICATE failed.'
Traceback (most recent call last):
File "/home/ubuntu/mini-oauth.py", line 21, in <module>
M.authenticate("XOAUTH2", lambda x: generate_auth_string('user#mydomain.com', result['access_token']))
File "/usr/lib/python3.10/imaplib.py", line 444, in authenticate
raise self.error(dat[-1].decode('utf-8', 'replace'))
imaplib.IMAP4.error: AUTHENTICATE failed.
Any idea where I might be going wrong, or how to get more robust information from the IMAP server about why the authentication is failing?
Things I've looked at
Note this answer no longer works as the suggested scopes fail to generate an Access Token.
The client credentials flow seems to mandate the https://graph.microsoft.com/.default grant. I'm not sure if that includes the scope required for the IMAP resource
https://outlook.office.com/IMAP.AccessAsUser.All?
Verified the code lifted from the Google thread produces the SASL XOAUTH2 string correctly, per example on the MS docs
import base64
user = 'test#contoso.onmicrosoft.com'
token = 'EwBAAl3BAAUFFpUAo7J3Ve0bjLBWZWCclRC3EoAA'
xoauth = "user=%s\1auth=Bearer %s\1\1" % (user, token)
xoauth = xoauth.encode('ascii')
xoauth = base64.b64encode(xoauth)
xoauth = xoauth.decode('ascii')
xsanity = 'dXNlcj10ZXN0QGNvbnRvc28ub25taWNyb3NvZnQuY29tAWF1dGg9QmVhcmVyIEV3QkFBbDNCQUFVRkZwVUFvN0ozVmUwYmpMQldaV0NjbFJDM0VvQUEBAQ=='
print(xoauth == xsanity) # prints True
This thread seems to suggest multiple tokens need to be fetched, one for graph, then another for the IMAP connection; could that be what I'm missing?

Try the below steps.
For Client Credentials Flow you need to assign “Application permissions” in the app registration, instead of “Delegated permissions”.
Add permission “Office 365 Exchange Online / IMAP.AccessAsApp” (application).
Grant admin consent to you application
Service Principals and Exchange.
Once a service principal is registered with Exchange Online, administrators can run the Add-Mailbox Permission cmdlet to assign receive permissions to the service principal.
Use scope 'https://outlook.office365.com/.default'.
Now you can generate the SALS authentication string by combining this access token and the mailbox username to authenticate with IMAP4.
#Python code
def get_access_token():
tenantID = 'abc'
authority = 'https://login.microsoftonline.com/' + tenantID
clientID = 'abc'
clientSecret = 'abc'
scope = ['https://outlook.office365.com/.default']
app = ConfidentialClientApplication(clientID,
authority=authority,
client_credential = clientSecret)
access_token = app.acquire_token_for_client(scopes=scope)
return access_token
def generate_auth_string(user, token):
auth_string = f"user={user}\1auth=Bearer {token}\1\1"
return auth_string
#IMAP AUTHENTICATE
imap = imaplib.IMAP4_SSL(imap_host, 993)
imap.debug = 4
access_token = get_access_token_to_authenticate_imap()
imap.authenticate("XOAUTH2", lambda x:generate_auth_string(
'useremail',
access_token['access_token']))
imap.select('inbox')

The imaplib.IMAP4.error: AUTHENTICATE failed Error occured because one point in the documentation is not that clear.
When setting up the the Service Principal via Powershell you need to enter the App-ID and an Object-ID. Many people will think, it is the Object-ID you see on the overview page of the registered App, but its not!
At this point you need the Object-ID from "Azure Active Directory -> Enterprise Applications --> Your-App --> Object-ID"
New-ServicePrincipal -AppId <APPLICATION_ID> -ServiceId <OBJECT_ID> [-Organization <ORGANIZATION_ID>]
Microsoft says:
The OBJECT_ID is the Object ID from the Overview page of the
Enterprise Application node (Azure Portal) for the application
registration. It is not the Object ID from the Overview of the App
Registrations node. Using the incorrect Object ID will cause an
authentication failure.
Ofcourse you need to take care for the API-permissions and the other stuff, but this was for me the point.
So lets go trough it again, like it is explained on the documentation page.
Authenticate an IMAP, POP or SMTP connection using OAuth
Register the Application in your Tenant
Setup a Client-Key for the application
Setup the API permissions, select the APIs my organization uses tab and search for "Office 365 Exchange Online" -> Application permissions -> Choose IMAP and IMAP.AccessAsApp
Setup the Service Principal and full access for your Application on the mailbox
Check if IMAP is activated for the mailbox
Thats the code I use to test it:
import imaplib
import msal
import pprint
conf = {
"authority": "https://login.microsoftonline.com/XXXXyourtenantIDXXXXX",
"client_id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX", #AppID
"scope": ['https://outlook.office365.com/.default'],
"secret": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-Value
"secret-id": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-ID
}
def generate_auth_string(user, token):
return f"user={user}\x01auth=Bearer {token}\x01\x01"
if __name__ == "__main__":
app = msal.ConfidentialClientApplication(conf['client_id'], authority=conf['authority'],
client_credential=conf['secret'])
result = app.acquire_token_silent(conf['scope'], account=None)
if not result:
print("No suitable token in cache. Get new one.")
result = app.acquire_token_for_client(scopes=conf['scope'])
if "access_token" in result:
print(result['token_type'])
pprint.pprint(result)
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id"))
imap = imaplib.IMAP4('outlook.office365.com')
imap.starttls()
imap.authenticate("XOAUTH2", lambda x: generate_auth_string("target_mailbox#example.com", result['access_token']).encode("utf-8"))
After setting up the Service Principal and giving the App full access on the mailbox, wait 15 - 30 minutes for the changes to take effect and test it.

Try with this script:
import json
import msal
import requests
client_id = '***'
client_secret = '***'
tenant_id = '***'
authority = f"https://login.microsoftonline.com/{tenant_id}"
app = msal.ConfidentialClientApplication(
client_id=client_id,
client_credential=client_secret,
authority=authority)
scopes = ["https://graph.microsoft.com/.default"]
result = None
result = app.acquire_token_silent(scopes, account=None)
if not result:
print(
"No suitable token exists in cache. Let's get a new one from Azure Active Directory.")
result = app.acquire_token_for_client(scopes=scopes)
# if "access_token" in result:
# print("Access token is " + result["access_token"])
if "access_token" in result:
userId = "***"
endpoint = f'https://graph.microsoft.com/v1.0/users/{userId}/sendMail'
toUserEmail = "***"
email_msg = {'Message': {'Subject': "Test Sending Email from Python",
'Body': {'ContentType': 'Text', 'Content': "This is a test email."},
'ToRecipients': [{'EmailAddress': {'Address': toUserEmail}}]
},
'SaveToSentItems': 'true'}
r = requests.post(endpoint,
headers={'Authorization': 'Bearer ' + result['access_token']}, json=email_msg)
if r.ok:
print('Sent email successfully')
else:
print(r.json())
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id"))
Source: https://kontext.tech/article/795/python-send-email-via-microsoft-graph-api

getting a "this program may not be used for excuting instructions" error when making a simple spl transfer

i'm trying to send an spl-token transaction but it's not working, it's a transacion on the devnet with a token that i newly created, my code is
from spl.token.instructions import transfer, TransferParams
from spl.token.client import Client
from solana.publickey import PublicKey
from solana.transaction import Transaction
from solana.keypair import Keypair
address = "address sending from"
my_token = "3hALJzSz2bx8gxgrHg7EQQtdiHxG7d7LNswxVMXrUApw" #token addredd on devnet
private_key= "64 bit key"
def send_payouts_spl(dest,amount):
source = address
transfer_params= TransferParams(
amount=amount,
dest=PublicKey(dest),
owner=PublicKey(source),
program_id=PublicKey(my_token),
source=PublicKey(source)
)
txn = Transaction()
txn.add(transfer(transfer_params))
solana_client = Client("https://api.devnet.solana.com")
owner = Keypair.from_secret_key(private_key)
tx_id = solana_client.send_transaction(txn, owner)
return tx_id
and also the error that i'm getting
solana.rpc.core.RPCException: {'code': -32002, 'message': 'Transaction simulation failed: This program may not be used for executing instructions', 'data': {'accounts': None, 'err': 'InvalidProgramForExecution', 'logs': [], 'unitsConsumed': 0}}
also if it helps, my devnet token address and my devnet address are
3hALJzSz2bx8gxgrHg7EQQtdiHxG7d7LNswxVMXrUApw, EckcvMCmpkKwF4hDhWxq8cm4qy8JBkb2vBVQDu4WvxmM respectively

In Solana, there's typically just one token program that is shared for all token types (mints).
When you provide the program_id, you shouldn't provide the address for your mint, but rather the id for the SPL Token Program, which is TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA. Also, the owner should be the key that owns the token account.
So instead, try:
owner = Keypair.from_secret_key(private_key)
transfer_params= TransferParams(
amount=amount,
dest=PublicKey(dest),
owner=PublicKey(owner.public_key),
program_id=PublicKey('TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA'),
source=PublicKey(source)
)

Solana SPL token transfer with python

So after having read a couple of articles I'm yet to understand how to create a Transaction and send custom SPL tokens across the Solana blockchain.
I've attached my code below.
I truly don't understand what each part of the transaction is supposed to be.
So I figured that owner is the account/wallet that is sending and paying for the transaction. And I'm assuming that dest is where I wish to send the tokens to.
This is the token (on devnet) that I wish to send, But I don't seem to be able.
from spl.token.constants import TOKEN_PROGRAM_ID
from spl.token.instructions import transfer_checked, TransferCheckedParams
from solana.rpc.commitment import Confirmed
from solana.rpc.api import Client
from solana.rpc.types import TxOpts
from solana.keypair import Keypair
from solana.publickey import PublicKey
from solana.transaction import Transaction
import os
from dotenv import load_dotenv
class TransferService:
def __init__(self, client: Client, service: SolanaService, token) -> None:
self.client = client
self.service = service
self.keypair = self.service.get_keypair(token)
def make_transaction(self, source, mint, dest, owner, amount=1, decimals=0) -> Transaction:
transaction = Transaction()
transaction.add(transfer_checked(
TransferCheckedParams(
program_id=TOKEN_PROGRAM_ID,
mint=PublicKey(mint),
source=PublicKey(source),
dest=PublicKey(dest),
owner=owner,
amount=amount,
decimals=decimals,
signers=[]
)))
return transaction
def send_transaction(self, transaction) -> None:
self.client.send_transaction(
transaction,
self.keypair,
opts=TxOpts(skip_confirmation=False, preflight_commitment=Confirmed)
)
load_dotenv()
if __name__ == "__main__":
token = os.getenv('TOKEN')
client = Client('https://api.devnet.solana.com')
service = SolanaService(client)
token = os.getenv('KEYPAIR')
transfer = TransferService(client, service, token)
a = client.get_account_info(transfer.keypair.public_key)
transaction = transfer.make_transaction(
source='CtURxXpzn9aredXse2KNtyDMeVW627tL3p7DCucdv8bc',
mint='DCzbhHu3YGnc8Vhez4YEMznQ38ad6WYGVYqeB4Wn3mie',
dest='sPkypr2LBtF5Go87zYSn5fBtWxCDEcobWeQQxXHpxJR',
owner=transfer.keypair.public_key,
amount=1,
decimals=9
)
transfer.send_transaction(transaction)

The destination sPkypr2LBtF5Go87zYSn5fBtWxCDEcobWeQQxXHpxJR is incorrect.
When you send SPL tokens, the source and dest must be addresses of token accounts, and in this case, sPkypr2LBtF5Go87zYSn5fBtWxCDEcobWeQQxXHpxJR is a wallet, so you'll need to create a recipient account for this wallet.
The preferred standard is to use an associated token account, created using something like create_associated_token_account from the SPL part of solana-py client: https://github.com/michaelhly/solana-py/blob/2c45353cb510bfeb7259fa19dacbaefe6b9ae3d1/src/spl/token/client.py#L173
For reference, the most important part is creating the instruction to create the associated token account:
def create_associated_token_account(payer: PublicKey, owner: PublicKey, mint: PublicKey) -> TransactionInstruction:
associated_token_address = PublicKey.find_program_address(
seeds=[bytes(owner), bytes(TOKEN_PROGRAM_ID), bytes(mint)], program_id=ASSOCIATED_TOKEN_PROGRAM_ID
)
return TransactionInstruction(
keys=[
AccountMeta(pubkey=payer, is_signer=True, is_writable=True),
AccountMeta(pubkey=associated_token_address, is_signer=False, is_writable=True),
AccountMeta(pubkey=owner, is_signer=False, is_writable=False),
AccountMeta(pubkey=mint, is_signer=False, is_writable=False),
AccountMeta(pubkey=SYS_PROGRAM_ID, is_signer=False, is_writable=False),
AccountMeta(pubkey=TOKEN_PROGRAM_ID, is_signer=False, is_writable=False),
AccountMeta(pubkey=SYSVAR_RENT_PUBKEY, is_signer=False, is_writable=False),
],
program_id=ASSOCIATED_TOKEN_PROGRAM_ID,
)
More background information about associated token accounts at https://spl.solana.com/associated-token-account

Passwordless authentication flow using Cognito & API Gateway & Lambda (Python)

I've been trying to implement passwordless authentication using AWS Cognito & API Gateway & Lambda (Python)
I have followed these articles:
https://medium.com/digicred/password-less-authentication-in-cognito-cafa016d4db7
https://medium.com/#pjatocheseminario/passwordless-api-using-cognito-and-serverless-framework-7fa952191352
I have configured Cognito (to accept CUSTOM_AUTH), added the Lambdas, and created the API endpoints:
/sign-up
/initiate-auth (aka initiate login)
/respond-to-auth-challenge (aka (verify login)
When calling initiateAuth I receive the following response:
An error occurred (NotAuthorizedException) when calling the InitiateAuth operation: Incorrect username or password."
I'm using CUSTOM_AUTH which doesn't require password, and the user name is definitely correct because it actually initiates the authentication flow and I receive a code, however because boto3 doesn't respond with a session I can't continue the authentication.
This is how I call Cognito:
res = cognito.initiate_auth(
ClientId=client_id,
AuthFlow="CUSTOM_AUTH",
AuthParameters={
"USERNAME": email,
"PASSWORD": random_password
}
)
It's probably something small I'm missing but I can't figure out what.

Your client code looks OK, mine has ClientId param in it but if your code is not raising an exception then it should be fine. Unless you had Generate client secret option checked when you created your app client.
If that is the case then you have to pass in SECRET_HASH in AuthParameters like the following:
import hmac
import hashlib
import base64
def get_secret_hash(email, client_id, client_secret):
"""
A keyed-hash message authentication code (HMAC) calculated using
the secret key of a user pool client and username plus the client
ID in the message.
"""
message = email + client_id
client_secret = str.encode(client_secret)
dig = hmac.new(client_secret, msg=message.encode('UTF-8'), digestmod=hashlib.sha256).digest()
return base64.b64encode(dig).decode()
client.admin_initiate_auth(
UserPoolId=COGNITO_USER_POOL_ID,
ClientId=CLIENT_ID,
AuthFlow='CUSTOM_AUTH',
AuthParameters={
'USERNAME': email,
'SECRET_HASH': get_secret_hash(email, CLIENT_ID, CLIENT_SECRET) # Omit if secret key option is disabled.
},
)
Next, double check the following:
Under App clients > * > Auth Flows Configuration, is ALLOW_CUSTOM_AUTH option enabled for your client?
Under App integration > App client settings > * > Enabled Identity Providers, is your user pool selected?
If you have Cognito setup correctly and your code still doesn't work then it is probably the lambda code. You probably know this but for password-less custom auth you need to use 3 lambda triggers: Define Auth Challenge, Create Auth Challenge, and Verify Auth Challenge.
Custom auth lambdas events are triggered in the following order:
DefineAuthChallenge_Authentication:
Technically, issueTokens can be set to True here to return tokens without going through the rest of the steps.
def lambda_handler(event, context):
if event['triggerSource'] == 'DefineAuthChallenge_Authentication':
event['response']['challengeName'] = 'CUSTOM_CHALLENGE'
event['response']['issueTokens'] = False
event['response']['failAuthentication'] = False
if event['request']['session']: # Needed for step 4.
# If all of the challenges are answered, issue tokens.
event['response']['issueTokens'] = all(
answered_challenge['challengeResult'] for answered_challenge in event['request']['session'])
return event
CreateAuthChallenge_Authentication:
def lambda_handler(event, context):
if event['triggerSource'] == 'CreateAuthChallenge_Authentication':
if event['request']['challengeName'] == 'CUSTOM_CHALLENGE':
event['response']['privateChallengeParameters'] = {}
event['response']['privateChallengeParameters']['answer'] = 'YOUR CHALLENGE ANSWER HERE'
event['response']['challengeMetadata'] = 'AUTHENTICATE_AS_CHALLENGE'
return event
Then your client must respond to the challenge:
client.respond_to_auth_challenge(
ClientId=CLIENT_ID,
ChallengeName='CUSTOM_CHALLENGE',
Session=session,
ChallengeResponses={
'USERNAME': email,
'ANSWER': 'Extra Protection!',
'SECRET_HASH': get_secret_hash(email, CLIENT_ID, CLIENT_SECRET) # Omit if secret key option is disabled.
}
)
VerifyAuthChallengeResponse_Authentication:
def lambda_handler(event, context):
if event['triggerSource'] == 'VerifyAuthChallengeResponse_Authentication':
if event['request']['challengeAnswer'] == event['request']['privateChallengeParameters']['answer']:
event['response']['answerCorrect'] = True
return event
DefineAuthChallenge_Authentication:
Set event['response']['issueTokens'] to True to return tokens (code shown in step 1), or issue another challenge to keep repeating steps 1-3.
Finally, make sure that if case-insensitivity option is enabled for your user pool too. Also, I can't exactly recall if CUSTOM_AUTH flow worked if the user is in FORCE_CHANGE_PASSWORD status. If the user is in that state, then try settings a permanent password with the sdk to set the status to CONFIRMED.

I was facing the same error, and I think that the error message is misleading.
When you did not respond correctly in Create-Auth-Challenge lambda, you will get this error. So make sure everything is right in your lambda.

multiuser management in O365 python

I was creating an application of office which contain multiple user credentials and do the tasks like emailing and adding calender events. I choosed O365. All things were great here except. I could not save the credentials. like in other google products we pickle the creds.
with open(f'account_data/{account_name}.pickle','wb') as stream:
pickle.dump(account, stream)
but I error as
AttributeError: Can't pickle local object 'OAuth2Session.__init__.<locals>.<lambda>'
I need to store multiple user keys and do some tasks. If you have any other module then tell me.

I figured it out myself.
from O365 import Account, MSGraphProtocol, message, FileSystemTokenBackend
def new_account(account_name):
account = Account(credentials, scopes=scopes, )
token_backend = FileSystemTokenBackend(token_path='account_data', token_filename=f'{account_name}.txt')
account.con.token_backend = token_backend
account.authenticate()
account.con.token_backend.save_token()
def load_account(account_name):
account = Account(credentials, scopes=scopes, )
token_backend = FileSystemTokenBackend(token_path='account_data', token_filename=f'{account_name}.txt')
account.con.token_backend = token_backend
account.con.token_backend.load_token()
if account.con.refresh_token():
return account