Develop Reference

Office 365 IMAP authentication via OAuth2 and python MSAL library

I'm trying to upgrade a legacy mail bot to authenticate via Oauth2 instead of Basic authentication, as it's now deprecated two days from now.
The document states applications can retain their original logic, while swapping out only the authentication bit
Application developers who have built apps that send, read, or
otherwise process email using these protocols will be able to keep the
same protocol, but need to implement secure, Modern authentication
experiences for their users. This functionality is built on top of
Microsoft Identity platform v2.0 and supports access to Microsoft 365
email accounts.
Note I've explicitly chosen the client credentials flow, because the documentation states
This type of grant is commonly used for server-to-server interactions
that must run in the background, without immediate interaction with a
user.
So I've got a python script that retrieves an Access Token using the MSAL python library. Now I'm trying to authenticate with the IMAP server, using that Access Token. There's some existing threads out there showing how to connect to Google, I imagine my case is pretty close to this one, except I'm connecting to a Office 365 IMAP server. Here's my script
import imaplib
import msal
import logging
app = msal.ConfidentialClientApplication(
'client-id',
authority='https://login.microsoftonline.com/tenant-id',
client_credential='secret-key'
)
result = app.acquire_token_for_client(scopes=['https://graph.microsoft.com/.default'])
def generate_auth_string(user, token):
return 'user=%s\1auth=Bearer %s\1\1' % (user, token)
# IMAP time!
mailserver = 'outlook.office365.com'
imapport = 993
M = imaplib.IMAP4_SSL(mailserver,imapport)
M.debug = 4
M.authenticate('XOAUTH2', lambda x: generate_auth_string('user#mydomain.com', result['access_token']))
print(result)
The IMAP authentication is failing and despite setting M.debug = 4, the output isn't very helpful
22:56.53 > b'DBDH1 AUTHENTICATE XOAUTH2'
22:56.53 < b'+ '
22:56.53 write literal size 2048
22:57.84 < b'DBDH1 NO AUTHENTICATE failed.'
22:57.84 NO response: b'AUTHENTICATE failed.'
Traceback (most recent call last):
File "/home/ubuntu/mini-oauth.py", line 21, in <module>
M.authenticate("XOAUTH2", lambda x: generate_auth_string('user#mydomain.com', result['access_token']))
File "/usr/lib/python3.10/imaplib.py", line 444, in authenticate
raise self.error(dat[-1].decode('utf-8', 'replace'))
imaplib.IMAP4.error: AUTHENTICATE failed.
Any idea where I might be going wrong, or how to get more robust information from the IMAP server about why the authentication is failing?
Things I've looked at
Note this answer no longer works as the suggested scopes fail to generate an Access Token.
The client credentials flow seems to mandate the https://graph.microsoft.com/.default grant. I'm not sure if that includes the scope required for the IMAP resource
https://outlook.office.com/IMAP.AccessAsUser.All?
Verified the code lifted from the Google thread produces the SASL XOAUTH2 string correctly, per example on the MS docs
import base64
user = 'test#contoso.onmicrosoft.com'
token = 'EwBAAl3BAAUFFpUAo7J3Ve0bjLBWZWCclRC3EoAA'
xoauth = "user=%s\1auth=Bearer %s\1\1" % (user, token)
xoauth = xoauth.encode('ascii')
xoauth = base64.b64encode(xoauth)
xoauth = xoauth.decode('ascii')
xsanity = 'dXNlcj10ZXN0QGNvbnRvc28ub25taWNyb3NvZnQuY29tAWF1dGg9QmVhcmVyIEV3QkFBbDNCQUFVRkZwVUFvN0ozVmUwYmpMQldaV0NjbFJDM0VvQUEBAQ=='
print(xoauth == xsanity) # prints True
This thread seems to suggest multiple tokens need to be fetched, one for graph, then another for the IMAP connection; could that be what I'm missing?

Try the below steps.
For Client Credentials Flow you need to assign “Application permissions” in the app registration, instead of “Delegated permissions”.
Add permission “Office 365 Exchange Online / IMAP.AccessAsApp” (application).
Grant admin consent to you application
Service Principals and Exchange.
Once a service principal is registered with Exchange Online, administrators can run the Add-Mailbox Permission cmdlet to assign receive permissions to the service principal.
Use scope 'https://outlook.office365.com/.default'.
Now you can generate the SALS authentication string by combining this access token and the mailbox username to authenticate with IMAP4.
#Python code
def get_access_token():
tenantID = 'abc'
authority = 'https://login.microsoftonline.com/' + tenantID
clientID = 'abc'
clientSecret = 'abc'
scope = ['https://outlook.office365.com/.default']
app = ConfidentialClientApplication(clientID,
authority=authority,
client_credential = clientSecret)
access_token = app.acquire_token_for_client(scopes=scope)
return access_token
def generate_auth_string(user, token):
auth_string = f"user={user}\1auth=Bearer {token}\1\1"
return auth_string
#IMAP AUTHENTICATE
imap = imaplib.IMAP4_SSL(imap_host, 993)
imap.debug = 4
access_token = get_access_token_to_authenticate_imap()
imap.authenticate("XOAUTH2", lambda x:generate_auth_string(
'useremail',
access_token['access_token']))
imap.select('inbox')

The imaplib.IMAP4.error: AUTHENTICATE failed Error occured because one point in the documentation is not that clear.
When setting up the the Service Principal via Powershell you need to enter the App-ID and an Object-ID. Many people will think, it is the Object-ID you see on the overview page of the registered App, but its not!
At this point you need the Object-ID from "Azure Active Directory -> Enterprise Applications --> Your-App --> Object-ID"
New-ServicePrincipal -AppId <APPLICATION_ID> -ServiceId <OBJECT_ID> [-Organization <ORGANIZATION_ID>]
Microsoft says:
The OBJECT_ID is the Object ID from the Overview page of the
Enterprise Application node (Azure Portal) for the application
registration. It is not the Object ID from the Overview of the App
Registrations node. Using the incorrect Object ID will cause an
authentication failure.
Ofcourse you need to take care for the API-permissions and the other stuff, but this was for me the point.
So lets go trough it again, like it is explained on the documentation page.
Authenticate an IMAP, POP or SMTP connection using OAuth
Register the Application in your Tenant
Setup a Client-Key for the application
Setup the API permissions, select the APIs my organization uses tab and search for "Office 365 Exchange Online" -> Application permissions -> Choose IMAP and IMAP.AccessAsApp
Setup the Service Principal and full access for your Application on the mailbox
Check if IMAP is activated for the mailbox
Thats the code I use to test it:
import imaplib
import msal
import pprint
conf = {
"authority": "https://login.microsoftonline.com/XXXXyourtenantIDXXXXX",
"client_id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX", #AppID
"scope": ['https://outlook.office365.com/.default'],
"secret": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-Value
"secret-id": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", #Key-ID
}
def generate_auth_string(user, token):
return f"user={user}\x01auth=Bearer {token}\x01\x01"
if __name__ == "__main__":
app = msal.ConfidentialClientApplication(conf['client_id'], authority=conf['authority'],
client_credential=conf['secret'])
result = app.acquire_token_silent(conf['scope'], account=None)
if not result:
print("No suitable token in cache. Get new one.")
result = app.acquire_token_for_client(scopes=conf['scope'])
if "access_token" in result:
print(result['token_type'])
pprint.pprint(result)
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id"))
imap = imaplib.IMAP4('outlook.office365.com')
imap.starttls()
imap.authenticate("XOAUTH2", lambda x: generate_auth_string("target_mailbox#example.com", result['access_token']).encode("utf-8"))
After setting up the Service Principal and giving the App full access on the mailbox, wait 15 - 30 minutes for the changes to take effect and test it.

Try with this script:
import json
import msal
import requests
client_id = '***'
client_secret = '***'
tenant_id = '***'
authority = f"https://login.microsoftonline.com/{tenant_id}"
app = msal.ConfidentialClientApplication(
client_id=client_id,
client_credential=client_secret,
authority=authority)
scopes = ["https://graph.microsoft.com/.default"]
result = None
result = app.acquire_token_silent(scopes, account=None)
if not result:
print(
"No suitable token exists in cache. Let's get a new one from Azure Active Directory.")
result = app.acquire_token_for_client(scopes=scopes)
# if "access_token" in result:
# print("Access token is " + result["access_token"])
if "access_token" in result:
userId = "***"
endpoint = f'https://graph.microsoft.com/v1.0/users/{userId}/sendMail'
toUserEmail = "***"
email_msg = {'Message': {'Subject': "Test Sending Email from Python",
'Body': {'ContentType': 'Text', 'Content': "This is a test email."},
'ToRecipients': [{'EmailAddress': {'Address': toUserEmail}}]
},
'SaveToSentItems': 'true'}
r = requests.post(endpoint,
headers={'Authorization': 'Bearer ' + result['access_token']}, json=email_msg)
if r.ok:
print('Sent email successfully')
else:
print(r.json())
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id"))
Source: https://kontext.tech/article/795/python-send-email-via-microsoft-graph-api

401 error code when trying to export powerbi reports

I am able to get a token using the default scope for powerbi ( "scope" :
"https://analysis.windows.net/powerbi/api/.default"] )
With that token, I am able to read the workspaces my user has access to, ( "https://api.powerbi.com/v1.0/myorg/groups") and the reports information inside each of those workspaces (
"https://api.powerbi.com/v1.0/myorg/reports/")
But it does not matter if I reuse the same token or just acquire a brand new, if I try to export a specific report, I got a 401 error code. This is the way I am issuing the requests.get
token_ = <new token or reused from previous get requests>
reports = requests.get( # Use token to call downstream service
config['reports']+report_id+'/Export',
headers={'Authorization': 'Bearer ' + token_ },)
Now, if I go to https://learn.microsoft.com/en-us/rest/api/power-bi/reports/getreportsingroup
and sign in (with the same user I am using on my python script). Get the token from that page and use it on my script. It works, If I use it in postman, it works
I I try to use the token acquired by my script in Postman, I also get a 401 error. So, yes, my script is not getting the correct token for this particular entry point but it is good enough for the groups and reports entry point.
Is there anything I need to add to the request for the token on this particular entry point?
Thank you very much,
Andres
Here is the full script I am using, there is also a params.json that looks like this:
{
"authority": "https://login.microsoftonline.com/1abcdefg-abcd-48b6-9b3c-bd5123456",
"client_id": "5d2545-abcd-4765-8fbb-53555f2fa91",
"username":"myusername#tenant",
"password": "mypass",
"scope" : ["https://analysis.windows.net/powerbi/api/.default"],
"workspaces" : "https://api.powerbi.com/v1.0/myorg/groups",
"reports": "https://api.powerbi.com/v1.0/myorg/reports/"
}
#script based on msal github library sample
import sys # For simplicity, we'll read config file from 1st CLI param sys.argv[1]
import json
import logging
import requests
import msal
def exportReport(report_id,token_):
result = app.acquire_token_by_username_password( config["username"], config["password"], scopes=config["scope"])
token_ = result['access_token']
print(f'Using token: {token_}')
reports = requests.get( # Use token to call downstream service
config['reports']+report_id+'/Export',
headers={'Authorization': 'Bearer ' + token_ },)
print(f'-reports: {reports.status_code}')
def list_reports(workspace_id,ws_id,ws_name,token_):
print(f'reports id for workspace {ws_name}')
for rp in workspace_id['value']:
if rp["id"] == "1d509119-76a1-42ce-8afd-bd3c420dd62d":
exportReport("1d509119-76a1-42ce-8afd-bd0c420dd62d",token_)
def list_workspaces(workspaces_dict):
for ws in workspaces_dict['value']:
yield (ws['id'],ws['name'])
config = json.load(open('params.json'))
app = msal.PublicClientApplication(
config["client_id"], authority=config["authority"],
)
result = None
if not result:
logging.info("No suitable token exists in cache. Let's get a new one from AAD.")
result = app.acquire_token_by_username_password(
config["username"], config["password"], scopes=config["scope"])
if "access_token" in result:
workspaces = requests.get( # Use token to call downstream service
config['workspaces'],
headers={'Authorization': 'Bearer ' + result['access_token']},).json()
ids=list_workspaces(workspaces) #prepare workspace generator
headers = {'Authorization': 'Bearer ' + result['access_token']}
while True:
try:
ws_id,ws_name=next(ids)
reports = requests.get( # Use token to call downstream service
config['workspaces']+'/'+ws_id+'/reports',
headers={'Authorization': 'Bearer ' + result['access_token']},).json()
list_reports(reports,ws_id,ws_name,result['access_token'])
except StopIteration:
exit(0)
else:
print(result.get("error"))
print(result.get("error_description"))
print(result.get("correlation_id")) # You may need this when reporting a bug
if 65001 in result.get("error_codes", []):
# AAD requires user consent for U/P flow
print("Visit this to consent:", app.get_authorization_request_url(config["scope"]))

From your description, I suppose you didn't grant the correct permission for your AD App used to login your user account in the code, please follow the steps below.
Navigate to the Azure portal -> Azure Active Directory -> App registrations -> find your AD App used in the code(filter with All applications) -> API permissions -> add the Report.Read.All Delegated permission in Power BI Service API(this permission is just for read action, if you need some further write operation, choose Report.ReadWrite.All) -> click the Grant admin consent for xxx button at last.
Update:
Use the application id of the access token got from Get-PowerBIAccessToken solve the issue.

Passwordless authentication flow using Cognito & API Gateway & Lambda (Python)

I've been trying to implement passwordless authentication using AWS Cognito & API Gateway & Lambda (Python)
I have followed these articles:
https://medium.com/digicred/password-less-authentication-in-cognito-cafa016d4db7
https://medium.com/#pjatocheseminario/passwordless-api-using-cognito-and-serverless-framework-7fa952191352
I have configured Cognito (to accept CUSTOM_AUTH), added the Lambdas, and created the API endpoints:
/sign-up
/initiate-auth (aka initiate login)
/respond-to-auth-challenge (aka (verify login)
When calling initiateAuth I receive the following response:
An error occurred (NotAuthorizedException) when calling the InitiateAuth operation: Incorrect username or password."
I'm using CUSTOM_AUTH which doesn't require password, and the user name is definitely correct because it actually initiates the authentication flow and I receive a code, however because boto3 doesn't respond with a session I can't continue the authentication.
This is how I call Cognito:
res = cognito.initiate_auth(
ClientId=client_id,
AuthFlow="CUSTOM_AUTH",
AuthParameters={
"USERNAME": email,
"PASSWORD": random_password
}
)
It's probably something small I'm missing but I can't figure out what.

Your client code looks OK, mine has ClientId param in it but if your code is not raising an exception then it should be fine. Unless you had Generate client secret option checked when you created your app client.
If that is the case then you have to pass in SECRET_HASH in AuthParameters like the following:
import hmac
import hashlib
import base64
def get_secret_hash(email, client_id, client_secret):
"""
A keyed-hash message authentication code (HMAC) calculated using
the secret key of a user pool client and username plus the client
ID in the message.
"""
message = email + client_id
client_secret = str.encode(client_secret)
dig = hmac.new(client_secret, msg=message.encode('UTF-8'), digestmod=hashlib.sha256).digest()
return base64.b64encode(dig).decode()
client.admin_initiate_auth(
UserPoolId=COGNITO_USER_POOL_ID,
ClientId=CLIENT_ID,
AuthFlow='CUSTOM_AUTH',
AuthParameters={
'USERNAME': email,
'SECRET_HASH': get_secret_hash(email, CLIENT_ID, CLIENT_SECRET) # Omit if secret key option is disabled.
},
)
Next, double check the following:
Under App clients > * > Auth Flows Configuration, is ALLOW_CUSTOM_AUTH option enabled for your client?
Under App integration > App client settings > * > Enabled Identity Providers, is your user pool selected?
If you have Cognito setup correctly and your code still doesn't work then it is probably the lambda code. You probably know this but for password-less custom auth you need to use 3 lambda triggers: Define Auth Challenge, Create Auth Challenge, and Verify Auth Challenge.
Custom auth lambdas events are triggered in the following order:
DefineAuthChallenge_Authentication:
Technically, issueTokens can be set to True here to return tokens without going through the rest of the steps.
def lambda_handler(event, context):
if event['triggerSource'] == 'DefineAuthChallenge_Authentication':
event['response']['challengeName'] = 'CUSTOM_CHALLENGE'
event['response']['issueTokens'] = False
event['response']['failAuthentication'] = False
if event['request']['session']: # Needed for step 4.
# If all of the challenges are answered, issue tokens.
event['response']['issueTokens'] = all(
answered_challenge['challengeResult'] for answered_challenge in event['request']['session'])
return event
CreateAuthChallenge_Authentication:
def lambda_handler(event, context):
if event['triggerSource'] == 'CreateAuthChallenge_Authentication':
if event['request']['challengeName'] == 'CUSTOM_CHALLENGE':
event['response']['privateChallengeParameters'] = {}
event['response']['privateChallengeParameters']['answer'] = 'YOUR CHALLENGE ANSWER HERE'
event['response']['challengeMetadata'] = 'AUTHENTICATE_AS_CHALLENGE'
return event
Then your client must respond to the challenge:
client.respond_to_auth_challenge(
ClientId=CLIENT_ID,
ChallengeName='CUSTOM_CHALLENGE',
Session=session,
ChallengeResponses={
'USERNAME': email,
'ANSWER': 'Extra Protection!',
'SECRET_HASH': get_secret_hash(email, CLIENT_ID, CLIENT_SECRET) # Omit if secret key option is disabled.
}
)
VerifyAuthChallengeResponse_Authentication:
def lambda_handler(event, context):
if event['triggerSource'] == 'VerifyAuthChallengeResponse_Authentication':
if event['request']['challengeAnswer'] == event['request']['privateChallengeParameters']['answer']:
event['response']['answerCorrect'] = True
return event
DefineAuthChallenge_Authentication:
Set event['response']['issueTokens'] to True to return tokens (code shown in step 1), or issue another challenge to keep repeating steps 1-3.
Finally, make sure that if case-insensitivity option is enabled for your user pool too. Also, I can't exactly recall if CUSTOM_AUTH flow worked if the user is in FORCE_CHANGE_PASSWORD status. If the user is in that state, then try settings a permanent password with the sdk to set the status to CONFIRMED.

I was facing the same error, and I think that the error message is misleading.
When you did not respond correctly in Create-Auth-Challenge lambda, you will get this error. So make sure everything is right in your lambda.

Accessing & Manipulating Azure AD B2C Users via Web Application

I'm struggling with basic user management while working with Azure's AD B2C framework.
I've successfully set up an Azure AD B2C resource, registered my consumer-facing web application (created a client secret and granted permission, both delegated and application, to User.ReadWrite.All), created custom attributes, and added out-of-the-box signup and signin user flows. Furthermore, I've successfully registered and signed-in users to my web application.
To get to this point, I followed the Python sample provided within the documentation (ms-identity-python-webapp-master):
app.py
#app.route("/login")
def login():
session["state"] = str(uuid.uuid4())
# Technically we could use empty list [] as scopes to do just sign in,
# here we choose to also collect end user consent upfront
auth_url = _build_auth_url(scopes=app_config_b2c.SCOPE, state=session["state"])
return render_template("templates/login.html", auth_url=auth_url, version=msal.__version__)
#app.route(app_config_b2c.REDIRECT_PATH) # Its absolute URL must match your app's redirect_uri set in AAD
def authorized():
if request.args.get('state') != session.get("state"):
return redirect(url_for("index")) # No-OP. Goes back to Index page
if "error" in request.args: # Authentication/Authorization failure
return render_template("auth_error.html", result=request.args)
if request.args.get('code'):
cache = _load_cache()
result = _build_msal_app(cache=cache).acquire_token_by_authorization_code(
request.args['code'],
scopes=app_config_b2c.SCOPE, # Misspelled scope would cause an HTTP 400 error here
redirect_uri=url_for("authorized", _external=True))
if "error" in result:
return render_template("auth_error.html", result=result)
session["user"] = result.get("id_token_claims")
_save_cache(cache)
return redirect(url_for("index"))
#app.route("/logout")
def logout():
session.clear() # Wipe out user and its token cache from session
return redirect( # Also logout from your tenant's web session
app_config_b2c.AUTHORITY + "/oauth2/v2.0/logout" +
"?post_logout_redirect_uri=" + url_for("index", _external=True))
#app.route("/graphcall")
def graphcall():
token = _get_token_from_cache(app_config_b2c.SCOPE)
if not token:
return redirect(url_for("login"))
graph_data = requests.get( # Use token to call downstream service
app_config_b2c.ENDPOINT,
headers={'Authorization': 'Bearer ' + token['access_token']},
).json()
return render_template('templates/display.html', result=graph_data)
def _load_cache():
cache = msal.SerializableTokenCache()
if session.get("token_cache"):
cache.deserialize(session["token_cache"])
return cache
def _save_cache(cache):
if cache.has_state_changed:
session["token_cache"] = cache.serialize()
def _build_msal_app(cache=None, authority=None):
return msal.ConfidentialClientApplication(
app_config_b2c.CLIENT_ID, authority=authority or app_config_b2c.AUTHORITY,
client_credential=app_config_b2c.CLIENT_SECRET, token_cache=cache)
def _build_auth_url(authority=None, scopes=None, state=None):
return _build_msal_app(authority=authority).get_authorization_request_url(
scopes or [],
state=state or str(uuid.uuid4()),
redirect_uri=url_for("authorized", _external=True))
def _get_token_from_cache(scope=None):
cache = _load_cache() # This web app maintains one cache per session
cca = _build_msal_app(cache=cache)
accounts = cca.get_accounts()
if accounts: # So all account(s) belong to the current signed-in user
result = cca.acquire_token_silent(scope, account=accounts[0])
_save_cache(cache)
return result
app_config_b2c.py
import os
b2c_tenant = "myapplication"
signupsignin_user_flow = "b2c_1_signupsignin1"
editprofile_user_flow = "b2c_1_profileediting1"
resetpassword_user_flow = "b2c_1_passwordreset1"
authority_template = "https://{tenant}.b2clogin.com/{tenant}.onmicrosoft.com/{user_flow}"
CLIENT_SECRET = "Enter_the_Client_Secret_Here" # Our Quickstart uses this placeholder
# In your production app, we recommend you to use other ways to store your secret,
# such as KeyVault, or environment variable as described in Flask's documentation here
# https://flask.palletsprojects.com/en/1.1.x/config/#configuring-from-environment-variables
# CLIENT_SECRET = os.getenv("CLIENT_SECRET")
# if not CLIENT_SECRET:
# raise ValueError("Need to define CLIENT_SECRET environment variable")
AUTHORITY = authority_template.format(
tenant=b2c_tenant, user_flow=signupsignin_user_flow)
B2C_PROFILE_AUTHORITY = authority_template.format(
tenant=b2c_tenant, user_flow=editprofile_user_flow)
B2C_RESET_PASSWORD_AUTHORITY = authority_template.format(
tenant=b2c_tenant, user_flow=resetpassword_user_flow)
CLIENT_ID = "xxx.xxxxxx"
REDIRECT_PATH = "/getAToken" # It will be used to form an absolute URL
# And that absolute URL must match your app's redirect_uri set in AAD
# This is the resource that you are going to access in your B2C tenant
ENDPOINT = 'https://graph.microsoft.com/v1.0/users'
# These are the scopes that you defined for the web API
SCOPE = ["User.ReadWrite.All"]
SESSION_TYPE = "filesystem" # So token cache will be stored in server-side session
The graphcall doesn't work within this framework (perhaps it's a b2c issue), which I'm sure is part of the problem, but ultimately I'd just like the application to consume the logged-in user's AD attributes (particularly the custom attributes I've enabled), and modify them when necessary. For instance, say a custom attribute is "paid_subscriber". When a user registers, the attribute is empty. When the user purchases content, I'd like to set the value of the attribute to something relevant (like "true").
Is this possible? Do I need other user flows? What am I missing here (theoretically and practically)?

Microsoft Graph does not support the tokens issued by the Azure AD B2C.
You need to have the access token generated by the Azure AD.
There is a process using Azure ad b2c custom policy where you can integrated the Microsoft Graph and add custom attributes to the claims.
This document helps you to get the Azure AD access token to call Graph, With the above implementation there wont be much changes in the phython code
for adding the custom attributes go through the document

Youtube API commentThreads.update return 400 error when replying to topLevelComment

Reporting a possible bug on Youtube Api commentThreads.update
Name of API affected: CommentThreads: update
Issue summary: CommentThreads: update When calling this api to update the topLevelComment returned by CommentThreads.list, I got a 400 error. I was using this API in an internal tool to allow me respond to comments. It was working fine for the last months... and in the last days (~2016-11-07) with no changes on any code on my side, the API started to return a 400 error. I have all my internal comment systems halted right now as I could not figure out how to fix this. Every thing I tried and tested brings me to the 400 error and the message below.
"While this can be a transient error, it usually indicates that the requests input is invalid."
Even using the API console, I got the same errror!
My bet? This is a transient error that I hope Google can fix.
Steps to reproduce issue:
Retrieve "heldForReview" threads using CommentThreads.list
Try to update the returned topLevelComment changing ["topLevelComment"]["snippet"]["textOriginal"] = "some text". Optionally, change the "moderationStatus" to "published". The error is the same.
This behaviour can be easily spotted using the Python API samples, specifically the file comment_threads.py from https://github.com/youtube/api-samples/blob/master/python/comment_threads.py
Expected output: The top level comment from Thread should become visible (after setting the moderationStatus to published) and have the "some text" as the first child comment as its answer.
Actual results:
{
"error": {
"errors": [
{
"domain": "youtube.commentThread",
"reason": "processingFailure",
"message": "The API server failed to successfully process the request. While this can be a transient error, it usually indicates that the requests input is invalid. Check the structure of the \u003ccode\u003ecommentThread\u003c/code\u003e resource in the request body to ensure that it is valid.",
"locationType": "other",
"location": "body"
}
],
"code": 400,
"message": "The API server failed to successfully process the request. While this can be a transient error, it usually indicates that the requests input is invalid. Check the structure of the \u003ccode\u003ecommentThread\u003c/code\u003e resource in the request body to ensure that it is valid."
}
}
Notes: I am providing a sample code below.
#!/usr/bin/python
# Usage example:
# python comment_threads.py --channelid='<channel_id>' --videoid='<video_id>' --text='<text>'
import httplib2
import os
import sys
from apiclient.discovery import build_from_document
from apiclient.errors import HttpError
from oauth2client.client import flow_from_clientsecrets
from oauth2client.file import Storage
from oauth2client.tools import argparser, run_flow
# The CLIENT_SECRETS_FILE variable specifies the name of a file that contains
# the OAuth 2.0 information for this application, including its client_id and
# client_secret. You can acquire an OAuth 2.0 client ID and client secret from
# the {{ Google Cloud Console }} at
# {{ https://cloud.google.com/console }}.
# Please ensure that you have enabled the YouTube Data API for your project.
# For more information about using OAuth2 to access the YouTube Data API, see:
# https://developers.google.com/youtube/v3/guides/authentication
# For more information about the client_secrets.json file format, see:
# https://developers.google.com/api-client-library/python/guide/aaa_client_secrets
CLIENT_SECRETS_FILE = "client_secrets.json"
# This OAuth 2.0 access scope allows for full read/write access to the
# authenticated user's account and requires requests to use an SSL connection.
YOUTUBE_READ_WRITE_SSL_SCOPE = "https://www.googleapis.com/auth/youtube.force-ssl"
YOUTUBE_API_SERVICE_NAME = "youtube"
YOUTUBE_API_VERSION = "v3"
# This variable defines a message to display if the CLIENT_SECRETS_FILE is
# missing.
MISSING_CLIENT_SECRETS_MESSAGE = """
WARNING: Please configure OAuth 2.0
To make this sample run you will need to populate the client_secrets.json file
found at:
%s
with information from the APIs Console
https://console.developers.google.com
For more information about the client_secrets.json file format, please visit:
https://developers.google.com/api-client-library/python/guide/aaa_client_secrets
""" % os.path.abspath(os.path.join(os.path.dirname(__file__),CLIENT_SECRETS_FILE))
# Authorize the request and store authorization credentials.
def get_authenticated_service(args):
flow = flow_from_clientsecrets(CLIENT_SECRETS_FILE, scope=YOUTUBE_READ_WRITE_SSL_SCOPE,message=MISSING_CLIENT_SECRETS_MESSAGE)
storage = Storage("%s-oauth2.json" % sys.argv[0])
credentials = storage.get()
if credentials is None or credentials.invalid:
credentials = run_flow(flow, storage, args)
# Trusted testers can download this discovery document from the developers page
# and it should be in the same directory with the code.
with open("youtube-v3-discoverydocument.json", "r") as f:
doc = f.read()
return build_from_document(doc, http=credentials.authorize(httplib2.Http()))
# Call the API's commentThreads.list method to list the existing comments.
def get_comments(youtube, video_id, channel_id):
results = youtube.commentThreads().list(
part="snippet",
videoId=video_id,
channelId=channel_id,
# ONLY FILTER COMMENTS THAT ARE HELD FOR REVIEW
# THOSE COMMENTS ONLY HAVE ONE TOP LEVEL COMMENT
moderationStatus="heldForReview",
textFormat="plainText").execute()
for item in results["items"]:
comment = item["snippet"]["topLevelComment"]
author = comment["snippet"]["authorDisplayName"]
text = comment["snippet"]["textDisplay"]
print "Comment by %s: %s" % (author, text)
return results["items"]
# Call the API's commentThreads.insert method to insert a comment.
def insert_comment(youtube, channel_id, video_id, text):
insert_result = youtube.commentThreads().insert(
part="snippet",
body=dict(
snippet=dict(
channelId=channel_id,
videoId=video_id,
topLevelComment=dict(
snippet=dict(
textOriginal=text
)
)
)
)
).execute()
comment = insert_result["snippet"]["topLevelComment"]
author = comment["snippet"]["authorDisplayName"]
text = comment["snippet"]["textDisplay"]
print "Inserted comment for %s: %s" % (author, text)
# Call the API's commentThreads.update method to update an existing comment.
def update_comment(youtube, comment):
comment["snippet"]["topLevelComment"]["snippet"]["textOriginal"] = 'updated'
update_result = youtube.commentThreads().update(
part="snippet",
body=comment
).execute()
comment = update_result["snippet"]["topLevelComment"]
author = comment["snippet"]["authorDisplayName"]
text = comment["snippet"]["textDisplay"]
print "Updated comment for %s: %s" % (author, text)
if __name__ == "__main__":
# The "channelid" option specifies the YouTube channel ID that uniquely
# identifies the channel for which the comment will be inserted.
argparser.add_argument("--channelid",
help="Required; ID for channel for which the comment will be inserted.")
# The "videoid" option specifies the YouTube video ID that uniquely
# identifies the video for which the comment will be inserted.
argparser.add_argument("--videoid",
help="Required; ID for video for which the comment will be inserted.")
# The "text" option specifies the text that will be used as comment.
argparser.add_argument("--text", help="Required; text that will be used as comment.")
args = argparser.parse_args()
if not args.channelid:
exit("Please specify channelid using the --channelid= parameter.")
if not args.videoid:
exit("Please specify videoid using the --videoid= parameter.")
if not args.text:
exit("Please specify text using the --text= parameter.")
youtube = get_authenticated_service(args)
try:
# All the available methods are used in sequence just for the sake of an example.
# Insert channel comment by omitting videoId
# REMOVED AS IT IS NOT RELEVANT TO THE ERRRO REPORT. insert_comment(youtube, args.channelid, None, args.text)
# Insert video comment
# REMOVED AS IT IS NOT RELEVANT TO THE ERRRO REPORT. insert_comment(youtube, args.channelid, args.videoid, args.text)
video_comments = get_comments(youtube, args.videoid, None)
if video_comments:
update_comment(youtube, video_comments[0])
except HttpError, e:
print "An HTTP error %d occurred:\n%s" % (e.resp.status, e.content)
else:
print "Inserted, listed and updated top-level comments."
Of course, you will need your authorization files inside the same directory.