I have an S3 upload tool that uses a combination of a bearer token provided through OIDC authentication for our service, as well as an STS token generated using that bearer token to perform uploads to S3 store.
To prevent large uploads from failing mid-upload due to token expiration, I implemented auto refreshable STS tokens using botocore's RefreshableCredentials class. The expectation is that once the STS token expires, it will attempt to refresh using the bearer token. If the bearer token is still live, it will refresh the STS token silently. Otherwise it will need to refresh the bearer token first using the longer term refresh token that is also received during the initial OIDC authentication.
My problem is that in all my testing, the times that the tokens get auto refreshed are not matching the expiration times that I am assigning the tokens. The minimum STS token life time is 15 minutes, so that's what I gave it, however when I assign the bearer token a life time of 1 minute, the STS token and bearer token will refresh after around 5 minutes. The frustrating part is that increasing the bearer token's life time is for some reason affecting when the STS token refreshes.
This is the stackoverflow post I used for the auto refresh STS token, and here is my implementation of it:
Class Session:
\```
(initial OIDC authentication to get bearer token and refresh token)
\```
self.s3 = (
RefreshableBotoSession(
Session=self, sts_arn=f"arn:aws:iam::{aws_account_id}:role/{s3_role}"
)
.refreshable_session()
.resource("s3")
)
def refresh_bearer_token(
self,
lookup_service_allowed_origin: str = "placeholder",
lookup_service_domain: str = "placeholder",
lookup_service_route: str = "placeholder",
lookup_service_auth_provider: str = "placeholder",
):
# generate user info
connection = http.client.HTTPSConnection(lookup_service_domain)
headers = {
"Content-type": "application/json",
"Origin": lookup_service_allowed_origin,
}
body = json.dumps(
{
"auth_provider": lookup_service_auth_provider,
"refresh_token": self.refresh_token,
"client_id": self.auth_client_id,
}
)
connection.request("PATCH", lookup_service_route, body, headers)
response = connection.getresponse().read().decode()
userdata = json.loads(response)
log.info("User successfully reauthenticated.")
self.bearer_token = userdata["access_token"]
self.user = userdata["username"]
self.refresh_token = userdata["refresh_token"]
self.jwt = _decode_bearer_token(self.bearer_token)
class RefreshableBotoSession:
"""
Boto Helper class which lets us create refreshable session, so that we can cache the client or resource.
Usage
-----
session = RefreshableBotoSession().refreshable_session()
client = session.client("s3") # we now can cache this client object without worrying about expiring credentials
"""
def __init__(
self,
Session,
sts_arn: str = None,
session_ttl: int = 900, #12 * 60 * 60,
):
"""
Initialize `RefreshableBotoSession`
Parameters
----------
sts_arn : str
The role arn to sts before creating session.
session_ttl : int (optional)
An integer number to set the TTL for each session. Beyond this session, it will renew the token.
50 minutes by default which is before the default role expiration of 1 hour
"""
self.Session = Session
self.sts_arn = sts_arn
self.session_ttl = session_ttl
def __get_session_credentials(self):
"""
Get session credentials
"""
sts_client = boto3.client(service_name="sts")
try:
sts_response = sts_client.assume_role_with_web_identity(
RoleArn=self.sts_arn,
RoleSessionName=self.Session.user,
WebIdentityToken=self.Session.bearer_token,
DurationSeconds=self.session_ttl,
).get("Credentials")
except botocore.exceptions.ClientError as error:
log.debug(error.response["Error"]["Code"])
if error.response["Error"]["Code"] == "ExpiredTokenException":
log.info("Bearer token has expired... Reauthenticating now")
self.Session.refresh_bearer_token()
sts_response = sts_client.assume_role_with_web_identity(
RoleArn=self.sts_arn,
RoleSessionName=self.Session.user,
WebIdentityToken=self.Session.bearer_token,
DurationSeconds=self.session_ttl,
).get("Credentials")
credentials = {
"access_key": sts_response.get("AccessKeyId"),
"secret_key": sts_response.get("SecretAccessKey"),
"token": sts_response.get("SessionToken"),
"expiry_time": sts_response.get("Expiration").isoformat(),
}
return credentials
def refreshable_session(self) -> boto3.Session:
"""
Get refreshable boto3 session.
"""
# get refreshable credentials
refreshable_credentials = RefreshableCredentials.create_from_metadata(
metadata=self.__get_session_credentials(),
refresh_using=self.__get_session_credentials,
method="sts-assume-role-with-web-identity",
)
# attach refreshable credentials current session
session = get_session()
session._credentials = refreshable_credentials
autorefresh_session = boto3.Session(botocore_session=session)
return autorefresh_session
I couldn't find any docs on the RefreshableToken class, so it's very hard to troubleshoot this.
Related
i want to trigger the dag externally
I was unable to find the solution , i'm new to programming
You can trigger a DAG externally in a several ways :
Solution 1 :
trigger a DAG with gcloud cli and gcloud composer command :
gcloud composer environments run ENVIRONMENT_NAME \
--location LOCATION \
dags trigger -- DAG_ID
Replace :
ENVIRONMENT_NAME with the name of the environment.
LOCATION with the region where the environment is located.
DAG_ID with the name of the DAG.
Solution 2 :
trigger a DAG with a Cloud function
from google.auth.transport.requests import Request
from google.oauth2 import id_token
import requests
IAM_SCOPE = 'https://www.googleapis.com/auth/iam'
OAUTH_TOKEN_URI = 'https://www.googleapis.com/oauth2/v4/token'
# If you are using the stable API, set this value to False
# For more info about Airflow APIs see https://cloud.google.com/composer/docs/access-airflow-api
USE_EXPERIMENTAL_API = True
def trigger_dag(data, context=None):
"""Makes a POST request to the Composer DAG Trigger API
When called via Google Cloud Functions (GCF),
data and context are Background function parameters.
For more info, refer to
https://cloud.google.com/functions/docs/writing/background#functions_background_parameters-python
To call this function from a Python script, omit the ``context`` argument
and pass in a non-null value for the ``data`` argument.
This function is currently only compatible with Composer v1 environments.
"""
# Fill in with your Composer info here
# Navigate to your webserver's login page and get this from the URL
# Or use the script found at
# https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/composer/rest/get_client_id.py
client_id = 'YOUR-CLIENT-ID'
# This should be part of your webserver's URL:
# {tenant-project-id}.appspot.com
webserver_id = 'YOUR-TENANT-PROJECT'
# The name of the DAG you wish to trigger
dag_name = 'composer_sample_trigger_response_dag'
if USE_EXPERIMENTAL_API:
endpoint = f'api/experimental/dags/{dag_name}/dag_runs'
json_data = {'conf': data, 'replace_microseconds': 'false'}
else:
endpoint = f'api/v1/dags/{dag_name}/dagRuns'
json_data = {'conf': data}
webserver_url = (
'https://'
+ webserver_id
+ '.appspot.com/'
+ endpoint
)
# Make a POST request to IAP which then Triggers the DAG
make_iap_request(
webserver_url, client_id, method='POST', json=json_data)
# This code is copied from
# https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/iap/make_iap_request.py
# START COPIED IAP CODE
def make_iap_request(url, client_id, method='GET', **kwargs):
"""Makes a request to an application protected by Identity-Aware Proxy.
Args:
url: The Identity-Aware Proxy-protected URL to fetch.
client_id: The client ID used by Identity-Aware Proxy.
method: The request method to use
('GET', 'OPTIONS', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE')
**kwargs: Any of the parameters defined for the request function:
https://github.com/requests/requests/blob/master/requests/api.py
If no timeout is provided, it is set to 90 by default.
Returns:
The page body, or raises an exception if the page couldn't be retrieved.
"""
# Set the default timeout, if missing
if 'timeout' not in kwargs:
kwargs['timeout'] = 90
# Obtain an OpenID Connect (OIDC) token from metadata server or using service
# account.
google_open_id_connect_token = id_token.fetch_id_token(Request(), client_id)
# Fetch the Identity-Aware Proxy-protected URL, including an
# Authorization header containing "Bearer " followed by a
# Google-issued OpenID Connect token for the service account.
resp = requests.request(
method, url,
headers={'Authorization': 'Bearer {}'.format(
google_open_id_connect_token)}, **kwargs)
if resp.status_code == 403:
raise Exception('Service account does not have permission to '
'access the IAP-protected application.')
elif resp.status_code != 200:
raise Exception(
'Bad response from application: {!r} / {!r} / {!r}'.format(
resp.status_code, resp.headers, resp.text))
else:
return resp.text
# END COPIED IAP CODE
I'm trying to add authentication to a FastAPI application using AWS Cognito. I can get valid JSON responses from Cognito, including AccessToken and RefreshToken. Using the FastAPI Oauth2 examples I've seen has led me to create code like this:
#router.post("/token")
async def get_token(form_data: OAuth2PasswordRequestForm = Depends()):
# get token from cognito
response = await concurrency.run_in_threadpool(
client.initiate_auth,
ClientId=settings.aws_cognito_app_client_id,
AuthFlow="USER_PASSWORD_AUTH",
AuthParameters={
"USERNAME": form_data.username,
"PASSWORD": form_data.password,
},
)
return response["AuthenticationResult"]["AccessToken"]
#router.post("/things")
async def things(token: str = Depends(oauth2_scheme)):
return {"token": token}
This seems to work as the "/things" endpoint is only accessible if authorized through the OpendAPI authentication popup. However, two things:
The token value is "undefined" in the things() handler, why is that?
How do I get the RefreshToken to the user?
Any suggestions or ideas are welcome.
I've been trying to implement passwordless authentication using AWS Cognito & API Gateway & Lambda (Python)
I have followed these articles:
https://medium.com/digicred/password-less-authentication-in-cognito-cafa016d4db7
https://medium.com/#pjatocheseminario/passwordless-api-using-cognito-and-serverless-framework-7fa952191352
I have configured Cognito (to accept CUSTOM_AUTH), added the Lambdas, and created the API endpoints:
/sign-up
/initiate-auth (aka initiate login)
/respond-to-auth-challenge (aka (verify login)
When calling initiateAuth I receive the following response:
An error occurred (NotAuthorizedException) when calling the InitiateAuth operation: Incorrect username or password."
I'm using CUSTOM_AUTH which doesn't require password, and the user name is definitely correct because it actually initiates the authentication flow and I receive a code, however because boto3 doesn't respond with a session I can't continue the authentication.
This is how I call Cognito:
res = cognito.initiate_auth(
ClientId=client_id,
AuthFlow="CUSTOM_AUTH",
AuthParameters={
"USERNAME": email,
"PASSWORD": random_password
}
)
It's probably something small I'm missing but I can't figure out what.
Your client code looks OK, mine has ClientId param in it but if your code is not raising an exception then it should be fine. Unless you had Generate client secret option checked when you created your app client.
If that is the case then you have to pass in SECRET_HASH in AuthParameters like the following:
import hmac
import hashlib
import base64
def get_secret_hash(email, client_id, client_secret):
"""
A keyed-hash message authentication code (HMAC) calculated using
the secret key of a user pool client and username plus the client
ID in the message.
"""
message = email + client_id
client_secret = str.encode(client_secret)
dig = hmac.new(client_secret, msg=message.encode('UTF-8'), digestmod=hashlib.sha256).digest()
return base64.b64encode(dig).decode()
client.admin_initiate_auth(
UserPoolId=COGNITO_USER_POOL_ID,
ClientId=CLIENT_ID,
AuthFlow='CUSTOM_AUTH',
AuthParameters={
'USERNAME': email,
'SECRET_HASH': get_secret_hash(email, CLIENT_ID, CLIENT_SECRET) # Omit if secret key option is disabled.
},
)
Next, double check the following:
Under App clients > * > Auth Flows Configuration, is ALLOW_CUSTOM_AUTH option enabled for your client?
Under App integration > App client settings > * > Enabled Identity Providers, is your user pool selected?
If you have Cognito setup correctly and your code still doesn't work then it is probably the lambda code. You probably know this but for password-less custom auth you need to use 3 lambda triggers: Define Auth Challenge, Create Auth Challenge, and Verify Auth Challenge.
Custom auth lambdas events are triggered in the following order:
DefineAuthChallenge_Authentication:
Technically, issueTokens can be set to True here to return tokens without going through the rest of the steps.
def lambda_handler(event, context):
if event['triggerSource'] == 'DefineAuthChallenge_Authentication':
event['response']['challengeName'] = 'CUSTOM_CHALLENGE'
event['response']['issueTokens'] = False
event['response']['failAuthentication'] = False
if event['request']['session']: # Needed for step 4.
# If all of the challenges are answered, issue tokens.
event['response']['issueTokens'] = all(
answered_challenge['challengeResult'] for answered_challenge in event['request']['session'])
return event
CreateAuthChallenge_Authentication:
def lambda_handler(event, context):
if event['triggerSource'] == 'CreateAuthChallenge_Authentication':
if event['request']['challengeName'] == 'CUSTOM_CHALLENGE':
event['response']['privateChallengeParameters'] = {}
event['response']['privateChallengeParameters']['answer'] = 'YOUR CHALLENGE ANSWER HERE'
event['response']['challengeMetadata'] = 'AUTHENTICATE_AS_CHALLENGE'
return event
Then your client must respond to the challenge:
client.respond_to_auth_challenge(
ClientId=CLIENT_ID,
ChallengeName='CUSTOM_CHALLENGE',
Session=session,
ChallengeResponses={
'USERNAME': email,
'ANSWER': 'Extra Protection!',
'SECRET_HASH': get_secret_hash(email, CLIENT_ID, CLIENT_SECRET) # Omit if secret key option is disabled.
}
)
VerifyAuthChallengeResponse_Authentication:
def lambda_handler(event, context):
if event['triggerSource'] == 'VerifyAuthChallengeResponse_Authentication':
if event['request']['challengeAnswer'] == event['request']['privateChallengeParameters']['answer']:
event['response']['answerCorrect'] = True
return event
DefineAuthChallenge_Authentication:
Set event['response']['issueTokens'] to True to return tokens (code shown in step 1), or issue another challenge to keep repeating steps 1-3.
Finally, make sure that if case-insensitivity option is enabled for your user pool too. Also, I can't exactly recall if CUSTOM_AUTH flow worked if the user is in FORCE_CHANGE_PASSWORD status. If the user is in that state, then try settings a permanent password with the sdk to set the status to CONFIRMED.
I was facing the same error, and I think that the error message is misleading.
When you did not respond correctly in Create-Auth-Challenge lambda, you will get this error. So make sure everything is right in your lambda.
I want to create/calculate a SECRET_HASH for AWS Cognito using boto3 and python. This will be incorporated in to my fork of warrant.
I configured my cognito app client to use an app client secret. However, this broke the following code.
def renew_access_token(self):
"""
Sets a new access token on the User using the refresh token.
NOTE:
Does not work if "App client secret" is enabled. 'SECRET_HASH' is needed in AuthParameters.
'SECRET_HASH' requires HMAC calculations.
Does not work if "Device Tracking" is turned on.
https://stackoverflow.com/a/40875783/1783439
'DEVICE_KEY' is needed in AuthParameters. See AuthParameters section.
https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_InitiateAuth.html
"""
refresh_response = self.client.initiate_auth(
ClientId=self.client_id,
AuthFlow='REFRESH_TOKEN',
AuthParameters={
'REFRESH_TOKEN': self.refresh_token
# 'SECRET_HASH': How to generate this?
},
)
self._set_attributes(
refresh_response,
{
'access_token': refresh_response['AuthenticationResult']['AccessToken'],
'id_token': refresh_response['AuthenticationResult']['IdToken'],
'token_type': refresh_response['AuthenticationResult']['TokenType']
}
)
When I run this I receive the following exception:
botocore.errorfactory.NotAuthorizedException:
An error occurred (NotAuthorizedException) when calling the InitiateAuth operation:
Unable to verify secret hash for client <client id echoed here>.
This answer informed me that a SECRET_HASH is required to use the cognito client secret.
The aws API reference docs AuthParameters section states the following:
For REFRESH_TOKEN_AUTH/REFRESH_TOKEN: USERNAME (required), SECRET_HASH
(required if the app client is configured with a client secret),
REFRESH_TOKEN (required), DEVICE_KEY
The boto3 docs state that a SECRET_HASH is
A keyed-hash message authentication code (HMAC) calculated using the
secret key of a user pool client and username plus the client ID in
the message.
The docs explain what is needed, but not how to achieve this.
The below get_secret_hash method is a solution that I wrote in Python for a Cognito User Pool implementation, with example usage:
import boto3
import botocore
import hmac
import hashlib
import base64
class Cognito:
client_id = app.config.get('AWS_CLIENT_ID')
user_pool_id = app.config.get('AWS_USER_POOL_ID')
identity_pool_id = app.config.get('AWS_IDENTITY_POOL_ID')
client_secret = app.config.get('AWS_APP_CLIENT_SECRET')
# Public Keys used to verify tokens returned by Cognito:
# http://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-with-identity-providers.html#amazon-cognito-identity-user-pools-using-id-and-access-tokens-in-web-api
id_token_public_key = app.config.get('JWT_ID_TOKEN_PUB_KEY')
access_token_public_key = app.config.get('JWT_ACCESS_TOKEN_PUB_KEY')
def __get_client(self):
return boto3.client('cognito-idp')
def get_secret_hash(self, username):
# A keyed-hash message authentication code (HMAC) calculated using
# the secret key of a user pool client and username plus the client
# ID in the message.
message = username + self.client_id
dig = hmac.new(self.client_secret, msg=message.encode('UTF-8'),
digestmod=hashlib.sha256).digest()
return base64.b64encode(dig).decode()
# REQUIRES that `ADMIN_NO_SRP_AUTH` be enabled on Client App for User Pool
def login_user(self, username_or_alias, password):
try:
return self.__get_client().admin_initiate_auth(
UserPoolId=self.user_pool_id,
ClientId=self.client_id,
AuthFlow='ADMIN_NO_SRP_AUTH',
AuthParameters={
'USERNAME': username_or_alias,
'PASSWORD': password,
'SECRET_HASH': self.get_secret_hash(username_or_alias)
}
)
except botocore.exceptions.ClientError as e:
return e.response
I also got a TypeError when I tried the above solution. Here is the solution that worked for me:
import hmac
import hashlib
import base64
# Function used to calculate SecretHash value for a given client
def calculateSecretHash(client_id, client_secret, username):
key = bytes(client_secret, 'utf-8')
message = bytes(f'{username}{client_id}', 'utf-8')
return base64.b64encode(hmac.new(key, message, digestmod=hashlib.sha256).digest()).decode()
# Usage example
calculateSecretHash(client_id, client_secret, username)
I am building an app on Google App Engine using Flask. I am implementing Google+ login from the server-side flow described in https://developers.google.com/+/web/signin/server-side-flow. Before switching to App Engine, I had a very similar flow working. Perhaps I have introduced an error since then. Or maybe it is an issue with my implementation in App Engine.
I believe the url redirected to by the Google login flow should have a GET argument set "gplus_id", however, I am not receiving this parameter.
I have a login button created by:
(function() {
var po = document.createElement('script');
po.type = 'text/javascript'; po.async = true;
po.src = 'https://plus.google.com/js/client:plusone.js?onload=render';
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(po, s);
})();
function render() {
gapi.signin.render('gplusBtn', {
'callback': 'onSignInCallback',
'clientid': '{{ CLIENT_ID }}',
'cookiepolicy': 'single_host_origin',
'requestvisibleactions': 'http://schemas.google.com/AddActivity',
'scope': 'https://www.googleapis.com/auth/plus.login',
'accesstype': 'offline',
'width': 'iconOnly'
});
}
In the javascript code for the page I have a function to initiate the flow:
var helper = (function() {
var authResult = undefined;
return {
onSignInCallback: function(authResult) {
if (authResult['access_token']) {
// The user is signed in
this.authResult = authResult;
helper.connectServer();
} else if (authResult['error']) {
// There was an error, which means the user is not signed in.
// As an example, you can troubleshoot by writing to the console:
console.log('GPlus: There was an error: ' + authResult['error']);
}
console.log('authResult', authResult);
},
connectServer: function() {
$.ajax({
type: 'POST',
url: window.location.protocol + '//' + window.location.host + '/connect?state={{ STATE }}',
contentType: 'application/octet-stream; charset=utf-8',
success: function(result) {
// After we load the Google+ API, send login data.
gapi.client.load('plus','v1',helper.otherLogin);
},
processData: false,
data: this.authResult.code,
error: function(e) {
console.log("connectServer: error: ", e);
}
});
}
}
})();
/**
* Calls the helper method that handles the authentication flow.
*
* #param {Object} authResult An Object which contains the access token and
* other authentication information.
*/
function onSignInCallback(authResult) {
helper.onSignInCallback(authResult);
}
This initiates the flow at "/connect" (See step 8. referenced in the above doc):
#app.route('/connect', methods=['GET', 'POST'])
def connect():
# Ensure that this is no request forgery going on, and that the user
# sending us this connect request is the user that was supposed to.
if request.args.get('state', '') != session.get('state', ''):
response = make_response(json.dumps('Invalid state parameter.'), 401)
response.headers['Content-Type'] = 'application/json'
return response
# Normally the state would be a one-time use token, however in our
# simple case, we want a user to be able to connect and disconnect
# without reloading the page. Thus, for demonstration, we don't
# implement this best practice.
session.pop('state')
gplus_id = request.args.get('gplus_id')
code = request.data
try:
# Upgrade the authorization code into a credentials object
oauth_flow = client.flow_from_clientsecrets('client_secrets.json', scope='')
oauth_flow.redirect_uri = 'postmessage'
credentials = oauth_flow.step2_exchange(code)
except client.FlowExchangeError:
app.logger.debug("connect: Failed to upgrade the authorization code")
response = make_response(
json.dumps('Failed to upgrade the authorization code.'), 401)
response.headers['Content-Type'] = 'application/json'
return response
# Check that the access token is valid.
access_token = credentials.access_token
url = ('https://www.googleapis.com/oauth2/v1/tokeninfo?access_token=%s'
% access_token)
h = httplib2.Http()
result = json.loads(h.request(url, 'GET')[1])
# If there was an error in the access token info, abort.
if result.get('error') is not None:
response = make_response(json.dumps(result.get('error')), 500)
response.headers['Content-Type'] = 'application/json'
return response
# Verify that the access token is used for the intended user.
if result['user_id'] != gplus_id:
response = make_response(
json.dumps("Token's user ID doesn't match given user ID."), 401)
response.headers['Content-Type'] = 'application/json'
return response
...
However, the flow stops at if result['user_id'] != gplus_id:, saying "Token's user ID doesn't match given user ID.". result['user_id'] is a valid users ID, but gplus_id is None.
The line gplus_id = request.args.get('gplus_id') is expecting the GET args to contain 'gplus_id', but they only contain 'state'. Is this a problem with my javascript connectServer function? Should I include 'gplus_id' there? Surely I don't know it at that point. Or something else?
Similar to this question, I believe this is an issue with incomplete / not up to date / inconsistent documentation.
Where https://developers.google.com/+/web/signin/server-side-flow suggests that gplus_id will be returned in the GET arguments, this is not the case for the flow I was using.
I found my answer in https://github.com/googleplus/gplus-quickstart-python/blob/master/signin.py, which includes this snippet:
# An ID Token is a cryptographically-signed JSON object encoded in base 64.
# Normally, it is critical that you validate an ID Token before you use it,
# but since you are communicating directly with Google over an
# intermediary-free HTTPS channel and using your Client Secret to
# authenticate yourself to Google, you can be confident that the token you
# receive really comes from Google and is valid. If your server passes the
# ID Token to other components of your app, it is extremely important that
# the other components validate the token before using it.
gplus_id = credentials.id_token['sub']