Develop Reference

Python - Boto3 STS Token refreshing too early using RefreshableCredentials

I have an S3 upload tool that uses a combination of a bearer token provided through OIDC authentication for our service, as well as an STS token generated using that bearer token to perform uploads to S3 store.
To prevent large uploads from failing mid-upload due to token expiration, I implemented auto refreshable STS tokens using botocore's RefreshableCredentials class. The expectation is that once the STS token expires, it will attempt to refresh using the bearer token. If the bearer token is still live, it will refresh the STS token silently. Otherwise it will need to refresh the bearer token first using the longer term refresh token that is also received during the initial OIDC authentication.
My problem is that in all my testing, the times that the tokens get auto refreshed are not matching the expiration times that I am assigning the tokens. The minimum STS token life time is 15 minutes, so that's what I gave it, however when I assign the bearer token a life time of 1 minute, the STS token and bearer token will refresh after around 5 minutes. The frustrating part is that increasing the bearer token's life time is for some reason affecting when the STS token refreshes.
This is the stackoverflow post I used for the auto refresh STS token, and here is my implementation of it:
Class Session:
\```
(initial OIDC authentication to get bearer token and refresh token)
\```
self.s3 = (
RefreshableBotoSession(
Session=self, sts_arn=f"arn:aws:iam::{aws_account_id}:role/{s3_role}"
)
.refreshable_session()
.resource("s3")
)
def refresh_bearer_token(
self,
lookup_service_allowed_origin: str = "placeholder",
lookup_service_domain: str = "placeholder",
lookup_service_route: str = "placeholder",
lookup_service_auth_provider: str = "placeholder",
):
# generate user info
connection = http.client.HTTPSConnection(lookup_service_domain)
headers = {
"Content-type": "application/json",
"Origin": lookup_service_allowed_origin,
}
body = json.dumps(
{
"auth_provider": lookup_service_auth_provider,
"refresh_token": self.refresh_token,
"client_id": self.auth_client_id,
}
)
connection.request("PATCH", lookup_service_route, body, headers)
response = connection.getresponse().read().decode()
userdata = json.loads(response)
log.info("User successfully reauthenticated.")
self.bearer_token = userdata["access_token"]
self.user = userdata["username"]
self.refresh_token = userdata["refresh_token"]
self.jwt = _decode_bearer_token(self.bearer_token)
class RefreshableBotoSession:
"""
Boto Helper class which lets us create refreshable session, so that we can cache the client or resource.
Usage
-----
session = RefreshableBotoSession().refreshable_session()
client = session.client("s3") # we now can cache this client object without worrying about expiring credentials
"""
def __init__(
self,
Session,
sts_arn: str = None,
session_ttl: int = 900, #12 * 60 * 60,
):
"""
Initialize `RefreshableBotoSession`
Parameters
----------
sts_arn : str
The role arn to sts before creating session.
session_ttl : int (optional)
An integer number to set the TTL for each session. Beyond this session, it will renew the token.
50 minutes by default which is before the default role expiration of 1 hour
"""
self.Session = Session
self.sts_arn = sts_arn
self.session_ttl = session_ttl
def __get_session_credentials(self):
"""
Get session credentials
"""
sts_client = boto3.client(service_name="sts")
try:
sts_response = sts_client.assume_role_with_web_identity(
RoleArn=self.sts_arn,
RoleSessionName=self.Session.user,
WebIdentityToken=self.Session.bearer_token,
DurationSeconds=self.session_ttl,
).get("Credentials")
except botocore.exceptions.ClientError as error:
log.debug(error.response["Error"]["Code"])
if error.response["Error"]["Code"] == "ExpiredTokenException":
log.info("Bearer token has expired... Reauthenticating now")
self.Session.refresh_bearer_token()
sts_response = sts_client.assume_role_with_web_identity(
RoleArn=self.sts_arn,
RoleSessionName=self.Session.user,
WebIdentityToken=self.Session.bearer_token,
DurationSeconds=self.session_ttl,
).get("Credentials")
credentials = {
"access_key": sts_response.get("AccessKeyId"),
"secret_key": sts_response.get("SecretAccessKey"),
"token": sts_response.get("SessionToken"),
"expiry_time": sts_response.get("Expiration").isoformat(),
}
return credentials
def refreshable_session(self) -> boto3.Session:
"""
Get refreshable boto3 session.
"""
# get refreshable credentials
refreshable_credentials = RefreshableCredentials.create_from_metadata(
metadata=self.__get_session_credentials(),
refresh_using=self.__get_session_credentials,
method="sts-assume-role-with-web-identity",
)
# attach refreshable credentials current session
session = get_session()
session._credentials = refreshable_credentials
autorefresh_session = boto3.Session(botocore_session=session)
return autorefresh_session
I couldn't find any docs on the RefreshableToken class, so it's very hard to troubleshoot this.

Failed Validation while setting up LinkedIn Webhook

#app.route('/webhooks/linkedin', methods=['GET'])
def webhook_challenge_linkedIn():
# creates HMAC SHA-256 hash from incoming token and your consumer secret
sha256_hash_digest = hmac.new(bytes({api_secret},'utf-8'), msg=bytes(request.args.get('challengeCode'),'utf-8'), digestmod=hashlib.sha256)
# construct response data with base64 encoded hash
print((sha256_hash_digest))
val = {
"challengeCode" : request.args.get('challengeCode'),
"challengeResponse" : sha256_hash_digest.hexdigest()
}
return json.dumps(val)
When I try to set up the server and send an authentication request from LinkedIn for webhook to this endpoint it says Failed Validation
all i need to do for the verification is return the challenge code in
challengeResponse = Hex-encoded(HMACSHA256(challengeCode, clientSecret))
I think this is what I did but the still validation gets failed.
I don't seem to see the issue.

Try the following:
#app.route('/webhooks/linkedin', methods=['GET'])
def webhook_challenge_linkedIn():
challenge_code = request.args.get('challengeCode')
# creates HMAC SHA-256 hash from incoming token and your consumer secret
sha256_hash_digest = hmac.new(api_secret.encode(), challenge_code.encode(), hashlib.sha256).hexdigest()
# construct response data with base64 encoded hash
# print(sha256_hash_digest)
val = {
"challengeCode" : challenge_code,
"challengeResponse" : sha256_hash_digest
}
return json.dumps(val)
See this answer from How to use SHA256-HMAC in python code?

Need some help on how to implement an restfull api app based on golang

My coding skills are a bit low :)
Recently i started learning golang and how to handle an Api communication app. Have been having a great time learning it by myself, golang is revealing itself as a challenging language with great rewards in the end (code sense ^^).
Have been trying to create a cryptsy api lib for golang based on their API V2 (BETA) which is a restfull api. They have a python lib on their api website https://github.com/ScriptProdigy/CryptsyPythonV2/blob/master/Cryptsy.py.
So far have been able to get the public access working but am having a really hard time at the private access because of the authentication part.. I find that the info they give on their website on how to implement it is a bit confusing :(
Authorization is performed by sending the following variables into the request header Key
Public API key.
All query data (nonce=blahblah&limit=blahblah) signed by a secret key according to HMAC-SHA512 method. Your secret key and public keys can be generated from your account settings page. Every request requires a unique nonce. (Suggested to use unix timestamp with microseconds)
For this authentication part the python code goes as:
def _query(self, method, id=None, action=None, query=[], get_method="GET"):
query.append(('nonce', time.time()))
queryStr = urllib.urlencode(query)
link = 'https://' + self.domain + route
sign = hmac.new(self.PrivateKey.encode('utf-8'), queryStr, hashlib.sha512).hexdigest()
headers = {'Sign': sign, 'Key': self.PublicKey.encode('utf-8')}
Got this far in golang:
package main
import(
"crypto/hmac"
"crypto/sha512"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"net/http"
"strings"
"time"
)
const (
API_BASE_CRY = "https://api.cryptsy.com/api/"
API_VERSION_CRY = "v2"
API_KEY_CRY = "xxxxx"
API_SECRET_CRY = "xxxxxxxxxxxx"
DEFAULT_HTTPCLIENT_TIMEOUT = 30 // HTTP client timeout
)
type clientCry struct {
apiKey string
apiSecret string
httpClient *http.Client
}
type Cryptsy struct {
clientCry *clientCry
}
type CryptsyApiRsp struct {
Success bool `json:"success"`
Data json.RawMessage `json:"data"`
}
func NewCry(apiKey, apiSecret string) *Cryptsy {
clientCry := NewClientCry(apiKey, apiSecret)
return &Cryptsy{clientCry}
}
func NewClientCry(apiKey, apiSecret string) (c *clientCry) {
return &clientCry{apiKey, apiSecret, &http.Client{}}
}
func ComputeHmac512Hex(secret, payload string) string {
h := hmac.New(sha512.New, []byte(secret))
h.Write([]byte(payload))
return hex.EncodeToString(h.Sum(nil))
}
func (c *clientCry) doTimeoutRequestCry(timer *time.Timer, req *http.Request) (*http.Response, error) {
type data struct {
resp *http.Response
err error
}
done := make(chan data, 1)
go func() {
resp, err := c.httpClient.Do(req)
done <- data{resp, err}
}()
select {
case r := <-done:
return r.resp, r.err
case <-timer.C:
return nil, errors.New("timeout on reading data from Bittrex API")
}
}
func (c *clientCry) doCry(method string, ressource string, payload string, authNeeded bool) (response []byte, err error) {
connectTimer := time.NewTimer(DEFAULT_HTTPCLIENT_TIMEOUT * time.Second)
var rawurl string
nonce := time.Now().UnixNano()
result := fmt.Sprintf("nonce=%d", nonce)
rawurl = fmt.Sprintf("%s%s/%s?%s", API_BASE_CRY ,API_VERSION_CRY , ressource, result )
req, err := http.NewRequest(method, rawurl, strings.NewReader(payload))
sig := ComputeHmac512Hex(API_SECRET_CRY, result)
req.Header.Add("Sign", sig)
req.Header.Add("Key", API_KEY_CRY )
resp, err := c.doTimeoutRequestCry(connectTimer, req)
defer resp.Body.Close()
response, err = ioutil.ReadAll(resp.Body)
fmt.Println(fmt.Sprintf("reponse %s", response), err)
return response, err
}
func main() {
crypsy := NewCry(API_KEY_CRY, API_SECRET_CRY)
r, _ := crypsy.clientCry.doCry("GET", "info", "", true)
fmt.Println(r)
}
and my output is :
response {"success":false,"error":["Must be authenticated"]} <nil>
not getting why :( im passing the public key and the signature in the header, the signature.. i think im doing it right in the hmac-sha512.
I'm quering the user info url https://www.cryptsy.com/pages/apiv2/user, which as stated in the api site doesn't have any extra query variables so the nonce is the only one needed..
Have googled about restfull api's but haven't been able to find any answer :( starting to not let me sleep at night since i think that what im doing is kinda right.. really cant spot the error..
Anyone out there that could try and help me with this?
Thxs a lot :)

I see the issue with result := fmt.Sprintf("%d", nonce). The code that corresponds to the Python code should be something like
result := fmt.Sprintf("nonce=%d", nonce)
Could you please check it with this fix?
I also can observe a major difference in how the request is sending. The Python version is (link):
ret = requests.get(link,
params=query,
headers=headers,
verify=False)
but your code is does not send params with added nonce, etc. I think it should be something like
rawurl = fmt.Sprintf("%s%s/%s?%s", API_BASE_CRY ,API_VERSION_CRY , ressource, queryStr)
where queryStr should contain nonce, etc.

How to limit Autobahn python subscriptions on a per session basis

I am using autobahnpython with twisted (wamp) on server side and autobahnjs in browser. Is there a straight-forward way to allow/restrict subscriptions on a per session basis? For example, a client should not be able to subscribe to topics relavant to other users.
While I am NOT using crossbar.io, I tried using the Python code shown in the 'Example' section at the end of this page http://crossbar.io/docs/Authorization/ where a RPC call is first used to give authorization to a client. Of course, I am using my own authorization logic. Once this authorization is successful, I'd like to give the client privileges to subscribe to topics related only to this client, like 'com.example.user_id'. My issue is that even if auth passes, however, I have not found a way to limit subscription requests in the ApplicationSession class which is where the authorization takes place. How can I prevent a client who authorizes with user_id=user_a from subscribing to 'com.example.user_b'?

You can authorize by creating your own router. To do that, subclass Router() and override (at a minumum) the authorize() method:
def authorize(self, session, uri, action):
return True
This method is pretty simple, if you return a True then the session is authorized to do whatever it is attempting. You could make a rule that all subscriptions must start with 'com.example.USER_ID', so, your python code would split the uri, take the third field, and compare it to the current session id, returning True if they match, false otherwise. This is where things get a little weird though. I have code that does a similar thing, here is my authorize() method:
#inlineCallbacks
def authorize(self, session, uri, action):
authid = session._authid
if authid is None:
authid = 1
log.msg("AuthorizeRouter.authorize: {} {} {} {} {}".format(authid,
session._session_id, uri, IRouter.ACTION_TO_STRING[action], action))
if authid != 1:
rv = yield self.check_permission(authid, uri, IRouter.ACTION_TO_STRING[action])
else:
rv = yield True
log.msg("AuthorizeRouter.authorize: rv is {}".format(rv))
if not uri.startswith(self.svar['topic_base']):
self.sessiondb.activity(session._session_id, uri, IRouter.ACTION_TO_STRING[action], rv)
returnValue(rv)
return
Note that I dive into the session to get the _authid, which is bad karma (I think) because I should not be looking at these private variables. I don't know where else to get it, though.
Also, of note, this goes hand in hand with Authentication. In my implementation, the _authid is the authenticated user id, which is similar to a unix user id (positive unique integer). I am pretty sure this can be anything, like a string, so you should be ok with your 'user_b' as the _auth_id if you wish.
-g

I found a relatively simple solution using a Node guest. Here's the code:
// crossbar setup
var autobahn = require('autobahn');
var connection = new autobahn.Connection({
url: 'ws://127.0.0.1:8080/ws',
realm: 'realm1'
}
);
// Websocket to Scratch setup
// pull in the required node packages and assign variables for the entities
var WebSocketServer = require('websocket').server;
var http = require('http');
var ipPort = 1234; // ip port number for Scratch to use
// this connection is a crossbar connection
connection.onopen = function (session) {
// create an http server that will be used to contain a WebSocket server
var server = http.createServer(function (request, response) {
// We are not processing any HTTP, so this is an empty function. 'server' is a wrapper for the
// WebSocketServer we are going to create below.
});
// Create an IP listener using the http server
server.listen(ipPort, function () {
console.log('Webserver created and listening on port ' + ipPort);
});
// create the WebSocket Server and associate it with the httpServer
var wsServer = new WebSocketServer({
httpServer: server
});
// WebSocket server has been activated and a 'request' message has been received from client websocket
wsServer.on('request', function (request) {
// accept a connection request from Xi4S
//myconnection is the WS connection to Scratch
myconnection = request.accept(null, request.origin); // The server is now 'online'
// Process Xi4S messages
myconnection.on('message', function (message) {
console.log('message received: ' + message.utf8Data);
session.publish('com.serial.data', [message.utf8Data]);
// Process each message type received
myconnection.on('close', function (myconnection) {
console.log('Client closed connection');
boardReset();
});
});
});
};
connection.open();

Google+ login - Server side flow - Python - Google App Engine

I am building an app on Google App Engine using Flask. I am implementing Google+ login from the server-side flow described in https://developers.google.com/+/web/signin/server-side-flow. Before switching to App Engine, I had a very similar flow working. Perhaps I have introduced an error since then. Or maybe it is an issue with my implementation in App Engine.
I believe the url redirected to by the Google login flow should have a GET argument set "gplus_id", however, I am not receiving this parameter.
I have a login button created by:
(function() {
var po = document.createElement('script');
po.type = 'text/javascript'; po.async = true;
po.src = 'https://plus.google.com/js/client:plusone.js?onload=render';
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(po, s);
})();
function render() {
gapi.signin.render('gplusBtn', {
'callback': 'onSignInCallback',
'clientid': '{{ CLIENT_ID }}',
'cookiepolicy': 'single_host_origin',
'requestvisibleactions': 'http://schemas.google.com/AddActivity',
'scope': 'https://www.googleapis.com/auth/plus.login',
'accesstype': 'offline',
'width': 'iconOnly'
});
}
In the javascript code for the page I have a function to initiate the flow:
var helper = (function() {
var authResult = undefined;
return {
onSignInCallback: function(authResult) {
if (authResult['access_token']) {
// The user is signed in
this.authResult = authResult;
helper.connectServer();
} else if (authResult['error']) {
// There was an error, which means the user is not signed in.
// As an example, you can troubleshoot by writing to the console:
console.log('GPlus: There was an error: ' + authResult['error']);
}
console.log('authResult', authResult);
},
connectServer: function() {
$.ajax({
type: 'POST',
url: window.location.protocol + '//' + window.location.host + '/connect?state={{ STATE }}',
contentType: 'application/octet-stream; charset=utf-8',
success: function(result) {
// After we load the Google+ API, send login data.
gapi.client.load('plus','v1',helper.otherLogin);
},
processData: false,
data: this.authResult.code,
error: function(e) {
console.log("connectServer: error: ", e);
}
});
}
}
})();
/**
* Calls the helper method that handles the authentication flow.
*
* #param {Object} authResult An Object which contains the access token and
* other authentication information.
*/
function onSignInCallback(authResult) {
helper.onSignInCallback(authResult);
}
This initiates the flow at "/connect" (See step 8. referenced in the above doc):
#app.route('/connect', methods=['GET', 'POST'])
def connect():
# Ensure that this is no request forgery going on, and that the user
# sending us this connect request is the user that was supposed to.
if request.args.get('state', '') != session.get('state', ''):
response = make_response(json.dumps('Invalid state parameter.'), 401)
response.headers['Content-Type'] = 'application/json'
return response
# Normally the state would be a one-time use token, however in our
# simple case, we want a user to be able to connect and disconnect
# without reloading the page. Thus, for demonstration, we don't
# implement this best practice.
session.pop('state')
gplus_id = request.args.get('gplus_id')
code = request.data
try:
# Upgrade the authorization code into a credentials object
oauth_flow = client.flow_from_clientsecrets('client_secrets.json', scope='')
oauth_flow.redirect_uri = 'postmessage'
credentials = oauth_flow.step2_exchange(code)
except client.FlowExchangeError:
app.logger.debug("connect: Failed to upgrade the authorization code")
response = make_response(
json.dumps('Failed to upgrade the authorization code.'), 401)
response.headers['Content-Type'] = 'application/json'
return response
# Check that the access token is valid.
access_token = credentials.access_token
url = ('https://www.googleapis.com/oauth2/v1/tokeninfo?access_token=%s'
% access_token)
h = httplib2.Http()
result = json.loads(h.request(url, 'GET')[1])
# If there was an error in the access token info, abort.
if result.get('error') is not None:
response = make_response(json.dumps(result.get('error')), 500)
response.headers['Content-Type'] = 'application/json'
return response
# Verify that the access token is used for the intended user.
if result['user_id'] != gplus_id:
response = make_response(
json.dumps("Token's user ID doesn't match given user ID."), 401)
response.headers['Content-Type'] = 'application/json'
return response
...
However, the flow stops at if result['user_id'] != gplus_id:, saying "Token's user ID doesn't match given user ID.". result['user_id'] is a valid users ID, but gplus_id is None.
The line gplus_id = request.args.get('gplus_id') is expecting the GET args to contain 'gplus_id', but they only contain 'state'. Is this a problem with my javascript connectServer function? Should I include 'gplus_id' there? Surely I don't know it at that point. Or something else?

Similar to this question, I believe this is an issue with incomplete / not up to date / inconsistent documentation.
Where https://developers.google.com/+/web/signin/server-side-flow suggests that gplus_id will be returned in the GET arguments, this is not the case for the flow I was using.
I found my answer in https://github.com/googleplus/gplus-quickstart-python/blob/master/signin.py, which includes this snippet:
# An ID Token is a cryptographically-signed JSON object encoded in base 64.
# Normally, it is critical that you validate an ID Token before you use it,
# but since you are communicating directly with Google over an
# intermediary-free HTTPS channel and using your Client Secret to
# authenticate yourself to Google, you can be confident that the token you
# receive really comes from Google and is valid. If your server passes the
# ID Token to other components of your app, it is extremely important that
# the other components validate the token before using it.
gplus_id = credentials.id_token['sub']