Develop Reference

How do I Authenticate my FTX_Client in Python

I have looked through the FTX api documentation found here: https://docs.ftx.us/#overview
And I've looked at example code found in this repo: https://github.com/ftexchange/ftx/tree/master/rest
I can't 'get' or 'post' anything that requires the Authentication. I am using the api key on my account that has 'full trade permissions', and when I look at: print(request.headers) the headers look like they are in the right format.
I've tried: using google colab instead of vs code, updating all my libraries, generating a new api key, restarting kernel and computer. I can pull something like 'markets' because it doesn't need the Authentication.
Let me know if you need any more information, below is a portion of the code I have that isolates the problem and returns {'success': False, 'error': 'Not logged in'}
import time
import urllib.parse
from typing import Optional, Dict, Any, List
from requests import Request, Session, Response
import hmac
ep = 'https://ftx.us/api/wallet/balances'
ts = int(time.time() * 1000)
s = Session()
request = Request('GET', ep)
prepared = request.prepare()
signature_payload = f'{ts}{prepared.method}{prepared.path_url}'.encode()
if prepared.body:
signature_payload += prepared.body
signature = hmac.new(secret.encode(), signature_payload, 'sha256').hexdigest()
request.headers['FTX-KEY'] = key
request.headers['FTX-SIGN'] = signature
request.headers['FTX-TS'] = str(ts)
response = s.send(prepared)
data = response.json()
print(data)

I've faced with the same problem.
You need to change this part:
prepared.headers['FTX-KEY'] = key
prepared.headers['FTX-SIGN'] = signature
prepared.headers['FTX-TS'] = str(ts)
PS. I believe that the FTX needs to fix their API documentation
PSS. I've checked the a part of https://github.com/ftexchange/ftx/tree/master/rest code. I beleave FTX guys just do a copy-paste into docs this code but originally it belongs to more a sophisticated object oriented solution that will work correctly because they pass into method an already created request and use a prepared variable just to calculate path_url and method

For ftx.us, you need to use different headers:
prepared.headers['FTXUS-KEY'] = key
prepared.headers['FTXUS-TS'] = str(ts)
prepared.headers['FTXUS-SIGN'] = signature

How do I send credentials so I can unit test MSAL for Python?

The flow to this admin app is basically:
User goes to /admin.
#azure/msal-react checks if the user is logged in and if not redirects them to login.
When the accessToken, idToken, and oid have been received on the FE, they are sent to my API (/api) where they are validated.
If the tokens are validated, and verified as belonging to the oid that came from the FE, then a JWT is issued for the user indicating they are authenticated with the API as well.
The JWT is sent back to the FE where having both the MSAL isAuthenticated() and their API JWT isApiAuthenticated() indicates they are fully authenticated and it renders the FE components.
I'm trying to write unit tests for the what I have in the aad.py (bottom).
The issue I'm seeing is that I need to retrieve and send it a legit MSAL access_token, id_token, and oid. That means I actually need to log in a user against our AAD.
I do have a dummy user to use for this purpose, but I'm not seeing in the Python MSAL documentation how to sign the user in without interaction and just submit username and password.
Suggestions for how to write unit tests to test? Do I need to mockup something else, or should I be able to send a dummy user's username and password with Python MSAL?
# Python libraries
import json
# Third-party dependencies
import jwt
import requests
from base64 import b64decode, b64encode
from cryptography import x509
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import serialization
from config import settings
def ms_graph(access_token):
"""
Takes the access_token from the FE and uses it query MS Graph for user
information.
Returns an object.
Takes one argument:
access_token: sent from FE.
"""
graph_response = requests.get( # Use token to call downstream service
settings.GRAPH_URI,
headers={'Authorization': 'Bearer ' + access_token},)
return json.loads(graph_response.text)
def validate_token(token):
"""
Used to decode the token sent from the FE. The result is
a verified signature that contains information about the user and tokens.
Returns an object.
Takes one argument:
token: The idToken sent from the FE that needs to be validated.
"""
# Get a list of the possible public keys from the JWKS_URI endpoint
jwkeys = requests.get(settings.JWKS_URI).json()['keys']
# Extract the 'kid' from the unverified header to get the public key
token_key_id = jwt.get_unverified_header(token)['kid']
# Get the object corresponding to the 'kid' key
jwk = [key for key in jwkeys if key['kid'] == token_key_id][0]
# Encode the 'x5c' to eventually get the public key for decoding
der_cert = b64decode(jwk['x5c'][0])
cert = x509.load_der_x509_certificate(der_cert, default_backend())
public_key = cert.public_key()
pem_key = public_key.public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo
)
# Finally, validate and decode the token
token_claims = jwt.decode(
token,
pem_key,
algorithms=['RS256'],
audience=settings.CLIENT_ID
)
return token_claims
def validate_user(access_token, id_token, client_oid):
"""
Takes the id_token sent from the FE and validates the signature.
The validated signature also contains an oid claim which is used to verify
the oid sent from the FE.
Returns an object containing:
{
"user_validated": bool,
"graph_response": object
}
Takes two arguments:
access_token: access_token sent from the FE.
id_token: the id_token sent from the FE.
client_oid: the oid sent from the FE.
"""
# Decode the token to retrieve claims
id_token_payload = validate_token(id_token)
# Check if the claim oid and matches the oid sent from the FE
if id_token_payload['oid'] == client_oid:
graph_response = ms_graph(access_token)
res = dict()
res['user_validated'] = graph_response['id'] == client_oid
res['graph_response'] = graph_response
return res

If you really really want to obtain tokens for unit tests with username and password then look at acquire_token_by_username_password() and its example in MS docs.
But there are reasons to not let unit tests use the real MS Graph API:
dependency on network connection can make unit tests run slow or fail, even if your code still works fine,
storing credentials in repository is dangerous, keeping them properly is difficult,
the dummy account can be locked,
numerous test calls can cause throttling,
it is kind of unpolite to abuse the infrastructure with non-productive requests.
The suggestion for unit tests is usual: isolate test cases, then mock and patch any external dependencies not related to code being tested.
MSAL uses requests for calling the cloud, and there is a great package for mocking its work: meet responses. Prevent MSAL for sending out any requests and mock OpenID configuration call:
import responses
import time
import unittest
MS_OPENID_CONFIG = {
"authorization_endpoint": "https://login.microsoftonline.com/common/"
"oauth2/v2.0/authorize",
"token_endpoint": "https://login.microsoftonline.com/common/oauth2/"
"v2.0/token",
}
MS_TOKENS = {
'id_token_claims': {
'exp': time.time() + 60,
'oid': '00000000-0000-0000-0000-000000000000',
'upn': 'test_user_1_'
},
'id_token': {},
'access_token': {}
}
class BaseTest(unittest.TestCase):
def afterSetUp(self):
self.responses = responses.RequestsMock()
self.responses.start()
self.responses.add(
responses.GET,
'https://login.microsoftonline.com/common/v2.0/'
'.well-known/openid-configuration',
json=MS_OPENID_CONFIG
)
# add_calback() allows modifying mock token in runtime
self.responses.add_callback(
responses.POST,
'https://login.microsoftonline.com/common/oauth2/v2.0/token',
callback=lambda request: (200, {}, json.dumps(MS_TOKENS)),
content_type='application/json',
)
self.addCleanup(self.responses.stop)
self.addCleanup(self.responses.reset)
For testing validate_token() you can bundle pre-generated test RSA key pair:
openssl genrsa -out test_rsa_private.pem 512
openssl rsa -in test_rsa_private.pem -pubout -outform PEM -out test_rsa_public.pem
Then you can mock response to settings.JWKS_URI in the same way as OpenID config, returning test public key with known kid.
If you'd prefer to save time on signing test tokens, then just mock jwt signature verification to always return True:
from unittest import Mock, patch
def test_validate_token(self)
...
with patch('jwt.algorithms.RSAAlgorithm.verify', new=Mock(return_value=True)):
# jwt.decode(at, key=open('test_rsa_public.pem').read(), algorithms=['RS256'], options={'verify_exp': False, 'verify_aud': False})
validate_token(test_token)
And so on, unit test should only test the code of unit itself, everything around can be mocked as needed. Read the source of MSAL and jwt. Don't mock what looks like implementation details, they might change in future versions.

Generate TSIG keyring (as encoded byte string) for DNS Update

I am trying to use python DNS module (dnspython) to create (add) new DNS record.
Documentation specifies how to create update http://www.dnspython.org/examples.html :
import dns.tsigkeyring
import dns.update
import sys
keyring = dns.tsigkeyring.from_text({
'host-example.' : 'XXXXXXXXXXXXXXXXXXXXXX=='
})
update = dns.update.Update('dyn.test.example', keyring=keyring)
update.replace('host', 300, 'a', sys.argv[1])
But it does not precise, how to actually generate keyring string that can be passed to dns.tsigkeyring.from_text() method in the first place.
What is the correct way to generate the key? I am using krb5 at my organization.

Server is running on Microsoft AD DNS with GSS-TSIG.
TSIG and GSS-TSIG are different beasts – the former uses a static preshared key that can be simply copied from the server, but the latter uses Kerberos (GSSAPI) to negotiate a session key for every transaction.
At the time when this thread was originally posted, dnspython 1.x did not have any support for GSS-TSIG whatsoever.
(The handshake does not result in a static key that could be converted to a regular TSIG keyring; instead the GSSAPI library itself must be called to build an authenticator – dnspython 1.x could not do that, although dnspython 2.1 finally can.)
If you are trying to update an Active Directory DNS server, BIND's nsupdate command-line tool supports GSS-TSIG (and sometimes it even works). You should be able to run it through subprocess and simply feed the necessary updates via stdin.
cmds = [f'zone {dyn_zone}\n',
f'del {fqdn}\n',
f'add {fqdn} 60 TXT "{challenge}"\n',
f'send\n']
subprocess.run(["nsupdate", "-g"],
input="".join(cmds).encode(),
check=True)
As with most Kerberos client applications, nsupdate expects the credentials to be already present in the environment (that is, you need to have already obtained a TGT using kinit beforehand; or alternatively, if a recent version of MIT Krb5 is used, you can point $KRB5_CLIENT_KTNAME to the keytab containing the client credentials).
Update: dnspython 2.1 finally has the necessary pieces for GSS-TSIG, but creating the keyring is currently a very manual process – you have to call the GSSAPI library and process the TKEY negotiation yourself. The code for doing so is included at the bottom.
(The Python code below can be passed a custom gssapi.Credentials object, but otherwise it looks for credentials in the environment just like nsupdate does.)
import dns.rdtypes.ANY.TKEY
import dns.resolver
import dns.update
import gssapi
import socket
import time
import uuid
def _build_tkey_query(token, key_ring, key_name):
inception_time = int(time.time())
tkey = dns.rdtypes.ANY.TKEY.TKEY(dns.rdataclass.ANY,
dns.rdatatype.TKEY,
dns.tsig.GSS_TSIG,
inception_time,
inception_time,
3,
dns.rcode.NOERROR,
token,
b"")
query = dns.message.make_query(key_name,
dns.rdatatype.TKEY,
dns.rdataclass.ANY)
query.keyring = key_ring
query.find_rrset(dns.message.ADDITIONAL,
key_name,
dns.rdataclass.ANY,
dns.rdatatype.TKEY,
create=True).add(tkey)
return query
def _probe_server(server_name, zone):
gai = socket.getaddrinfo(str(server_name),
"domain",
socket.AF_UNSPEC,
socket.SOCK_DGRAM)
for af, sf, pt, cname, sa in gai:
query = dns.message.make_query(zone, "SOA")
res = dns.query.udp(query, sa[0], timeout=2)
return sa[0]
def gss_tsig_negotiate(server_name, server_addr, creds=None):
# Acquire GSSAPI credentials
gss_name = gssapi.Name(f"DNS#{server_name}",
gssapi.NameType.hostbased_service)
gss_ctx = gssapi.SecurityContext(name=gss_name,
creds=creds,
usage="initiate")
# Name generation tips: https://tools.ietf.org/html/rfc2930#section-2.1
key_name = dns.name.from_text(f"{uuid.uuid4()}.{server_name}")
tsig_key = dns.tsig.Key(key_name, gss_ctx, dns.tsig.GSS_TSIG)
key_ring = {key_name: tsig_key}
key_ring = dns.tsig.GSSTSigAdapter(key_ring)
token = gss_ctx.step()
while not gss_ctx.complete:
tkey_query = _build_tkey_query(token, key_ring, key_name)
response = dns.query.tcp(tkey_query, server_addr, timeout=5)
if not gss_ctx.complete:
# Original comment:
# https://github.com/rthalley/dnspython/pull/530#issuecomment-658959755
# "this if statement is a bit redundant, but if the final token comes
# back with TSIG attached the patch to message.py will automatically step
# the security context. We dont want to excessively step the context."
token = gss_ctx.step(response.answer[0][0].key)
return key_name, key_ring
def gss_tsig_update(zone, update_msg, creds=None):
# Find the SOA of our zone
answer = dns.resolver.resolve(zone, "SOA")
soa_server = answer.rrset[0].mname
server_addr = _probe_server(soa_server, zone)
# Get the GSS-TSIG key
key_name, key_ring = gss_tsig_negotiate(soa_server, server_addr, creds)
# Dispatch the update
update_msg.use_tsig(keyring=key_ring,
keyname=key_name,
algorithm=dns.tsig.GSS_TSIG)
response = dns.query.tcp(update_msg, server_addr)
return response

implementing USER_SRP_AUTH with python boto3 for AWS Cognito

Amazon provides iOS, Android, and Javascript Cognito SDKs that offer a high-level authenticate-user operation.
For example, see Use Case 4 here:
https://github.com/aws/amazon-cognito-identity-js
However, if you are using python/boto3, all you get are a pair of primitives: cognito.initiate_auth and cognito.respond_to_auth_challenge.
I am trying to use these primitives along with the pysrp lib authenticate with the USER_SRP_AUTH flow, but what I have is not working.
It always fails with "An error occurred (NotAuthorizedException) when calling the RespondToAuthChallenge operation: Incorrect username or password." (The username/password pair work find with the JS SDK.)
My suspicion is I'm constructing the challenge response wrong (step 3), and/or passing Congito hex strings when it wants base64 or vice versa.
Has anyone gotten this working? Anyone see what I'm doing wrong?
I am trying to copy the behavior of the authenticateUser call found in the Javascript SDK:
https://github.com/aws/amazon-cognito-identity-js/blob/master/src/CognitoUser.js#L138
but I'm doing something wrong and can't figure out what.
#!/usr/bin/env python
import base64
import binascii
import boto3
import datetime as dt
import hashlib
import hmac
# http://pythonhosted.org/srp/
# https://github.com/cocagne/pysrp
import srp
bytes_to_hex = lambda x: "".join("{:02x}".format(ord(c)) for c in x)
cognito = boto3.client('cognito-idp', region_name="us-east-1")
username = "foobar#foobar.com"
password = "123456"
user_pool_id = u"us-east-1_XXXXXXXXX"
client_id = u"XXXXXXXXXXXXXXXXXXXXXXXXXX"
# Step 1:
# Use SRP lib to construct a SRP_A value.
srp_user = srp.User(username, password)
_, srp_a_bytes = srp_user.start_authentication()
srp_a_hex = bytes_to_hex(srp_a_bytes)
# Step 2:
# Submit USERNAME & SRP_A to Cognito, get challenge.
response = cognito.initiate_auth(
AuthFlow='USER_SRP_AUTH',
AuthParameters={ 'USERNAME': username, 'SRP_A': srp_a_hex },
ClientId=client_id,
ClientMetadata={ 'UserPoolId': user_pool_id })
# Step 3:
# Use challenge parameters from Cognito to construct
# challenge response.
salt_hex = response['ChallengeParameters']['SALT']
srp_b_hex = response['ChallengeParameters']['SRP_B']
secret_block_b64 = response['ChallengeParameters']['SECRET_BLOCK']
secret_block_bytes = base64.standard_b64decode(secret_block_b64)
secret_block_hex = bytes_to_hex(secret_block_bytes)
salt_bytes = binascii.unhexlify(salt_hex)
srp_b_bytes = binascii.unhexlify(srp_b_hex)
process_challenge_bytes = srp_user.process_challenge(salt_bytes,
srp_b_bytes)
timestamp = unicode(dt.datetime.utcnow().strftime("%a %b %d %H:%m:%S +0000 %Y"))
hmac_obj = hmac.new(process_challenge_bytes, digestmod=hashlib.sha256)
hmac_obj.update(user_pool_id.split('_')[1].encode('utf-8'))
hmac_obj.update(username.encode('utf-8'))
hmac_obj.update(secret_block_bytes)
hmac_obj.update(timestamp.encode('utf-8'))
challenge_responses = {
"TIMESTAMP": timestamp.encode('utf-8'),
"USERNAME": username.encode('utf-8'),
"PASSWORD_CLAIM_SECRET_BLOCK": secret_block_hex,
"PASSWORD_CLAIM_SIGNATURE": hmac_obj.hexdigest()
}
# Step 4:
# Submit challenge response to Cognito.
response = cognito.respond_to_auth_challenge(
ClientId=client_id,
ChallengeName='PASSWORD_VERIFIER',
ChallengeResponses=challenge_responses)

There are many errors in your implementation. For example:
pysrp uses SHA1 algorithm by default. It should be set to SHA256.
_ng_const length should be 3072 bits and it should be copied from amazon-cognito-identity-js
There is no hkdf function in pysrp.
The response should contain secret_block_b64, not secret_block_hex.
Wrong timestamp format. %H:%m:%S means "hour:month:second" and +0000 should be replaced by UTC.
Has anyone gotten this working?
Yes. It's implemented in the warrant.aws_srp module.
https://github.com/capless/warrant/blob/master/warrant/aws_srp.py
from warrant.aws_srp import AWSSRP
USERNAME='xxx'
PASSWORD='yyy'
POOL_ID='us-east-1_zzzzz'
CLIENT_ID = '12xxxxxxxxxxxxxxxxxxxxxxx'
aws = AWSSRP(username=USERNAME, password=PASSWORD, pool_id=POOL_ID,
client_id=CLIENT_ID)
tokens = aws.authenticate_user()
id_token = tokens['AuthenticationResult']['IdToken']
refresh_token = tokens['AuthenticationResult']['RefreshToken']
access_token = tokens['AuthenticationResult']['AccessToken']
token_type = tokens['AuthenticationResult']['TokenType']
authenticate_user method supports only PASSWORD_VERIFIER challenge. If you want to respond to other challenges, just look into the authenticate_user and boto3 documentation.

Unfortunately it's a hard problem since you don't get any hints from the service with regards to the computations (it mainly says not authorized as you mentioned).
We are working on improving the developer experience when users are trying to implement SRP on their own in languages where we don't have an SDK. Also, we are trying to add more SDKs.
As daunting as it sounds, what I would suggest is to take the Javascript or the Android SDK, fix the inputs (SRP_A, SRP_B, TIMESTAMP) and add console.log statements at various points in the implementation to make sure your computations are similar. Then you would run these computations in your implementation and make sure you are getting the same. As you have suggested, the password claim signature needs to be passed as a base64 encoded string to the service so that might be one of the issues.
Some of the issues I encountered while implementing this was related to BigInteger library differences (the way they do byte padding and transform negative numbers to byte arrays and inversely).

Unable utilize a methodname(from webservice)present in the form of a string.I tried utilizing getattr,but then not unable to extend it for suds

from suds.client import Client ##UnresolvedImport
from suds.transport.https import HttpAuthenticated ##UnresolvedImport
import urllib2
class methodinvokeclass():
def methodinvokemethod(self,*args):
method=args[1]
c=args[2]
print c
response=c.service.method("90210")# I know this wont work,coz of method, but even I cant get me way thru with getattr
#response=c.service.LatLonListZipCode("90210")
print response
if __name__=="__main__":
invokemethodname="LatLonListZipCode"#Webservice name which I want to invoke...later !
f=open("C:\wsdllocation.txt",'r')# picks up the WSDL file location from the above file
webservwsdl=f.readline()
f.close()
y=methodinvokeclass()#dummy object
z=methodinvokeclass()#dummy object
t = HttpAuthenticated(username='x', password='x')#helps me getting thru my corporate firewall
t.handler = urllib2.HTTPBasicAuthHandler(t.pm)#helps me getting thru my corporate firewall
t.urlopener = urllib2.build_opener(t.handler)#helps me getting thru my corporate firewall
c = Client(url=webservwsdl,transport=t)#SUDs client !!!!!
x=y.methodinvokemethod(z,invokemethodname,c)# invoking the code above

You can use getattr to retrieve the SOAP method from the service. Use this instead:
impl = getattr(c.service, method)
response = impl('90210')
print response
Here is a working example, using a webservicex test SOAP service:
url = 'http://www.webservicex.net/stockquote.asmx?WSDL'
client = Client(url=url)
name = 'GetQuote'
impl = getattr(client.service, name)
print impl('IBM')
Output:
<StockQuotes>
<Stock>
<Symbol>IBM</Symbol>
<Last>163.81</Last>
<Date>4/7/2011</Date>
<Time>11:47am</Time>
<Change>-0.23</Change>
<Open>164.10</Open>
<High>164.5463</High>
<Low>163.44</Low>
<Volume>1573461</Volume>
<MktCap>199.8B</MktCap>
<PreviousClose>164.04</PreviousClose>
<PercentageChange>-0.14%</PercentageChange>
<AnnRange>116.00 - 167.72</AnnRange>
<Earns>11.52</Earns>
<P-E>14.24</P-E>
<Name>International Bus</Name>
</Stock>
</StockQuotes>