Develop Reference

How do I Authenticate my FTX_Client in Python

I have looked through the FTX api documentation found here: https://docs.ftx.us/#overview
And I've looked at example code found in this repo: https://github.com/ftexchange/ftx/tree/master/rest
I can't 'get' or 'post' anything that requires the Authentication. I am using the api key on my account that has 'full trade permissions', and when I look at: print(request.headers) the headers look like they are in the right format.
I've tried: using google colab instead of vs code, updating all my libraries, generating a new api key, restarting kernel and computer. I can pull something like 'markets' because it doesn't need the Authentication.
Let me know if you need any more information, below is a portion of the code I have that isolates the problem and returns {'success': False, 'error': 'Not logged in'}
import time
import urllib.parse
from typing import Optional, Dict, Any, List
from requests import Request, Session, Response
import hmac
ep = 'https://ftx.us/api/wallet/balances'
ts = int(time.time() * 1000)
s = Session()
request = Request('GET', ep)
prepared = request.prepare()
signature_payload = f'{ts}{prepared.method}{prepared.path_url}'.encode()
if prepared.body:
signature_payload += prepared.body
signature = hmac.new(secret.encode(), signature_payload, 'sha256').hexdigest()
request.headers['FTX-KEY'] = key
request.headers['FTX-SIGN'] = signature
request.headers['FTX-TS'] = str(ts)
response = s.send(prepared)
data = response.json()
print(data)

I've faced with the same problem.
You need to change this part:
prepared.headers['FTX-KEY'] = key
prepared.headers['FTX-SIGN'] = signature
prepared.headers['FTX-TS'] = str(ts)
PS. I believe that the FTX needs to fix their API documentation
PSS. I've checked the a part of https://github.com/ftexchange/ftx/tree/master/rest code. I beleave FTX guys just do a copy-paste into docs this code but originally it belongs to more a sophisticated object oriented solution that will work correctly because they pass into method an already created request and use a prepared variable just to calculate path_url and method

For ftx.us, you need to use different headers:
prepared.headers['FTXUS-KEY'] = key
prepared.headers['FTXUS-TS'] = str(ts)
prepared.headers['FTXUS-SIGN'] = signature

Python requests session does not rotate proxies

I am using private rotating proxy provided by (https://proxy.webshare.io/proxy/rotating?) in which each request to rotating proxy receives a new IP address. when I am using
requests.get('https://httpbin.org/get', headers=headers, proxies=get_proxy())
it returns a new IP each time whenever I make request. but when using
session = requests.Session()
session.headers = headers
session.proxies = get_proxy()
session.get('https://httpbin.org/get')
It returns same IP each time whenever I make request.
How does session object behaves different from requests.get() function in case of proxies.

Session uses previously set up variables/values for each subsequent request, like Cookies. If you want to change the proxy for each request in the session, then use Prepared Requests to set it each time or just put it in a function:
def send(session, url):
return session.get(url, proxy=get_proxy())
sess = requests.Session()
sess.headers = headers
resp = send(sess, 'https://httpbin.org/get')
print(resp.status_code)
But if you're trying to hide your origin IP for scraping or something, you probably don't want to persist cookies, etc. so you shouldn't use sessions.

The following code works, and it take a proxylistfile.txt file to check every proxy:
from requests import *
import bs4
import sys
if len(sys.argv) < 2:
print('Usage: ./testproxy.py <proxylistfile.txt>')
sys.exit()
ifco = 'http://ifconfig.co'
PROXIES_FILE = sys.argv[1]
proxy = dict()
with open(PROXIES_FILE) as file:
for line in file:
if line[0] == '#' or line == "\n":
continue
line_parts = line.replace('\n', '').split(':')
proxy['http'] = f'{line_parts[0]}://{line_parts[1]}:{line_parts[2]}'
try:
i = get(ifco, proxies=proxy, timeout=11)
print(f"{proxy['http']} - successfull - IP ---> ", end='')
zu = bs4.BeautifulSoup(i.text, 'html.parser')
testo = zu.findAll('p', text=True)[0].get_text()
print(testo)
except:
print(f"{proxy['http']} - unsuccessfull")
pass
It connect ot ifconfig.co site and return its real ip to check if the proxy works.
The output will be something like:
http://proxy:port - successfull - IP ---> your.real.ip
the input file format should be like:
http:1.1.1.1:3128

I finally switch to another rotating proxy provider (https://www.proxyegg.com) and the issue has been resolved now.

HSM integration with Python requests module

So I'm writting an application that needs to authenticate to a server using a client certificate (Mutual Authentication). The client certificate key is stored in an HSM (Gemalto's). I did the OpenSSL integration with the Luna Client but the requests module requires a file:
from requests import Session
session: Session = Session()
session.cert = (
"/ssl/client.pem",
"/ssl/client.key"
)
session.verify = "/ssl/server.pem"
My issue is that I could not find a way to bind the private key when it's in the HSM. Here's what I tried so far with the pycryptoki library:
from pycryptoki.session_management import (
c_initialize_ex,
c_open_session_ex,
login_ex,
c_logout_ex,
c_close_session_ex,
c_finalize_ex,
)
from requests import Session
c_initialize_ex()
auth_session = c_open_session_ex(0)
login_ex(auth_session, 0, "some-pass")
session: Session = Session()
session.cert = (
"/ssl/client.pem",
"rsa-private-156405640312",
)
session.verify = "/ssl/server.pem"
...
c_logout_ex(auth_session)
c_close_session_ex(auth_session)
c_finalize_ex()
I have opened an issue on there here a while back, I had to finish the app implementation so I put the HSM integration on the ice, but I need to make that work before going to production: https://github.com/gemalto/pycryptoki/issues/17
I also tried using py-hsm but it is a low level api library, I also opened an issue there with an example of my code:
from pyhsm.hsmclient import HsmClient
from requests import Session
c = HsmClient(pkcs11_lib="/usr/lib/libCryptoki2_64.so")
c.open_session(slot=0)
c.login(pin="some-code")
session: Session = Session()
session.cert = "/ssl/client.pem"
c.logout()
c.close_session()
Anyone can provide an example of Mutual authentication with the certificate pair in an HSM? If you have something in C/C++ it would be great too, I could implement my request function and just wrap it in my python code.
Thank you in advance!

I have tested almost all wrappers for Python to do the same.
PyKCS11 is not really solid.
I recommend to use one of these two possibilities:
1. PyCurl
When you configure correctly your OpenSSL and so your cURL, the Python wrapper to cUrl can do this.
Here is a simple implementation:
import pycurl
from io import BytesIO
import pprint
import json
c = pycurl.Curl()
url = 'https://yourserver/endpoint'
c.setopt(pycurl.URL, url)
# set proxy-insecure
c.setopt(c.SSL_VERIFYPEER, 0)
c.setopt(c.SSL_VERIFYHOST, False)
c.setopt(c.VERBOSE, 0)
c.setopt(pycurl.SSLENGINE, 'pkcs11')
c.setopt(pycurl.SSLCERTTYPE, 'eng')
c.setopt(pycurl.SSLKEYTYPE, 'eng')
c.setopt(pycurl.SSLCERT, 'pkcs11:model=XXX;manufacturer=YYYYY;serial=ZZZZ;'
'token=AAAAA;id=BBBBBBBBB;'
'object=CCCCCC;type=cert;pin-value=pin-pin')
c.setopt(pycurl.SSLKEY, 'pkcs11:model=XXX;manufacturer=YYYYY;serial=ZZZZ;'
'token=AAAAA;id=BBBBBBBBB;'
'object=CCCCCC;type=private;pin-value=pin-pin')
# set headers
c.setopt(pycurl.HEADER, True)
# c.setopt(pycurl.USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0')
c.setopt(pycurl.HTTPHEADER, ("HEADER_TO_ADD:VALUE",))
buffer = BytesIO()
c.setopt(c.WRITEDATA, buffer)
c.perform()
# HTTP response code, e.g. 200.
print('>>> Status: %d' % c.getinfo(c.RESPONSE_CODE))
# Elapsed time for the transfer.
print('>>> Time: %f' % c.getinfo(c.TOTAL_TIME))
# getinfo must be called before close.
c.close()
body = buffer.getvalue().decode('utf-8')
print('>>> Body:\n', body)
if body.find('{') >= 0:
body = body[body.find('{'):]
dictionary = json.loads(body)
pprint.pprint(dictionary)
Note that the pkcs#11 URI Scheme must be compliant to the RFC7512
It can be discovered using this command:
p11tool --provider=/usr/lib/libeTPkcs11.so --list-all
All fields including the pin must be url encoded. Use an online website to encode/decode string(pin)<->url_encoded
2. M2Crypto
This is the best pkcs#11 implementation done for Python in my point of view.
It allows to override urllib2 and so requests calls thanks to the requests.Session.mount method and the requests.adapters.BaseAdapter.
I put here some code to use it with urllib2:
from M2Crypto import m2urllib2 as urllib2
from M2Crypto import m2, SSL, Engine
# load dynamic engine
e = Engine.load_dynamic_engine("pkcs11", "/usr/lib/x86_64-linux-gnu/engines-1.1/libpkcs11.so")
pk = Engine.Engine("pkcs11")
pk.ctrl_cmd_string("MODULE_PATH", "/usr/lib/libeTPkcs11.so")
m2.engine_init(m2.engine_by_id("pkcs11"))
pk.ctrl_cmd_string("PIN", 'pin-pin')
cert = pk.load_certificate('pkcs11:model=XXX;manufacturer=YYYYY;serial=ZZZZ;'
'token=AAAAA;id=BBBBBBBBB;'
'object=CCCCCC')
key = pk.load_private_key('pkcs11:model=XXX;manufacturer=YYYYY;serial=ZZZZ;'
'token=AAAAA;id=BBBBBBBBB;'
'object=CCCCCC', pin='pin-pin')
ssl_context = SSL.Context('tls')
ssl_context.set_cipher_list('EECDH+AESGCM:EECDH+aECDSA:EECDH+aRSA:EDH+AESGCM:EDH+aECDSA:EDH+aRSA:!SHA1:!SHA256:!SHA384:!MEDIUM:!LOW:!EXP:!aNULL:!eNULL:!PSK:!SRP:#STRENGTH')
ssl_context.set_default_verify_paths()
ssl_context.set_allow_unknown_ca(True)
SSL.Connection.postConnectionCheck = None
m2.ssl_ctx_use_x509(ssl_context.ctx, cert.x509)
m2.ssl_ctx_use_pkey_privkey(ssl_context.ctx, key.pkey)
opener = urllib2.build_opener(ssl_context)
urllib2.install_opener(opener)
url = 'https://yourserver/endpoint'
content = urllib2.urlopen(url=url).read()
# content = opener.open(url)
print(content)
Note that, we don't indicate type and pin in the pkcs#11 URI as for PyCurl.
The PIN is indicated as string and not url encoded when passed to load_private_key.
Finally, I found a good implementation for the HttpAdapter to mount on requests to do the calls.
Here is the original implementation.
I added some lines to support the PIN code and to disable checking hostname with this:
M2Crypto.SSL.Connection.postConnectionCheck = None
And here is how to mount the adapter:
from requests import Session
from m2requests import M2HttpsAdapter
import pprint, json
request = Session()
m2httpsadapter = M2HttpsAdapter()
# Added by me to set a pin when loading private key
m2httpsadapter.pin = 'pin-pin'
# Need this attribute set to False else we cannot use direct IP (added by me to disable checking hostname)
m2httpsadapter.check_hostname = False
request.mount("https://", m2httpsadapter)
request.cert=('pkcs11:model=XXX;manufacturer=YYYYY;serial=ZZZZ;'
'token=AAAAA;id=BBBBBBBBB;'
'object=CCCCCC',
'pkcs11:model=XXX;manufacturer=YYYYY;serial=ZZZZ;'
'token=AAAAA;id=BBBBBBBBB;'
'object=CCCCCC')
headers = {'HEADER_TO_ADD_IF_WANTED': 'VALUE', }
r = request.get("https://yourserver/endpoint", headers=headers, verify=False)
print(r.status_code)
pprint.pprint(json.loads(r.raw.data.decode('utf-8')))

A private key in an HSM can be used to sign some data, which is what you are trying to accomplish for mutual authentication. Signing some data provided by the other party proves that you control the Private Key corresponding to the certificate.
From the HSM you can get a handle to the private key and use that handle to perform private key operations without exporting the key (the whole reason for using an HSM is to prevent the content of the private key from being seen).
HSMs can expose a lot of different interfaces but PKCS#11 is by far the most common. You don't really need to use OpenSSL or to program in C/C++. Since you are using Python requests, there are some PKCS#11 Python libraries that can be used.
Please take a look at this Python PKCS11 library: https://github.com/danni/python-pkcs11.
You can then do things like (assuming the key is RSA)
import pkcs11
lib = pkcs11.lib("/usr/lib/libCryptoki2_64.so")
token = lib.get_token(token_label='DEMO')
# Open a session on our token
with token.open(user_pin='1234') as session:
# Get the key from the HSM by label
priv = session.get_key(
object_class=pkcs11.constants.ObjectClass.PRIVATE_KEY,
label='key_label')
# sign with that key using the required mechanism:
signature = priv.sign(my_to_be_signed_data, mechanism=pkcs11.Mechanism.SHA256_RSA_PKCS)
You do not provide many details about this mutual authentication but hopefully, this should get you on the right track. The code above retrieves the key based on a label, but this PKCS#11 library also supports finding a key in the HSM based on a lot of other properties.

How to set the return value of a get request in an unit test?

I am trying to set the return value of a get request in python in order to do a unit test, which tests if the post request is called with the correct arguments. Assume I have the following code to test
# main.py
import requests
from django.contrib.auth.models import User
def function_with_get():
client = requests.session()
some_data = str(client.get('https://cool_site.com').content)
return some_data
def function_to_test(data):
for user in User.objects.all():
if user.username in data:
post_data = dict(data=user.username)
else:
post_data = dict(data='Not found')
client.post('https://not_cool_site.com', data=post_data)
#test.py
from unittest import mock
from unittest import TestCase
from main import function_with_get, function_to_test
class Testing(TestCase):
#mock.patch('main.requests.session')
def test_which_fails_because_of_get(self, mock_sess):
mock_sess.get[0].return_value.content = 'User1'
data = function_with_get()
function_to_test(data)
assertIn('Not Found', mock_sess.retrun_value.post.call_args_list[1])
This, sadly, does not work and I have also tried to set it without content, however, I get an error AttributeError: 'str' object has no attribute 'content'
What would be the correct way to set the return_value of the get request, so that I can test the arguments of the post request?

I think you almost have it, except you're missing the return value for session() - because session is instantiated to create the client instance. I think you can drop the [0] too.
Try:
mock_sess.return_value.get.return_value.content = 'User1'

Try with .text because this should work for strings.
s = requests.Session()
s.get('https://httpbin.org/cookies/ set/sessioncookie/123456789')
r = s.get('https://httpbin.org/ cookies')
print(r.text)
http://docs.python-requests.org/en/master/user/advanced/

requests - Gateway Timeout

this is a test script to request data from Rovi API, provided by the API itself.
test.py
import requests
import time
import hashlib
import urllib
class AllMusicGuide(object):
api_url = 'http://api.rovicorp.com/data/v1.1/descriptor/musicmoods'
key = 'my key'
secret = 'secret'
def _sig(self):
timestamp = int(time.time())
m = hashlib.md5()
m.update(self.key)
m.update(self.secret)
m.update(str(timestamp))
return m.hexdigest()
def get(self, resource, params=None):
"""Take a dict of params, and return what we get from the api"""
if not params:
params = {}
params = urllib.urlencode(params)
sig = self._sig()
url = "%s/%s?apikey=%s&sig=%s&%s" % (self.api_url, resource, self.key, sig, params)
resp = requests.get(url)
if resp.status_code != 200:
# THROW APPROPRIATE ERROR
print ('unknown err')
return resp.content
from another script I import the module:
from roviclient.test import AllMusicGuide
and create an instance of the class inside a mood function:
def mood():
test = AllMusicGuide()
print (test.get('[moodids=moodids]'))
according to documentation, the following is the syntax for requests:
descriptor/musicmoods?apikey=apikey&sig=sig [&moodids=moodids] [&format=format] [&country=country] [&language=language]
but running the script I get the following error:
unknown err
<h1>Gateway Timeout</h1>:
what is wrong?

"504, try once more. 502, it went through."
Your code is fine, this is a network issue. "Gateway Timeout" is a 504. The intermediate host handling your request was unable to complete it. It made its own request to another server on your behalf in order to handle yours, but this request took too long and timed out. Usually this is because of network congestion in the backend; if you try a few more times, does it sometimes work?
In any case, I would talk to your network administrator. There could be any number of reasons for this and they should be able to help fix it for you.