Develop Reference

Simple JSON Decode Error in Gemini API Started Out of Nowhere

I am trying to play with the Gemini trading API. I have issued myself an API key and a secret, and after configuring my environment, in which I had a lot of issues setting up and installing requests through pip, I used their example code to create a simple script to read my recent trades. Here is the script, minus my API keys:
#!/usr/bin/env/ python
import requests
import json
import base64
import hmac
import hashlib
import datetime, time
url = "https://api.sandbox.gemini.com"
gemini_api_key = "master-xxx"
gemini_api_secret = "xxx".encode()
t = datetime.datetime.now()
payload_nonce = str(int(time.mktime(t.timetuple())*1000))
payload = {"request": "/v1/mytrades", "nonce": payload_nonce}
encoded_payload = json.dumps(payload).encode()
b64 = base64.b64encode(encoded_payload)
signature = hmac.new(gemini_api_secret, b64, hashlib.sha384).hexdigest()
request_headers = {
'Content-Type': "text/plain",
'Content-Length': "0",
'X-GEMINI-APIKEY': gemini_api_key,
'X-GEMINI-PAYLOAD': b64,
'X-GEMINI-SIGNATURE': signature,
'Cache-Control': "no-cache"
}
response = requests.post(url, headers=request_headers)
my_trades = response.json()
print(my_trades)
Now at first, it would run, but give me an error saying I hadn't specified an account. Then, without changing ANYTHING AT ALL, it suddenly quit working altogether. So while I still have some sort of issue accessing the API, I can't even get to the errors anymore to try to figure out why. Now what I get is a JSON decode error, which looks like the following:
Traceback (most recent call last):
File "c:\Users\david\Desktop\Code Projects\GeminiTrader\GeminiTrader-v0.1.py",
line 33, in <module>
my_trades = response.json()
File "C:\Users\david\AppData\Local\Programs\Python\Python39\lib\site-packages\requests-2.25.1-py3.9.egg\requests\models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\david\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\david\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\david\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
What is causing this json decode issue? Why was it not coming up before, when the API was simply rejecting my request due to an account parameter error? Why did it suddenly change to this error without me modifying anything in the code? How can I fix it? I kept having issues with installing requests and getting it to work, maybe I messed something up in that process?
Certainly after fixing this I will have a new host of issues to fix because the documentation on this API is abysmal. Any help in progressing this project would be greatly appreciated! Thanks!

As you are calling an API, there are chances that your API call fails and returns just string or empty response. I would suggest that you first add a check on the status code of your response something like below and then process the json data.
data = requests.post()
if data.status_code != 200:
raise Exception("Error", data.reason)
json_data = data.json()

Compatibility issue (python 2 vs python 3)

I have a program that will auto create a Github issue via the API. It works in Python 2.7, but when I run it with Python 3 I get the following error:
Traceback (most recent call last):
File "/home/baal/bin/python/zeus-scanner/var/auto_issue/github.py", line 92, in request_issue_creation
urllib2.urlopen(req, timeout=10).read()
File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.5/urllib/request.py", line 464, in open
req = meth(req)
File "/usr/lib/python3.5/urllib/request.py", line 1183, in do_request_
raise TypeError(msg)
TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str.
I have the following method that creates a github issue (successful in python 2 unsuccessful in python 3):
def request_issue_creation():
logger.info(set_color(
"Zeus got an unexpected error and will automatically create an issue for this error, please wait..."
))
def __extract_stacktrace(file_data):
logger.info(set_color(
"extracting traceback from log file..."
))
retval, buff_mode, _buffer = [], False, ""
with open(file_data, "r+") as log:
for line in log:
if "Traceback" in line:
buff_mode = True
if line and len(line) < 5:
buff_mode = False
retval.append(_buffer)
_buffer = ""
if buff_mode:
if len(line) > 400:
line = line[:400] + "...\n"
_buffer += line
return "".join(retval)
logger.info(set_color(
"getting authorization..."
))
encoded = __get_encoded_string()
n = get_decode_num(encoded)
token = decode(n, encoded)
current_log_file = get_latest_log_file(CURRENT_LOG_FILE_PATH)
stacktrace = __extract_stacktrace(current_log_file)
issue_title = stacktrace.split("\n")[-2]
issue_data = {
"title": issue_title,
"body": "Error info:\n```{}````\n\n"
"Running details:\n`{}`\n\n"
"Commands used:\n`{}`\n\n"
"Log file info:\n```{}```".format(
str(stacktrace),
str(platform.platform()),
" ".join(sys.argv),
open(current_log_file).read()
),
}
try:
req = urllib2.Request(
url="https://api.github.com/repos/<API-REPO>/issues", data=json.dumps(issue_data),
headers={"Authorization": "token {}".format(token)}
)
urllib2.urlopen(req, timeout=10).read()
logger.info(set_color(
"issue has been created successfully with the following name '{}'...".format(issue_title)
))
except Exception as e:
logger.exception(set_color(
"failed to auto create the issue, got exception '{}', "
"you may manually create an issue...".format(e), level=50
))
I read online that encoding the string to utf-8 will fix the issue, however I'm not sure if that is possible here? Any help would be greatly appreciated, thank you.

You need to encode your JSON payload:
data = json.dumps(issue_data)
if sys.version_info > (3,): # Python 3
data = data.encode('utf8')
req = urllib2.Request(
url="https://api.github.com/repos/<API-REPO>/issues", data=data,
headers={"Authorization": "token {}".format(token),
"Content-Type": "application/json; charset=utf-8"}
)
I added a Content-Type header with a charset parameter to communicate the codec used to the server. This is not always needed, JSON's default codec is UTF-8. If you don't specify the header, a (wrong) default would be supplied; it depends on the server whether or not that matters.

GAE Python URL Fetch throws InvalidURLError [duplicate]

This question already has an answer here:
Python GAE urlfetch credentials
(1 answer)
Closed 7 years ago.
GAE Python URL Fetch throws InvalidURLError while the same URL works perfectly with Postman ( Google Chrome App ).
CODE
url = "https://abcdefgh:28dfd95928dfd95928dfd95928dfd95928dfd95928dfd959#twilix.exotel.in/v1/Accounts/abcdefgh/Sms/send"
form_fields = {
"From": "08039511111",
"To": "+919844100000",
"Body": "message for you"
}
form_data = urllib.urlencode (form_fields)
try:
result = urlfetch.fetch(url=url,
payload=form_data,
method=urlfetch.POST,
headers={'Content-Type': 'application/x-www-form-urlencoded' }
)
logging.info ("result = ["+repr (result)+"] ")
except Exception:
logging.error ("Exception. ["+traceback.format_exc ()+"] ")
OUTPUT LOGS
2016-01-21 15:48:23.368 +0530 E Exception. [
Traceback (most recent call last): File "main.py", line 27, in get method=urlfetch.POST,
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 271, in fetch return rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py", line 389, in _get_fetch_result 'Invalid request URL: ' + url + error_detail) InvalidURLError: Invalid request URL: https://abcdefgh:28dfd95928dfd95928dfd95928dfd95928dfd95928dfd959#twilix.exotel.in/v1/Accounts/abcdefgh/Sms/send ]
For security purpose, I have replaced sensitive text in the URL with similar different characters.

The code indicates an INVALID_URL RPC error code was received from the urlfetch service.
The most common occurence seems to be due to the URL length limit (check if your unedited URL hits that): Undocumented max length for urlfetch URL?
A long time ago it was also seen for very slow URLs (in Go land, but I suspect the urlfetch service itself is the same serving all language sandboxes) - unsure if this still stands, I also see a DEADLINE_EXCEEDED error code as well which might have been introduced specifically for such case in the meantime): Google App Engine Go HTTP request to a slow page
The failure might also be related to incorrect parsing of the rather unusual "host" portion of your URL foo:blah#hostname. Check if it you're getting the same error if dropping the foo:blah# portion. If it's indeed the case you might want to file an issue with Google - the URL seems valid, works with curl as well.

I found the problem and the solution.
We need to specify the HTTP auth info using headers.
urlfetch.make_fetch_call ( rpc,
url,
method = urlfetch.POST,
headers = { "Authorization" : "Basic %s" % base64.b64encode ( URL_USERNAME+":"+URL_PASSOWRD ) },
)
Courtesy
https://stackoverflow.com/a/8454580/1443563 by raugfer

http.client.BadStatusLine: '' when attempting app-only twitter oauth with python3

I am trying to create a script that will process Twitter streams. Unfortunately, the OAuth process has stymied me. Adopting some code I found on the internet, I receive a blank response from https://api.twitter.com/oauth/token. In order to better understand the process, I am trying to do this without special modules. Here is my code, what am missing? Any help would be greatly appreciated.
Code:
import http.client
import urllib
import base64
CONSUMER_KEY = 'yadayadayada'
CONSUMER_SECRET = 'I am really tired today'
encoded_CONSUMER_KEY = urllib.parse.quote(CONSUMER_KEY)
encoded_CONSUMER_SECRET = urllib.parse.quote(CONSUMER_SECRET)
concat_consumer_url = encoded_CONSUMER_KEY + ':' + encoded_CONSUMER_SECRET
host = 'api.twitter.com'
url = '/oauth2/token/'
params = urllib.parse.urlencode({'grant_type' : 'client_credentials'})
req = http.client.HTTPSConnection(host, timeout = 100)
req.set_debuglevel(1)
req.putrequest("POST", url)
req.putheader("Host", host)
req.putheader("User-Agent", "My Twitter 1.1")
req.putheader("Authorization", "Basic %s" % base64.b64encode(b'concat_consumer_url'))
req.putheader("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8")
req.putheader("Content-Length", "29")
req.putheader("Accept-Encoding", "identity")
req.endheaders()
req.send(b'params')
resp = req.getresponse()
print ("{} {}".format(resp.status, resp.reason))
Error message:
C:\Python33>app_only_test_klug.py
Traceback <most recent call last>:
File "C:\Python33\app_only_test_klug.py", line 31, in <module>
resp = req.getresponse()
File "C:\Python33\lib\http\client.py", line 1131, in getresponse
response.being()
File "C:\Python33\lib\http\client.py", line 354, in begin
version, status, reason = self._read_status()
File "C:\Python33\lib\http\client.py", line 324, in _read_status
raise BadStatusLine(line)
http.client.BadStatusLine: ''
Any help would be greatly appreciated.
UPDATE:
After some more tinkering, I believe that the issue lies with my base64 encoding:
req.putheader("Authorization", "Basic %s" % base64.b64encode(b'concat_consumer_url'))
When I decode the resulting encoding of the above, I get "b'concat_consumer_url'" rather than a concatenation of the encoded_CONSUMER_KEY and encoded_CONSUMER_SECRET combined around a colon. How do I get the base64 to b64encode the value that concat_comsumer_url represents rather than the string "concat_consumer_url" so that I can move forward? Thanks in advance.

I believe the issue is there as well--you should just pass the variable to the encoding function, rather than the name of the variable as bytes, like this:
req.putheader("Authorization", "Basic %s" % base64.b64encode(concat_consumer_url))
Try it again with that change.

How to query a restful webservice using Python

Writing a Python script that uses Requests lib to fire off a request to a remote webservice. Here is my code (test.py):
import logging.config
from requests import Request, Session
logging.config.fileConfig('../../resources/logging.conf')
logr = logging.getLogger('pyLog')
url = 'https://158.74.36.11:7443/hqu/hqapi1/user/get.hqu'
token01 = 'hqstatus_python'
token02 = 'ytJFRyV7g'
response_length = 351
def main():
try:
logr.info('start SO example')
s = Session()
prepped = Request('GET', url, auth=(token01, token02), params={'name': token01}).prepare()
response = s.send(prepped, stream=True, verify=False)
logr.info('status: ' + str(response.status_code))
logr.info('elapsed: ' + str(response.elapsed))
logr.info('headers: ' + str(response.headers))
logr.info('content: ' + response.raw.read(response_length).decode())
except Exception:
logr.exception("Exception")
finally:
logr.info('stop')
if __name__ == '__main__':
main()
I get the following successful output when i run this:
INFO test - start SO example
INFO test - status: 200
INFO test - elapsed: 0:00:00.532053
INFO test - headers: CaseInsensitiveDict({'server': 'Apache-Coyote/1.1', 'set-cookie': 'JSESSIONID=8F87A69FB2B92F3ADB7F8A73E587A10C; Path=/; Secure; HttpOnly', 'content-type': 'text/xml;charset=UTF-8', 'transfer-encoding': 'chunked', 'date': 'Wed, 18 Sep 2013 06:34:28 GMT'})
INFO test - content: <?xml version="1.0" encoding="utf-8"?>
<UserResponse><Status>Success</Status> .... </UserResponse>
INFO test - stop
As you can see, there is this weird variable 'response_length' that i need to pass to the response object (optional argument) to be able to read the content. This variable has to be set to a numeric value that is equal to length of the 'content'. This obviously means that i need to know the response-content-length before hand, which is unreasonable.
If i don't pass that variable or set it to a value greater than the content length, I get the following error:
Traceback (most recent call last):
File "\Python33\lib\http\client.py", line 590, in _readall_chunked
chunk_left = self._read_next_chunk_size()
File "\Python33\lib\http\client.py", line 562, in _read_next_chunk_size
return int(line, 16)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 22, in main
logr.info('content: ' + response.raw.read().decode())
File "\Python33\lib\site-packages\requests\packages\urllib3\response.py", line 167, in read
data = self._fp.read()
File "\Python33\lib\http\client.py", line 509, in read
return self._readall_chunked()
File "\Python33\lib\http\client.py", line 594, in _readall_chunked
raise IncompleteRead(b''.join(value))
http.client.IncompleteRead: IncompleteRead(351 bytes read)
How do i make this work without this 'response_length' variable?
Also, are there any better options than 'Requests' lib?
PS: this code is an independent script, and does not run in the Django framework.

Use the public API instead of internals and leave worrying about content length and reading to the library:
import requests
s = requests.Session()
s.verify = False
s.auth = (token01, token02)
resp = s.get(url, params={'name': token01}, stream=True)
content = resp.content
or, since stream=True, you can use the resp.raw file object:
for line in resp.iter_lines():
# process a line
or
for chunk in resp.iter_content():
# process a chunk
If you must have a file-like object, then resp.raw can be used (provided stream=True is set on the request, like done above), but then just use .read() calls without a length to read to EOF.
If you are however, not querying a resource that requires you to stream (anything but a large file request, a requirement to test headers first, or a web service that is explicitly documented as a streaming service), just leave off the stream=True and use resp.content or resp.text for byte or unicode response data.
In the end, however, it appears your server is sending chunked responses that are malformed or incomplete; a chunked transfer encoding includes length information for each chunk and the server appears to be lying about a chunk length or sending too little data for a given chunk. The decode error is merely the result of incomplete data having been sent.

The server you request use "chunked" transfer encoding so there is not a content-length header. A raw response in chunked transfer encoding contains not only actual content but also chunks, a chunk is a number in hex followed by "\r\n" and it always cause xml or json parser error.
try use:
response.raw.read(decode_content=True)