Develop Reference

How to capture the response based on the Content-Type sent by the server using requests in Python?

I am pretty new to python and learning how to make HTTP request and store the response in a variable.
Below is the similar kind of code snippet that I am trying to make the POST request.
import requests
import simplejson as json
api_url = https://jsonplaceholder.typicode.com/tickets
raw_body = {"searchBy":"city","searchValue":"1","processed":9,"size":47,"filter":{"cityCode":["BA","KE","BE"],"tickets":["BLUE"]}}
raw_header = {"X-Ticket-id": "1234567", "X-Ticket-TimeStamp": "11:01:1212", "X-Ticket-MessageId": "123", 'Content-Type': 'application/json'}
result = requests.post(api_url, headers=json.loads(raw_header), data=raw_body)
#Response Header
response_header_contentType = result.headers['Content-Type'] #---> I am getting response_header_contentType as "text/html; charset=utf-8"
#Trying to get the result in json format
response = result.json() # --> I am getting error at this line. May be because the server is sending the content type as "text/html" and I am trying to capture the json response.
Error in console :
Traceback (most recent call last):
File "C:\sam\python-project\v4\makeApiRequest.py", line 45, in make_API_request
response = result.json()
File "C:\Users\userName\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\userName\AppData\Local\Programs\Python\Python37\lib\site-packages\simplejson\__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "C:\Users\userName\AppData\Local\Programs\Python\Python37\lib\site-packages\simplejson\decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "C:\Users\userName\AppData\Local\Programs\Python\Python37\lib\site-packages\simplejson\decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
So, how can I store the response in a variable based on the content-type sent by the server using requests.
Can somebody please help me here. I tried googling too but did not find any helpful documentation on how to capture the response based on the content-type.

as you already said your contentType is 'text/html' not 'application/json' that normally means that it can not be decoded as json.
If you look at the documentation
https://2.python-requests.org/en/master/user/quickstart/#response-content you can find that there are different ways to decode the body, if you already know you have 'text/html' it makes sense to decode it with response.text.
Hence it makes sense to distinquish based on the content type how to decode your data:
if result.headers['Content-Type'] == 'application/json':
data = result.json()
elif result.headers['Content-Type'] == 'text/html':
data = result.text
else:
data = result.raw

API returned JSON data wrapped in a function, breaking json.loads

I'm trying to work with JSON data that is pulled from USGS Earthquake API. If you follow that link, you can see the raw JSON data.
The JSON looks great; however, the returned request is wrapped in an eqfeed_callback(); that is breaking the JSON deserializer in Python.
A quick look at the code I have so far:
import requests
import json
from pprint import pprint
URL = "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week.geojsonp"
response = requests.get(URL)
raw_json = str(response.content)
json = json.loads(raw_json)
print(json)
I get the errors:
Traceback (most recent call last):
File "run.py", line 11, in <module>
json = json.loads(raw_json)
File "C:\Program Files\Anaconda3\lib\json\__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "C:\Program Files\Anaconda3\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\Anaconda3\lib\json\decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Although I'm positive the issue is that it's wrapped in that function and the JSON decoder doesn't like it. So how would I go about removing the function wrapper to leave me with the clean JSON inside.

You're using the wrong URL.
JSON wrapped in a function call is JSONP, which is needed for getting around CORS when calling an API from web browsers.
The URL to get normal JSON is
URL = "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week.geojson"

Can not parse response from sg.media-imdb in python

I'm trying to parse response from https://sg.media-imdb.com/suggests/a/a.json in Python 3.6.8.
Here is my code:
import requests
url = 'https://sg.media-imdb.com/suggests/a/a.json'
data = requests.get(url).json()
I get this error:
$ /usr/bin/python3 /home/livw/Python/test_scrapy/phase_1.py
Traceback (most recent call last):
File "/home/livw/Python/test_scrapy/phase_1.py", line 33, in <module>
data = requests.get(url).json()
File "/home/livw/.local/lib/python3.6/site-packages/requests/models.py", line 889, in json
self.content.decode(encoding), **kwargs
File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 518, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
It seems like the response format is not JSON format, although I can parse the response at JSON Formatter & Validator
How to fix it and store the response in a json object?

This probably happend because its not a complete json, it have a prefix
you can see that the response start with imdb$a( and ends with )
json parsing doesn't know how to handle it and he fails, you can remove those values and just parse the json itself
you can do this:
import json
import requests
url = 'https://sg.media-imdb.com/suggests/a/a.json'
data = requests.get(url).text
json.loads(data[data.index('{'):-1])

Microsoft Bing Speech API using Python: No JSON Object

I'm trying to implement the Microsoft Bing Speech REST API with Python and I've found some code online.
https://www.taygan.co/blog/2018/02/09/getting-started-with-speech-to-text
import json
import requests
YOUR_API_KEY = 'ENTER_YOUR_KEY_HERE'
YOUR_AUDIO_FILE = 'ENTER_PATH_TO_YOUR_AUDIO_FILE_HERE'
MODE = 'interactive'
LANG = 'en-US'
FORMAT = 'simple'
def handler():
# 1. Get an Authorization Token
token = get_token()
# 2. Perform Speech Recognition
results = get_text(token, YOUR_AUDIO_FILE)
# 3. Print Results
print(results)
def get_token():
# Return an Authorization Token by making a HTTP POST request to Cognitive Services with a valid API key.
url = 'https://api.cognitive.microsoft.com/sts/v1.0/issueToken'
headers = {
'Ocp-Apim-Subscription-Key': YOUR_API_KEY
}
r = requests.post(url, headers=headers)
token = r.content
return(token)
def get_text(token, audio):
# Request that the Bing Speech API convert the audio to text
url = 'https://speech.platform.bing.com/speech/recognition/{0}/cognitiveservices/v1?language={1}&format={2}'.format(MODE, LANG, FORMAT)
headers = {
'Accept': 'application/json',
'Ocp-Apim-Subscription-Key': YOUR_API_KEY,
'Transfer-Encoding': 'chunked',
'Content-type': 'audio/wav; codec=audio/pcm; samplerate=16000',
'Authorization': 'Bearer {0}'.format(token)
}
r = requests.post(url, headers=headers, data=stream_audio_file(audio))
results = json.loads(r.content)
return results
def stream_audio_file(speech_file, chunk_size=1024):
# Chunk audio file
with open(speech_file, 'rb') as f:
while 1:
data = f.read(1024)
if not data:
break
yield data
if __name__ == '__main__':
handler()
I've followed the above code (and changed the key and audio file name) but keep getting this:
Traceback (most recent call last):
File "/Users/emi/Desktop/proj/taygan tutorial/handler.py", line 55, in <module>
handler()
File "/Users/emi/Desktop/proj/taygan tutorial/handler.py", line 15, in handler
results = get_text(token, YOUR_AUDIO_FILE)
File "/Users/emi/Desktop/proj/taygan tutorial/handler.py", line 40, in get_text
results = json.loads(r.content)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Complete newbie so really don't understand what's going on. Thanks!
NOTE: I've also tried the code from this
GitHub Repos
but I receive many errors with this one.

UnicodeDecodeError utf8 codec Python 2.7

I have built a scraper which reads artistnames from a csv file and collects artistdata via the Songkick api from these artists. However, after running my code for a while I get the following error:
File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 64-65: invalid continuation byte
Sample data can be downloaded here:
I am relatively new to coding and I was wondering how can I solve this error? Below you can find my code.
import urllib2
import requests
import json
import csv
from tinydb import TinyDB, Query
db = TinyDB('spotify_artists.json')
#read csv
def wait_for_internet():
while True:
try:
resp = urllib2.urlopen('http://google.com', timeout=1)
return
except:
pass
def load_artists():
f = open('artistnames.csv', 'r').readlines();
for a in f:
artist = a.strip()
print(artist)
url = 'http://api.songkick.com/api/3.0/search/artists.json?query='+artist+'&apikey='
# wait_for_internet()
r = requests.get(url)
resp = r.json()
# print(resp)
try :
if(resp['resultsPage']['totalEntries']):
# print(json.dumps(resp['resultsPage']['results']['artist'], indent=4, sort_keys=True))
results = resp['resultsPage']['results']['artist'];
for x in results:
# print('rxx')
# print(json.dumps(x, indent=4, sort_keys=True))
if(x['displayName'] == artist):
print(x)
db.insert(x)
except:
print('cannot fetch url',url);
load_artists()
db.close()
Traceback (most recent call last):
File "C:\Users\rmlj\Dropbox\songkick\scrapers\Data\Scraper.py", line 45, in <module>
load_artists()
File "C:C:\Users\rmlj\Dropbox\songkick\scrapers\Data\Scraper.py".py", line 25, in load_artists
r = requests.get(url)
File "C:\Python27\lib\site-packages\requests\api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "C:\Python27\lib\site-packages\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 474, in request
prep = self.prepare_request(req)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 407, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "C:\Python27\lib\site-packages\requests\models.py", line 302, in prepare
self.prepare_url(url, params)
File "C:\Python27\lib\site-packages\requests\models.py", line 358, in prepare_url
url = url.decode('utf8')
File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 64-65: invalid continuation byte

The problem is in your formation of a URL where you pass the query string as bytes (regular str on Python 2.x) with characters in a non-utf-8 encoding to the requests module, which in turn tries to turn it into an utf-8 unicode string, and fails.
First of all, you should let the requests module form your query string and deal with the creation of your final URL:
url = "http://api.songkick.com/api/3.0/search/artists.json"
r = requests.get(url, params={"query": artist, "apikey": ""})
# etc.
But second, you should not mix encodings least you want to be in a world of hurt. Unfortunately, the built-in csv module doesn't work with Unicode which is probably why you end up with invalid characters. To remedy that, install unicodecsv and use it as a drop-in replacement (just replace your import csv with import unicodecsv as csv).
update: Wait, on a second look you're not even using csv. You're reading your file line by line and trying to pass that off as a query. Is that your intended behavior? If that's the case, keeping with the idea of staying with same encoding:
import codecs
URL = "http://api.songkick.com/api/3.0/search/artists.json" # no need to redefine this
with codecs.open("artistnames.csv", "r", "utf-8") as f:
for a in f:
artist = a.strip()
r = requests.get(URL, params={"query": artist, "apikey": ""})
# etc.

You should use unicode whenever possible. Requests should convert any non-ascii characters in the url to the correct encoding.
>>> import requests
>>> requests.get(u'http://Motörhead.com/?q=Motörhead').url
u'http://xn--motrhead-p4a.com/?q=Mot%C3%B6rhead'
As you can see, the domain name is encoded as punycode, and the querystring uses percent-encoding.
as long as artist is a valid unicode string, this should work.
url = u'http://api.songkick.com/api/3.0/search/artists.json?query='+artist
if artist is a byte string, you must decode it into unicode using the correct encoding, which depends on what your original input file was encoded as.
artist = artist.decode('SHIFT-JIS')