Develop Reference

How do I combine several JSON API responses into a single variable/object?

I am pulling data in from an API that limits the number of records per request to 100. There are 7274 records in total and everything is returned as JSON.
I want to concatenate all 7274 records into a single variable/object and eventually export to a JSON file.
The response JSON objects are structured like this:
{"data":[{"key1":87,"key2":"Ottawa",..."key21":"ReggieWatts"}],"total":7274}
I just want the objects inside the "data" array so that the output looks like this:
{'key1': 87, 'key2': 'Ottawa', 'key21': 'ReggieWatts'},{'key1': 23, 'key2': 'Cincinnati', 'key21': 'BabeRuth'},...
I’ve tried without success to use the dict.update() method to concatenate the new values to a variable that’s collecting all the records.
I am getting this error:
ValueError: dictionary update sequence element #0 has length 21; 2 is required
Here’s the stripped down code.
import json
import time
import random
import requests
from requests.exceptions import HTTPError
api_request_limit = 100
# total_num_players = 7274
total_num_players = 201 # only using 201 for now so that we aren't hammering the api while testing
start_index = 0
base_api_url = "https://api.nhle.com/stats/rest/en/skater/bios?isAggregate=true&isGame=false&sort=[{%22property%22:%22playerId%22,%22direction%22:%22ASC%22}]&limit=100&factCayenneExp=gamesPlayed%3E=1&cayenneExp=gameTypeId=2%20and%20seasonId%3C=20202021%20and%20seasonId%3E=19171918&start="
player_data = {}
curr_data = {}
for curr_start_index in range(start_index, total_num_players, api_request_limit):
api_url = base_api_url + str(curr_start_index)
try:
response = requests.get(api_url)
# if successful, no exceptions
response.raise_for_status()
except HTTPError as http_err:
print(f'HTTP error occurred: {http_err}')
except Exception as err:
print(f'Other error occurred: {err}')
else:
# print('Success!')
curr_data = response.json()
player_data.update(curr_data['data'])
# player_data = {**player_data, **curr_data['data']} # Does not work either
# print(player_data)
# for i in curr_skaters['data']:
# print(str(i['firstSeasonForGameType']) + ": " + str(i['skaterFullName']) + " " + str(i['playerId']))
set_delay = (random.random() * 2) + 1
time.sleep(set_delay)
Should I be iterating through each of the 100 records individually to add them to player_data?
The ValueError implies that the issue is with the number of key:value pairs in each object which says to me I'm using the .update() method incorrectly here.
Thanks

If you want them all in a single dictionary which can be exported to a json file, you'll need to have unique keys for each response. Perhaps the following will accomplish what you want:
response0 = {"data":[{"key1":87,"key2":"Ottawa","key21":"ReggieWatts"}],"total":7274}
response1 = {"data":[{"key1":23,"key2":"Cincinnati","key21":"BabeRuth"}],"total":4555}
all_data = {}
for i, resp in enumerate([response0, response1]):
all_data[f'resp{i}'] = resp['data'][0]
This returns
all_data = {'resp0': {'key1': 87, 'key2': 'Ottawa', 'key21': 'ReggieWatts'},
'resp1': {'key1': 23, 'key2': 'Cincinnati', 'key21': 'BabeRuth'}}
Edit: I went for a dictionary object initially since I think it saves more naturally as json, but to get it as a python list, you can use the following:
all_data = []
for resp in [response0, response1]:
all_data.append(resp['data'][0])
Finally, this object is easily saveable as json:
import json
with open('saved_responses.json', 'w') as file:
json.dump(all_data, file)

I figured it out with a big thanks to William.
I also found this post very helpful.
https://stackoverflow.com/a/26853961/610406
Here's the fix I eventually landed on:
import json
import time
import random
import requests
from requests.exceptions import HTTPError
api_request_limit = 100
# total_num_players = 7274 # skaters
total_num_players = 201 # only using 201 for now so that we aren't hammering the api while testing
start_index = 0
base_api_url_skaters = "https://api.nhle.com/stats/rest/en/skater/bios?isAggregate=true&isGame=false&sort=[{%22property%22:%22playerId%22,%22direction%22:%22ASC%22}]&limit=100&factCayenneExp=gamesPlayed%3E=1&cayenneExp=gameTypeId=2%20and%20seasonId%3C=20202021%20and%20seasonId%3E=19171918&start="
player_data = [] # Needs to be a list.
curr_data = {}
for curr_start_index in range(start_index, total_num_players, api_request_limit):
api_url = base_api_url_skaters + str(curr_start_index)
try:
response = requests.get(api_url)
# if successful, no exceptions
response.raise_for_status()
except HTTPError as http_err:
print(f'HTTP error occurred: {http_err}')
except Exception as err:
print(f'Other error occurred: {err}')
else:
# print('Success!')
curr_data = response.json()
# *** >>>This line is what I needed! So simple in retrospect.<<< ***
player_data = [*player_data, *curr_data['data']]
set_delay = (random.random() * 3) + 1
time.sleep(set_delay)
print(f'Counter: {curr_start_index}. Delay: {set_delay}. Record Count: {len(player_data)}.')
with open('nhl_skaters_bios_1917-2021.json', 'w') as f:
json.dump(player_data,f)
As a Gist:
https://gist.github.com/sspboyd/68ec8f5c5cd15ee7467d4326e3b74111

How to troubleshoot and resolve "TypeError: 'NoneType' object is not subscriptable" in python

I have a piece of code that fetches data from the ticketmaster API using a function I've named get_event_info. The first revision of the code worked as desired, subsequently I modified the original function to make use of header based authentication instead of URL based. I also added a few lines to the function which were intended to validate the response status code. After making these changes the code began producing the following TypeError:
Traceback (most recent call last):
File "ticketmaster_only_w_headers.py", line 146, in <module>
for event in ticket_search["_embedded"]["events"].items():
TypeError: 'NoneType' object is not subscriptable
I've read quite a bit about this type of error but I'm still unable to determine why my code is producing it in this instance. I would really appreciate an explanation on why my code is producing this error and what troubleshooting methods I should have used to uncover the source error. I'm fairly comfortable with programming but certainly no expert so the simpler the language used the better.
(Function Definition)
def get_event_info(search):
if search in CACHE_DICTION:
d = CACHE_DICTION[search]
else:
api_url = '{0}events/'.format(api_url_base)
payload = {"keyword": search, "apikey": api_token,
"format": "json", "dmaId": "366", "size": 200, "radius": "2"}
response = requests.get(api_url, headers=headers, params=payload)
if response.status_code == 200:
d = json.loads(response.text)
CACHE_DICTION[search] = d
f = open(CACHE_FNAME, 'w')
f.write(json.dumps(CACHE_DICTION))
f.close()
else:
d = None
return d
(Code snippet that triggers the error)
ticket_search = get_event_info("")
for event in ticket_search["_embedded"]["events"]:
a = event["id"]
b = event["name"]
if "dateTime" in event["dates"]["start"]:
c = event["dates"]["start"]["dateTime"].replace(
"T", " ").replace("Z", "")
else:
c = "NONE"
if "end" in event["dates"] and "dateTime" in event["dates"]["end"]:
j = event["dates"]["end"]["dateTime"].replace(
"T", " ").replace("Z", "")
else:
j = "NONE"
(Code that creates, opens, and writes to the cache used in the above code)
CACHE_FNAME = "ticketmaster_cache.json"
try:
cache_file = open(CACHE_FNAME, "r")
cache_contents = cache_file.read()
CACHE_DICTION = json.loads(cache_contents)
cache_file.close()
except:
CACHE_DICTION = {}
The previous revision of the get_event_info function shown below which does not produce any TypeError:
def get_event_info(search, ticketmaster_key = ticketmaster_key):
if search in CACHE_DICTION:
d = CACHE_DICTION[search]
else:
data = requests.get("https://app.ticketmaster.com/discovery/v2/events",
params = {"keyword": search, "apikey": ticketmaster_key,
"format":"json", "dmaId": "366", "size": 200, "radius": "2"})
print(data.url)
d = json.loads(data.text)
CACHE_DICTION[search] = d
f = open(CACHE_FNAME, 'w')
f.write(json.dumps(CACHE_DICTION))
f.close()
return d
Traceback & Error message I see when I run the latest revision of the code:
Traceback (most recent call last):
File "ticketmaster_only_w_headers.py", line 146, in <module>
for event in ticket_search["_embedded"]["events"]:
TypeError: 'NoneType' object is not subscriptable

Whenever you have a function that can explicitly return None, you should always check the return value first:
def func(a):
if a == 1:
return list(range(10)) # could return a list
else:
return None # or it could return None
a = 10
f = func(a)
f[1]
# raises TypeError: NoneType is not subscriptable
# check for NoneType first
if f is not None:
print(f[1])
# otherwise, kick out different result
else:
print('Got "None" for f!')
# Got "None" for f!
Your ticket_search is returned as None, but because your for loop is trying to do a key-lookup, it's failing, because None doesn't support that operation. Your logic, following from the above, should look like:
if ticket_search is not None:
for event in ticket_search["_embedded"]["events"]:
a = event["id"]
else:
raise TypeError
# or do something else

Well, the interpreter is explicitly telling you that you are trying to evaluate something like a[i], where a is None (instead of the intended type, like a list or a dict). In your case, it is either ticket_search itself, or ticket_search["_embedded"].
In any case, if you can rerun your code at all, putting a print(ticket_search) under ticket_search = get_event_info("") should make everything clear.

Python code in Zapier (invalid syntax (usercode.py, line 42))

This code is pre-made in a Zapier forum to pull failed responses from another piece of software called iAuditor. When I plug in the code and update the API token and webhook URL this error pops up:
Traceback (most recent call last):
SyntaxError: invalid syntax (usercode.py, line 42)
Here is the code:
[code]
import json
import requests
auth_header = {'Authorization': 'a4fca847d3f203bd7306ef5d1857ba67a2b3d66aa455e06fac0ad0be87b9d226'}
webhook_url = 'https://hooks.zapier.com/hooks/catch/3950922/efka9n/'
api_url = 'https://api.safetyculture.io/audits/'
audit_id = input['audit_id']
audit_doc = requests.get(api_url + audit_id, headers=auth_header).json()
failed_items = []
audit_author = audit_doc['audit_data']['authorship']['author']
conducted_on = audit_doc['audit_data']['date_completed']
conducted_on = conducted_on[:conducted_on.index('T')]
audit_title = audit_doc['template_data']['metadata']['name']
for item in audit_doc['items']:
if item.get('responses') and item['responses'].get('failed') == True:
label = item.get('label')
if label is None:
label = 'no_label'
responses = item['responses']
response_label = responses['selected'][0]['label']
notes = responses.get('text')
if notes is None:
notes = ''
failed_items.append({'label': label,
'response_label': response_label,
'conducted_on': conducted_on,
'notes': notes,
'author': audit_author
})
for item in failed_items:
r = requests.post(webhook_url, data = item)
return response.json()
[/code]

This looks like an error from the platform. It looks like Zapier uses a script called usercode.py to bootstrap launching your script and the error seems to be coming from that part.

Value error: view didn't return an HttpResponse object. It returned None instead Django

I'm yet to understand why I'm getting 'HttpResponse' error.
Traceback (most recent call last):
File "C:\Python27\Scripts\covaenv\lib\site-packages\django\core\handlers\exception.py", line 42, in inner
response = get_response(request)
File "C:\Python27\Scripts\covaenv\lib\site-packages\django\core\handlers\base.py", line 198, in _get_response
"returned None instead." % (callback.__module__, view_name)
ValueError: The view exampleapp.views.get_recieve_update didn't return an HttpResponse object. It returned None instead.
This view is responsible for getting a POST request from an API and load the data and do things with it.
Views:
#csrf_exempt
def get_recieve_update(request):
if request.method=="POST":
man= json.loads(request.body)
txId = man['hash']
uri = bgo_main_base_url + '/wallet/{}/tx/{}'.format(WALLETID, txId)
rexa = requests.get(uri, headers=headers)
vd = rexa.json()
isMine = vd['outputs'][0]['isMine']
confirmations = vd['confirmations']
if isMine == True and confirmations > 1:
address = vd['outputs'][0]['account']
value = vd['outputs'][0]['value']
try:
get_adr = CPro.objects.get(address = address)
except CPro.DoesNotExist:
get_adr = None
if not get_adr.is_used==True and get_adr.is_active==False:
update_cw = CW.objects.filter(user =
get_adr.user).update(current_btc_balance=F('current_btc_balance') + value , modified_date=datetime.datetime.now())
return HttpResponse('done')
elif get_adr.is_used==True and get_adr.is_active==False:
address = vd['outputs'][0]['account']
value = vd['outputs'][0]['value']
send_mail('Recieved on Used Address','failed to credit for {} with {} and id {}'.format(address, value, txId), DEFAULT_FROM_EMAIL,[DE_MAIL,])
else:
address = vd['outputs'][0]['account']
value = vd['outputs'][0]['value']
send_mail('Recieved Callback Error','failed to credit for {} with {}'.format(address, value), DEFAULT_FROM_EMAIL,[DE_MAIL,])
What am I missing here?

You need to return an HttpResponse on every condition.In last if else statement you can see you are not returning anything from the view so you have to return an appropriate http response for every case in your view. See updated code below.
#csrf_exempt
def get_recieve_update(request):
if request.method=="POST":
man= json.loads(request.body)
txId = man['hash']
uri = bgo_main_base_url + '/wallet/{}/tx/{}'.format(WALLETID, txId)
rexa = requests.get(uri, headers=headers)
vd = rexa.json()
isMine = vd['outputs'][0]['isMine']
confirmations = vd['confirmations']
if isMine == True and confirmations > 1:
address = vd['outputs'][0]['account']
value = vd['outputs'][0]['value']
try:
get_adr = CPro.objects.get(address = address)
except CPro.DoesNotExist:
get_adr = None
if not get_adr.is_used==True and get_adr.is_active==False:
update_cw = CW.objects.filter(user =
get_adr.user).update(current_btc_balance=F('current_btc_balance') + value , modified_date=datetime.datetime.now())
return HttpResponse('done')
elif get_adr.is_used==True and get_adr.is_active==False:
address = vd['outputs'][0]['account']
value = vd['outputs'][0]['value']
send_mail('Recieved on Used Address','failed to credit for {} with {} and id {}'.format(address, value, txId), DEFAULT_FROM_EMAIL,[DE_MAIL,])
return HttpResponse("Some appropriate response")
else:
# return something. If both condition from does not get true then there will be no return from view
else:
address = vd['outputs'][0]['account']
value = vd['outputs'][0]['value']
send_mail('Recieved Callback Error','failed to credit for {} with {}'.format(address, value), DEFAULT_FROM_EMAIL,[DE_MAIL,])
return HttpResponse("Some appropriate response") # <-- here you were not returning a response
Another Helpful answer

Exception Value: No JSON object could be decoded

I am using postman to test if my django app call is working as expected but it is not.
I get the following error:
ValueError at /mobile/emulate_create</title>
Exception Value: No JSON object could be decoded
I have the following code:
#csrf_exempt
def emulate_create(request):
args = json.loads(request.body, object_hook=utils._datetime_decoder)
resourceId, count, hours = args['resourceId'], args['count'], args['hours']
now = datetime.now()
remainder = now.minute % 15
timeFrom = now + timedelta(minutes=remainder)
timeTo = timeFrom + timedelta(hours=hours)
reservation = ReservationProspect(byUser=request.user, forUser=request.user, resource=get_object_or_404(Resource, uuid=resourceId), modality=get_object_or_404(Modality, name="online"), timeFrom=timeFrom, timeTo=timeTo, count=count)
return HttpResponse(json.dumps[reservation.toDict()])
Does anyone see what might be happening here?