I'm trying to run this code to create a data frame from a JSON link. Sometimes, the code will run. Other times, I will get an error message (below). I'm not sure why this occurs, even though the code is the same.
import requests
import json
url = "http://stats.nba.com/stats/leaguedashplayerstats?College=&Conference=&Country=&DateFrom=&DateTo=&Division=&DraftPick=&DraftYear=&GameScope=&GameSegment=&Height=&LastNGames=0&LeagueID=00&Location=&MeasureType=Advanced&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=Totals&Period=0&PlayerExperience=&PlayerPosition=&PlusMinus=N&Rank=N&Season=2016-17&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=&Weight="
jd = requests.get(url).json()
df = []
for item in requests.get(url).json()['resultSets']:
print("got here")
row_df = []
for row in item['rowSet']:
row_df.append(str(row).strip('[]'))
df.append("\n")
df.append(row_df)
print(df)
Error Message:
Traceback (most recent call last):
File "/Users/K/PycharmProjects/mousefun/fun", line 8, in <module>
jd = requests.get(url).json()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/models.py", line 812, in json return complexjson.loads(self.text, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/__init__.py", line 318, in loads return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 343, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 361, in raw_decode raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)
Change your request logic to this and try again:
r = requests.get(url)
r.raise_for_status()
df = []
for item in r.json()["resultSets"]:
# ...
r.raise_for_status() will raise if the status is not OK .
Also, this does not do the request two times like your code does.
Related
I am trying to get data from the below seen URLs
import requests
import json
service_a = requests.get('https:xxxxx')
service_b = requests.get('https:xxxxx')
service_a.json()
service_b.json()
While for the case of service_a, service_a.json() gives me the expected data, for service_b, the call service_b.json() gives me the following error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3/dist-packages/requests/models.py", line 892, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 518, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
File "/usr/lib/python3/dist-packages/simplejson/scanner.py", line 79, in scan_once
return _scan_once(string, idx)
File "/usr/lib/python3/dist-packages/simplejson/scanner.py", line 70, in _scan_once
raise JSONDecodeError(errmsg, string, idx)
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Why is this happening?
ps. I notced that when I try to visit the https link of service_a, the data it contains are shown in the browser, while for sevice_b, no data are shown in the browser, but when I click it a file with its data is downloaded in my computer.
Try to check the Content-Type of service_b response.
import csv
response = requests.get(API_URL)
decoded_data = response.content.decode("utf-8")
csv_data = csv.reader(decoded_data.splitlines(), delimiters=",")
csv_list = list(csv_data)
in most cases your
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
error is due to:
non-JSON conforming quoting
XML/HTML output (that is, a string starting with <), or
incompatible character encoding
I've looked for similar questions, but I couldn't find anything what can help me. When i execute this code (which should work) I get "json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)"
TOKEN = '_EXAMPLE_TOKEN_'
def getPagedData(start,size):
url = 'http://EXAMPLE_URL'.format(start, size)
response = requests.get(url, headers={'token': TOKEN}, verify=False)
return response
def getDataInBatches():
start = 0
size = 1000
allData = []
data = getPagedData(start, size)
allData.extend(data.json()['data'])
total = data.json()['count']
for i in range(1, round(total/size) + 1):
print(allData[len(allData) - 1])
allData.extend(getPagedData(i * size, size).json()['data'])
return allData
print(getDataInBatches())
Here full traceback:
Traceback (most recent call last):
File "C:/Users/userxy/Documents/workProject/API/Get_Information.py", line 30, in <module>
print(getDataInBatches())
File "C:/Users/userxy/Documents/workProject/API/Get_Information.py", line 23, in getDataInBatches
allData.extend(data.json()['data'])
File "C:\Users\userxy\Anaconda3\lib\site-packages\requests\models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\userxy\Anaconda3\lib\json\__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "C:\Users\userxy\Anaconda3\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\userxy\Anaconda3\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
What can it be?
Make sure the response is in json format. The standard way to ensure the response is in json format is to add a headers={'Content-Type': 'application/json'}
But to be sure, check the documentation of an API you're accessing.
I tried many different ways to fix this error, but it failed.
Could you fix this error?
p.s.
formatted_url --> There's a problem that comma turns %2c when it calls request, and this way works.
https://opendart.fss.or.kr/api/fnlttMultiAcnt.json?crtfc_key=cd155118bef391723b04f8879878921476a97b9f&corp_code=01015160,00288495,00365590,01035678,00625942,00287812,00220622,00173351,00186559,00476036,00899556&bsns_year=2020&reprt_code=11013
request post --> params text is really long, and request get method occurs 414 error, so I changed it to post.
print(resp) :
<urllib3.response.HTTPResponse object at 0x7fb822c79ee0>
params.items() :
dict_items([('crtfc_key', 'cd155118bef391723b04f8879878921476a97b9f'), ('corp_code', '00260985,...........00173351,00186559,00476036,00899556'), ('bsns_year', '2020'), ('reprt_code', '11013')])
error message
enter image description here
def convertFnltt(url, items, item_names, params):
formatted_url = url + '?' + '&'.join(["{}={}".format(k,v) for k,v in params.items()])
print(formatted_url)
http = urllib3.PoolManager(ca_certs=certifi.where())
resp = http.request('POST',formatted_url)
print(resp)
print('1')
json_dict = json.load(resp) <--here!!
print('2')
data = []
#if self.get_status(json_dict['status']):
print(json_dict['status'])
if json_dict['status']=='000':
for line in json_dict['list']:
data.append([])
for itm in items:
if itm in line.keys():
data[-1].append(line[itm])
else:
data[-1].append('')
else:
return json_dict['status']
df = pd.DataFrame(data)
df.columns = item_names
return df #df.to_excel('data.xlsx')
1
Traceback (most recent call last):
File "main.py", line 106, in <module>
print(get_fnlttMultiAcnt(crtfc_key, get_corpcode(crtfc_key), bsns_year, reprt_code))
File "main.py", line 103, in get_fnlttMultiAcnt
return convertFnltt(url,items,item_names,params)
File "main.py", line 61, in convertFnltt
json_dict = json.load(resp)
File "/usr/lib/python3.8/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
It was the issue of the format of 'params'.
I used 'append' to make 'params', which caused a series of problems.
I changed 'append' to 'extend' , and it worked.
res = requests.get(url, params) <---'params format issue'
json_dict = json.loads(res.text)
data = []
for line in json_dict["list"]:
data.append([])
for itm in items:
if itm in line.keys():
data[-1].append(line[itm])
else:
data[-1].append('')
dfp = pd.DataFrame(data, columns = item_names)
i am a little bit new to coding, but i am trying to get data stored in a json file by another script though it keeps on giving me errors
This is my code
import requests
URL = "http://127.0.0.1:5000/predict"
TEST_AUDIO_FILE_PATH = "test/soma.wav"
if __name__ =="__main__":
audio_file = open(TEST_AUDIO_FILE_PATH, "rb")
values = {"file":(TEST_AUDIO_FILE_PATH, audio_file, "audio/wav")}
response = requests.post(URL, files=values)
data = response.json()
print(f"Predicted keyword is: {data['keyword']}")
This is the error i keep on getting
File "C:\Users\Tatooine\Desktop\FYP\client.py", line 11, in
response.json()
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "C:\ProgramData\Anaconda3\lib\json__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\ProgramData\Anaconda3\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\ProgramData\Anaconda3\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
JSONDecodeError: Expecting value
i think you should decode the data variable first
data = response.decode()
then after you can convert it to a json object
import json
data = json.loads(data)
then print
print(f"Predicted keyword is: {data['keyword']}")
please let me know, if this works for you
I am making API calls and getting back nested JSON response for every ID.
If I run the API call for one ID the JSON looks like this.
u'{"id":26509,"name":"ORD.00001","order_type":"sales","consumer_id":415372,"order_source":"in_store","is_submitted":0,"fulfillment_method":"in_store","order_total":150,"balance_due":150,"tax_total":0,"coupon_total":0,"order_status":"cancelled","payment_complete":null,"created_at":"2017-12-02 19:49:15","updated_at":"2017-12-02 20:07:25","products":[{"id":48479,"item_master_id":239687,"name":"QA_FacewreckHaze","quantity":1,"pricing_weight_id":null,"category_id":1,"subcategory_id":8,"unit_price":"150.00","original_unit_price":"150.00","discount_total":"0.00","created_at":"2017-12-02 19:49:45","sold_weight":10,"sold_weight_uom":"GR"}],"payments":[],"coupons":[],"taxes":[],"order_subtotal":150}'
I can successfully parse this one JSON string into a dataframe using this line of code:
order_detail = json.loads(r.text)
order_detail = json_normalize(order_detail_staging)
I can iterate all my IDs through the API using this code:
lists = []
for id in df.id:
r = requests.get("URL/v1/orders/{id}".format(id=id), headers = headers_order)
lists.append(r.text)
Now that all my JSON responses are stored in the list. How do I write all the elements into the list into a dataframe?
The code I have been trying is this:
for x in lists:
order_detail = json.loads(x)
order_detail = json_normalize(x)
print(order_detail)
I get error:
AttributeError: 'unicode' object has no attribute 'itervalues'
I know this is happening at line:
order_detail = json_normalize(x)
Why does this line work for a single JSON string but not for the list? What can I do get the list of nested JSON into a dataframe?
Thank you in advance for the help.
edit:
Traceback (most recent call last):
File "<ipython-input-108-5051d2ceb18b>", line 3, in <module>
for id in df.id
File "/Users/bob/anaconda/lib/python2.7/site-packages/requests/models.py", line 802, in json
return json.loads(self.text, **kwargs)
File "/Users/bob/anaconda/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/Users/bob/anaconda/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Users/bob/anaconda/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Traceback (most recent call last):
File "<ipython-input-108-5051d2ceb18b>", line 3, in <module>
for id in df.id
File "/Users/bob/anaconda/lib/python2.7/site-packages/requests/models.py", line 802, in json
return json.loads(self.text, **kwargs)
File "/Users/bob/anaconda/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/Users/bob/anaconda/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Users/bob/anaconda/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
use response .json() method
feed it directly to json_normalize
Example:
df = json_normalize([
requests.get("URL/v1/orders/{id}".format(id=id), headers = headers_order).json()
for id in df.id
])
UPD:
failsaife version to handle incorrect responses:
def gen():
for id in df.id:
try:
yield requests.get("URL/v1/orders/{id}".format(id=id), headers = headers_order).json()
except ValueError: # incorrect API response
pass
df = json_normalize(list(gen()))
Try this:
In [28]: lst = list(set(order_detail) - set(['products','coupons','payments','taxes']))
In [29]: pd.io.json.json_normalize(order_detail, ['products'], lst, meta_prefix='p_')
Out[29]:
category_id created_at discount_total id item_master_id name original_unit_price pricing_weight_id \
0 1 2017-12-02 19:49:45 0.00 48479 239687 QA_FacewreckHaze 150.00 None
quantity sold_weight ... p_tax_total p_order_source p_consumer_id p_payment_complete p_coupon_total \
0 1 10 ... 0 in_store 415372 None 0
p_fulfillment_method p_order_type p_is_submitted p_balance_due p_updated_at
0 in_store sales 0 150 2017-12-02 20:07:25
[1 rows x 29 columns]