I am working on a project where I have to scrape data from a website.I get 200 response while i run the code without json. But i am facing raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) while displaying the json response.
here is my code:
import requests
import json
PARAMS = {"customCrawlParam": True, "categoryId": -11, "pageNumber": 1, "pageSize": 60, "crawlerInfo": "0aoAfanUmycYY9dVkR6C_tawcS4PTcat2tyeioefYJvTz-K_x_HTVGslqTelRkU5hNZUrTG6C2zGn-abo8Er2xr4oH-1xPuM7nyLIAJYOdY9lYQ3tCpr2VOBopWuu2iTCrAWW-nJ2I2nqdWxLrtzvWtwtAJkQgWNR7v6DA_Xg_8_bEjpDUezOkiknnz-17XSLeTXUzaO5EaIYv1epxQef3rsGabBJyl4TsJZGfd2Rj70huRosXUJxmjSNTFaBDX3jQ1c1WgOkF0HXXEuwCmS7_VCoWo0tYqPiJhDmsZ6VQB476mxPUQVmXm6UzmdIZ6t16Ov7wmaTlY18KoO00V9jIZJs8wx_q5s5lampf7saWD5wsX0EfnWBkSo1vMacbCNkAVKrIMVViGjrtxXmstaGZ_uDoHuOeV4_RTWMX_F-NjlT3G0XRBPEJBiW_5D6U_76LcqHApmDJ130DHuynsOrYu7_k0IGoet8SAA6wAVElfMY8-Hjc3rZLr061S7SGxHC7y0uJAy3NMHr_RWBRu4lsWFzzy0ZLlUN6S6i7eWgP"}
r = requests.get(
"https://youhui.pinduoduo.com/search/landing?catId=-11", json=PARAMS)
print(r.status_code)
js1 = r.json()
# # data1 = json.dumps(js1, indent=4)
# #data2 = json.loads(r.decode("utf-8"))
print(js1)
here is the output i get:
200
Traceback (most recent call last):
File "e:\PROJECTS\first earn\test.py", line 13, in <module>
js1 = r.json()
File "C:\Users\Asus\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\Asus\AppData\Local\Programs\Python\Python38\lib\json\__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "C:\Users\Asus\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Asus\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
In params I wrote True instead of JSON true . Is it causing the problem?
It's not working because the response is not a json. So you can't load it as a json. As simple as that.
Do r.text to see the response yourself.
I've looked for similar questions, but I couldn't find anything what can help me. When i execute this code (which should work) I get "json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)"
TOKEN = '_EXAMPLE_TOKEN_'
def getPagedData(start,size):
url = 'http://EXAMPLE_URL'.format(start, size)
response = requests.get(url, headers={'token': TOKEN}, verify=False)
return response
def getDataInBatches():
start = 0
size = 1000
allData = []
data = getPagedData(start, size)
allData.extend(data.json()['data'])
total = data.json()['count']
for i in range(1, round(total/size) + 1):
print(allData[len(allData) - 1])
allData.extend(getPagedData(i * size, size).json()['data'])
return allData
print(getDataInBatches())
Here full traceback:
Traceback (most recent call last):
File "C:/Users/userxy/Documents/workProject/API/Get_Information.py", line 30, in <module>
print(getDataInBatches())
File "C:/Users/userxy/Documents/workProject/API/Get_Information.py", line 23, in getDataInBatches
allData.extend(data.json()['data'])
File "C:\Users\userxy\Anaconda3\lib\site-packages\requests\models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\userxy\Anaconda3\lib\json\__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "C:\Users\userxy\Anaconda3\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\userxy\Anaconda3\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
What can it be?
Make sure the response is in json format. The standard way to ensure the response is in json format is to add a headers={'Content-Type': 'application/json'}
But to be sure, check the documentation of an API you're accessing.
I am streaming data from a sensor using socket library in Json format, and trying to parse it and load it into Database.
When I print the stream I get this Json in this format:
b'[{"metadata":{"timezone":{"location":"Etc/UTC"},"serial_number":"00:07:32:52:09:fc","device_type":"SPIDER"},"timestamp":"2019-08-29T13:53:05.895Z","framenumber":"2290718","tracked_objects":[{"id":2592,"is_at_border":true,"type":"PERSON","position":{"x":233,"y":262,"type":"FOOT","coordinate_system":"PROCESSING_IN_PIXEL"},"person_data":{"height":1728}}]}]'
Based on my research prefix b stands for the byte type. So when I try to parse it with code below:
while True:
message, address = server_socket.recvfrom(1024)
message = message.upper()
# loading json file.
objs_json = json.loads(message)
# using if looop to prevent script of trying to to parse data without any object being tracked.
if "tracked_objects" in objs_json:
# Parsing json file with json_normalize object
objs_df = json_normalize(
objs_json, record_path='tracked_objects',
meta=[['metadata', 'serial_number'], 'timestamp']
)
# Renaming columns
objs_df = objs_df.rename(
columns={
"id": "object_id", "position.x": "x_pos",
"position.y": "y_pos", "person_data.height": "height",
"metadata.serial_number": "serial_number",
"timestamp": "timestamp"
}
)
# Selecting columns of interest
objs_df = objs_df.loc[:, ["timestamp", "serial_number", "object_id", "x_pos", "y_pos", "height"]]
# Writting the data into SQlite db
objs_df.to_sql('data_object', con=engine, if_exists='append', index=False)
# In case there is no tracks, print No Tracks in console.
else:
print("No Tracks")
I get this error message:
Traceback (most recent call last):
File "/home/pi/ProRail-PMS/Test_Spider2.py", line 20, in <module>
objs_json = json.loads(message)
File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 215 (char 214)
However if I save that data into json file and remove the prefix b my parsing code works.
How do i go around this so when I receive the data from socket library I want to be able to parse it and feed it into database?
At first I wanted to comment that it works for me, but then I noticed how you get the message and what you do with it:
Remove message = message.upper():
>>> message = b'[{"metadata":{"timezone":{"location":"Etc/UTC"},"serial_number":"00:07:32:52:09:fc","device_type":"SPIDER"},"timestamp":"2019-08-29T13:53:05.895Z","framenumber":"2290718","tracked_objects":[{"id":2592,"is_at_border":true,"type":"PERSON","position":{"x":233,"y":262,"type":"FOOT","coordinate_system":"PROCESSING_IN_PIXEL"},"person_data":{"height":1728}}]}]'
>>> json.loads(message)
[{'metadata': {'timezone': {'location': 'Etc/UTC'}, 'serial_number': '00:07:32:52:09:fc', 'device_type': 'SPIDER'}, 'timestamp': '2019-08-29T13:53:05.895Z', 'framenumber': '2290718', 'tracked_objects': [{'id': 2592, 'is_at_border': True, 'type': 'PERSON', 'position': {'x': 233, 'y': 262, 'type': 'FOOT', 'coordinate_system': 'PROCESSING_IN_PIXEL'}, 'person_data': {'height': 1728}}]}]
>>
>>
>>> message = message.upper()
b'[{"METADATA":{"TIMEZONE":{"LOCATION":"ETC/UTC"},"SERIAL_NUMBER":"00:07:32:52:09:FC","DEVICE_TYPE":"SPIDER"},"TIMESTAMP":"2019-08-29T13:53:05.895Z","FRAMENUMBER":"2290718","TRACKED_OBJECTS":[{"ID":2592,"IS_AT_BORDER":TRUE,"TYPE":"PERSON","POSITION":{"X":233,"Y":262,"TYPE":"FOOT","COORDINATE_SYSTEM":"PROCESSING_IN_PIXEL"},"PERSON_DATA":{"HEIGHT":1728}}]}]'
>>> json.loads(message)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 217 (char 216)
>>> message[217:]
b'RUE,"TYPE":"PERSON","POSITION":{"X":233,"Y":262,"TYPE":"FOOT","COORDINATE_SYSTEM":"PROCESSING_IN_PIXEL"},"PERSON_DATA":{"HEIGHT":1728}}]}]'
Your upper breaks the True value that is unquoted (because it's a boolean, not a string). ;)
Imported it into Notepad++ since the JSON file is of type byte. Saved it out as JSON and it saved normally as a standard JSON file. Used Python json.load(f) to bring it in from my drive and then it can be used.
If I go for example from urlib2
import urllib.request
response = urllib.request.urlopen('http://python.org/')
html = response.read()
then it works fine.
I was suprised to see that my code
req = urllib.request.Request('https://www.tehnomanija.rs/it-shop/laptop-racunari')
response = urllib.request.urlopen(req)
the_page = response.read()
data = json.loads(the_page)
print (data)
produces error like this
data = json.loads(the_page)
File "/home/mm/anaconda3/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/home/mm/anaconda3/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/home/mm/anaconda3/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Except the other protocol I do not see why this would not work.
json.load() is for loading from file. You need json.loads():
data = json.loads(the_page)
I'm trying to run this code to create a data frame from a JSON link. Sometimes, the code will run. Other times, I will get an error message (below). I'm not sure why this occurs, even though the code is the same.
import requests
import json
url = "http://stats.nba.com/stats/leaguedashplayerstats?College=&Conference=&Country=&DateFrom=&DateTo=&Division=&DraftPick=&DraftYear=&GameScope=&GameSegment=&Height=&LastNGames=0&LeagueID=00&Location=&MeasureType=Advanced&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PaceAdjust=N&PerMode=Totals&Period=0&PlayerExperience=&PlayerPosition=&PlusMinus=N&Rank=N&Season=2016-17&SeasonSegment=&SeasonType=Regular+Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=&Weight="
jd = requests.get(url).json()
df = []
for item in requests.get(url).json()['resultSets']:
print("got here")
row_df = []
for row in item['rowSet']:
row_df.append(str(row).strip('[]'))
df.append("\n")
df.append(row_df)
print(df)
Error Message:
Traceback (most recent call last):
File "/Users/K/PycharmProjects/mousefun/fun", line 8, in <module>
jd = requests.get(url).json()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/models.py", line 812, in json return complexjson.loads(self.text, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/__init__.py", line 318, in loads return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 343, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/json/decoder.py", line 361, in raw_decode raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)
Change your request logic to this and try again:
r = requests.get(url)
r.raise_for_status()
df = []
for item in r.json()["resultSets"]:
# ...
r.raise_for_status() will raise if the status is not OK .
Also, this does not do the request two times like your code does.