I am streaming data from a sensor using socket library in Json format, and trying to parse it and load it into Database.
When I print the stream I get this Json in this format:
b'[{"metadata":{"timezone":{"location":"Etc/UTC"},"serial_number":"00:07:32:52:09:fc","device_type":"SPIDER"},"timestamp":"2019-08-29T13:53:05.895Z","framenumber":"2290718","tracked_objects":[{"id":2592,"is_at_border":true,"type":"PERSON","position":{"x":233,"y":262,"type":"FOOT","coordinate_system":"PROCESSING_IN_PIXEL"},"person_data":{"height":1728}}]}]'
Based on my research prefix b stands for the byte type. So when I try to parse it with code below:
while True:
message, address = server_socket.recvfrom(1024)
message = message.upper()
# loading json file.
objs_json = json.loads(message)
# using if looop to prevent script of trying to to parse data without any object being tracked.
if "tracked_objects" in objs_json:
# Parsing json file with json_normalize object
objs_df = json_normalize(
objs_json, record_path='tracked_objects',
meta=[['metadata', 'serial_number'], 'timestamp']
)
# Renaming columns
objs_df = objs_df.rename(
columns={
"id": "object_id", "position.x": "x_pos",
"position.y": "y_pos", "person_data.height": "height",
"metadata.serial_number": "serial_number",
"timestamp": "timestamp"
}
)
# Selecting columns of interest
objs_df = objs_df.loc[:, ["timestamp", "serial_number", "object_id", "x_pos", "y_pos", "height"]]
# Writting the data into SQlite db
objs_df.to_sql('data_object', con=engine, if_exists='append', index=False)
# In case there is no tracks, print No Tracks in console.
else:
print("No Tracks")
I get this error message:
Traceback (most recent call last):
File "/home/pi/ProRail-PMS/Test_Spider2.py", line 20, in <module>
objs_json = json.loads(message)
File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 215 (char 214)
However if I save that data into json file and remove the prefix b my parsing code works.
How do i go around this so when I receive the data from socket library I want to be able to parse it and feed it into database?
At first I wanted to comment that it works for me, but then I noticed how you get the message and what you do with it:
Remove message = message.upper():
>>> message = b'[{"metadata":{"timezone":{"location":"Etc/UTC"},"serial_number":"00:07:32:52:09:fc","device_type":"SPIDER"},"timestamp":"2019-08-29T13:53:05.895Z","framenumber":"2290718","tracked_objects":[{"id":2592,"is_at_border":true,"type":"PERSON","position":{"x":233,"y":262,"type":"FOOT","coordinate_system":"PROCESSING_IN_PIXEL"},"person_data":{"height":1728}}]}]'
>>> json.loads(message)
[{'metadata': {'timezone': {'location': 'Etc/UTC'}, 'serial_number': '00:07:32:52:09:fc', 'device_type': 'SPIDER'}, 'timestamp': '2019-08-29T13:53:05.895Z', 'framenumber': '2290718', 'tracked_objects': [{'id': 2592, 'is_at_border': True, 'type': 'PERSON', 'position': {'x': 233, 'y': 262, 'type': 'FOOT', 'coordinate_system': 'PROCESSING_IN_PIXEL'}, 'person_data': {'height': 1728}}]}]
>>
>>
>>> message = message.upper()
b'[{"METADATA":{"TIMEZONE":{"LOCATION":"ETC/UTC"},"SERIAL_NUMBER":"00:07:32:52:09:FC","DEVICE_TYPE":"SPIDER"},"TIMESTAMP":"2019-08-29T13:53:05.895Z","FRAMENUMBER":"2290718","TRACKED_OBJECTS":[{"ID":2592,"IS_AT_BORDER":TRUE,"TYPE":"PERSON","POSITION":{"X":233,"Y":262,"TYPE":"FOOT","COORDINATE_SYSTEM":"PROCESSING_IN_PIXEL"},"PERSON_DATA":{"HEIGHT":1728}}]}]'
>>> json.loads(message)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 217 (char 216)
>>> message[217:]
b'RUE,"TYPE":"PERSON","POSITION":{"X":233,"Y":262,"TYPE":"FOOT","COORDINATE_SYSTEM":"PROCESSING_IN_PIXEL"},"PERSON_DATA":{"HEIGHT":1728}}]}]'
Your upper breaks the True value that is unquoted (because it's a boolean, not a string). ;)
Imported it into Notepad++ since the JSON file is of type byte. Saved it out as JSON and it saved normally as a standard JSON file. Used Python json.load(f) to bring it in from my drive and then it can be used.
Related
I am working on a project where I have to scrape data from a website.I get 200 response while i run the code without json. But i am facing raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) while displaying the json response.
here is my code:
import requests
import json
PARAMS = {"customCrawlParam": True, "categoryId": -11, "pageNumber": 1, "pageSize": 60, "crawlerInfo": "0aoAfanUmycYY9dVkR6C_tawcS4PTcat2tyeioefYJvTz-K_x_HTVGslqTelRkU5hNZUrTG6C2zGn-abo8Er2xr4oH-1xPuM7nyLIAJYOdY9lYQ3tCpr2VOBopWuu2iTCrAWW-nJ2I2nqdWxLrtzvWtwtAJkQgWNR7v6DA_Xg_8_bEjpDUezOkiknnz-17XSLeTXUzaO5EaIYv1epxQef3rsGabBJyl4TsJZGfd2Rj70huRosXUJxmjSNTFaBDX3jQ1c1WgOkF0HXXEuwCmS7_VCoWo0tYqPiJhDmsZ6VQB476mxPUQVmXm6UzmdIZ6t16Ov7wmaTlY18KoO00V9jIZJs8wx_q5s5lampf7saWD5wsX0EfnWBkSo1vMacbCNkAVKrIMVViGjrtxXmstaGZ_uDoHuOeV4_RTWMX_F-NjlT3G0XRBPEJBiW_5D6U_76LcqHApmDJ130DHuynsOrYu7_k0IGoet8SAA6wAVElfMY8-Hjc3rZLr061S7SGxHC7y0uJAy3NMHr_RWBRu4lsWFzzy0ZLlUN6S6i7eWgP"}
r = requests.get(
"https://youhui.pinduoduo.com/search/landing?catId=-11", json=PARAMS)
print(r.status_code)
js1 = r.json()
# # data1 = json.dumps(js1, indent=4)
# #data2 = json.loads(r.decode("utf-8"))
print(js1)
here is the output i get:
200
Traceback (most recent call last):
File "e:\PROJECTS\first earn\test.py", line 13, in <module>
js1 = r.json()
File "C:\Users\Asus\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "C:\Users\Asus\AppData\Local\Programs\Python\Python38\lib\json\__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "C:\Users\Asus\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\Asus\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
In params I wrote True instead of JSON true . Is it causing the problem?
It's not working because the response is not a json. So you can't load it as a json. As simple as that.
Do r.text to see the response yourself.
i am trying to create blog page using flask and i want to take input file as a config.json which i have created. Please help me i am getting json decoder error
i also tried to convert str to utf8 but its showing the error
with open('config.json', 'r', encoding ='utf-8') as c:
params = json.load(c)["params"]
json content:
{
"params":
{
"local_server":"True",
"local_uri":"mysql://root:#localhost/codingthunder",
"prod_uri":"mysql://root:#localhost/codingthunder",
"fb_url":"https://facebook.com/codingthunder",
"tw_url":"https://twitter.com/codingthunder",
"gh_url":"https://github.com/codingthunder",
"blog_name":"Coding Thunder",
"tag_line":"A Blog liked by Programmers"
}
}
output log:
PS C:\Users\ASHISH\Desktop\Coding\Flask> python -u "c:\Users\ASHISH\Desktop\Coding\Flask\Blog Page\main.py"
Traceback (most recent call last):
File "c:\Users\ASHISH\Desktop\Coding\Flask\Blog Page\main.py", line 7, in <module>
params = json.load(c)["params"]
File "C:\Users\ASHISH\AppData\Local\Programs\Python\Python37\lib\json\__init__.py", line 296, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "C:\Users\ASHISH\AppData\Local\Programs\Python\Python37\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\ASHISH\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\ASHISH\AppData\Local\Programs\Python\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0) hints that there is no value at the beginning of the file, so no content in the parsed file.
This could mean, that
the file is empty
you opened the wrong or a non existing file
you are using a relative path and your working direktory is wrong (e.g. the direktory you execute your program in)
I want to read a JSON file with Python :
Here is part of my JSON file :
{ "Jointure":[ { "IDJointure":1, "societe":"S.R.T.K", "date":"2019/01/01", "heure":"05:47:00"}, { "IDJointure":2, "societe":"S.R.T.K", "date":"2019/01/01", "heure":"05:50:00"}]}
This is the code :
import json
data = json.loads('Data2019.json')
for i in data['Jointure']:
print(i)
But, here is the error that was displayed
Traceback (most recent call last):
File "C:\Users\HP\Desktop\readJSON.py", line 4, in <module>
data = json.loads('Data2019.json')
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\json\__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\HP\AppData\Local\Programs\Python\Python38\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
>>>
json.loads() expects the json data to already be a string -- so it's trying to interpret the filename Data2019.json as actual json data.
Open the file, and then pass the file object to json.load():
with open('Data2019.json') as fp:
data = json.load(fp)
don't read the file directly. Open the file it's only the contents of the file that works with the json module. Try this:
import json
with open('path_to_file/person.json') as f:
data = json.load(f)
Try pandas
import pandas as pd
patients_df = pd.read_json('E:/datasets/patients.json')
patients_df.head()
i am a little bit new to coding, but i am trying to get data stored in a json file by another script though it keeps on giving me errors
This is my code
import requests
URL = "http://127.0.0.1:5000/predict"
TEST_AUDIO_FILE_PATH = "test/soma.wav"
if __name__ =="__main__":
audio_file = open(TEST_AUDIO_FILE_PATH, "rb")
values = {"file":(TEST_AUDIO_FILE_PATH, audio_file, "audio/wav")}
response = requests.post(URL, files=values)
data = response.json()
print(f"Predicted keyword is: {data['keyword']}")
This is the error i keep on getting
File "C:\Users\Tatooine\Desktop\FYP\client.py", line 11, in
response.json()
File "C:\ProgramData\Anaconda3\lib\site-packages\requests\models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "C:\ProgramData\Anaconda3\lib\json__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\ProgramData\Anaconda3\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\ProgramData\Anaconda3\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
JSONDecodeError: Expecting value
i think you should decode the data variable first
data = response.decode()
then after you can convert it to a json object
import json
data = json.loads(data)
then print
print(f"Predicted keyword is: {data['keyword']}")
please let me know, if this works for you
I'm a newbie into Python and i'm trying to open a json file that looks like this:
{
"empty_char":"",
"empty_number": ,
"char":"i'm a char",
"number":0
}
Let's say that this is the test.json file:
When trying to open it in python I'm using:
import json
with open('test.json') as f:
data = json.load(f)
and it raises the following error:
Traceback (most recent call last):
File "<input>", line 2, in <module>
File "C:\ProgramData\Anaconda3\Lib\json\__init__.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "C:\ProgramData\Anaconda3\Lib\json\__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "C:\ProgramData\Anaconda3\Lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\ProgramData\Anaconda3\Lib\json\decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 3 column 19 (char 39)
It gives an error because of "empty_number": ,
How can I deal with this error if I can't modify my .json file?
You can use the more flexible yaml to read in values not provided as None.
import yaml
from io import StringIO
mystr = StringIO("""{
"empty_char":"",
"empty_number": ,
"char":"i'm a char",
"number":0
}""")
# replace mystr with open('test.json', 'r')
with mystr as stream:
res = yaml.load(stream)
print(res, type(res))
{'empty_char': '', 'empty_number': None, 'char': "i'm a char", 'number': 0}
<class 'dict'>
Note io.StringIO lets us read from a string as if it were a file.