Develop Reference

Reading in JSON file in Python

Keep having issue reading in JSON file. I've tried everything from changing cwd to specifying absolute path... but nothing seems to work. Here is the current code:
from dataclasses import dataclass
import json
import os
os.chdir('C:/Users/NEHRU/Desktop/New Project')
cwd = os.getcwd() # Get the current working directory (cwd)
files = os.listdir(cwd) # Get all the files in that directory
print("Files in %r: %s" % (cwd, files))
with open('C:\\Users\\NEHRU\\Desktop\\New Project\\identities.json') as f:
data = json.load(f)
print(data)
And this is the output I keep getting:
PS C:\Users\NEHRU> & C:/Users/NEHRU/AppData/Local/Microsoft/WindowsApps/python3.10.exe "c:/Users/NEHRU/Desktop/New Project/import json.py"
Files in 'C:\\Users\\NEHRU\\Desktop\\New Project': ['collection.json', 'identities.json', 'import json.py', 'Project Instructions']
Traceback (most recent call last):
File "c:\Users\NEHRU\Desktop\New Project\import json.py", line 12, in <module>
data = json.load(f)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\json\__init__.py", line 293, in load
return loads(fp.read(),
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 27 column 3 (char 238)
PS C:\Users\NEHRU>

Rospy JSONDecodeError when loading from file with json.load()

I'm trying to store the data published on a topic to a JSON file, but I keep getting a JSONDecodeError.
DB = '/home/path/data.json'
f = open(DB, 'w+')
json_array = json.load(fp=f)
json_array.append(data)
json.dump(json_array, f)
f.close()
The open() command successfully creates the file. But loading fails. I have tried running it with the file contents: [] and {}, both gave the same Exception:
[ERROR] [1637699609.562673]: bad callback: <function callback at 0x7fe36196b1f0>
Traceback (most recent call last):
File "/opt/ros/noetic/lib/python3/dist-packages/rospy/topics.py", line 750, in _invoke_callback
cb(msg)
File "/home/path/scripts/extract_info_node.py", line 43, in callback
json_array = json.load(fp=f)
File "/usr/lib/python3.8/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
It's not really descriptive and I have nothing to go on. Is there something about the way subscribers are run that doesn't allow writing? Can you even do this from a callback function? I have just started working in ROS so it could be something simple or obvious to someone more experienced.

.load() takes a .read() supporting file. You're only opening the file for reading. Instead try this:
f = open(DB, 'r+')
json_array = json.load(f)
As another note, if you're storing and re-reading topic data I'd suggest looking potentially using rosbag. This is, however, dependent on your actual application.

`r+`, `a+`, `w+` for doing both reading and writing file in python

I have a file named dict_file.json in current folder with empty dict content {}
I want to open it such that I can do both read and modify the content. That is I will read the json file as dict, modify that dict and write it back to json file.
I tried r and r+ below:
import json
f = open('dict_file.json', 'r') # same output for r+
json.loads(list_file.read())
This prints
{}
When I tried w+:
f = open('dict_file.json', 'r')
this first clears the file. Then,
json.loads(list_file.read())
gives error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "F:\ProgramFiles\Python37\lib\json\__init__.py", line 296, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "F:\ProgramFiles\Python37\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "F:\ProgramFiles\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "F:\ProgramFiles\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
When I tried a+:
f = open('dict_file.json', 'a+')
json.loads(list_file.read())
gives error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "F:\ProgramFiles\Python37\lib\json\__init__.py", line 296, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "F:\ProgramFiles\Python37\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "F:\ProgramFiles\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "F:\ProgramFiles\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Though, it does not clear the file.
So, I guess I should be using r+ for my usecase scenario. Also I tried reading writing both :
f = open('dict_file.json', 'r+')
a = json.loads(list_file.read())
a = {'key':'value'} # this involves complex logic instead of plain assignment
json.dump(a,f)
f.flush()
Now when I open the file, its contents are weird:
{}{'key':'value'}
whereas I want it to be:
{'key':'value'}
So I have few questions:
Q1. Why w+ and a+ gives error?
Q2. Why file contained {}{'key':'value'} instead of {'key':'value'}?
Q3. How to correctly do reading and writing (with or without with block)?
PS: I am reading file, then running some loop which computes new dict and write it to file. Then loop sleeps for some time and repeats the same. Thats why I felt flush() will be correct here. That is I open file once outside loop and only flush inside the loop. No need to open file in each iteration.

Why w+ and a+ gives error?
Because the file reader is pointed at the end of the file, where there's no json object to read
its contents are weird
Seems like you're not resetting the file to an actual valid JSON object throughout your tests, and so you've managed to clear the file, read nothing, maybe write some other object, clear it again, then append one or two objects to it, etc etc.
Keep it simple. Start over
filename = "data.json"
with open(filename) as f:
data = json.load(f)
data["foo"] = "bar"
with open(filename, "w") as f:
json.dump(data, f)

You have to think in terms of what the operations do to a file when they open it:
r and r+ open a file for reading. This means that the contents of the file is intact, and the file pointer is the beginning of the file. The entire file is visible to the reader. After reading, the file pointer will be at the end, meaning that you're effectively appending at that point.
w and w+ open a file for writing. That means that the contents of the file is truncated. There is nothing to be read in, since the original contents is destroyed.
a and a+ open the file for appending. The contents of the file are unchanged. However, the file pointer is at the end, so a reader will see no data and raise an error.
The correct way to do this is to open the file twice, and use with blocks to do it:
with open('dict_file.json', 'r') as f:
a = json.load(f)
# you don't need the file to be open here
a = {'key': 'value'}
with open('dict_file.json', 'w') as f:
json.dump(a, f)
You might be tempted to try to open the file in read-write mode (r+), read it, then rewind it, so the pointer returns to the beginning, and then overwrite. There are two reasons not to do this:
Not every stream is rewindable
If the contents of the file decreases in size, you will end up with trash at the end. You would need to truncate to the current pointer after writing, which is an unnecessary layer of complexity

How can I open a json file from another folder?

I am trying to make a code in python that imports a JSON file from a folder into my program in order for it to access the data inside. However, I am facing errors
global bees
with open('data/bees.json') as f:
bees = json.load(f)["bees"]
Where in the data/bees.json I have this:
{
"bees": []
}
The error I get
Traceback (most recent call last):
File "d:/Entertainment/Coding/Python/Pygame/BUG WORLD/main.py", line 70, in <module>
bees = json.load(f)
File "C:\Users\ernes\AppData\Local\Programs\Python\Python37-32\lib\json\__init__.py", line 296, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "C:\Users\ernes\AppData\Local\Programs\Python\Python37-32\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\ernes\AppData\Local\Programs\Python\Python37-32\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\ernes\AppData\Local\Programs\Python\Python37-32\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Is there a way I am able to import JSON files from another folder without causing any errors?? Please help

Looks like you're on Windows, so:
with open(r'c:\path\to\file\bees.json') as json_file:
bees_js = json.load(json_file)

The relative path should be open("data/bees.json") without the /, starting a path with / means absolute path from the root.

You might want to try the following:
import json
with open('path_to_file/bees.json') as json_file:
bees_js = json.load(json_file)

Python json.load(file) error with valid JSON

I have a question concerning an issue I ran into while using the json lib in Python.
I'm tying to read a json file using the json.load(file) command using the following code:
import json
filename= '../Data/exampleFile.json'
histFile= open(filename, 'w+')
print(json.load(histFile))
The JSON file I am trying to read is valid according to some website I found: a screenshot of that validation, because I'm new and still lack the reputation...
The error message I'm getting is the following:
File ".\testLoad.py", line 5, in <module>
print(json.load(histFile))
File "C:\Users\...\Python\Python37\lib\json\__init__.py", line 296, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "C:\Users\...\Python\Python37\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\...\Python\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\...\Python\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Alright, so I believe it is not the file that is the issue, but the json.load(file) works for me in other cases.
Sadly I was not able to figure this error-message out on my own, so it would be amazing if someone with some more experience dealing with Python-JSON interaction could maybe help me out.

You opened the file for writing:
histFile= open(filename, 'w+')
# ^^^^
The w mode first truncates the file, so the file is empty (it doesn't matter here that the file can also be read from, the + sees to that but the file is truncated nonetheless). See the open() function documentation:
'w': open for writing, truncating the file first)
There is no JSON data in it to parse. This is why the exception tells you that parsing failed at the very start of the file:
Expecting value: line 1 column 1 (char 0)
There is no data in line one, column one.
If you wanted to open a file for both reading and writing without truncating it first, use 'r+' as the file mode.