Develop Reference

How do I parse faulty json file from python using module json?

I have big size of json file to parse with python, however it's incomplete (i.e., missing parentheses in the end). The json file consist of one big json object which contains json objects inside. All json object in outer json object is complete, just finishing parenthese are missing.
for example, its structure is like this.
{bigger_json_head:value, another_key:[{small_complete_json1},{small_complete_json2}, ...,{small_complete_json_n},
So final "]}" are missing. however, each small json forms a single row so when I tried to print each line of the json file I have, I get each json object as a single string.
so I've tried to use:
with open("file.json","r",encoding="UTF-8") as f:
for line in f.readlines()
line_arr.append(line)
I expected to have a list with line of json object as its element
and then I tried below after the process:
for json_line in line_arr:
try:
json_str = json.loads(json_line)
print(json_str)
except json.decoder.JSONDecodeError:
continue
I expected from this code block, except first and last string, this code would print json string to console. However, it printed nothing and just got decode error.
Is there anyone who solved similar problem? please help. Thank you

If the faulty json file only miss the final "]}", then you can actually fix it before parse it.
Here is an example code to illustrate:
with open("file.json","r",encoding="UTF-8") as f:
faulty_json_str = f.read()
fixed_json_str = faulty_json_str + ']}'
json_obj = json.loads(fixed_json_str)

I'm having difficulty parsing this json file

I'm trying to parse this json file that I recieved from an API call.
"[{\"ip\":\"xx.xx.xxx.xx\",\"dns\":\"xxx.net\",\"netbios\":\"xxxxx\",....
I dumped it to a file like so:
with open('jayo.json', 'w') as j:
json.dump(r.text, j) #r.text being the API response
json should just be a straightforward dictionary, right? why does mine have all the back-slashes?
How would I print each value on it's own? IP/DNS etc.

You are receiving the API response as a str you need to load it using json before dumping it. json.dump is usually used with collections not strings as it does the conversion for you.
data = json.loads(r.text)
with open('jayo.json', 'w') as j:
json.dump(data, j)
If you need the data in the file before overwriting it load it use
with open('jayo.json', 'r') as j:
data = json.load(j)

Are you trying to load the JSON in Python, or dump it to a file? (or both?)
json.dump is for writing a Python object to a JSON file. r.text is just a string, so the resulting format will look like a single string in JSON (including all of the escaped quotes) instead of a full object.
Presumably you want to use json.loads to load the JSON string into a Python object before using json.dump. Or if you want to dump the JSON string straight to a file, you can just use j.write(r.text).

How to convert python Dictionary to JSON Dicitonary

I'm using python 3.x. I have a python dictionary
my_dict={'key1':'A12b','Key2':'cd12'}
I want to convert it in JSON format. I need it in this format
my_dict1={"key1":"A12b","Key2":"cd12"}
I tried my_dict1=json.dumps(my_dict) & I'm getting '{"key1": "A12b", "Key2": "cd12"}' which is of type string. Can you suggest me how do I convert?

That is as close to JSON as you can get in Python. Send that object to a webserver and it will be interpreted as JSON which is probably what you want.

dict in python is a kind of data structure, while json is a kind of data format;
I suppose you want to convert a dict into a json-formated file? Then you should use json.dump(obj, fp) instead of json.dumps(obj). The first one dumps a python dict obj to a file fp, and the second function make obj into a json formatted string.
In your case, to dump a dict obj into a file example.json, code it like below:
import json
my_dict={'key1':'A12b','Key2':'cd12'}
with open('example.json', 'w') as file:
json.dump(my_dict, file)
Good luck!

Ijson parse from list

I have a list in which each item contains JSON data, so I am trying to parse the data using Ijson since the data load will be huge.
This is what I am trying to achieve:
article_data=#variable which contains the list
parser = ijson.parse(article_data)
for id in ijson.items(parser, 'item'):
if(id['article_type'] != "Monthly Briefing" and id['article_type']!="Conference"):
data_article_id.append(id['article_id'])
data_article_short_desc.append(id['short_desc'])
data_article_long_desc.append(id['long_desc'])
This is the error I get:
AttributeError: 'generator' object has no attribute 'read'
I thought of converting the list into string and then try to parse in Ijson, but it fails and gives me the same error.
Any suggestions please?
data_article_id=[]
data_article_short_desc=[]
data_article_long_desc=[]
for index in article_data:
parser = ijson.parse(index)
for id in ijson.items(parser, 'item'):
if(id['article_type'] != "Monthly Briefing" and id['article_type']!="Conference"):
data_article_id.append(id['article_id'])
data_article_short_desc.append(id['short_desc'])
data_article_long_desc.append(id['long_desc'])
since it is in list, i tried this one also .. but it is giving me the same error.
'generator' object has no attribute 'read'

I am assuming that you have a list of byte string json object that you want to parse.
ijson.items(JSON, prefix) takes a readable byte object as input. That is it takes a opened file or file-like object as input. Specifically, the input should be bytes file-like objects.
If you are using Python 3, you can use io module with
io.BytesIO to create a in-memory binary stream.
Example
Suppose input is [b'{"id": "ab"}', b'{"id": "cd"}']
list_json = [b'{"id": "ab"}', b'{"id": "cd"}']
for json in list_json:
item = ijson.items(io.BytesIO(json), "")
for i in item:
print(i['id'])
Output:
ab
cd

Writing Json in for loop in Python

I am downloading Json files from an API, I use the following code to write the JSON. Each item the loop gives me a JSON file. I need to save it and extract entities from the appended JSON file using a loop.
for item in style_ls:
dat = get_json(api, item)
specs_dict[item] = dat
with open("specs_append.txt", "a") as myfile:
json.dump(dat, myfile)
myfile.close()
print item
with open ("specs_data.txt", "w") as my file:
json.dump(spec_dict, myfile)
myfile.close()
I know that I cannot get a valid JSON format from the specs_append.txt, but I can get one from the specs_data.txt. I am doing the first one just because my program needs atleast 3-4 days to complete and there are high chances that my system may shutdown. So is there anyway I can do this efficiently ?
If not is there anyway I can extract it from specs_append.txt <{JSON}{JSON}> format (which is not a valid JSON format)?
If not should I write specs_dict to a txt file every time in the loop, so that even if program gets terminated i can start if from that point in loop and still get a valid json format?

I suggest several possible solutions.
One solution is to write custom code to slurp in the input file. I would suggest putting a special line before each JSON object in the file, such as: ###
Then you could write code like this:
import json
def json_get_objects(f):
temp = ''
line = next(f) # pull first line
assert line == SPECIAL_LINE
for line in f:
if line != SPECIAL_LINE:
temp += line
else:
# found special marker, temp now contains a complete JSON object
j = json.loads(temp)
yield j
temp = ''
# after loop done, yield up last JSON object
if temp:
j = json.loads(temp)
yield j
with open("specs_data.txt", "r") as f:
for j in json_get_objects(f):
pass # do something with JSON object j
Two notes on this. First, I am simply appending to a string over and over; this used to be a very slow way to do this in Python, so if you are using a very old version of Python, don't do it this way unless your JSON objects are very small. Second, I wrote code to split the input and yield up JSON objects one at a time, but you could also use a guaranteed-unique string, slurp in all the data with a single call to f.read() and then split on your guaranteed-unique string using the str.split() method function.
Another solution would be to write the whole file as a valid JSON list of valid JSON objects. Write the file like this:
{"mylist":[
# first JSON object, followed by a comma
# second JSON object, followed by a comma
# third JSON object
]}
This would require your file appending code to open the file with writing permission, and seek to the last ] in the file before writing a comma plus newline, then the new JSON object on the end, and then finally writing ]} to close out the file. If you do it this way, you can use json.loads() to slurp the whole thing in and have a list of JSON objects.
Finally, I suggest that maybe you should just use a database. Use SQLite or something and just throw the JSON strings in to a table. If you choose this, I suggest using an ORM to make your life simple, rather than writing SQL commands by hand.
Personally, I favor the first suggestion: write in a special line like ###, then have custom code to split the input on those marks and then get the JSON objects.
EDIT: Okay, the first suggestion was sort of assuming that the JSON was formatted for human readability, with a bunch of short lines:
{
"foo": 0,
"bar": 1,
"baz": 2
}
But it's all run together as one big long line:
{"foo":0,"bar":1,"baz":2}
Here are three ways to fix this.
0) write a newline before the ### and after it, like so:
###
{"foo":0,"bar":1,"baz":2}
###
{"foo":0,"bar":1,"baz":2}
Then each input line will alternately be ### or a complete JSON object.
1) As long as SPECIAL_LINE is completely unique (never appears inside a string in the JSON) you can do this:
with open("specs_data.txt", "r") as f:
temp = f.read() # read entire file contents
lst = temp.split(SPECIAL_LINE)
json_objects = [json.loads(x) for x in lst]
for j in json_objects:
pass # do something with JSON object j
The .split() method function can split up the temp string into JSON objects for you.
2) If you are certain that each JSON object will never have a newline character inside it, you could simply write JSON objects to the file, one after another, putting a newline after each; then assume that each line is a JSON object:
import json
def json_get_objects(f):
for line in f:
if line.strip():
yield json.loads(line)
with open("specs_data.txt", "r") as f:
for j in json_get_objects(f):
pass # do something with JSON object j
I like the simplicity of option (2), but I like the reliability of option (0). If a newline ever got written in as part of a JSON object, option (0) would still work, but option (2) would error.
Again, you can also simply use an actual database (SQLite) with an ORM and let the database worry about the details.
Good luck.

Append json data to a dict on every loop.
In the end dump this dict as a json and write it to a file.
For getting you an idea for appending data to dict:
>>> d1 = {'suku':12}
>>> t1 = {'suku1':212}
>>> d1.update(t1)
>>> d1
{'suku1': 212, 'suku': 12}