Develop Reference

filter json file with python

How to filter a json file to show only the information I need?
To start off I want to say I'm fairly new to python and working with JSON so sorry if this question was asked before and I overlooked it.
I have a JSON file that looks like this:
[
{
"Store": 417,
"Item": 10,
"Name": "Burger",
"Modifiable": true,
"Price": 8.90,
"LastModified": "09/02/2019 21:30:00"
},
{
"Store": 417,
"Item": 15,
"Name": "Fries",
"Modifiable": false,
"Price": 2.60,
"LastModified": "10/02/2019 23:00:00"
}
]
I need to filter this file to only show Item and Price, like
[
{
"Item": 10,
"Price": 8.90
},
{
"Item": 15,
"Price": 2.60
}
]
I have a code that looks like this:
# Transform json input to python objects
with open("StorePriceList.json") as input_file:
input_dict = json.load(input_file)
# Filter python objects with list comprehensions
output_dict = [x for x in input_dict if ] #missing logical test here.
# Transform python object back into json
output_json = json.dumps(output_dict)
# Show json
print(output_json)
What logical test I should be doing here to do that?

Let's say we can use dict comprehension, then it will be
output_dict = [{k:v for k,v in x.items() if k in ["Item", "Price"]} for x in input_dict]

You can also do it like this :)
>>> [{key: d[key] for key in ['Item', 'Price']} for d in input_dict] # you should rename it to `input_list` rather than `input_dict` :)
[{'Item': 10, 'Price': 8.9}, {'Item': 15, 'Price': 2.6}]

import pprint
with open('data.json', 'r') as f:
qe = json.load(f)
list = []
for item in qe['<your data>']:
query = (f'{item["Item"]} {item["Price"]}')
print("query")

How to get a flat JSON from a nested one?

I have a nested JSON and I need "Max" & "Remaining" percentage values from it.
This is sample formula I am thinking of 100-(Remaining/Max)*100=(Value)
Sample JSON:
{
"ConcurrentAsyncGetReportInstances":
{
"Max": 5,
"Remaining": 3
},
"DailyApiRequests":
{
"Max":15000,"Remaining":14108
}
}
This is the JSON output.
I need to add the % value to the key
Sample output:
{
"ConcurrentAsyncGetReportInstances":40,(%value) 100-(5/3)*100
"DailyApiRequests": 5.95(%value) 100-(14108/15000)*100
}
Workarounds:
Tried to do make it a flat JSON and worked but didn't helped me
Worked on converting JSON into CSV and tried some but it was hard
Can someone suggest the best to do this? If possible provide some examples. Some help would also be appreciated.
Note: I am using Python 2.7

There is now a Python package for this called flatten_json. An introduction is provided here.
An example from that page--
In your shell:
> pip install flatten_json
In your Python console:
from flatten_json import flatten
input_dict = {
"a": 1,
"b": 2,
"c": [{"d": [2, 3, 4], "e": [{"f": 1, "g": 2}]}]
}
print(flatten(input_dict))
Results:
{'a': 1,
'b': 2,
'c_0_d_0': 2,
'c_0_d_1': 3,
'c_0_d_2': 4,
'c_0_e_0_f': 1,
'c_0_e_0_g': 2}
I've tested this in both Python 3.6 and 2.7.

Firstly receive your json and convert it to dictionary
import json
input_dict = json.loads(<your received son string>)
Then work on the input dict like below through recursive calls:
input_dict = {
"ConcurrentAsyncGetReportInstances":
{
"Max": 200,"Remaining":200
},
"DailyApiRequests":
{
"Max": 15000, "Remaining": 14108,
"Ant Migration Tool": {"Max": 0, "Remaining": 0},
"Chatter Desktop": {"Max": 0, "Remaining": 0},
"Chatter Mobile for BlackBerry":
{"Max": 0, "Remaining": 0},
"Chemical Equipment And Processing":
{"Max": 0,"Remaining": 0}
}
}
def flattenjson(input_dict, odict):
for ky in input_dict.keys():
if isinstance(input_dict[ky], dict):
if set(['Max', 'Remaining']).issubset(input_dict[ky].keys()):
if input_dict[ky]["Max"] != 0:
odict[ky] = 100-(float(input_dict[ky]["Remaining"])/input_dict[ky]["Max"])*100
else:
odict[ky] = 0
for iky in input_dict[ky].keys():
if isinstance(input_dict[ky][iky], dict):
tmp = {iky : input_dict[ky][iky]}
odict = flattenjson(tmp, odict)
return odict
odict = flattenjson(input_dict, dict())
print json.dumps(odict)
flattenjson helps you recursively work on to get your desired output for all Max and Remaining entries

You can retrieve nested values using the json library like so:
import json
sample_json = '{"ConcurrentAsyncGetReportInstances":{"Max": 5,"Remaining": 3},"DailyApiRequests": {"Max":15000,"Remaining":14108}}'
jason = json.loads(sample_json)
cagri_max = jason['ConcurrentAsyncGetReportInstances']['Max']
cagri_rem = jason['ConcurrentAsyncGetReportInstances']['Remaining']

You don't need to flatten the data structure. Just reference that pieces of it you want—so, for example, I think the following does essentially what you want:
import json
json_data = {
"ConcurrentAsyncGetReportInstances": {
"Max": 5,
"Remaining": 3
},
"DailyApiRequests": {
"Max": 15000,
"Remaining": 14108
}
}
def percentage_values(remaining, maximum):
return 100 - (float(remaining)/float(maximum)) * 100
# build output dictionary
json_out = {}
for key in json_data:
value = percentage_values(json_data[key]["Remaining"], json_data[key]["Max"])
json_out.update([(key, value)])
print(json.dumps(json_out, indent=4))
The resulting output showing the contents of json_out is:
{
"ConcurrentAsyncGetReportInstances": 40.0,
"DailyApiRequests": 5.9466666666666725
}
There are more succinct ways to write this in Python, but they all would do what is done above in a very simple manner.

Show certain parameter JSON in python

I have some JSON output, for example:
«somethings»:
{
"id": 1,
"param1": 11,
"param2": 12
},
{
"id": 2,
"param1": 21,
"param2": 22
}
I want to show only param1 for each "id". I can do it for use circle for:
for i in range(0, 50):
print parsed_string["somethings"][i]["param1"]
but I don't now how many ids i can get.

In the JSON it's just an array of objects (and Python list once parsed), so you can iterate over all of them like so:
for obj in parsed_string["somethings"]:
print(obj["param1"])

If you need to display key-value, use following code
import json as json
json_string = '[{"id":"A 0","param1":"SOME DATA 1","param2":"SOME DATA 2"},' \
'{"id":"A 1","param1":"SOME DATA 2","param2":"SOME DATA 2"},' \
'{"id":"A 2","param1":"SOME DATA 2","param2":"SOME DATA 2"}]'
json_data = json.loads(json_string)
for d in json_data:
for key, value in d.iteritems():
if (key in ("param1")): # ("id", "param1")
print key, value

Convert following data to JSON format in python

I have a file with following content which is generated by some other program. Please note that each one has a new line character at the end.
{'a':1,'b':534}
{'a':4,'b':882}
{'a':2,'b':964}
.
.
so on...
How do I convert this into valid JSON format?
The above should be printed something like this
{
"Sale": [
{
"a": 1,
"b": 534
},
{
"a": 4,
"b": 882
},
{
"a": 2,
"b": 964
}
]
}
I can do it in JQuery but I need to do this in python as i am not good at python but i need to make it runnable through bash script hence the need to make it to a python script.
I am gonna share the jquery solution just in case you need to refer.
result = replaceAll(result,"'","\"");
result = replaceAll(result,"}, {",",");
result = replaceAll(result,"}{","},{");
result = replaceAll(result,"}\n{","},{");
result = "{\"Sale\":["+result+"]}";
//alert(result);
replaceAll function is
function replaceAll(str, find, replace) {
var i = str.indexOf(find);
if (i > -1){
str = str.replace(find, replace);
i = i + replace.length;
var st2 = str.substring(i);
if(st2.indexOf(find) > -1){
str = str.substring(0,i) + replaceAll(st2, find, replace);
}
}
return str;
}
The above works but i need a python alternative to this.

from ast import literal_eval
with open('data.txt') as f:
sale = [literal_eval(line) for line in f]
data = {'Sale': sale}
print(data)
Output
{'Sale': [{'a': 1, 'b': 534}, {'a': 4, 'b': 882}, {'a': 2, 'b': 964}]}
From there, you can use the json library to write this to file in JSON format.
import json
json.dumps(data, indent=4)

from ast import literal_eval
import json
sale = []
with open(filepath) as f:
lines = f.read()
for line in lines:
sale.append(literal_eval(line))
json.dumps({"sale":sale}, indent=4)

How to filter JSON data using Python?

How to convert JSON data from input.json to output.json using Python? In general, what data structures are used for filtering JSON data?
File: input.json
[
{
"id":1,
"a":22,
"b":11
},
{
"id":1,
"e":44,
"c":77,
"f":55,
"d":66
},
{
"id":3,
"b":11,
"a":22
},
{
"id":3,
"d":44,
"c":88
}
]
File: output.json
[
{
"id":1,
"a":22,
"b":11,
"e":44,
"c":77,
"f":55,
"d":66
},
{
"id":3,
"b":11,
"a":22,
"d":44,
"c":88
}
]
Any pointers would be appreciated!

The idea is to:
use json.load() to load the JSON content from file to a Python list
regroup the data by the id, using collections.defaultdict and .update() method
use json.dump() to dump the result into the JSON file
Implementation:
import json
from collections import defaultdict
# read JSON data
with open("input.json") as input_file:
old_data = json.load(input_file)
# regroup data
d = defaultdict(dict)
for item in old_data:
d[item["id"]].update(item)
# write JSON data
with open("output.json", "w") as output_file:
json.dump(list(d.values()), output_file, indent=4)
Now the output.json would contain:
[
{
"d": 66,
"e": 44,
"a": 22,
"b": 11,
"c": 77,
"id": 1,
"f": 55
},
{
"b": 11,
"id": 3,
"d": 44,
"c": 88,
"a": 22
}
]

from collections import defaultdict
input_list=[{"id":1, ...}, {...}]
result_dict=defaultdict(dict)
for d in input_list:
result_dict[d['id']].update(d)
output_list=result_dict.values()
result_dict is a default dictionary which uses a dict for every access without a available key. So we iterate through the input_list and update our result_dict with key equals id with the new values from the corresponding dictionary.
The output list is a transformation of the result_dict and uses only its values.
Use the json module to work directly with the json data.