I have a file with following content which is generated by some other program. Please note that each one has a new line character at the end.
{'a':1,'b':534}
{'a':4,'b':882}
{'a':2,'b':964}
.
.
so on...
How do I convert this into valid JSON format?
The above should be printed something like this
{
"Sale": [
{
"a": 1,
"b": 534
},
{
"a": 4,
"b": 882
},
{
"a": 2,
"b": 964
}
]
}
I can do it in JQuery but I need to do this in python as i am not good at python but i need to make it runnable through bash script hence the need to make it to a python script.
I am gonna share the jquery solution just in case you need to refer.
result = replaceAll(result,"'","\"");
result = replaceAll(result,"}, {",",");
result = replaceAll(result,"}{","},{");
result = replaceAll(result,"}\n{","},{");
result = "{\"Sale\":["+result+"]}";
//alert(result);
replaceAll function is
function replaceAll(str, find, replace) {
var i = str.indexOf(find);
if (i > -1){
str = str.replace(find, replace);
i = i + replace.length;
var st2 = str.substring(i);
if(st2.indexOf(find) > -1){
str = str.substring(0,i) + replaceAll(st2, find, replace);
}
}
return str;
}
The above works but i need a python alternative to this.
from ast import literal_eval
with open('data.txt') as f:
sale = [literal_eval(line) for line in f]
data = {'Sale': sale}
print(data)
Output
{'Sale': [{'a': 1, 'b': 534}, {'a': 4, 'b': 882}, {'a': 2, 'b': 964}]}
From there, you can use the json library to write this to file in JSON format.
import json
json.dumps(data, indent=4)
from ast import literal_eval
import json
sale = []
with open(filepath) as f:
lines = f.read()
for line in lines:
sale.append(literal_eval(line))
json.dumps({"sale":sale}, indent=4)
Related
I am trying to convert CSV file to JSON file based on a column value. The csv file looks somewhat like this.
ID Name Age
CSE001 John 18
CSE002 Marie 20
ECE001 Josh 22
ECE002 Peter 23
currently I am using the following code to obtain json file.
import csv
import json
def csv_to_json(csv_file_path, json_file_path):
data_dict = {}
with open(csv_file_path, encoding = 'utf-8') as csv_file_handler:
csv_reader = csv.DictReader(csv_file_handler)
for rows in csv_reader:
key = rows['ID']
data_dict[key] = rows
with open(json_file_path, 'w', encoding = 'utf-8') as json_file_handler:
json_file_handler.write(json.dumps(data_dict, indent = 4))
OUTPUT:
**{
"CSE001":{
"ID":"CSE001",
"Name":"John",
"Age":18
}
"CSE002":{
"ID":"CSE002",
"Name":"Marie",
"Age":20
}
"ECE001":{
"ID":"ECE001",
"Name":"Josh",
"Age":22
}
"ECE002":{
"ID":"ECE002",
"Name":"Peter",
"Age":23
}
}**
I want my output to generate two separate json files for CSE and ECE based on the ID value. Is there a way to achieve this output.
Required Output:
CSE.json:
{
"CSE001":{
"ID":"CSE001",
"Name":"John",
"Age":18
}
"CSE002":{
"ID":"CSE002",
"Name":"Marie",
"Age":20
}
}
ECE.json:
{
"ECE001":{
"ID":"ECE001",
"Name":"Josh",
"Age":22
}
"ECE002":{
"ID":"ECE002",
"Name":"Peter",
"Age":23
}
}
I would suggest you to use pandas, that way will be more easier.
Code may look like:
import pandas as pd
def csv_to_json(csv_file_path):
df = pd.read_csv(csv_file_path)
df_CSE = df[df['ID'].str.contains('CSE')]
df_ECE = df[df['ID'].str.contains('ECE')]
df_CSE.to_json('CSE.json')
df_ECE.to_json('ESE.json')
You can create dataframe and then do the following operation
import pandas as pd
df = pd.DataFrame.from_dict({
"CSE001":{
"ID":"CSE001",
"Name":"John",
"Age":18
},
"CSE002":{
"ID":"CSE002",
"Name":"Marie",
"Age":20
},
"ECE001":{
"ID":"ECE001",
"Name":"Josh",
"Age":22
},
"ECE002":{
"ID":"ECE002",
"Name":"Peter",
"Age":23
}
},orient='index')
df["id_"] = df["ID"].str[0:2] # temp column for storing first two chars
grps = df.groupby("id_")[["ID", "Name", "Age"]]
for k, v in grps:
print(v.to_json(orient="index")) # you can create json file as well
You could store each row into two level dictionary with the top level being the first 3 characters of the ID.
These could then be written out into separate files with the key being part of the filename:
from collections import defaultdict
import csv
import json
def csv_to_json(csv_file_path, json_base_path):
data_dict = defaultdict(dict)
with open(csv_file_path, encoding = 'utf-8') as csv_file_handler:
csv_reader = csv.DictReader(csv_file_handler)
for row in csv_reader:
key = row['ID'][:3]
data_dict[key][row['ID']] = row
for key, values in data_dict.items():
with open(f'{json_base_path}_{key}.json', 'w', encoding='utf-8') as json_file_handler:
json_file_handler.write(json.dumps(values, indent = 4))
csv_to_json('input.csv', 'output')
The defaultdict is used to avoid needing to first test if a key is already present before using it.
This would create output_CSE.json and output_ECE.json, e.g.
{
"ECE001": {
"ID": "ECE001",
"Name": "Josh",
"Age": "22"
},
"ECE002": {
"ID": "ECE002",
"Name": "Peter",
"Age": "23"
}
}
I am trying to delete an element in a json file,
here is my json file:
before:
{
"names": [
{
"PrevStreak": false,
"Streak": 0,
"name": "Brody B#3719",
"points": 0
},
{
"PrevStreak": false,
"Streak": 0,
"name": "XY_MAGIC#1111",
"points": 0
}
]
}
after running script:
{
"names": [
{
"PrevStreak": false,
"Streak": 0,
"name": "Brody B#3719",
"points": 0
}
]
}
how would I do this in python? the file is stored locally and I am deciding which element to delete by the name in each element
Thanks
I would load the file, remove the item, and then save it again. Example:
import json
with open("filename.json") as f:
data = json.load(f)
f.pop(data["names"][1]) # or iterate through entries to find matching name
with open("filename.json", "w") as f:
json.dump(data, f)
You will have to read the file, convert it to python native data type (e.g. dictionary), then delete the element and save the file. In your case something like this could work:
import json
filepath = 'data.json'
with open(filepath, 'r') as fp:
data = json.load(fp)
del data['names'][1]
with open(filepath, 'w') as fp:
json.dump(data, fp)
Try this:
# importing the module
import ast
# reading the data from the file
with open('dictionary.txt') as f:
data = f.read()
print("Data type before reconstruction : ", type(data))
# reconstructing the data as a dictionary
a_dict = ast.literal_eval(data)
{"names":[a for a in a_dict["names"] if a.get("name") !="XY_MAGIC#1111"]}
import json
with open("test.json",'r') as f:
data = json.loads(f.read())
names=data.get('names')
for idx,name in enumerate(names):
if name['name']=='XY_MAGIC#1111':
del names[idx]
break
print(names)
In order to read the file best approach would be using the with statement after which you could just use pythons json library and convert json string to python dict. once you get dict you can access the values and do your operations as required. you could convert it as json using json.dumps() then save it
This does the right thing useing the python json module, and prettyprints the json back to the file afterwards:
import json
jsonpath = '/path/to/json/file.json'
with open(jsonpath) as file:
j = json.loads(file.read())
names_to_remove = ['XY_MAGIC#1111']
for element in j['names']:
if element['name'] in names_to_remove:
j['names'].remove(element)
with open(jsonpath, 'w') as file:
file.write(json.dumps(j, indent=4))
I have string containing multiple jsons. I want to convert that string to single json object.
For example,
Assuming the following input,
input = """
{
"a" : {
"x":"y",
"w":"z"
}
}
{
"b" : {
"v":"w",
"z":"l"
}
}
"""
The expected output will be:
Output:
{
"a" : {
"x":"y",
"w":"z"
}
"b" : {
"v":"w",
"z":"l"
}
}
if we treat them as dictionaries and have
>>> a = {'a':{'a':1}}
>>> b = {'b':{'b':1}}
we can simply
>>> a.update(b)
>>> a
{'a': {'a': 1}, 'b': {'b': 1}}
you can take advantage of the fact that you can see when a dictionary begins by looking if a line starts with '{':
import json
input = """
{
"a" : {
"x":"y",
"w":"z"
}
}
{
"b" : {
"v":"w",
"z":"l"
}
}"""
my_dicts = {}
start_dict = False
one_json = ''
for line in input.split('\n'):
if line.startswith('{'):
# check if we pass a json
if start_dict:
my_dicts.update(json.loads(one_json))
one_json = ''
else:
start_dict = True
one_json = f'{one_json}\n{line}'
# take the last json
my_dicts.update(json.loads(one_json))
print(my_dicts)
output:
{'a': {'w': 'z', 'x': 'y'}, 'b': {'v': 'w', 'z': 'l'}}
Build up a list of dictionaries parsing each character. One could also parse each line.
There is good probability of finding a user library that already does this function but here is a way to go
import json
braces = []
dicts = []
dict_chars = []
for line in inp: # input is a builtin so renamed from input to inp
char = line.strip()
dict_chars.append(line)
if '{' == char:
braces.append('}')
elif '}' == char:
braces.pop()
elif len(braces) == 0 and dict_chars:
text = ''.join(dict_chars)
if text.strip():
dicts.append(json.loads(text))
dict_chars = []
Then, merge dictionaries in the list.
merged_dict = {}
for dct in dicts:
merged_dict.update(dct)
> print(merged_dict)
{u'a': {u'x': u'y', u'w': u'z'}, u'b': {u'z': u'l', u'v': u'w'}}
Output merged dictionary as json string with indentation.
merged_output = json.dumps(merged_dict, indent=4)
I am attempting to pass in to a function, a string which will be interpreted to determine the desired dictionary call required to update a dictionary.
Here is an example of what I have so far, hard-coded:
import json
from collections import defaultdict
def default_dict():
return defaultdict(default_dict)
def build_dict():
d["a"]["b"]["c"]["d"]["e"]["f"].update({})
d["a"]["b"]["c1"]["d1"].update({})
return json.dumps(d)
d = default_dict()
print build_dict()
But to be useful to me I want to pass in strings to the build_dict() function. Lets call it 's':
for s in ["a/b/c/d/e/f", "a/b/c1/d1"]:
print build_dict(s)
Which should print the following (exactly as it does in the example I hard-coded:
{
"a": {
"b": {
"c": {
"d": {
"e": {
"f": {}
}
}
},
"c1": {
"d1": {}
}
}
}
}
I have to make sure that multiple branches are supported in the way they are (as far as I have tested) in my hard-coded example.
What I am currently attempting:
Midway through constructing this question I found out about dpath, "A python library for accessing and searching dictionaries via /slashed/paths ala xpath". It looks exactly what I need so if I successfully work it out, I will post an answer to this question.
I worked out a solution to my own question.
import json
import dpath.util
def build_dict(viewsDict, viewsList):
for views in viewsList:
viewsDict = new_keys(viewsDict, views)
return viewsDict
def new_keys(viewsDict, views):
dpath.util.new(viewsDict, views, {})
return viewsDict
viewsDict = {}
viewsList = [
"a/b/c/d/e/f",
"a/b/c1/d1"
]
print json.dumps(build_dict(viewsDict, viewsList), indent=4, sort_keys=True)
This builds a dict based on sequence of paths and passes your test case.
It builds a dictionary from up to down, adding a new keys if they are missing, and updating an existing dictionary when they are present.
def build_dict(string_seq):
d = {}
for s in string_seq:
active_d = d
parts = s.split("/")
for p in parts:
if p not in active_d:
active_d[p] = {}
active_d = active_d[p]
return d
expected = {
"a": {
"b": {
"c": {
"d": {
"e": {
"f": {}
}
}
},
"c1": {
"d1": {}
}
}
}
}
string_seq = ["a/b/c/d/e/f", "a/b/c1/d1"]
result = build_dict(string_seq)
assert result == expected
I have a dict like this:
sample = {'ObjectInterpolator': 1629, 'PointInterpolator': 1675, 'RectangleInterpolator': 2042}
I can't figure out how to dump the dict to a JSON file as showed below:
{
"name": "interpolator",
"children": [
{"name": "ObjectInterpolator", "size": 1629},
{"name": "PointInterpolator", "size": 1675},
{"name": "RectangleInterpolator", "size": 2042}
]
}
Is there a pythonic way to do this?
You may guess that I want to generate a d3 treemap.
import json
with open('result.json', 'w') as fp:
json.dump(sample, fp)
This is an easier way to do it.
In the second line of code the file result.json gets created and opened as the variable fp.
In the third line your dict sample gets written into the result.json!
Combine the answer of #mgilson and #gnibbler, I found what I need was this:
d = {
"name": "interpolator",
"children": [{
'name': key,
"size": value
} for key, value in sample.items()]
}
j = json.dumps(d, indent=4)
with open('sample.json', 'w') as f:
print >> f, j
It this way, I got a pretty-print json file.
The tricks print >> f, j is found from here:
http://www.anthonydebarros.com/2012/03/11/generate-json-from-sql-using-python/
d = {"name":"interpolator",
"children":[{'name':key,"size":value} for key,value in sample.items()]}
json_string = json.dumps(d)
Since python 3.7 the ordering of dicts is retained https://docs.python.org/3.8/library/stdtypes.html#mapping-types-dict
Dictionaries preserve insertion order. Note that updating a key does not affect the order. Keys added after deletion are inserted at the end
Also wanted to add this (Python 3.7)
import json
with open("dict_to_json_textfile.txt", 'w') as fout:
json_dumps_str = json.dumps(a_dictionary, indent=4)
print(json_dumps_str, file=fout)
Update (11-04-2021): So the reason I added this example is because sometimes you can use the print() function to write to files, and this also shows how to use the indentation (unindented stuff is evil!!). However I have recently started learning about threading and some of my research has shown that the print() statement is not always thread-safe. So if you need threading you might want to be careful with this one.
This should give you a start
>>> import json
>>> print json.dumps([{'name': k, 'size': v} for k,v in sample.items()], indent=4)
[
{
"name": "PointInterpolator",
"size": 1675
},
{
"name": "ObjectInterpolator",
"size": 1629
},
{
"name": "RectangleInterpolator",
"size": 2042
}
]
with pretty-print format:
import json
with open(path_to_file, 'w') as file:
json_string = json.dumps(sample, default=lambda o: o.__dict__, sort_keys=True, indent=2)
file.write(json_string)
If you're using Path:
example_path = Path('/tmp/test.json')
example_dict = {'x': 24, 'y': 25}
json_str = json.dumps(example_dict, indent=4) + '\n'
example_path.write_text(json_str, encoding='utf-8')