How can I add multiple dictionary in a JSON file?
I want to add 1 or 2 dictionaries once and after a while to add 1 or 2 or 3 dictionaries in same JSON file.
Exemple:
dict1 = {'a': 1, 'b':2}
-> I want to add it to a 'test.json' file and after a while I want to add the dictionary
dict2 = {'c': 1, 'd':2}
dict3 = {'e': 1, 'f':2}
-> and after a while I want to add this 2 for example
EDIT
import json
dict1 = {'a': 1, 'b': 1}
dict2 = {'c': 2, 'd': 2}
dict3 = {'e': 3, 'f': 3}
list1 = []
list1.append(dict1)
with open('testjson_dict.json', 'a') as f:
json.dump(list1, f)
-> this is first output
[
{
"a": 1,
"b": 1
}
]
-> than I append dict2 to list1, and this is the output, it create a second list and put dict2 in it, how can i change the code to put dict2 in my first list?
[
{
"a": 1,
"b": 1
}
][
{
"c": 2,
"d": 2
}
]
I am assuming you want to store these dicts as a list in json, so the final result would be:
[
{'a': 1, 'b':2},
{'c': 1, 'd':2},
{'e': 1, 'f':2}
]
Here is a possible workflow. Start with dict_list = [dict1].
Make sure you import json. Write dict_list to test.json
with open('test.json', 'w', encoding='utf-8') as json_file:
json.dump(dict_list, json_file)
Read the contents of test.json into a Python list.
with open('test.json', encoding='utf-8') as json_file:
dicts = json.load(json_file)
Add dict2 and dict3 to the list you just read in.
Overwrite test.json with the resulting list (like in step 1).
Now test.json should incude a list of the 3 dicts.
You can concat the new data as a list with + in that way:
import json
# write first file
dict_list = [{'a': 1, 'b':2}]
with open('test.json', 'w', encoding='utf-8') as json_file:
json.dump(dict_list, json_file)
# concat to readed file and overvwrite
with open('test.json', encoding='utf-8') as json_file:
dicts = json.load(json_file)
dicts += [{'c': 1, 'd':2}, {'e': 1, 'f':2}] # list concatenation operation
with open('test.json', 'w', encoding='utf-8') as json_file:
json.dump(dicts, json_file)
Related
I have two json objects coming from a file. Those two objects make one record. They are of different length. I was using pandas.read_json(), but didnt work.
Here is an example:
input:
{"a":1,"b":2,"c":3}{"x":[100],"y":"123"}
expected output:
{
"a":1,
"b":2,
"c":3,
"x":[100],
"y":"123"
}
IIUC, You want to read two JSON and create a new JSON from them.
import json
new_json = {}
for json_file in ['js1.json', 'js2.json']:
with open(json_file) as f:
d = json.load(f)
new_json.update(d)
print(new_json)
# {'a': 1, 'b': 2, 'c': 3, 'x': [100], 'y': '123'}
# create a new json that contains two old json
res = json.dumps(new_json)
Update You can use ast.literal_eval, If two JSON in one file.
import json
import ast
# jss.json -> {"a":1,"b":2,"c":3}{"x":[100],"y":"123"}
new_json = {}
for json_file in ['jss.json']:
with open(json_file) as f:
jsons = f.read()
for js in jsons.split('}')[:-1]:
st = js+'}'
d = ast.literal_eval(st)
new_json.update(d)
print(new_json)
# {'a': 1, 'b': 2, 'c': 3, 'x': [100], 'y': '123'}
# create a new json that contains two old json
res = json.dumps(new_json)
I have multiple text files that have this inside them(with diferent values):
{"abandon": {"R1F2V0CQAYJOZZ": 2, "R3NUFWQ9SPGVJO": 1}, "abduct": {"R1F2V0CQAYJOZZ": 1, "R3376OQSHCTV1A": 1, "R14BW4EQZNVKKG": 1, "R233CMES8RCOCU": 1},
If i format it online it becomes like this:
"abandon":{
"R1F2V0CQAYJOZZ":2,
"R3NUFWQ9SPGVJO":1
},
"abduct":{
"R1F2V0CQAYJOZZ":1,
"R3376OQSHCTV1A":1,
"R14BW4EQZNVKKG":1,
"R233CMES8RCOCU":1
},
What this JSON mean is:
"word":{
"Document name":"value"
},
But there are repeated words in diferent files.
What i want to do is:
Read all files and store everything in one dictionary, but:
If the "word" exists in the dictionary check if the "document" is there;
If the "document exists", then increment the "value", else put the document there and the "value = 1"
If the "word" doesnt exist, store the "word", "document" and "value = 1"
EDIT:
So imagine this 2 files:
File1.txt = {"abandon": {"doc1": 2, "doc2": 1}, "abduct": {"doc1": 1, "doc2": 1, "doc8": 1},
File1.txt = {"abandon": {"doc1": 1, "doc3": 1}, "abduct": {"doc5": 1, "doc8": 1},
I want my dictionary to end like this:
{"abandon": {"doc1": 3, "doc2": 1, "doc3": 1}, "abduct": {"doc1": 1, "doc2": 1, "doc5": 1, "doc8": 2},
EDIT2:
it can also be a nested List
IIUC, try:
import os
import json
files = [f for f in os.listdir() if f.endswith(".txt")]
result = dict()
for file in files:
d = json.load(open(file))
for word in d:
if word not in result:
result[word] = dict()
for doc in d[word]:
if doc not in result[word]:
result[word][doc] = d[word][doc]
else:
result[word][doc] += d[word][doc]
>>> result
{'abandon': {'doc1': 3, 'doc2': 1, 'doc3': 1},
'abduct': {'doc1': 1, 'doc2': 1, 'doc8': 2, 'doc5': 1}}
Input files:
file1.txt:
{"abandon": {"doc1": 2, "doc2": 1}, "abduct": {"doc1": 1, "doc2": 1, "doc8": 1}}
file2.txt:
{"abandon": {"doc1": 1, "doc3": 1}, "abduct": {"doc5": 1, "doc8": 1}}
import json
files = ["input", "list", "of", "files"]
outDict = {}
for file in files: # iterate over the files
with open(file) as fn:
newDict = json.load(fn)
for word in newDict: # iterate over each word from the file
inWord = newDict[word]
outWord = outDict.get(word, {}) # returns an empty dict if word isn't already in the output dictionary
for docName in inWord: # iterate over each document name from the file
value = outWord.get(docName, 0) # returns 0 if the document name isn't already in the output dictionary
value += 1 # increment the value
outWord[docName] = value # update the output dictionary
It's straight-forward to merge using .setdefault:
import json
import glob
merged = {}
for file in glob.glob('*.txt'): # match *.txt files in current directory
with open(file) as f:
in_dict = json.load(f)
for word, docs in in_dict.items():
for doc, value in docs.items():
merged.setdefault(word,{}) # create word with empty dict value if it doesn't exist
merged[word].setdefault(doc, 0) # create value of 0 for document if it doesn't exist
merged[word][doc] += value # add the doc's value.
print(json.dumps(merged,indent=2))
Or using defaultdict. The parameter to defaultdict must be a function that returns the default value, hence the lambda that returns a default integer dictionary:
import json
import glob
from collections import defaultdict
merged = defaultdict(lambda: defaultdict(int))
for file in glob.glob('*.txt'): # match *.txt files in current directory
with open(file) as f:
in_dict = json.load(f)
for word, docs in in_dict.items():
for doc,value in docs.items():
merged[word][doc] += value # add the doc's value.
print(json.dumps(merged,indent=2))
I have huge text file which I have to parse.
individual line of the file contains some text and dict. I only care about dict data.
file contain logs in the following format
my data : {"a":1, "b":2, "c": 3}
my data : {"a":23, "b": 44, "c": 565}
my_data : {"a":1233, "b": 21, "c":544}
so, from above data I am only looking for dict.
I tried with
f = open(‘text.file’,'r’)
my_dict = eval(f.read())
but it gives me error as the initial part of the line is string.
So, my question is what is the best way to extract dict from the file.
You can use the re module
import re
text = """my data : {"a":1, "b":2, "c": 3}
my data : {"a":23, "b": 44, "c": 565}
my_data : {"a":1233, "b": 21, "c":544}"""
dict = re.compile(r"{[^}]*?}", re.I)
matches = dict.finditer(text)
for match in matches:
my_dict = eval(match.group())
print(my_dict)
which gives you
{'b': 2, 'c': 3, 'a': 1}
{'b': 44, 'c': 565, 'a': 23}
{'b': 21, 'c': 544, 'a': 1233}
It looks like you've got some delimator between the strings, so str.split() is your friend there.
Afterwards, consider using the AST module instead of the eval. It presents less of a security risk than blindly eval'ing.
>>>import ast
>>> a = ast.literal_eval("{'a':1}")
>>> type(a)
<class 'dict'>
>>> a
{'a': 1}
eval is bad
here's what I would do:
import json
dicts = []
with open('text.file', 'r') as f:
for line in f.readlines():
if not line: continue
_, dict_str = line.split(':', 1)
dict_str = dict_str.strip()
dict = json.load(dict_str)
dicts.append(dict)
I've already searched for a solution to my problem but with no success. Part of the solution for my problem is here, but this does not solve it all.
I have two lists of dictionaries like this - each dictionary is written to a csv file but I read the contents to the following variables:
list1 = [{a:1, b:2, c:3}, {a:4, b:5, c:6}, {a:7, b:8, c:9}]
list2 = [{b:2, a:1, c:3}, {c:6, b:5, a:4}, {b:8, a:7, c:9}]
Using the solution of the link above, ie:
>>> import itertools
>>> a = [{'a': '1'}, {'c': '2'}]
>>> b = [{'a': '1'}, {'b': '2'}]
>>> intersec = [item for item in a if item in b]
>>> sym_diff = [item for item in itertools.chain(a,b) if item not in intersec]
I get no matches because the order of the dictionary is different. But in fact, both lists are the same. How can I check this? Do I have to sort the dictionaries before writing them to the csv file? Can this be a solution?
This is my major problem at the moment but I have another issue also. It would be great to be able to make this match check but ignoring one or more keys defined by me. Is this also possible?
EDIT: I have the dicitonaries in a csv file and I'm reading them with the following code:
def read_csv_file(self, filename):
'''Read CSV file and return its content as a Python list.'''
f = open(filename, 'r')
csvfile = csv.reader(f)
f.close
return [row for row in csvfile]
This is very important because I think the problem is that after reading the values from the csv it's not dictionaries anymore, so the order has to be the same.
EDIT2: sample of the csv file (3 lines, it's creating an empty line but that's not an issue...)
"{u'Deletion': '0', u'Source': 'Not Applicable', u'Status': ''}"
"{u'Deletion': '0', u'Source': 'Not Applicable', u'Status': ''}"
Part of this solution was found by OP as per our last CHAT conversation, it was to convert a string into dictionary using ast module.
Now using this module to convert every row read by the csv.reader() as it returns a list of strings, which would be a list of one string in case of OP's CVS file, then append this dictionary into a list. After that using list comprehension with itertools.chain, we can get the difference between the two lists.
import csv
import ast
import itertools
def csvToList(myCSVFile):
'''This function is used to convert strings returned by csv.reader() into List of dictionaries'''
f = open(myCSVFile, 'r')
l = []
try:
reader = csv.reader(f)
for row in reader:
if row: #as you mentioned in your 2nd edit that you could have empty rows.
l.append(ast.literal_eval(row[0]))
finally:
f.close()
return l
list1 = csvToList('myCSV1.csv')
list2 = csvToList('myCSV2.csv')
l1_sub_l2 = [d for d in list1 if d not in list2]
l2_sub_l1 = [d for d in list2 if d not in list1]
list_difference = list(itertools.chain(l1_sub_l2, l2_sub_l1))
You need to double check your code. I'm not getting the issue you're bringing up.
list1 = [{a:1, b:2, c:3}, {a:4, b:5, c:6}, {a:7, b:8, c:9}]
list2 = [{b:2, a:1, c:3}, {c:6, b:5, a:4}, {b:8, a:7, c:9}]
list1 = [{'a':1, 'b':2, 'c':3}, {'a':4, 'b':5, 'c':7}, {'a':7, 'b':8, 'c':9}]
list2 = [{'b':2, 'a':1, 'c':3}, {'c':6, 'b':2, 'a':4}, {'b':8, 'a':7, 'c':9}]
intersec = [item for item in list1 if item in list2]
sym_diff = [item for item in itertools.chain(list1,list2) if item not in intersec]
print(intersec)
print(sym_diff)
>>>[{'a': 1, 'c': 3, 'b': 2}, {'a': 4, 'c': 6, 'b': 5}, {'a': 7, 'c': 9, 'b': 8}]
>>>>[]
If I change list1 and list 2 (middle dictionary):
list1 = [{'a':1, 'b':2, 'c':3}, {'a':7, 'b':5, 'c':2}, {'a':7, 'b':8, 'c':9}]
list2 = [{'b':2, 'a':1, 'c':3}, {'c':6, 'b':5, 'a':4}, {'b':8, 'a':7, 'c':9}]
Running same code:
[{'a': 1, 'c': 3, 'b': 2}, {'a': 7, 'c': 9, 'b': 8}]
[{'a': 7, 'c': 2, 'b': 5}, {'a': 4, 'c': 6, 'b': 5}]
The provided code in the link seems to be working fine. The order of the dictionary or a list does not matter in python.
Use a dictionary comprehension instead of a list comprehension in your return.
So my problem is this... I have multiple Pickle object files (which are Pickled Dictionaries) and I want to load them all, but essentially merge each dictionary into a single larger dictionary.
E.g.
I have pickle_file1 and pickle_file2 both contain dictionaries. I would like the contents of pickle_file1 and pickle_file2 loaded into my_dict_final.
EDIT
As per request here is what i have so far:
for pkl_file in pkl_file_list:
pickle_in = open(pkl_file,'rb')
my_dict = pickle.load(pickle_in)
pickle_in.close()
In essence, it works, but just overwrites the contents of my_dict rather than append each pickle object.
Thanks in advance for the help.
my_dict_final = {} # Create an empty dictionary
with open('pickle_file1', 'rb') as f:
my_dict_final.update(pickle.load(f)) # Update contents of file1 to the dictionary
with open('pickle_file2', 'rb') as f:
my_dict_final.update(pickle.load(f)) # Update contents of file2 to the dictionary
print my_dict_final
You can use the dict.update function.
pickle_dict1 = pickle.load(picke_file1)
pickle_dict2 = pickle.load(picke_file2)
my_dict_final = pickle_dict1
my_dict_final.update(pickle_dict2)
Python Standard Library Docs
#Nunchux, #Vikas Ojha If the dictionaries happen to have common keys, the update method will, unfortunately, overwrite the values for those common keys. Example:
>>> dict1 = {'a': 4, 'b': 3, 'c': 0, 'd': 4}
>>> dict2 = {'a': 1, 'b': 8, 'c': 5}
>>> All_dict = {}
>>> All_dict.update(dict1)
>>> All_dict.update(dict2)
>>> All_dict
{'a': 1, 'b': 8, 'c': 5, 'd': 4}
If you'd like to avoid this and keep adding the counts of common keys, one option is to use the following strategy. Applied to your example, here is a minimal working example:
import os
import pickle
from collections import Counter
dict1 = {'a': 4, 'b': 3, 'c': 0, 'd': 4}
dict2 = {'a': 1, 'b': 8, 'c': 5}
# just creating two pickle files:
pickle_out = open("dict1.pickle", "wb")
pickle.dump(dict1, pickle_out)
pickle_out.close()
pickle_out = open("dict2.pickle", "wb")
pickle.dump(dict2, pickle_out)
pickle_out.close()
# Here comes:
pkl_file_list = ["dict1.pickle", "dict2.pickle"]
All_dict = Counter({})
for pkl_file in pkl_file_list:
if os.path.exists(pkl_file):
pickle_in = open(pkl_file, "rb")
dict_i = pickle.load(pickle_in)
All_dict = All_dict + Counter(dict_i)
print (dict(All_dict))
This will happily give you:
{'a': 5, 'b': 11, 'd': 4, 'c': 5}