I have two json objects coming from a file. Those two objects make one record. They are of different length. I was using pandas.read_json(), but didnt work.
Here is an example:
input:
{"a":1,"b":2,"c":3}{"x":[100],"y":"123"}
expected output:
{
"a":1,
"b":2,
"c":3,
"x":[100],
"y":"123"
}
IIUC, You want to read two JSON and create a new JSON from them.
import json
new_json = {}
for json_file in ['js1.json', 'js2.json']:
with open(json_file) as f:
d = json.load(f)
new_json.update(d)
print(new_json)
# {'a': 1, 'b': 2, 'c': 3, 'x': [100], 'y': '123'}
# create a new json that contains two old json
res = json.dumps(new_json)
Update You can use ast.literal_eval, If two JSON in one file.
import json
import ast
# jss.json -> {"a":1,"b":2,"c":3}{"x":[100],"y":"123"}
new_json = {}
for json_file in ['jss.json']:
with open(json_file) as f:
jsons = f.read()
for js in jsons.split('}')[:-1]:
st = js+'}'
d = ast.literal_eval(st)
new_json.update(d)
print(new_json)
# {'a': 1, 'b': 2, 'c': 3, 'x': [100], 'y': '123'}
# create a new json that contains two old json
res = json.dumps(new_json)
Related
How can I add multiple dictionary in a JSON file?
I want to add 1 or 2 dictionaries once and after a while to add 1 or 2 or 3 dictionaries in same JSON file.
Exemple:
dict1 = {'a': 1, 'b':2}
-> I want to add it to a 'test.json' file and after a while I want to add the dictionary
dict2 = {'c': 1, 'd':2}
dict3 = {'e': 1, 'f':2}
-> and after a while I want to add this 2 for example
EDIT
import json
dict1 = {'a': 1, 'b': 1}
dict2 = {'c': 2, 'd': 2}
dict3 = {'e': 3, 'f': 3}
list1 = []
list1.append(dict1)
with open('testjson_dict.json', 'a') as f:
json.dump(list1, f)
-> this is first output
[
{
"a": 1,
"b": 1
}
]
-> than I append dict2 to list1, and this is the output, it create a second list and put dict2 in it, how can i change the code to put dict2 in my first list?
[
{
"a": 1,
"b": 1
}
][
{
"c": 2,
"d": 2
}
]
I am assuming you want to store these dicts as a list in json, so the final result would be:
[
{'a': 1, 'b':2},
{'c': 1, 'd':2},
{'e': 1, 'f':2}
]
Here is a possible workflow. Start with dict_list = [dict1].
Make sure you import json. Write dict_list to test.json
with open('test.json', 'w', encoding='utf-8') as json_file:
json.dump(dict_list, json_file)
Read the contents of test.json into a Python list.
with open('test.json', encoding='utf-8') as json_file:
dicts = json.load(json_file)
Add dict2 and dict3 to the list you just read in.
Overwrite test.json with the resulting list (like in step 1).
Now test.json should incude a list of the 3 dicts.
You can concat the new data as a list with + in that way:
import json
# write first file
dict_list = [{'a': 1, 'b':2}]
with open('test.json', 'w', encoding='utf-8') as json_file:
json.dump(dict_list, json_file)
# concat to readed file and overvwrite
with open('test.json', encoding='utf-8') as json_file:
dicts = json.load(json_file)
dicts += [{'c': 1, 'd':2}, {'e': 1, 'f':2}] # list concatenation operation
with open('test.json', 'w', encoding='utf-8') as json_file:
json.dump(dicts, json_file)
I am merging two json in python
I'm doing
import json
json_obj = json.dumps({"a": [1,2]})
json_obj1 = json.dumps({"a": [3,4]})
json_obj += json_obj1
print(json_obj)
I am expecting the output as
{"a": [1, 2,3,4]}
but i got
{"a": [1, 2]}{"a": [3, 4]}
How to get the earlier one?
In json module, dumps convert python object to a string, and loads convert a string into python object. So in your original codes, you just try to concat two json-string. Try to code like this:
import json
from collections import defaultdict
def merge_dict(d1, d2):
dd = defaultdict(list)
for d in (d1, d2):
for key, value in d.items():
if isinstance(value, list):
dd[key].extend(value)
else:
dd[key].append(value)
return dict(dd)
if __name__ == '__main__':
json_str1 = json.dumps({"a": [1, 2]})
json_str2 = json.dumps({"a": [3, 4]})
dct1 = json.loads(json_str1)
dct2 = json.loads(json_str2)
combined_dct = merge_dict(dct1, dct2)
json_str3 = json.dumps(combined_dct)
# {"a": [1, 2, 3, 4]}
print(json_str3)
json.dumps() converts a dictionary to str object, not a json(dict) object.
So, adding some dumps statement in your code shows that the type is changed to str after using json.dumps() and with + you are effectively concatenating the two string and hence you get the concatenated output.
Further, to merge the two dictionaries for your simple case, you can just use the append:
import json
json_obj = json.dumps({"a": [1,2]})
json_obj1 = json.dumps({"a": [3,4]})
print(type(json_obj1)) # the type is `str`
json_obj += json_obj1 # this concatenates the two str objects
json_obj = {"a": [1,2]}
json_obj1 = {"a": [3,4]}
json_obj["a"].extend(json_obj1["a"])
print(json_obj)
I suggest you to study basic fundamental of Python for your own sake as you don't seem to understand why your code wouldn't work.
import json
# We have two dictionaries to combine
json_obj_1 = {"a": [1,2], "b":[2,3], 'c': [1,2,3]}
json_obj_2 = {"a": [3,4], 'd':[4,2], 'e': [4,2,2]}
Merged dictionary will be stored here
hold_json_obj = {}
Don't worry, it's not actually that complicated. Read the code line by line with comments attached and you'll understand.
# We'll loop through every item in the json_obj_1 dictionary
for item_1 in json_obj_1:
# We'll also loop through every item in the json_obj_2 dictionary
for item_2 in json_obj_2:
# Now let's compare whether they are the same KEYS (not values)
if item_1 == item_2:
# if they match, we create a list to store the array
hold_array = []
hold_array.extend(json_obj_1[item_1])
hold_array.extend(json_obj_2[item_1])
# finally putting the array to our hold_json_obj
hold_json_obj[item_1] = hold_array
else:
# if they don't match, check if the key already exists in the
# hold_json_obj because we might be iterating json_obj_2 for the second time.
if item_2 not in hold_json_obj:
#add the ummatched array to hold_json_obj
hold_json_obj[item_2] = json_obj_2[item_2]
Now simply update json_obj_1 with the update method. The update function is required because if json_obj_1 has keys that json_obj_2 doesn't then we may have missed them out in the above loops.
json_obj_1.update(hold_json_obj)
print(json_obj_1)
This is what the print displays.
{'a': [1, 2, 3, 4], 'b': [2, 3], 'c': [1, 2, 3], 'd': [4, 2], 'e': [4, 2, 2]}
So I get a response and print it. The result is bytes:
payload = request.body
print (payload)
b'a=123&b=345&c=678&d=910'
I decode it, and the result is:
dataform = payload.decode('utf-8').replace("'", '"')
print(dataform, 'dataform')
a=123&b=345&c=678&d=910
I dumps it, and the result is:
result = json.dumps(dataform, indent=4, sort_keys=True)
print(result, 'result')
"a=123&b=345&c=678&d=910"
I loads it, and the result is:
jason = json.loads(result)
print(jason, 'jason')
a=123&b=345&c=678&d=910
I just want a normal json dictionary that I can refer to like data['string']. What am I doing wrong or not doing?
There's a few errors here.
First off, dumping to JSON and then loading it again does absolutely nothing (it does have a few side-effects, but that's not important here).
Secondly, and mainly, your input data isn't JSON - it's either a query string or, more likely, form-data.
You can try to parse it using the standard parse_qs in urllib.parse, but if that fails you'll have to look around for a library that can handle proper form data.
In [1]: from urllib.parse import parse_qs
In [2]: payload = b'a=123&b=345&c=678&d=910'
In [3]: dataform = payload.decode('utf-8').replace("'", '"')
In [4]: result = parse_qs(dataform)
In [5]: print(result)
{'a': ['123'], 'b': ['345'], 'c': ['678'], 'd': ['910']}
At first, you need to convert the string (here, as the example, to the array, but you can use that you want)
data = [x.split('=') for x in data.split('&')]
>>> data
[['a', '123'], ['b', '345'], ['c', '678'], ['d', '910']]
And after this, you can easily create the dictionary.
dict = {key: value for (key,value) in data}
>>> dict
{'a': '123', 'c': '678', 'b': '345', 'd': '910'}
Or if you want to store numbers as int:
dict = {key: int(value) for (key,value) in data}
>>> dict
{'a': 123, 'c': 678, 'b': 345, 'd': 910}
import json
from urllib.parse import parse_qs
payload = request.body
# b'a=123&b=345&c=678&d=910'
qs = parse_qs(payload.decode())
# {'a': ['123'], 'b': ['345'], 'c': ['678'], 'd': ['910']}
Convert list values and convert data into JSON
json.dumps({k: v[0] for k, v in qs.items()})
# '{"a": "123", "b": "345", "c": "678", "d": "910"}'
I have huge text file which I have to parse.
individual line of the file contains some text and dict. I only care about dict data.
file contain logs in the following format
my data : {"a":1, "b":2, "c": 3}
my data : {"a":23, "b": 44, "c": 565}
my_data : {"a":1233, "b": 21, "c":544}
so, from above data I am only looking for dict.
I tried with
f = open(‘text.file’,'r’)
my_dict = eval(f.read())
but it gives me error as the initial part of the line is string.
So, my question is what is the best way to extract dict from the file.
You can use the re module
import re
text = """my data : {"a":1, "b":2, "c": 3}
my data : {"a":23, "b": 44, "c": 565}
my_data : {"a":1233, "b": 21, "c":544}"""
dict = re.compile(r"{[^}]*?}", re.I)
matches = dict.finditer(text)
for match in matches:
my_dict = eval(match.group())
print(my_dict)
which gives you
{'b': 2, 'c': 3, 'a': 1}
{'b': 44, 'c': 565, 'a': 23}
{'b': 21, 'c': 544, 'a': 1233}
It looks like you've got some delimator between the strings, so str.split() is your friend there.
Afterwards, consider using the AST module instead of the eval. It presents less of a security risk than blindly eval'ing.
>>>import ast
>>> a = ast.literal_eval("{'a':1}")
>>> type(a)
<class 'dict'>
>>> a
{'a': 1}
eval is bad
here's what I would do:
import json
dicts = []
with open('text.file', 'r') as f:
for line in f.readlines():
if not line: continue
_, dict_str = line.split(':', 1)
dict_str = dict_str.strip()
dict = json.load(dict_str)
dicts.append(dict)
So my problem is this... I have multiple Pickle object files (which are Pickled Dictionaries) and I want to load them all, but essentially merge each dictionary into a single larger dictionary.
E.g.
I have pickle_file1 and pickle_file2 both contain dictionaries. I would like the contents of pickle_file1 and pickle_file2 loaded into my_dict_final.
EDIT
As per request here is what i have so far:
for pkl_file in pkl_file_list:
pickle_in = open(pkl_file,'rb')
my_dict = pickle.load(pickle_in)
pickle_in.close()
In essence, it works, but just overwrites the contents of my_dict rather than append each pickle object.
Thanks in advance for the help.
my_dict_final = {} # Create an empty dictionary
with open('pickle_file1', 'rb') as f:
my_dict_final.update(pickle.load(f)) # Update contents of file1 to the dictionary
with open('pickle_file2', 'rb') as f:
my_dict_final.update(pickle.load(f)) # Update contents of file2 to the dictionary
print my_dict_final
You can use the dict.update function.
pickle_dict1 = pickle.load(picke_file1)
pickle_dict2 = pickle.load(picke_file2)
my_dict_final = pickle_dict1
my_dict_final.update(pickle_dict2)
Python Standard Library Docs
#Nunchux, #Vikas Ojha If the dictionaries happen to have common keys, the update method will, unfortunately, overwrite the values for those common keys. Example:
>>> dict1 = {'a': 4, 'b': 3, 'c': 0, 'd': 4}
>>> dict2 = {'a': 1, 'b': 8, 'c': 5}
>>> All_dict = {}
>>> All_dict.update(dict1)
>>> All_dict.update(dict2)
>>> All_dict
{'a': 1, 'b': 8, 'c': 5, 'd': 4}
If you'd like to avoid this and keep adding the counts of common keys, one option is to use the following strategy. Applied to your example, here is a minimal working example:
import os
import pickle
from collections import Counter
dict1 = {'a': 4, 'b': 3, 'c': 0, 'd': 4}
dict2 = {'a': 1, 'b': 8, 'c': 5}
# just creating two pickle files:
pickle_out = open("dict1.pickle", "wb")
pickle.dump(dict1, pickle_out)
pickle_out.close()
pickle_out = open("dict2.pickle", "wb")
pickle.dump(dict2, pickle_out)
pickle_out.close()
# Here comes:
pkl_file_list = ["dict1.pickle", "dict2.pickle"]
All_dict = Counter({})
for pkl_file in pkl_file_list:
if os.path.exists(pkl_file):
pickle_in = open(pkl_file, "rb")
dict_i = pickle.load(pickle_in)
All_dict = All_dict + Counter(dict_i)
print (dict(All_dict))
This will happily give you:
{'a': 5, 'b': 11, 'd': 4, 'c': 5}