Develop Reference

How to merge two json objects coming from a file

I have two json objects coming from a file. Those two objects make one record. They are of different length. I was using pandas.read_json(), but didnt work.
Here is an example:
input:
{"a":1,"b":2,"c":3}{"x":[100],"y":"123"}
expected output:
{
"a":1,
"b":2,
"c":3,
"x":[100],
"y":"123"
}

IIUC, You want to read two JSON and create a new JSON from them.
import json
new_json = {}
for json_file in ['js1.json', 'js2.json']:
with open(json_file) as f:
d = json.load(f)
new_json.update(d)
print(new_json)
# {'a': 1, 'b': 2, 'c': 3, 'x': [100], 'y': '123'}
# create a new json that contains two old json
res = json.dumps(new_json)
Update You can use ast.literal_eval, If two JSON in one file.
import json
import ast
# jss.json -> {"a":1,"b":2,"c":3}{"x":[100],"y":"123"}
new_json = {}
for json_file in ['jss.json']:
with open(json_file) as f:
jsons = f.read()
for js in jsons.split('}')[:-1]:
st = js+'}'
d = ast.literal_eval(st)
new_json.update(d)
print(new_json)
# {'a': 1, 'b': 2, 'c': 3, 'x': [100], 'y': '123'}
# create a new json that contains two old json
res = json.dumps(new_json)

Merge two json object in python

I am merging two json in python
I'm doing
import json
json_obj = json.dumps({"a": [1,2]})
json_obj1 = json.dumps({"a": [3,4]})
json_obj += json_obj1
print(json_obj)
I am expecting the output as
{"a": [1, 2,3,4]}
but i got
{"a": [1, 2]}{"a": [3, 4]}
How to get the earlier one?

In json module, dumps convert python object to a string, and loads convert a string into python object. So in your original codes, you just try to concat two json-string. Try to code like this:
import json
from collections import defaultdict
def merge_dict(d1, d2):
dd = defaultdict(list)
for d in (d1, d2):
for key, value in d.items():
if isinstance(value, list):
dd[key].extend(value)
else:
dd[key].append(value)
return dict(dd)
if __name__ == '__main__':
json_str1 = json.dumps({"a": [1, 2]})
json_str2 = json.dumps({"a": [3, 4]})
dct1 = json.loads(json_str1)
dct2 = json.loads(json_str2)
combined_dct = merge_dict(dct1, dct2)
json_str3 = json.dumps(combined_dct)
# {"a": [1, 2, 3, 4]}
print(json_str3)

json.dumps() converts a dictionary to str object, not a json(dict) object.
So, adding some dumps statement in your code shows that the type is changed to str after using json.dumps() and with + you are effectively concatenating the two string and hence you get the concatenated output.
Further, to merge the two dictionaries for your simple case, you can just use the append:
import json
json_obj = json.dumps({"a": [1,2]})
json_obj1 = json.dumps({"a": [3,4]})
print(type(json_obj1)) # the type is `str`
json_obj += json_obj1 # this concatenates the two str objects
json_obj = {"a": [1,2]}
json_obj1 = {"a": [3,4]}
json_obj["a"].extend(json_obj1["a"])
print(json_obj)

I suggest you to study basic fundamental of Python for your own sake as you don't seem to understand why your code wouldn't work.
import json
# We have two dictionaries to combine
json_obj_1 = {"a": [1,2], "b":[2,3], 'c': [1,2,3]}
json_obj_2 = {"a": [3,4], 'd':[4,2], 'e': [4,2,2]}
Merged dictionary will be stored here
hold_json_obj = {}
Don't worry, it's not actually that complicated. Read the code line by line with comments attached and you'll understand.
# We'll loop through every item in the json_obj_1 dictionary
for item_1 in json_obj_1:
# We'll also loop through every item in the json_obj_2 dictionary
for item_2 in json_obj_2:
# Now let's compare whether they are the same KEYS (not values)
if item_1 == item_2:
# if they match, we create a list to store the array
hold_array = []
hold_array.extend(json_obj_1[item_1])
hold_array.extend(json_obj_2[item_1])
# finally putting the array to our hold_json_obj
hold_json_obj[item_1] = hold_array
else:
# if they don't match, check if the key already exists in the
# hold_json_obj because we might be iterating json_obj_2 for the second time.
if item_2 not in hold_json_obj:
#add the ummatched array to hold_json_obj
hold_json_obj[item_2] = json_obj_2[item_2]
Now simply update json_obj_1 with the update method. The update function is required because if json_obj_1 has keys that json_obj_2 doesn't then we may have missed them out in the above loops.
json_obj_1.update(hold_json_obj)
print(json_obj_1)
This is what the print displays.
{'a': [1, 2, 3, 4], 'b': [2, 3], 'c': [1, 2, 3], 'd': [4, 2], 'e': [4, 2, 2]}

Python Conditional Operator Useage

I'm trying to better understand a way to implement a conditional variable into a python requests' http request.
I'm doing the following:
def make_post(parameter1_value, parameter2='None'):
payload = {
"parameter1": parameter1_value,
"parameter2": "None"
}
r = requests.post('https://myrequesturl.com/location/', params=payload)
I'm trying to find the best way to NOT include the "parameter2": "None" value in the params unless the value is not equal to None.
I realize I could have several conditional statements to select a proper format of request depending on the parameters, but then it would become sticky to scale for each additional parameterN needed in the function.
I'm wondering if there is a way to include a conditional type of variable in the payload that would have the effect of including that parameter only if it isn't equal to the default value set; in this case 'None'.
So if the the value of parameter2 = None:
payload = {
"parameter1": parameter1_value
}
But if the value of parameter2 = anything other than None
payload = {
"parameter1": parameter1_value,
"parameter2": "Non_default_value"
}
I'm trying to avoid the following type of approach:
if some_parameter != 'None':
payload = {
"one": "arrangement"
}
if some_some_other_parameter != 'None':
payload = {
"one": "arrangement"
}
It seems a bit impractical in an example of only two parameters, but if I were to have a function with many parameters it would seem that a one line include/exclude type expression would greatly reduce the amount of overall code required. It's been my experience when I can't figure out how to get python to do something clever it's only because I don't know how, and not that python can't.

Something like this?
def foo(v1, v2=None):
params = {k:v for k,v in locals().items() if v!=None}
print params
foo('hello')
foo('hello', 'world')
Output:
{'v1': 'hello'}
{'v1': 'hello', 'v2': 'world'}
Or
Something like this?
def foo(a={}, url=''):
params = {k:v for k,v in a.items() if v!=None}
print params
foo({'k1':'v1','k2':2,'k3':None, 'k4':'v4'})
foo()
Output:
{'k2': 2, 'k1': 'v1', 'k4': 'v4'}
{}

I think something like this would work for you, especially if the default values are specific to the key:
def optional_include_mut(payload, key, value, default=None):
"""
Mutate the given payload dict by adding the given key/value pair if the value is not equal to default.
:param payload: Dict to mutate
:param key: Key to insert into dict
:param value: Value to insert into dict
:param default: Default value to control insertion
:return: A mutated version of the payload dict
"""
if value != default:
payload[key] = value
return payload
>>> payload = dict(good=1)
>>> payload = optional_include_mut(payload, 'foo', 'bar', default='bar')
{'good': 1}
>>> payload = optional_include_mut(payload, 'foo', 'bar', default='not bar')
{'good': 1, 'foo': 'bar'}

You could use a dictionary comprehension to filter the values:
>>> payload = dict(a='value1',b='value2',c='None',d='None',e='value3')
>>> payload
{'a': 'value1', 'b': 'value2', 'c': 'None', 'd': 'None', 'e': 'value3'}
>>> {k:v for k,v in payload.items() if v != 'None'}
{'a': 'value1', 'b': 'value2', 'e': 'value3'}

how about something like this
>>> def fun(*param):
return {"parameter{}".format(k): v for k,v in enumerate(param,1) if v not in (None,"None")}
>>> fun("hello","None","world",None,42)
{'parameter1': 'hello', 'parameter5': 42, 'parameter3': 'world'}
>>>
(*param is the way to said you want a variable numbers of arguments)
this way you can pass as many parameter as you want, and filter those with invalid values
with python 3 syntax you can also do it as
def fun(*param, invalid=(None,"None")):
return {"parameter{}".format(k): v for k,v in enumerate(param,1) if v not in invalid}
for python 2 to accomplished the same would be
def fun(*param, **karg):
invalid = karg.get("invalid", (None,"None") )
return {"parameter{}".format(k): v for k,v in enumerate(param,1) if v not in invalid}
another example
>>> fun("hello","None","world",None,42,32, invalid=("None",None,32))
{'parameter1': 'hello', 'parameter5': 42, 'parameter3': 'world'}
>>>

Python dict.fromkeys return same id element

When i do this on python i update all keys in one time.
>>> base = {}
>>> keys = ['a', 'b', 'c']
>>> base.update(dict.fromkeys(keys, {}))
>>> base.get('a')['d'] = {}
>>> base
{'a': {'d': {}}, 'c': {'d': {}}, 'b': {'d': {}}}
>>> map(id, base.values())
[140536040273352, 140536040273352, 140536040273352]
If instead of .get i use [] operator this not happen:
>>> base['a']['d'] = {}
>>> base
{'a': {'d': {}}, 'c': {}, 'b': {}}
Why?

When you initialize the value for the new keys as {} a new dictionary is created and a reference to this dictionary is becoming the values. There is only one dictionary and so if you change one, you will change "all".

I tried it with both Python 2.7.6 and 3.4.3. I get the same answer when either get('a') or ['a'] is used. Appreciate if you can verify this at your end. Python does object reuse. Thus, dict.fromkeys() reuses the same empty dict is to initialize. To make each one a separate object, you can do this:
base.update(zip(keys, ({} for _ in keys)))

How to search for dict from a big text file using python

I have huge text file which I have to parse.
individual line of the file contains some text and dict. I only care about dict data.
file contain logs in the following format
my data : {"a":1, "b":2, "c": 3}
my data : {"a":23, "b": 44, "c": 565}
my_data : {"a":1233, "b": 21, "c":544}
so, from above data I am only looking for dict.
I tried with
f = open(‘text.file’,'r’)
my_dict = eval(f.read())
but it gives me error as the initial part of the line is string.
So, my question is what is the best way to extract dict from the file.

You can use the re module
import re
text = """my data : {"a":1, "b":2, "c": 3}
my data : {"a":23, "b": 44, "c": 565}
my_data : {"a":1233, "b": 21, "c":544}"""
dict = re.compile(r"{[^}]*?}", re.I)
matches = dict.finditer(text)
for match in matches:
my_dict = eval(match.group())
print(my_dict)
which gives you
{'b': 2, 'c': 3, 'a': 1}
{'b': 44, 'c': 565, 'a': 23}
{'b': 21, 'c': 544, 'a': 1233}

It looks like you've got some delimator between the strings, so str.split() is your friend there.
Afterwards, consider using the AST module instead of the eval. It presents less of a security risk than blindly eval'ing.
>>>import ast
>>> a = ast.literal_eval("{'a':1}")
>>> type(a)
<class 'dict'>
>>> a
{'a': 1}

eval is bad
here's what I would do:
import json
dicts = []
with open('text.file', 'r') as f:
for line in f.readlines():
if not line: continue
_, dict_str = line.split(':', 1)
dict_str = dict_str.strip()
dict = json.load(dict_str)
dicts.append(dict)