on a nested JSON object, I would like to modify values and adding a JSON Object.
Assume a JSON Object like this:
{
"key1": "value1",
"key2": {
"key2_1": "value2_2 ",
"key2_2": {
"key2_2_1 ": "value2_2_1"
},
"key2_3": "value2_3",
"key2_4": {
"key2_4_1": [{
"key2_4_1_1a": "value2_4_1_1a",
"key2_4_1_2a": "value2_4_1_2a"
}, {
"key2_4_1_1b": "value2_4_1_1b",
"key2_4_1_2b": "value2_4_1_2b"
}]
}
},
"key3": {
"key3_1": "value3_2 ",
"key3_2": {
"key3_2_1 ": "value3_2_1"
},
"key3_3": "value3_3",
"key3_4": {
"key3_4_1": {
"key3_4_1_1": "value3_4_1_1"
}
}
}
}
now the JSON will be recursive iterated to search for a specific value.
The replacement value can be a string
repl = 'MyString'
a dict string
repl = '''{"MyKey": [{"MyKey1": "MyValye1"},{"MyKey2": "MyValye2"}]}'''
or a list
repl = '''[{"MyKey1": "MyValye1"},{"MyKey2": "MyValye2"}]'''
so after I found the key where the replacement to add, I would like to replace the existing value for the given key.
eg for the string:
a[key] = repl
How I can do this for dict or list replacements?
The result could be depending on the replacement variable, the string eg in "key2_1", the dict in "key2_2_1" or the list in "key2_3". The keys where string,dict or list are inserted are examples.
{
"key1": "value1",
"key2": {
"key2_1": "MyString",
"key2_2": {
"key2_2_1 ": {"MyKey": [{"MyKey1": "MyValye1"},{"MyKey2": "MyValye2"}]}
},
"key2_3": [{"MyKey1": "MyValye1"},{"MyKey2": "MyValye2"}],
"key2_4": {
"key2_4_1": [{
"key2_4_1_1a": "value2_4_1_1a",
"key2_4_1_2a": "value2_4_1_2a"
}, {
"key2_4_1_1b": "value2_4_1_1b",
"key2_4_1_2b": "value2_4_1_2b"
}]
}
}
}
i have a search function:
def searchNreplace(data, search_val, replace_val):
if isinstance(data, list):
return [searchNreplace(listd, search_val, replace_val) for listd in data]
if isinstance(data, dict):
return {dictkey: searchNreplace(dictvalue, search_val, replace_val) for dictkey, dictvalue in data.items()}
return replace_val if data == search_val else data
print(searchNreplace(data, "key3", repl))
If you really don't struggle with finding a key, you can use json library to parse your string to object and just assign it as str.
import json
repl = """{"MyKey": [{"MyKey1": "MyValye1"},{"MyKey2": "MyValye2"}]}"""
a[key] = json.loads(repl)
After that you can dump content back to file
with open("my_file", "w+") as f:
json.dump(a, f)
Related
I have created a var that is equal to t.json. The JSON file is a follows:
{
"groups": {
"customerduy": {
"nonprod": {
"name": "customerduynonprod",
"id": "529646781943",
"owner": "cloudops#coerce.com",
"manager_email": ""
},
"prod": {
"name": "phishing_duyaccountprod",
"id": "241683454720",
"owner": "cloudops#coerce.com",
"manager_email": ""
}
},
"customerduyprod": {
"nonprod": {
"name": "phishing_duyaccountnonprod",
"id": "638968214142",
"owner": "cloudops#coerce.com",
"manager_email": ""
}
},
"ciasuppliergenius": {
"prod": {
"name": "ciasuppliergeniusprod",
"id": "220753788760",
"owner": "cia_developers#coerce.com",
"manager_email": "jarks#coerce.com"
}
}
}
}
my goal was to pars this JSON file and get value for "owner" and output it to a new var. Example below:
t.json = group_map
group_id_aws = group(
group.upper(),
"accounts",
template,
owner = group_map['groups']['prod'],
manager_description = "Groups for teams to access their product accounts.",
The error I keep getting is: KeyError: 'prod'
Owner occurs 4 times, so here is how to get all of them.
import json
# read the json
with open("C:\\test\\test.json") as f:
data = json.load(f)
# get all 4 occurances
owner_1 = data['groups']['customerduy']['nonprod']['owner']
owner_2 = data['groups']['customerduy']['prod']['owner']
owner_3 = data['groups']['customerduyprod']['nonprod']['owner']
owner_4 = data['groups']['ciasuppliergenius']['prod']['owner']
# print results
print(owner_1)
print(owner_2)
print(owner_3)
print(owner_4)
the result:
cloudops#coerce.com
cloudops#coerce.com
cloudops#coerce.com
cia_developers#coerce.com
You get a key error since the key 'prod' is not in 'groups'
What you have is
group_map['groups']['customerduy']['prod']
group_map['groups']['ciasuppliergenius']['prod']
So you will have to extract the 'owner' from each element in the tree:
def s(d,t):
for k,v in d.items():
if t == k:
yield v
try:
for i in s(v,t):
yield i
except:
pass
print(','.join(s(j,'owner')))
If your JSON is loaded in variable data, you can use a recursive function
that deals with the two containers types (dict and list) that can occur
in a JSON file, recursively:
def find_all_values_for_key(d, key, result):
if isinstance(d, dict):
if key in d:
result.append(d[key])
return
for k, v in d.items():
find_all_values_for_key(v, key, result)
elif isinstance(d, list):
for elem in d:
find_all_values_for_key(elem, key, result)
owners = []
find_all_values_for_key(data, 'owner', owners)
print(f'{owners=}')
which gives:
owners=['cloudops#coerce.com', 'cloudops#coerce.com', 'cloudops#coerce.com', 'cia_developers#coerce.com']
This way you don't have to bother with the names of intermediate keys, or in general the structure of your JSON file.
You don't have any lists in your example, but it is trivial to recurse through
them to any dict with an owner key that might "lurk" somewhere nested
under a a list element, so it is better to deal with potential future changes
to the JSON.
I need to get the name information, inside the "object" list.
In this example I need this information (10.0.0.19)
"sourceNetworks":{
"objects":[
{
"type":"Host",
"overridable":false,
"id":"005056BF-7C6E-0ed3-0000-012884911113",
"name":"**10.0.0.19**"
}
]
}
I can get any information that is not in the "objects" lists with the command example_json[['metadata']['accessPolicy']['name']
and I get the "mb-test-01" information correctly from the json, but I don't know the syntax to get the items inside the "object" list.
to create this json I use in GET request this way
example_json = requests.get(f"https://{hostname}/api/fmc_config/v1/domain/{uuid}/policy/accesspolicies/{acp_id}/accessrules?expanded=true",headers=header_acp, verify=False).json()
follow the full json.
{
"metadata":{
"ruleIndex":1,
"section":"Mandatory",
"category":"--Undefined--",
"accessPolicy":{
"type":"AccessPolicy",
"name":"mb-test-01",
"id":"005056BF-7C6E-0ed3-0000-012884914323"
},
"timestamp":1635219651530,
"domain":{
"name":"Global",
"id":"e276abec-e0f2-11e3-8169-6d9ed49b625f",
"type":"Domain"
}
},
"links":{
"self":"https://fmcrestapisandbox.cisco.com/api/fmc_config/v1/domain/e276abec-e0f2-11e3-8169-6d9ed49b625f/policy/accesspolicies/005056BF-7C6E-0ed3-0000-012884914323/accessrules/005056BF-7C6E-0ed3-0000-000268434442"
},
"enabled":true,
"action":"ALLOW",
"type":"AccessRule",
"id":"005056BF-7C6E-0ed3-0000-000268434442",
"sourceNetworks":{
"objects":[
{
"type":"Host",
"overridable":false,
"id":"005056BF-7C6E-0ed3-0000-012884911113",
"name":"10.0.0.19"
}
]
},
"destinationNetworks":{
"objects":[
{
"type":"Host",
"overridable":false,
"id":"005056BF-7C6E-0ed3-0000-012884911491",
"name":"192.168.0.39"
}
]
},
"logBegin":false,
"logEnd":false,
"variableSet":{
"name":"Default-Set",
"id":"76fa83ea-c972-11e2-8be8-8e45bb1343c0",
"type":"VariableSet"
},
"logFiles":false,
"enableSyslog":false,
"vlanTags":{
},
"sendEventsToFMC":false,
"name":"rule-1"
}
Presumably you want to retrieve all "name"s under "objects" keys so you could use a recursive function:
def get_name(d):
for k,v in d.items():
if k=='objects':
for i in v:
yield i.get('name')
elif isinstance(v, dict):
yield from get_name(v)
names = list(get_name(data))
Output:
['10.0.0.19', '192.168.0.39']
I'm building json format and the expected is as below:
{
accounts:[ {
"acctnum": "acct1",
"key2":"value2"
"key3": []
"summary" : {
//nested dict
}
}
] //if we have 1 account for given customer
}
If we have more than one account for customer:
{
accounts:[ {
"acctnum": "acct1", // for acct1
"key2":"value2"
"key3": []
"summary" : {
//nested dict
}
},
{
"acctnum": "acct2", //for acct2
"key2":"value3"
"key3": []
"summary" : {
//nested dict
}
}
]
}
After building the dictionary with required attributes for acct1, my code towards end is (and have question if I should be using collections module):
acctlist = []
acctlist = results //results is dict for acct1 (with nested dict)
print(acctlist)
accounts = {}
accounts["accounts"] = acctlist
j = json.dumps(accounts, indent=4)
print(j)
But the actual json format comes out as:
{
accounts: {
"acctnum": "acct1",
"key2":"value2"
"key3": []
"summary1" : {
//nested dict
}
}
}
Your print statement likely answers your question. Is it a list?
acctlist = []
acctlist = results //results is dict for acct1 (with nested dict)
print(acctlist)
I suspect you meant to do acctlist.append(results)
I'm comparing json files between two different API endpoints to see which json records need an update, which need a create and what needs a delete. So, by comparing the two json files, I want to end up with three json files, one for each operation.
The json at both endpoints is structured like this (but they use different keys for same sets of values; different problem):
{
"records": [{
"id": "id-value-here",
"c": {
"d": "eee"
},
"f": {
"l": "last",
"f": "first"
},
"g": ["100", "89", "9831", "09112", "800"]
}, {
…
}]
}
So the json is represented as a list of dictionaries (with further nested lists and dictionaries).
If a given json endpoint (j1) id value ("id":) exists in the other endpoint json (j2), then that record should be added to j_update.
So far I have something like this, but I can see that .values() doesn't work because it's trying to operate on the list instead of on all the listed dictionaries(?):
j_update = {r for r in j1['records'] if r['id'] in
j2.values()}
This doesn't return an error, but it creates an empty set using test json files.
Seems like this should be simple, but tripping over the nesting I think of dictionaries in a list representing the json. Do I need to flatten j2, or is there a simpler dictionary method python has to achieve this?
====edit j1 and j2====
have same structure, use different keys; toy data
j1
{
"records": [{
"field_5": 2329309841,
"field_12": {
"email": "cmix#etest.com"
},
"field_20": {
"last": "Mixalona",
"first": "Clara"
},
"field_28": ["9002329309999", "9002329309112"],
"field_44": ["1002329309832"]
}, {
"field_5": 2329309831,
"field_12": {
"email": "mherbitz345#test.com"
},
"field_20": {
"last": "Herbitz",
"first": "Michael"
},
"field_28": ["9002329309831", "9002329309112", "8002329309999"],
"field_44": ["1002329309832"]
}, {
"field_5": 2329309855,
"field_12": {
"email": "nkatamaran#test.com"
},
"field_20": {
"first": "Noriss",
"last": "Katamaran"
},
"field_28": ["9002329309111", "8002329309112"],
"field_44": ["1002329309877"]
}]
}
j2
{
"records": [{
"id": 2329309831,
"email": {
"email": "mherbitz345#test.com"
},
"name_primary": {
"last": "Herbitz",
"first": "Michael"
},
"assign": ["8003329309831", "8007329309789"],
"hr_id": ["1002329309877"]
}, {
"id": 2329309884,
"email": {
"email": "yinleeshu#test.com"
},
"name_primary": {
"last": "Lee Shu",
"first": "Yin"
},
"assign": ["8002329309111", "9003329309831", "9002329309111", "8002329309999", "8002329309112"],
"hr_id": ["1002329309832"]
}, {
"id": 23293098338,
"email": {
"email": "amlouis#test.com"
},
"name_primary": {
"last": "Maxwell Louis",
"first": "Albert"
},
"assign": ["8002329309111", "8007329309789", "9003329309831", "8002329309999", "8002329309112"],
"hr_id": ["1002329309877"]
}]
}
If you read the json it will output a dict. You are looking for a particular key in the list of the values.
if 'records' in j2:
r = j2['records'][0].get('id', []) # defaults if id does not exist
It it prettier to do a recursive search but i dunno how you data is organized to quickly come up with a solution.
To give an idea for recursive search consider this example
def resursiveSearch(dictionary, target):
if target in dictionary:
return dictionary[target]
for key, value in dictionary.items():
if isinstance(value, dict):
target = resursiveSearch(value, target)
if target:
return target
a = {'test' : 'b', 'test1' : dict(x = dict(z = 3), y = 2)}
print(resursiveSearch(a, 'z'))
You tried:
j_update = {r for r in j1['records'] if r['id'] in j2.values()}
Aside from the r['id'/'field_5] problem, you have:
>>> list(j2.values())
[[{'id': 2329309831, ...}, ...]]
The id are buried inside a list and a dict, thus the test r['id'] in j2.values() always return False.
The basic solution is fairly simple.
First, create a set of j2 ids:
>>> present_in_j2 = set(record["id"] for record in j2["records"])
Then, rebuild the json structure of j1 but without the j1 field_5 that are not present in j2:
>>> {"records":[record for record in j1["records"] if record["field_5"] in present_in_j2]}
{'records': [{'field_5': 2329309831, 'field_12': {'email': 'mherbitz345#test.com'}, 'field_20': {'last': 'Herbitz', 'first': 'Michael'}, 'field_28': ['9002329309831', '9002329309112', '8002329309999'], 'field_44': ['1002329309832']}]}
It works, but it's not totally satisfying because of the weird keys of j1. Let's try to convert j1 to a more friendly format:
def map_keys(json_value, conversion_table):
"""Map the keys of a json value
This is a recursive DFS"""
def map_keys_aux(json_value):
"""Capture the conversion table"""
if isinstance(json_value, list):
return [map_keys_aux(v) for v in json_value]
elif isinstance(json_value, dict):
return {conversion_table.get(k, k):map_keys_aux(v) for k,v in json_value.items()}
else:
return json_value
return map_keys_aux(json_value)
The function focuses on dictionary keys: conversion_table.get(k, k) is conversion_table[k] if the key is present in the conversion table, or the key itself otherwise.
>>> j1toj2 = {"field_5":"id", "field_12":"email", "field_20":"name_primary", "field_28":"assign", "field_44":"hr_id"}
>>> mapped_j1 = map_keys(j1, j1toj2)
Now, the code is cleaner and the output may be more useful for a PUT:
>>> d1 = {record["id"]:record for record in mapped_j1["records"]}
>>> present_in_j2 = set(record["id"] for record in j2["records"])
>>> {"records":[record for record in mapped_j1["records"] if record["id"] in present_in_j2]}
{'records': [{'id': 2329309831, 'email': {'email': 'mherbitz345#test.com'}, 'name_primary': {'last': 'Herbitz', 'first': 'Michael'}, 'assign': ['9002329309831', '9002329309112', '8002329309999'], 'hr_id': ['1002329309832']}]}
In python, I'm having trouble figuring out how to output the JSON object (expressed below) as a string wherein the contents of Baseball are ordered based on "key1" (descending). When I receive the JSON (from the datasources) it's got the players out of order. Ultimately, my code needs to order the players, and then pass it along to the next function ordered. Please assume that I cannot modify the format of the JSON to be/have arrays as the consuming function can't handle that (as it's currently written).
Example JSON:
{
"DataSource1":{
"Baseball":{
"Sean":{
"key1":"10",
},
"Gene":{
"key1":"100",
},
"Alan":{
"key1":"1",
}
}
},
"DataSource2":{
"Baseball":{
"Bob_Smith":{
"key1":"1"
},
"Adam_Filmore":{
"key1":"100"
},
"Joe_Allen":{
"key1":"10"
}
}
}
"DataSource3":{
"Baseball":{
"Jake":{
"key1":"10"
},
"Huck":{
"key1":"1"
},
"Eric":{
"key1":"100"
}
}
}
}
Example of how I would like JSON to output:
{
"DataSource1":{
"Baseball":{
"Alan":{
"key1":"1",
},
"Sean":{
"key1":"10",
},
"Gene":{
"key1":"100",
}
}
},
"DataSource2":{
"Baseball":{
"Bob_Smith":{
"key1":"1"
},
"Joe_Allen":{
"key1":"10"
},
"Adam_Filmore":{
"key1":"100"
}
}
}
"DataSource3":{
"Baseball":{
"Huck":{
"key1":"1"
},
"Jake":{
"key1":"10"
},
"Eric":{
"key1":"100"
}
}
}
}
Use sorted() to establish the sort order you want, then store the results in a collections.OrderedDict.
Try this:
import json
from collections import OrderedDict
with open('data.json') as f:
data = json.load(f)
for data_source in data:
data[data_source]["Baseball"] = OrderedDict(
sorted(data[data_source]["Baseball"].items(),
key=lambda x: x[1]["key1"]))
with open('new_data.json', 'w') as f:
json.dump(data, f, indent=4)