Develop Reference

Python combine JSON arrays if value matches within objects irrespective of order

I wish to merge elements from a secondary JSON array into a primary array if values match.
Primary JSON:
primary = {
"#odata.context": "https://api.securitycenter.microsoft.com/api/$metadata#Collection(microsoft.windowsDefenderATP.api.PublicAssetVulnerabilityDto)",
"value": [
{
"id": "5afa3afc92a7c63d4b70129e0a6f33f63a427e21-_-CVE-2020-6494-_-microsoft-_-edge_chromium-based-_-81.0.416.77-_-",
"cveId": "CVE-2020-6494",
"machineId": "e5bc9d7e413ddd7902c2932e418702b84d0cc07",
"fixingKbId": null,
"productName": "edge_chromium-based",
"productVendor": "microsoft",
"productVersion": "81.0.416.77",
"severity": "Low"
},
{
"id": "7a704e17d1c2977c0e7b665fb18ae6e1fe7f3283-_-CVE-2016-3348-_-microsoft-_-windows_server_2012_r2-_-6.3.9600.19728-_-3185911",
"cveId": "CVE-2016-3348",
"machineId": "7a704e17d1c2977c0e7b665fb18ae6e1fe7f3283",
"fixingKbId": "3185911",
"productName": "windows_server_2012_r2",
"productVendor": "microsoft",
"productVersion": "6.3.9600.19728",
"severity": "Low"
}]}
Secondary JSON:
secondary = {
"#odata.context": "https://api.securitycenter.microsoft.com/api/$metadata#Machines",
"value": [
{
"id": "e5bc9d7e413ddd7902c2932e418702b84d0cc07",
"computerDnsName": "mymachine1.contoso.com",
"firstSeen": "2018-08-02T14:55:03.7791856Z",
"lastSeen": "2018-08-02T14:55:03.7791856Z",
"osPlatform": "Windows10" "Windows11",
"version": "1709",
"osProcessor": "x64",
"lastIpAddress": "172.17.230.209",
"lastExternalIpAddress": "167.220.196.71",
"osBuild": 18209,
"healthStatus": "Active",
"rbacGroupId": 140,
"rbacGroupName": "The-A-Team",
"riskScore": "Low",
"exposureLevel": "Medium",
"isAadJoined": true,
"aadDeviceId": "80fe8ff8-2624-418e-9591-41f0491218f9",
"machineTags": [ "test tag 1", "test tag 2" ]
}
]
}
I would like to merge based off these key values:
machineId (primary json)
id (secondary json)
Within my json, you can see that the 0th element in Primary JSON has the same "machineId" as the 0th element in the secondary JSON's "id". Therefore I would like to append the lastSeen key and value from the secondary JSON into the primary JSON as shown beneath:
Merged:
merged = {
"id": "5afa3afc92a7c63d4b70129e0a6f33f63a427e21-_-CVE-2020-6494-_-microsoft-_-edge_chromium-based-_-81.0.416.77-_-",
"cveId": "CVE-2020-6494",
"machineId": "e5bc9d7e413ddd7902c2932e418702b84d0cc07",
"fixingKbId": null,
"productName": "edge_chromium-based",
"productVendor": "microsoft",
"productVersion": "81.0.416.77",
"severity": "Low"
"lastSeen": "2018-08-02T14:55:03.7791856Z"
}
I was making use of this beneath solution to merge:
for element in a:
for lastSeen in s:
print(primary['machineId'])
print(secondary['id'])
if primary['machineId'] == secondary['id']:
element["lastSeen"] = secondary["lastSeen"]
print(element)
This wasn't working to well for me. I understand there may be a more optimized solution opposed to making use of embedded for loops that I'm not aware of.
If any clarify is required, please get in touch. Thanks.
Essentially in pseudo-code:
For each value in primary:
For each value in secondary:
If primary["id"] equals secondary["machineId"]
Append secondary["lastSeen"] to primary

Python list of dictionaries - access keys

I need to change name of the keys in dictionary using item.replace(" ", "_").lower()
How could I access these keys?
{
"environment": [
{
"Branch Branching": "97/97(100%)",
"Test Status": "TC39",
},
{
"Branch Branching": "36/36(100%)",
"Test Status": "TC29",
}
],
}

One way is to use:
dictionary[new_key] = dictionary.pop(old_key)
In your example:
env = {
"environment": [
{
"Branch Coverage": "97/97(100%)",
"Test Environment": "REGISTERHANDLING",
"Test Configuration": "TC39",
},
{
"Branch Coverage": "36/36(100%)",
"Test Environment": "PRA",
"Test Configuration": "TC29",
}
],
}
# Looping over each index in the env['environment'] list,
# this way we can edit the original dictionary.
# Note that enumerate returns a tuple of values (idx, val)
# And _ is commonly used to demonstrate that we will not be using val, only the index.
for index, _ in enumerate(env['environment']):
# For each key, we want to create a new key and delete the old one.
for key in env['environment'][index].keys():
# Calculate the new key
new_key = key.replace(" ", "_").lower()
# .pop deletes the old key and returns the result, and the left hand side of this operation creates the new key in the correct index.
env['environment'][index][new_key] = env['environment'][index].pop(key)
This question was already solved before, if you'd like to explore other answers, click here.

Printing pair of a dict

Im new in python but always trying to learn.
Today I got this error while trying select a key from dictionary:
print(data['town'])
KeyError: 'town'
My code:
import requests
defworld = "Pacera"
defcity = 'Svargrond'
requisicao = requests.get(f"https://api.tibiadata.com/v2/houses/{defworld}/{defcity}.json")
data = requisicao.json()
print(data['town'])
The json/dict looks this:
{
"houses": {
"town": "Venore",
"world": "Antica",
"type": "houses",
"houses": [
{
"houseid": 35006,
"name": "Dagger Alley 1",
"size": 57,
"rent": 2665,
"status": "rented"
}, {
"houseid": 35009,
"name": "Dream Street 1 (Shop)",
"size": 94,
"rent": 4330,
"status": "rented"
},
...
]
},
"information": {
"api_version": 2,
"execution_time": 0.0011,
"last_updated": "2017-12-15 08:00:00",
"timestamp": "2017-12-15 08:00:02"
}
}
The question is, how to print the pairs?
Thanks

You have to access the town object by accessing the houses field first, since there is nesting.
You want print(data['houses']['town']).

To avoid your first error, do
print(data["houses"]["town"])
(since it's {"houses": {"town": ...}}, not {"town": ...}).
To e.g. print all of the names of the houses, do
for house in data["houses"]["houses"]:
print(house["name"])

As answered, you must do data['houses']['town']. A better approach so that you don't raise an error, you can do:
houses = data.get('houses', None)
if houses is not None:
print(houses.get('town', None))
.get is a method in a dict that takes two parameters, the first one is the key, and the second parameter is ghe default value to return if the key isn't found.
So if you do in your example data.get('town', None), this will return None because town isn't found as a key in data.

Proper way to iterate Python dictionary and build new one

I'm trying to find a cogent way to check to see whether certain keys exist in a dictionary and use those to build a new one.
Here is my example json:
"dmarc": {
"record": "v=DMARC1; p=none; rua=mailto:dmarc.spc#test.domain; adkim=s; aspf=s",
"valid": true,
"location": "test.domain",
"warnings": [
"DMARC record at root of test.domain has no effect"
],
"tags": {
"v": {
"value": "DMARC1",
"explicit": true
},
"p": {
"value": "none",
"explicit": true
},
"rua": {
"value": [
{
"scheme": "mailto",
"address": "ssc.dmarc.spc#canada.ca",
"size_limit": null
}
],
"explicit": true
},
"adkim": {
"value": "s",
"explicit": true
},
"aspf": {
"value": "s",
"explicit": true
},
"fo": {
"value": [
"0"
],
"explicit": false
},
"pct": {
"value": 100,
"explicit": false
},
"rf": {
"value": [
"afrf"
],
"explicit": false
},
"ri": {
"value": 86400,
"explicit": false
},
"sp": {
"value": "none",
"explicit": false
}
}
}
}
What I'm specifically looking to do, is pull record, valid, location, tags-p, tags-sp, and tags-pct in a programmatic way, instead of doing a bunch of try/excepts. For example, to get valid, I do:
try:
res_dict['valid'] = jsonData['valid']
except KeyError:
res_dict['valid'] = None
Now, this is easy enough to loop/repeat for top level key/values, but how would I accomplish this for the nested key/values?

No, you don't need a try-except block for the same. You can check if the key exists using:
if jsonData.get("valid"):
res_dict["valid"] = jsonData.get("valid")
The .get("key") method returns the value for the given key, if present in the dictionary. If not, then it will return None (if get() is used with only one argument).
If you want it to return something else if it doesn't find the key then suppose:
jsonData.get("valid", "invalid_something_else")

One way of handling this is by taking advantage of the fact that the result of dict.keys can be treated as a set. See the following code.
my_keys = {'record', 'valid', 'location'} # you can add more here
new_dict = {}
available_keys = my_keys & jsonData.keys()
for key in available_keys:
new_dict[key] = jsonData[key]
Above, we define the keys we are interested in within the my_keys set. We then get the available keys by taking the intersection of the keys in the dictionary and the keys we are interested in. This, in effect, only gets the keys that we are interested in that are also defined in the dictionary. Finally, we just iterate through the available_keys and build the new dictionary.
However, this does not set keys to None if they do not exist in the input dictionary. For that, it may be best to use the get method as mentioned in other answers, like so:
my_keys = ['record', 'valid', 'location'] # you can add more here
new_dict = {}
for key in my_keys:
new_dict[key] = jsonData.get(key)
The get method allows us to attempt to get the value for a key in the dictionary. If that key is not defined, it returns None. You can also change the returned default by adding an extra argument to the get method like so new_dict[key] = jsonData.get(key, "some other default value")

Simple: instead of dict['key'] use
dict.get('key', {}) for all nodes that are not leaves, and
dict.get('key', DEFAULT) for leaves, where DEFAULT is whatever you need.
If you omit DEFAULT and 'key' is absent, you get None. See the docs.
E.g.:
jsonData.get('record', "") # empty string if no 'record' key
jsonData.get('valid', False) # False if no 'valid' key
jsonData.get('location') # None if no 'location'
jsonData.get('tags', {}).get('p') # None if no 'tags' and/or no 'p'
jsonData.get('tags', {}).get('p', {}) # {} if no 'tags' and/or no 'p'
jsonData.get('tags', {}).get('p', {}).get('explicit', False) # and so on
The above presumes that you don't traverse lists (JSON arrays). If you do, you can still use
dict.get('key', [])
but if you have to dive deeper from there, you will probably have to loop over list items.

modifying json - deleting certain elements within a json structure using python

My json structure is as follows :
"AGENT": {
"pending": [],
"active": null,
"completed": [
**{
"result": {
"job1.AGENT": "SUCCESS",
"job2.AGENT": "SUCCESS"
},
"return_value": {
"job1.AGENT": "",
"job2.AGENT": ""
},
"visible": true,
"global": true,
"locale": [
"en_US"
],
"complete_time": "2018-01-24T17:44:33.484Z",
"persist": true,
"type": "script",
"script": "<script_name>.py",
"preset_status": "CONFIGURING",
"parameters": {},
"submit_time": "2018-01-24T17:44:26.747Z"
}**,
{
"result": {
..
},
"return_value": {
..
},
"visible": true,
"global": true,
"locale": [
"en_US"
],
"complete_time": "2018-04-2T17:44:40.049Z",
"submit_time": "2018-04-2T17:44:26.817Z"
}
I need to delete the entire result block based on complete_time, like delete the result block before 2018-04-03
How can i acheive this in python ?
I have tried the following so far :
json_data = json.dumps(data)
item_dict = json.loads(data)
print item_dict["AGENT"]["completed"][0]["complete_time"]
This prints the complete time. However my problem is "AGENT" is not a constant string. The string can vary. Also I will need to figure out the logic to remove the entire json block based on complete_time

Ok, I assume that you were able to correctly load the json into a Python dictionnary, let call it item_dict, but the key may vary.
What you need now it to walk inside that Python object, and decode the complete_time field. Unfortunately, Python strptime does not know about the Z time zone, so we will have to skip that last character.
Additionaly, you should never modify a collection object while iterating it, so the bullet proof way is to store indices to remove and later remove them. Code could be:
datelimit = datetime.datetime(2018, 4, 1) # limit date for completed_time
to_remove = []
dateformat = '%Y-%m-%dT%H:%M:%S.%f'
for k, v in item_dict.items(): # enumerate top_level objects
for i, block in enumerate(v['completed']): # enumerate inner blocks
complete_time = datetime.datetime.strptime( # skip last char from complete_time
block["complete_time"][:-1], dateformat)
# print(k, i, complete_time) # uncomment for tests
if complete_time < datelimit: # too old
to_remove.append((k, i)) # store the index for later processing
for k, i in reversed(to_remove): # start from the end to keep consistent indices
del item_dict[k]["completed"][i] # actual deletion