Develop Reference

Python combine JSON arrays if value matches within objects irrespective of order

I wish to merge elements from a secondary JSON array into a primary array if values match.
Primary JSON:
primary = {
"#odata.context": "https://api.securitycenter.microsoft.com/api/$metadata#Collection(microsoft.windowsDefenderATP.api.PublicAssetVulnerabilityDto)",
"value": [
{
"id": "5afa3afc92a7c63d4b70129e0a6f33f63a427e21-_-CVE-2020-6494-_-microsoft-_-edge_chromium-based-_-81.0.416.77-_-",
"cveId": "CVE-2020-6494",
"machineId": "e5bc9d7e413ddd7902c2932e418702b84d0cc07",
"fixingKbId": null,
"productName": "edge_chromium-based",
"productVendor": "microsoft",
"productVersion": "81.0.416.77",
"severity": "Low"
},
{
"id": "7a704e17d1c2977c0e7b665fb18ae6e1fe7f3283-_-CVE-2016-3348-_-microsoft-_-windows_server_2012_r2-_-6.3.9600.19728-_-3185911",
"cveId": "CVE-2016-3348",
"machineId": "7a704e17d1c2977c0e7b665fb18ae6e1fe7f3283",
"fixingKbId": "3185911",
"productName": "windows_server_2012_r2",
"productVendor": "microsoft",
"productVersion": "6.3.9600.19728",
"severity": "Low"
}]}
Secondary JSON:
secondary = {
"#odata.context": "https://api.securitycenter.microsoft.com/api/$metadata#Machines",
"value": [
{
"id": "e5bc9d7e413ddd7902c2932e418702b84d0cc07",
"computerDnsName": "mymachine1.contoso.com",
"firstSeen": "2018-08-02T14:55:03.7791856Z",
"lastSeen": "2018-08-02T14:55:03.7791856Z",
"osPlatform": "Windows10" "Windows11",
"version": "1709",
"osProcessor": "x64",
"lastIpAddress": "172.17.230.209",
"lastExternalIpAddress": "167.220.196.71",
"osBuild": 18209,
"healthStatus": "Active",
"rbacGroupId": 140,
"rbacGroupName": "The-A-Team",
"riskScore": "Low",
"exposureLevel": "Medium",
"isAadJoined": true,
"aadDeviceId": "80fe8ff8-2624-418e-9591-41f0491218f9",
"machineTags": [ "test tag 1", "test tag 2" ]
}
]
}
I would like to merge based off these key values:
machineId (primary json)
id (secondary json)
Within my json, you can see that the 0th element in Primary JSON has the same "machineId" as the 0th element in the secondary JSON's "id". Therefore I would like to append the lastSeen key and value from the secondary JSON into the primary JSON as shown beneath:
Merged:
merged = {
"id": "5afa3afc92a7c63d4b70129e0a6f33f63a427e21-_-CVE-2020-6494-_-microsoft-_-edge_chromium-based-_-81.0.416.77-_-",
"cveId": "CVE-2020-6494",
"machineId": "e5bc9d7e413ddd7902c2932e418702b84d0cc07",
"fixingKbId": null,
"productName": "edge_chromium-based",
"productVendor": "microsoft",
"productVersion": "81.0.416.77",
"severity": "Low"
"lastSeen": "2018-08-02T14:55:03.7791856Z"
}
I was making use of this beneath solution to merge:
for element in a:
for lastSeen in s:
print(primary['machineId'])
print(secondary['id'])
if primary['machineId'] == secondary['id']:
element["lastSeen"] = secondary["lastSeen"]
print(element)
This wasn't working to well for me. I understand there may be a more optimized solution opposed to making use of embedded for loops that I'm not aware of.
If any clarify is required, please get in touch. Thanks.
Essentially in pseudo-code:
For each value in primary:
For each value in secondary:
If primary["id"] equals secondary["machineId"]
Append secondary["lastSeen"] to primary

Campaign Manager 360 API Issue

I have developed a python script that allows users to create trackers, assign them, and change their URLs very quickly and automatically. When I tried to develop further into banners, I have faced several issues that were all resolved except for 2
The first error is: Required parameter: profileID
The second error is: the creative must have "Include in Rotation" set to "Yes' - even though I have the parameter "active": True literally everywhere in the code
The below is a snippet of the code if anyone can help.. Thanks in advance
ad_tracker = [{
"active": "true",
"archived": False,
"campaignId": campaign_id,
"creativeRotation": creative_rotation_tracker,
"deliverySchedule": delivery_schedule,
"endTime": datetime.datetime.strftime(
datetime.datetime.strptime(
"%sT04:59:59Z" % campaign["endDate"], "%Y-%m-%dT%H:%M:%SZ"
)
+ datetime.timedelta(days=1),
"%Y-%m-%dT%H:%M:%SZ",
),
"name": f"{creative_names_ids_URLs[0][0]}-{placement_name}"[0:255],
"placementAssignments": [placement_assignment],
"startTime": "%sT05:00:00Z" % time.strftime("%Y-%m-%d"),
"type": "AD_SERVING_TRACKING",
}]
ad_non_tracker = [{
"active": True,
"archived": False,
"campaignId": campaign_id,
"creativeRotation": creative_rotation_non_tracker,
"deliverySchedule": delivery_schedule_non_tracker,
"endTime": datetime.datetime.strftime(
datetime.datetime.strptime(
"%sT04:59:59Z" % campaign["endDate"], "%Y-%m-%dT%H:%M:%SZ"
)
+ datetime.timedelta(days=1),
"%Y-%m-%dT%H:%M:%SZ",
),
"name": f"{creative_names_ids_URLs[0][0]}-{placement_name}"[0:255],
"placementAssignments": [placement_assignment],
"startTime": "%sT05:00:00Z" % time.strftime("%Y-%m-%d"),
"type": "AD_SERVING_STANDARD_AD",
}]
request = service.ads().insert(profileId=profile_id, body=ad_tracker)
request = service.ads().insert(profileId=profile_id, body=ad_non_tracker)
Note: Profile ID, Advertiser ID and Campaign ID are all inputs
I am looping on an excel sheet to get them altogether at once.. if any additional information is needed, it will surely be provided
Thanks a lot

I am unable to find attribute values from JSON data in python

I want to find id and options in this JSON data.
Here's What I did so far.
data = """
"list": null,
"promotionID": "",
"isFreeShippingApplicable": true,
"html": "\n\n\n<div class=\"b-product-tile-price\">\n \n \n\n\n\n<span class=\"b-product-tile-price-outer\">\n <span class=\"b-product-tile-price-item\">\n 1200 €\n\n\n </span>\n</span>\n\n</div>\n\n"
},
"longDescription": "<ul>\n\t<li>STYLE: BQ4420-100</li>\n\t<li>Laufsohle: Gummi</li>\n\t<li>Obermaterial: beschichtetes Leder, Textil</li>\n\t<li>Innenmaterial: Textil</li>\n</ul>\n",
"shortDescription": null,
"availability": {
"messages": [
"Sofort lieferbar"
],
"inStockDate": null,
"custom": {
"code": null,
"label": null,
"orderable": true,
"sizeSelectable": true,
"badge": false
"""
find_values = json.loads(data)
id = find_values["id"]
variables = find_product_data["variables"]
print(id, variables)
The output is an erro but when I try to get the values of first the attribute action, it gets returned but not the others.

You can't access the id directly, because it is nested inside another dictionary. What you have to do is get that dict first and then access the id.
find_values = json.loads(data)
product = find_values["product"]
id_value = product("id")
If you are working with an IDE it could help to debug your code and see how the dict is actually nested.

Getting KeyError when parsing JSON in Python for following response

TL;DR:
Confused on how to parse following JSON response and get the value of [status of 12345 of dynamicValue_GGG of payload] in this case.
Full question:
I get the following as (sanitized) response upon hitting a REST API via Python code below:
response = requests.request("POST", url, data=payload,
headers=headers).json()
{
"payload": {
"name": "asdasdasdasd",
"dynamicValue_GGG": {
"12345": {
"model": "asad",
"status": "active",
"subModel1": {
"dynamicValue_67890": {
"model": "qwerty",
"status": "active"
},
"subModel2": {
"dynamicValue_33445": {
"model": "gghjjj",
"status": "active"
},
"subModel3": {
"dynamicValue_66778": {
"model": "tyutyu",
"status": "active"
}
}
}
},
"date": "2016-02-04"
},
"design": "asdasdWWWsaasdasQ"
}
If I do a type(response['payload']), it gives me 'dict'.
Now, I'm trying to parse the response above and fetch certain keys and values out of it. The problem is that I'm not able to iterate through using "index" and rather have to specify the "key", but then the response has certain "keys" that are dynamically generated and sent over. For instance, the keys called "dynamicValue_GGG", "dynamicValue_66778" etc are not static unlike the "status" key.
I can successfully parse by mentioning like:
print response['payload']['dynamicValue_GGG']['12345'][status]
in which case I get the expected output = 'active'.
However, since I have no control on 'dynamicValue_GGG', it would work only if I can specify something like this instead:
print response['payload'][0][0][status]
But the above line gives me error: " KeyError: 0 " when the python code is executed.
Is there someway in which I can use the power of both keys as well as index together in this case?

The order of values in a dictionary in Python are random, so you cannot use indexing. You'll have to iterate over all elements, potentially recursive, and test to see if it's the thing you're looking for. For example:
def find_submodels(your_dict):
for item_key, item_values in your_dict.items():
if 'status' in item_values:
print item_key, item_values['status']
if type(item_values) == dict:
find_submodels(item_values)
find_submodels(your_dict)
Which would output:
12345 active
dynamicValue_67890 active
dynamicValue_66778 active
dynamicValue_33445 active

How to access a possibly existing element derived from a JSON data tree?

I have a rather complex data structure derived from JSON with json.load(<filehandle>), e.g.
{
"sensors": {
"measurements": [
[
null,
null,
null
],
[
{
"sensor": "Fields 341",
"remote": null,
"mark": "reliable",
"duration": 81.54296875
},
{
"sensor": "Fields 341",
"remote": null,
"mark": "reliable",
"duration": 61.54534875
},
{
"sensor": "Fields 341",
"remote": null,
"mark": "reliable",
"duration": 61.89378945
}
]
]
}
}
The result are nested dictionaries and lists, e.g.
{u'sensors': {u'measurements': [[None, None, None],
[{u'duration': 81.54296875,
u'mark': u'reliable',
u'remote': None,
u'sensor': u'Fields 341'},
{u'duration': 61.545348750000002,
u'mark': u'reliable',
u'remote': None,
u'sensor': u'Fields 341'},
{u'duration': 61.89378945,
u'mark': u'reliable',
u'remote': None,
u'sensor': u'Fields 341'}]]}}
The goal is to take the data and stuff it in a database via sqlalchemy. Like this:
for item in sensors["measurements"]:
for j in range(len(sensors["measurements"])):
DBTable.DBField = item[j]["sensor"]
...
The problem with this is: Any parts in the data could be replaced with null/None in case the measurements where erroneous (like the first one). If so the DBFields should be assigned None. If this is the case the assignment raises an exception:
TypeError: 'NoneType' object is unsubscriptable
Is there a sane way to get the for loop to run without raising the exception (maybe using .get() of somekind)? If not, what is the best way to handle this non-existent data structures?

You can preconstruct a dictionary with defaults that you like:
defaults = {"sensor": None, "remote": None", "mark": None, "duration": None }
Then you can do:
DBTable.DBField = item.get(j, defaults)["sensor"]
You could also use a collections.defaultdict for the defaults item, which will allow you to provide a default for any key accessed.

May be this would work
for item in sensors["measurements"]:
for j in range(len(sensors["measurements"])):
try:
DBTable.DBField = item[j]["sensor"]
except:
DBTable.DBField = None