Develop Reference

Python combine JSON arrays if value matches within objects irrespective of order

I wish to merge elements from a secondary JSON array into a primary array if values match.
Primary JSON:
primary = {
"#odata.context": "https://api.securitycenter.microsoft.com/api/$metadata#Collection(microsoft.windowsDefenderATP.api.PublicAssetVulnerabilityDto)",
"value": [
{
"id": "5afa3afc92a7c63d4b70129e0a6f33f63a427e21-_-CVE-2020-6494-_-microsoft-_-edge_chromium-based-_-81.0.416.77-_-",
"cveId": "CVE-2020-6494",
"machineId": "e5bc9d7e413ddd7902c2932e418702b84d0cc07",
"fixingKbId": null,
"productName": "edge_chromium-based",
"productVendor": "microsoft",
"productVersion": "81.0.416.77",
"severity": "Low"
},
{
"id": "7a704e17d1c2977c0e7b665fb18ae6e1fe7f3283-_-CVE-2016-3348-_-microsoft-_-windows_server_2012_r2-_-6.3.9600.19728-_-3185911",
"cveId": "CVE-2016-3348",
"machineId": "7a704e17d1c2977c0e7b665fb18ae6e1fe7f3283",
"fixingKbId": "3185911",
"productName": "windows_server_2012_r2",
"productVendor": "microsoft",
"productVersion": "6.3.9600.19728",
"severity": "Low"
}]}
Secondary JSON:
secondary = {
"#odata.context": "https://api.securitycenter.microsoft.com/api/$metadata#Machines",
"value": [
{
"id": "e5bc9d7e413ddd7902c2932e418702b84d0cc07",
"computerDnsName": "mymachine1.contoso.com",
"firstSeen": "2018-08-02T14:55:03.7791856Z",
"lastSeen": "2018-08-02T14:55:03.7791856Z",
"osPlatform": "Windows10" "Windows11",
"version": "1709",
"osProcessor": "x64",
"lastIpAddress": "172.17.230.209",
"lastExternalIpAddress": "167.220.196.71",
"osBuild": 18209,
"healthStatus": "Active",
"rbacGroupId": 140,
"rbacGroupName": "The-A-Team",
"riskScore": "Low",
"exposureLevel": "Medium",
"isAadJoined": true,
"aadDeviceId": "80fe8ff8-2624-418e-9591-41f0491218f9",
"machineTags": [ "test tag 1", "test tag 2" ]
}
]
}
I would like to merge based off these key values:
machineId (primary json)
id (secondary json)
Within my json, you can see that the 0th element in Primary JSON has the same "machineId" as the 0th element in the secondary JSON's "id". Therefore I would like to append the lastSeen key and value from the secondary JSON into the primary JSON as shown beneath:
Merged:
merged = {
"id": "5afa3afc92a7c63d4b70129e0a6f33f63a427e21-_-CVE-2020-6494-_-microsoft-_-edge_chromium-based-_-81.0.416.77-_-",
"cveId": "CVE-2020-6494",
"machineId": "e5bc9d7e413ddd7902c2932e418702b84d0cc07",
"fixingKbId": null,
"productName": "edge_chromium-based",
"productVendor": "microsoft",
"productVersion": "81.0.416.77",
"severity": "Low"
"lastSeen": "2018-08-02T14:55:03.7791856Z"
}
I was making use of this beneath solution to merge:
for element in a:
for lastSeen in s:
print(primary['machineId'])
print(secondary['id'])
if primary['machineId'] == secondary['id']:
element["lastSeen"] = secondary["lastSeen"]
print(element)
This wasn't working to well for me. I understand there may be a more optimized solution opposed to making use of embedded for loops that I'm not aware of.
If any clarify is required, please get in touch. Thanks.
Essentially in pseudo-code:
For each value in primary:
For each value in secondary:
If primary["id"] equals secondary["machineId"]
Append secondary["lastSeen"] to primary

Fastest way to generate a nested JSON using pandas

This is a sample of a real-world problem that I cannot find a way to solve.
I need to create a nested JSON from a pandas dataframe. Considering this data, I need to create a JSON object like that:
[
{
"city": "Belo Horizonte",
"by_rooms": [
{
"rooms": 1,
"total price": [
{
"total (R$)": 499,
"details": [
{
"animal": "acept",
"area": 22,
"bathroom": 1,
"parking spaces": 0,
"furniture": "not furnished",
"hoa (R$)": 30,
"rent amount (R$)": 450,
"property tax (R$)": 13,
"fire insurance (R$)": 6
}
]
}
]
},
{
"rooms": 2,
"total price": [
{
"total (R$)": 678,
"details": [
{
"animal": "not acept",
"area": 50,
"bathroom": 1,
"parking spaces": 0,
"furniture": "not furnished",
"hoa (R$)": 0,
"rent amount (R$)": 644,
"property tax (R$)": 25,
"fire insurance (R$)": 9
}
]
}
]
}
]
},
{
"city": "Campinas",
"by_rooms": [
{
"rooms": 1,
"total price": [
{
"total (R$)": 711,
"details": [
{
"animal": "acept",
"area": 42,
"bathroom": 1,
"parking spaces": 0,
"furniture": "not furnished",
"hoa (R$)": 0,
"rent amount (R$)": 690,
"property tax (R$)": 12,
"fire insurance (R$)": 9
}
]
}
]
}
]
}
]
each level can have one or more items.
Based on this answer, I have a snippet like that:
data = pd.read_csv("./houses_to_rent_v2.csv")
cols = data.columns
data = (
data.groupby(['city', 'rooms', 'total (R$)'])[['animal', 'area', 'bathroom', 'parking spaces', 'furniture',
'hoa (R$)', 'rent amount (R$)', 'property tax (R$)', 'fire insurance (R$)']]
.apply(lambda x: x.to_dict(orient='records'))
.reset_index(name='details')
.groupby(['city', 'rooms'])[['total (R$)', 'details']]
.apply(lambda x: x.to_dict(orient='records'))
.reset_index(name='total price')
.groupby(['city'])[['rooms', 'total price']]
.apply(lambda x: x.to_dict(orient='records'))
.reset_index(name='by_rooms')
)
data.to_json('./jsondata.json', orient='records', force_ascii=False)
but all those groupbys don't look very Pythonic and it's pretty slow.
Before use this method, I tried split this big dataframe into smaller ones to use individual groupbys for each level, but it's even slower than doing that way.
I tried dask, with no improvement at all.
I read about numba and cython, but I have no idea how to implement in this case. All docs that I find use only numeric data and I have string and date/datetime data too.
In my real-world problem, this data is processed to response to http request. My dataframe has 30+ columns and ~35K rows per request and it takes 45 seconds to process just this snippet.
So, there is a faster way to do that?

This can be done as list / dict comprehensions. Have not timed this, but I'm not waiting for it.
import kaggle.cli
import sys, requests
import pandas as pd
from pathlib import Path
from zipfile import ZipFile
import urllib
# fmt: off
# download data set
url = "https://www.kaggle.com/rubenssjr/brasilian-houses-to-rent"
sys.argv = [sys.argv[0]] + f"datasets download {urllib.parse.urlparse(url).path[1:]}".split(" ")
kaggle.cli.main()
zfile = ZipFile(f'{urllib.parse.urlparse(url).path.split("/")[-1]}.zip')
dfs = {f.filename: pd.read_csv(zfile.open(f)) for f in zfile.infolist()}
# fmt: on
js = [
{
"city": g[0],
"by_room": [
{
"rooms": r["rooms"],
"total_price": [
{
"total (R$)": r["total (R$)"],
"details": [
{
k: v
for k, v in r.items()
if k not in ["city", "rooms", "total (R$)"]
}
],
}
],
}
for r in g[1].to_dict("records")
],
}
for g in dfs["houses_to_rent_v2.csv"].groupby("city")
]
print(len(js), len(js[0]["by_room"]))

I needed to adapt #RobRaymond answer, because I need the inner data grouped too. So I take his code, did some adjustments and this is the final result:
import kaggle.cli
import sys, requests
import pandas as pd
from pathlib import Path
from zipfile import ZipFile
import urllib
# fmt: off
# download data set
url = "https://www.kaggle.com/rubenssjr/brasilian-houses-to-rent"
sys.argv = [sys.argv[0]] + f"datasets download {urllib.parse.urlparse(url).path[1:]}".split(" ")
kaggle.cli.main()
zfile = ZipFile(f'{urllib.parse.urlparse(url).path.split("/")[-1]}.zip')
dfs = {f.filename: pd.read_csv(zfile.open(f)) for f in zfile.infolist()}
# fmt: on
js = [
{
"city": g[0],
"by_room": [
{
"rooms": r["rooms"],
"total_price": [
{
"total (R$)": r["total (R$)"],
"details": [
{
k: v
for k, v in r.items()
if k not in ["city", "rooms", "total (R$)"]
}
],
}
],
}
for r in g[1].to_dict("records")
],
}
for g in dfs["houses_to_rent_v2.csv"].groupby("city")
]
for city in js:
rooms_qty = list(set([r['rooms'] for r in city['by_room']]))
newRooms = [{'rooms': x, 'total_price': []} for x in rooms_qty]
for r in city['by_room']:
newRooms[rooms_qty.index(r['rooms'])]'total_price'].extend(r['total_price'])
for r in newRooms:
prices = list(set([p['total (R$)'] for p in r['total_price']]))
newPrices = [{'total (R$)': x, 'details': []} for x in prices]
for price in r['total_price']:
newPrices[prices.index(price['total (R$)'])]['details'].extend(price['details'])
r['total_price'] = newPrices
city['by_room'] = newRooms
And the execution time drops to 5 seconds.

How can I define a structure of a json to transform it to csv

I have a json structured as this:
{
"data": [
{
"groups": {
"data": [
{
"group_name": "Wedding planning - places and bands (and others) to recommend!",
"date_joined": "2009-03-12 01:01:08.677427"
},
{
"group_name": "Harry Potter and the Deathly Hollows",
"date_joined": "2009-01-15 01:38:06.822220"
},
{
"group_name": "Xbox , Playstation, Wii - console fans",
"date_joined": "2010-04-02 04:02:58.078934"
}
]
},
"id": "0"
},
{
"groups": {
"data": [
{
"group_name": "Lost&Found (Strzegom)",
"date_joined": "2010-02-01 14:13:34.551920"
},
{
"group_name": "Tennis, Squash, Badminton, table tennis - looking for sparring partner (Strzegom)",
"date_joined": "2008-09-24 17:29:43.356992"
}
]
},
"id": "1"
}
]
}
How does one parse jsons in this form? Should i try building a class resembling this format? My desired output is a csv where index is an "id" and in the first column I have the most recently taken group, in the second column the second most recently taken group and so on.
Meaning the result of this would be:
most recent second most recent
0 Xbox , Playstation, Wii - console fans Wedding planning - places and bands (and others) to recommend!
1 Lost&Found (Strzegom) Tennis, Squash, Badminton, table tennis - looking for sparring partner (Strzegom)

solution could be like this:
data = json.load(f)
result = []
# it's max element in there for each id. Helping how many group_name here for this example [3,2]
max_element_group_name = [len(data['data'][i]['groups']['data']) for i in range(len(data['data']))]
max_element_group_name.sort()
for i in range(len(data['data'])):
# get id for each groups
id = data['data'][i]['id']
# sort data_joined in groups
sorted_groups_by_date = sorted(data['data'][i]['groups']['data'],key=lambda x : time.strptime(x['date_joined'],'%Y-%m-%d %H:%M:%S.%f'),reverse=True)
# get groups name using minumum value in max_element_group_name for this example [2]
group_names = [sorted_groups_by_date[j]['group_name'] for j in range(max_element_group_name[0])]
# add result list with id
result.append([id]+group_names)
# create df for list
df = pd.DataFrame(result, columns = ['id','most recent', 'second most recent'])
# it could be better.

how to extract specific data from json and put in to csv using python

I have a JSON which is in nested form. I would like to extract specific data from json and put into csv using pandas python.
data = {
"class":"hudson.model.Hudson",
"jobs":[
{
"_class":"hudson.model.FreeStyleProject",
"name":"git_checkout",
"url":"http://localhost:8080/job/git_checkout/",
"builds":[
{
"_class":"hudson.model.FreeStyleBuild",
"duration":1201,
"number":6,
"result":"FAILURE",
"url":"http://localhost:8080/job/git_checkout/6/"
}
]
},
{
"_class":"hudson.model.FreeStyleProject",
"name":"output",
"url":"http://localhost:8080/job/output/",
"builds":[
]
},
{
"_class":"org.jenkinsci.plugins.workflow.job.WorkflowJob",
"name":"pipeline_test",
"url":"http://localhost:8080/job/pipeline_test/",
"builds":[
{
"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun",
"duration":9274,
"number":85,
"result":"SUCCESS",
"url":"http://localhost:8080/job/pipeline_test/85/"
},
{
"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun",
"duration":4251,
"number":84,
"result":"SUCCESS",
"url":"http://localhost:8080/job/pipeline_test/84/"
}
]
}
]
}
From the above JSON i want to fetch jobs name value and builds result value . I am new to python any help will be appreciated .
Till now i have tried
main_data = data['jobs]
json_normalize(main_data,['builds'],
record_prefix='jobs_', errors='ignore')
which gives information only build key values and not the name of job .
Can anyone help ?
Expected Output:

Considering only first build result value you can need to be in csv column you can achieve this using pandas.
data = {
"class": "hudson.model.Hudson",
"jobs": [
{
"_class": "hudson.model.FreeStyleProject",
"name": "git_checkout",
"url": "http://localhost:8080/job/git_checkout/",
"builds": [
{
"_class": "hudson.model.FreeStyleBuild",
"duration": 1201,
"number": 6,
"result": "FAILURE",
"url": "http://localhost:8080/job/git_checkout/6/"
}
]
},
{
"_class": "hudson.model.FreeStyleProject",
"name": "output",
"url": "http://localhost:8080/job/output/",
"builds": []
},
{
"_class": "org.jenkinsci.plugins.workflow.job.WorkflowJob",
"name": "pipeline_test",
"url": "http://localhost:8080/job/pipeline_test/",
"builds": [
{
"_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
"duration": 9274,
"number": 85,
"result": "SUCCESS",
"url": "http://localhost:8080/job/pipeline_test/85/"
},
{
"_class": "org.jenkinsci.plugins.workflow.job.WorkflowRun",
"duration": 4251,
"number": 84,
"result": "SUCCESS",
"url": "http://localhost:8080/job/pipeline_test/84/"
}
]
}
]
}
main_data = data.get('jobs')
res = {'name':[], 'result':[]}
for name_dict in main_data:
res['name'].append(name_dict.get('name','NA'))
resultval = name_dict['builds'][0].get('result') if len(name_dict['builds'])>0 else 'NA'
res['result'].append(resultval)
print(res)
import pandas as pd
df = pd.DataFrame(res)
df.to_csv("/home/file_timer/jobs.csv", index=False)
Check the csv file output
name,result
git_checkout,FAILURE
output,NA
pipeline_test,SUCCESS
If 'NA' result want to skip then
main_data = data.get('jobs')
res = {'name':[], 'result':[]}
for name_dict in main_data:
if len(name_dict['builds'])==0:
continue
res['name'].append(name_dict.get('name', 'NA'))
resultval = name_dict['builds'][0].get('result')
res['result'].append(resultval)
print(res)
import pandas as pd
df = pd.DataFrame(res)
df.to_csv("/home/akash.pagar/shell_learning/file_timer/jobs.csv", index=False)
Output will bw like
name,result
git_checkout,FAILURE
pipeline_test,SUCCESS

Simply with build number,
for job in data.get('jobs'):
for build in job.get('builds'):
print(job.get('name'), build.get('number'), build.get('result'))
gives the result
git_checkout 6 FAILURE
pipeline_test 85 SUCCESS
pipeline_test 84 SUCCESS
If you want to get the result of latest build, and pretty sure about the build number always in decending order,
for job in data.get('jobs'):
if job.get('builds'):
print(job.get('name'), job.get('builds')[0].get('result'))
and if you are not sure the order,
for job in data.get('jobs'):
if job.get('builds'):
print(job.get('name'), sorted(job.get('builds'), key=lambda k: k.get('number'))[-1].get('result'))
then the result will be:
git_checkout FAILURE
pipeline_test SUCCESS

Assuming last build is the last element of its list and you don't care about jobs with no builds, this does:
import pandas as pd
#data = ... #same format as in the question
z = [(job["name"], job["builds"][-1]["result"]) for job in data["jobs"] if len(job["builds"])]
df = pd.DataFrame(data=z, columns=["name", "result"])
#df.to_csv #TODO
Also we don't necessarily need pandas to create the csv file.
You could do:
import csv
#z = ... #see previous code block
with open("f.csv", 'w') as fp:
csv.writer(fp).writerows([("name", "result")] + z)

JSON parse Python

I have a json data that i got from VK.
{
"response": [{
"id": 156603484,
"name": "Equestria in the Space",
"screen_name": "equestriaspace",
"is_closed": 0,
"type": "group",
"is_admin": 1,
"admin_level": 3,
"is_member": 1,
"description": "Официально сообщество Equestria in the Space!",
"photo_50": "https://pp.userap...089/u0_mBSE4E34.jpg",
"photo_100": "https://pp.userap...088/O6vENP0IW_w.jpg",
"photo_200": "https://pp.userap...086/rwntMz6YwWM.jpg"
}]
}
So i wanted to print only "name" but when i did it it gave me an error
TypeError: list indices must be integers or slices, not str
My code is:
method_url = 'https://api.vk.com/method/groups.getById?'
data = dict(access_token=access_token, gid=group_id)
response = requests.post(method_url, data)
result = json.loads(response.text)
print (result['response']['name'])
Any idea how can i fix it? In google i found how to parse json with one array. But here is two or something
P.S dont beat me so much. I am new in Python, just learning

What sort of data structure is the value of the key response?
i.e. how would you get it if I gave you the following instead?
"response": [{
"id": 156603484,
"name": "Equestria in the Space",
"screen_name": "equestriaspace",
"is_closed": 0,
"type": "group",
"is_admin": 1,
"admin_level": 3,
"is_member": 1,
"description": "Официально сообщество Equestria in the Space!",
"photo_50": "https://pp.userap...089/u0_mBSE4E34.jpg",
"photo_100": "https://pp.userap...088/O6vENP0IW_w.jpg",
"photo_200": "https://pp.userap...086/rwntMz6YwWM.jpg"
},
{
"not_a_real_response": "just some garbage actually"
}]
You would need to pick out the first response in that array of responses. As nice people in the comments have already told you.
name = result['response'][0]['name']