I want to remove multiple key from my json and I'm using dictionary comprehensions like this
remove_key = ['name', 'description']
data = {'full-workflow': {'task_defaults': {'retry': {'count': 3, 'delay': 2}}, 'tasks': {'t1': {'action': 'nn.postgres.export-db', 'description': 'Dump prod DB with default settings', 'input': {'database': 'prod', 'filepath': '/var/tmp/prod-dump.pgsql', 'host': 'postgres.local', 'password': 'mypass', 'username': 'myuser'}, 'name': 'db export', 'on-success': ['t2']}, 't2': {'action': 'nn.aws.upload-to-s3', 'description': 'Upload to S3 bucket for development', 'input': {'sourcepath': '{{ tasks(t1).result.filepath }}', 'targetpath': 's3://mybucket/prod-dump.pgsql'}, 'name': 'Upload to S3', 'on-success': ['t3'], 'retry': {'count': 5, 'delay': 5}}, 't3': {'action': 'nn.shell.command', 'description': 'Remove temp file from batch folder ', 'input': {'cmd': 'rm {{ tasks(t1).result.filepath }}'}, 'name': 'Remove temp file', 'on-complete': ['t4']}, 't4': {'action': 'nn.notify.send-mail', 'description': 'Send email to admin containing target path', 'input': {'from': 'bot#nn.io', 'message': 'DB Dump {{ tasks(t1).result.filepath }} was stored to S3', 'subject': 'Prod DB Backup', 'to': 'admin#nn.io'}, 'name': 'Send email', 'target': 'nn'}}}, 'version': '2'}
def remove_additional_key(data):
return {
key: data[key] for key in data if key not in remove_key
}
then just
new_data = remove_additional_key(data)
Because this is nested dict, I want to remove_key from tasks dict, so what am I doing wrong?
Your data are nested dictionaries. If you want to remove any data with a key contained in remove_key, then I suggest a recursive approach. This can be achieved, based on your exiting function remove_additional_key, with ease:
def remove_additional_key(data):
sub_data = {}
for key in data:
if key not in remove_key:
if isinstance(data[key], dict): # if is dictionary
sub_data[key] = remove_additional_key(data[key])
else:
sub_data[key] = data[key]
return sub_data
new_data = remove_additional_key(data)
Note, if a entry is a dictionary can be checked by isinstance(data[key], dict). See How to check if a variable is a dictionary in Python?
You have a dictionary with a few nested dictionaries. If you know exactly which subdictionary you have those keys to remove, you can use:
data['full-workflow']['tasks']['t1'].pop('name')
Using the lookup approach (key: data[key]) in a dictionary comprehension is inefficient however, on such a small data amount you won't notice a difference.
If you don't know the exact path to your nested dictionary, you can use a function (posting another answer for your convenience)
def delete_keys_from_dict(d, lst_keys):
for k in lst_keys:
try:
del dict_del[k]
except KeyError:
pass
for v in dict_del.values():
if isinstance(v, dict):
delete_keys_from_dict(v, lst_keys)
return dict_del
Then you could call
delete_keys_from_dict(data, ['name', 'description'])
Needless to say, should you have name key in multiple nested dictionaries, all of them would be deleted, so be careful.
Related
I'm getting members from a Google group using the directory API. This is what is returned:
{'etag': '"..."',
'kind': 'admin#directory#members',
'members': [{'email': 'a#a.com',
'etag': '"..."',
'id': '....',
'kind': 'admin#directory#member',
'role': 'MEMBER',
'type': 'USER'},
{'email': 'b#a.com',
'etag': '"..."',
'id': '...',
'kind': 'admin#directory#member',
'role': 'MEMBER',
'type': 'USER'},
...],
'nextPageToken': '...'}
There are multiple pages being returned and I want to append the next set of members to the members key of this dictionary...I think.
Here is my code so far:
for group in groups_list:
members_service = service.members()
member_request = members_service.list(groupKey=group['email'], maxResults=200, roles='MEMBER')
memberEmails = {}
while member_request is not None:
member_results = member_request.execute()
memberEmails.update(member_results)
member_request = members_service.list_next(member_request, member_results)
I don't think memberEmails.update(member_results) will work here because it overwrites and updates existing keys.
How can I accomplish this? or what are some alternatives?
I wonder if you want to treat memberEmails like a list and extend it with the value associated with the members key in the result. For example:
for group in groups_list:
members_service = service.members()
member_request = members_service.list(groupKey=group['email'], maxResults=200, roles='MEMBER')
member_emails = list()
while member_request is not None:
member_results = member_request.execute()
member_emails.extend(member_request['members'])
member_request = members_service.list_next(member_request, member_results)
I have a following dictionary:
dict1 = {'Diacto Group': {'id': 7547,
'type': 'user',
'name': 'Diacto Group',
'filters': [{'column': 'Core Area',
'values': ['Missiles & Aviation Technology'],
'operator': 'EQUALS',
'not': False}],
'users': [],
'virtualUsers': [],
'groups': [360305499]},
'Diacto People': {'id': 7548,
'type': 'user',
'name': 'Diacto People',
'filters': [{'column': 'Core Area',
'values': ['Aircraft Company', 'Aviation Technology'],
'operator': 'EQUALS',
'not': False}],
'users': [326197441, 1293859642],
'virtualUsers': [],
'groups': []},
}
Basically I want to extract either one of the lists from 'users' or 'groups' if they have list containing atleast one value. I want the final output to look like this:
l1 = [# Extracted list as value from 'group' key from Diacto Group key as users key was blank
# list.
[360305499],
# Extracted list as value from 'users' key from Diacto People key as groups key was
# blank list.
[326197441, 1293859642]
]
List comprehension would be more preferable if possible.
Thank you for the efforts and time you put into this.
The simplest I can think of with comprehension, provided either 'users' or 'groups' is not empty:
[v['users']+v['groups'] for v in dict1.values()]
I think it would be possible to do with list comprehension, but extremely difficult to understand.
This would be my approach:
l1 = []
for k in dict1:
if len(dict1[k]['users']) > 0:
l1.append(dict1[k]['users'])
if len(dict1[k]['groups']) > 0:
l1.append(dict1[k]['groups'])
print(l1)
Using list comprehension and filtering out cases where both "users" and "groups" are empty:
l1 = [v["users"]+v["groups"] for _, v in dict1.items() if v["users"]+v["groups"]]
I call an API and this my JSON response:
{'data': [{
'id': 'd-1225959',
'startTime': '2022-12-30T00:00:00.000Z',
'endTime': '2022-12-30T23:59:00.000Z',
'checkedInAt': None,
'checkedOutAt': None,
'status': 'PENDING',
'space': {
'id': 'd-4063963',
'name': '082',
'type': 'DESK',
'createdAt': '2021-07-06T11:48:57.000Z',
'updatedAt': '2021-07-06T11:48:57.000Z',
'isAvailable': False,
'assignedTo': None,
'locationId': '133778',
'floorId': '41681',
'floorName': 'Car Park',
'neighborhoodId': '92267',
'neighborhoodName': 'NEI1'}}
I'm struggling to get the 'space' 'id' and 'name' extracted out if I do a nested python loop like so it only returns the headers like 'id' and 'name' not the values held within.
for order in response['data']:
print(order['id'])
print(order['startTime'])
print(order['endTime'])
print(order['checkedInAt'])
print(order['checkedOutAt'])
print(order['status'])
print(order['space'])
for doc in response['space']:
print(doc['id'], doc['name'])
Any help with this would be much appreciated!
for doc in response['space']: will iterate over the keys in response['space'] dict, i.e. doc will be str.
You want to do doc = response['space'] instead and then print(doc['id']). or directly print(response['space']['id']).
Note, you may want to use dict.get() method to avoid KeyError.
# if response dict has no 'space' key, return empty dict.
# if no 'id' key - return None
space_id = response.get('space', {}).get('id')
I want to convert this nested json into a df.
Tried different functions but none works correctly.
The encoding that worked for my was -
encoding = "utf-8-sig"
[{'replayableActionOperationState': 'SKIPPED',
'replayableActionOperationGuid': 'RAO_1037351',
'failedMessage': 'Cannot replay action: RAO_1037351: com.ebay.sd.catedor.core.model.DTOEntityPropertyChange; local class incompatible: stream classdesc serialVersionUID = 7777212484705611612, local class serialVersionUID = -1785129380151507142',
'userMessage': 'Skip all mode',
'username': 'gfannon',
'sourceAuditData': [{'guid': '24696601-b73e-43e4-bce9-28bc741ac117',
'operationName': 'UPDATE_CATEGORY_ATTRIBUTE_PROPERTY',
'creationTimestamp': 1563439725240,
'auditCanvasInfo': {'id': '165059', 'name': '165059'},
'auditUserInfo': {'id': 1, 'name': 'gfannon'},
'externalId': None,
'comment': None,
'transactionId': '0f135909-66a7-46b1-98f6-baf1608ffd6a',
'data': {'entity': {'guid': 'CA_2511202',
'tagType': 'BOTH',
'description': None,
'name': 'Number of Shelves'},
'propertyChanges': [{'propertyName': 'EntityProperty',
'oldEntity': {'guid': 'CAP_35',
'name': 'DisableAsVariant',
'group': None,
'action': 'SET',
'value': 'true',
'tagType': 'SELLER'},
'newEntity': {'guid': 'CAP_35',
'name': 'DisableAsVariant',
'group': None,
'action': 'SET',
'value': 'false',
'tagType': 'SELLER'}}],
'entityChanges': None,
'primary': True}}],
'targetAuditData': None,
'conflictedGuids': None,
'fatal': False}]
This is what i tried so far, there are more tries but that got me as close as i can.
with open(r"Desktop\Ann's json parsing\report.tsv", encoding='utf-8-sig') as data_file:
data = json.load(data_file)
df = json_normalize(data)
print (df)
pd.DataFrame(df) ## The nested lists are shown as a whole column, im trying to parse those colums - 'failedMessage' and 'sourceAuditData'`I also tried json.loads/json(df) but the output isnt correct.
pd.DataFrame.from_dict(a['sourceAuditData'][0]['data']['propertyChanges'][0]) ##This line will retrive one of the outputs i need but i dont know how to perform it on the whole file.
The expected result should be a csv/xlsx file with a column and value for each row.
For your particular example:
def unroll_dict(d):
data = []
for k, v in d.items():
if isinstance(v, list):
data.append((k, ''))
data.extend(unroll_dict(v[0]))
elif isinstance(v, dict):
data.append((k, ''))
data.extend(unroll_dict(v))
else:
data.append((k,v))
return data
And given the data in your question is stored in the variable example:
df = pd.DataFrame(unroll_dict(example[0])).set_index(0).transpose()
I'm trying to convert dict for current view to format that I can use it in AngularJS object:
data = "{'root': {'host': {'hostname1': {'10.0.0.1': {}}, 'hostname2': {'10.0.0.2': {}}}, 'monitor': {'bandwidth': {'hostname1': {'10.0.0.1': {'hostname1': {'10.0.0.1': {'10': {}}}, 'hostname2': {'10.0.0.2': {'10': {}}}}}, 'hostname2': {'10.0.0.2': {'hostname1': {'10.0.0.1': {'10': {}}}, 'hostname2': {'10.0.0.2': {'10': {}}}}}}}}}"
to format with names and children values, like:
[{
name: "Node 1",
children: [{
name: "Node 1.1",
children:[{name:"Node 1.1.1"},{name: "Node 1.1.2"}]
}]},{
name: "Node 2",
children: [{name: "Node 2.1"},{name: "Node 2.2"}]
}]
I tried few different approaches, but always received partial results. For example I tried to use recursion, it went till the depth value and then ignored all other tree.
def modifydict2(data):
for key, value in data.items():
return [{'name': key, 'children':modifydict2(value)}]
As a result I received only part of my dict back. I understood that my loop never worked cause I returned value before next iteration, but not sure how to fix that:
[{'name': 'root', 'children': [{'name': 'host', 'children': [{'name': 'ctest1.prod01.weave.local', 'children': [{'name': '10.32.62.1', 'children': None}]}]}]}]
You need to append those individual values that you're currently returning to a list and then return that list. Or, using a list comprehension:
def modify_dict(d):
return [{'name': key, 'children': modify_dict(value)}
for key, value in d.items()]