Develop Reference

Storing List of Dict in a DynamoDB Table

I want to store a list of Tags of an Elasticsearch domain in a DynamoDB and i'm facing some errors.
I'm getting the list of tags using list_tags() function :
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/es.html#ElasticsearchService.Client.list_tags
response = client.list_tags(
ARN='string'
)
It returns that :
{
'TagList': [
{
'Key': 'string',
'Value': 'string'
},
]
}
Here's what they say in the doc :
Response Structure
(dict) --
The result of a ListTags operation. Contains tags for all requested Elasticsearch domains.
TagList (list) --
List of Tag for the requested Elasticsearch domain.
(dict) --
Specifies a key value pair for a resource tag.
Now i tried to insert the list in DynamoDB using various ways but i'm always getting errors :
':TagList': {
'M': response_list_tags['TagList']
},
Invalid type for parameter ExpressionAttributeValues.:TagList.M, value: [{'Key': 'Automation', 'Value': 'None'}, {'Key': 'Owner', 'Value': 'owner'}, {'Key': 'BU', 'Value': 'DS'}, {'Key': 'Support', 'Value': 'teamA'}, {'Key': 'Note', 'Value': ''}, {'Key': 'Environment', 'Value': 'dev'}, {'Key': 'Creator', 'Value': ''}, {'Key': 'SubProject', 'Value': ''}, {'Key': 'DateTimeTag', 'Value': 'nodef'}, {'Key': 'ApplicationCode', 'Value': ''}, {'Key': 'Criticity', 'Value': '3'}, {'Key': 'Name', 'Value': 'dev'}], type: , valid types: : ParamValidationError
Tried with L instead of M and got this :
Unknown parameter in ExpressionAttributeValues.:TagList.L[11]: "Value", must be one of: S, N, B, SS, NS, BS, M, L, NULL, BOOL: ParamValidationError

The specific error you are getting is because you are using the native DynamoDB document item JSON format which requires that any attribute value (including key-values in a map, nested in a list) to be fully qualified with a type as a key-value.
There are two ways you can do that and from your question I'm not sure if you wanted to store those key-value tag objects as a list, or you wanted to store that as an actual map in Dynamo.
Either way, I recommend you JSON encode you list and just store it in DynamoDB as a string value. There's no really good reason why you would want to go through the trouble of storing that as a map or list.
However, if you really wanted to you could do the conversion to the DynamoDB native JSON and store as a map. You will end up with something like this:
':TagList': {
'M': {
'Automation': { 'S': 'None' },
'Owner': {'S': 'owner'},
'BU': {'S': 'DS'},
'Support': {'S': 'teamA'}
...
}
}
Another possibility would be using a list of maps:
':TagList': {
'L': [
'M': {'Key': {'S': 'Automation'}, 'Value': { 'S': 'None' }},
'M': {'Key': {'S': 'Owner'}, 'Value' : {'S': 'owner'}},
'M': {'Key': {'S': 'BU'}, 'Value': {'S': 'DS'}},
'M': {'Key': {'S': 'Support'}, 'Value': {'S': 'teamA'}}
...
]
}
But in my experience I have never gotten any real value out of storing data like this in Dynamo. Instead, storing those tags as a JSON string is both easier and less error prone. You end up with this:
':TagList': {
'S': '{\'Key\': \'Automation\', \'Value\': \'None\'}, {\'Key\': \'Owner\', \'Value\': \'owner\'}, {\'Key\': \'BU\', \'Value\': \'DS\'}, {\'Key\': \'Support\', \'Value\': \'teamA\'}, ... }'
}
And all you have to do is writhe the equivalent of:
':TagList': {
'S': json.dumps(response_list_tags['TagList'])
}

Thank you Mike, i eneded up with a similar solution. I stored the Tag List as String like that :
':TagList': {
'S': str(response_list_tags['TagList'])
}
Then to convert the string to a list for a later use i did this :
import ast
...
TagList= ast.literal_eval(db_result['Item']['TagList']['S'])

Parsing dict with nested dict of lists

I am trying to parse the following dict
{'IsTruncated': False,
'MaxItems': '100',
'ResourceRecordSets': [{'Name': 'test.com.',
{'Name': '1.test.com.',
'ResourceRecords': [{'Value': '10.0.0.1'}],
{'Name': '2.test.com.',
'ResourceRecords': [{'Value': '10.0.0.2'}],
}
The output I am looking for is:
1.test.com 10.0.0.1
2.test.com 10.0.0.2
I have tried:
for resource in response['ResourceRecordSets']:
print("{} {}".format(resource['Name'], resource['ResourceRecords'] ))
and
for resource in response['ResourceRecordSets']:
print("{} {}".format(resource['Name'], resource['ResourceRecords'][0] ))
Is there a simple way to access this dict key/values within the nested list?

Assuming that your dictionary should look like this:
{'IsTruncated': False,
'MaxItems': '100',
'ResourceRecordSets': [
{'Name': 'test.com.',
'ResourceRecords' : [{'Value' : '<mising ip addr>'}]},
{'Name': '1.test.com.',
'ResourceRecords': [{'Value': '10.0.0.1'}]},
{'Name': '2.test.com.',
'ResourceRecords': [{'Value': '10.0.0.2'}]},
]
Try this:
for resource in response['ResourceRecordSets']:
print("{} {}".format(resource['Name'], resource['ResourceRecords'][0]['Value']))

Removing mulitple key items with dict comprehension python

I want to remove multiple key from my json and I'm using dictionary comprehensions like this
remove_key = ['name', 'description']
data = {'full-workflow': {'task_defaults': {'retry': {'count': 3, 'delay': 2}}, 'tasks': {'t1': {'action': 'nn.postgres.export-db', 'description': 'Dump prod DB with default settings', 'input': {'database': 'prod', 'filepath': '/var/tmp/prod-dump.pgsql', 'host': 'postgres.local', 'password': 'mypass', 'username': 'myuser'}, 'name': 'db export', 'on-success': ['t2']}, 't2': {'action': 'nn.aws.upload-to-s3', 'description': 'Upload to S3 bucket for development', 'input': {'sourcepath': '{{ tasks(t1).result.filepath }}', 'targetpath': 's3://mybucket/prod-dump.pgsql'}, 'name': 'Upload to S3', 'on-success': ['t3'], 'retry': {'count': 5, 'delay': 5}}, 't3': {'action': 'nn.shell.command', 'description': 'Remove temp file from batch folder ', 'input': {'cmd': 'rm {{ tasks(t1).result.filepath }}'}, 'name': 'Remove temp file', 'on-complete': ['t4']}, 't4': {'action': 'nn.notify.send-mail', 'description': 'Send email to admin containing target path', 'input': {'from': 'bot#nn.io', 'message': 'DB Dump {{ tasks(t1).result.filepath }} was stored to S3', 'subject': 'Prod DB Backup', 'to': 'admin#nn.io'}, 'name': 'Send email', 'target': 'nn'}}}, 'version': '2'}
def remove_additional_key(data):
return {
key: data[key] for key in data if key not in remove_key
}
then just
new_data = remove_additional_key(data)
Because this is nested dict, I want to remove_key from tasks dict, so what am I doing wrong?

Your data are nested dictionaries. If you want to remove any data with a key contained in remove_key, then I suggest a recursive approach. This can be achieved, based on your exiting function remove_additional_key, with ease:
def remove_additional_key(data):
sub_data = {}
for key in data:
if key not in remove_key:
if isinstance(data[key], dict): # if is dictionary
sub_data[key] = remove_additional_key(data[key])
else:
sub_data[key] = data[key]
return sub_data
new_data = remove_additional_key(data)
Note, if a entry is a dictionary can be checked by isinstance(data[key], dict). See How to check if a variable is a dictionary in Python?

You have a dictionary with a few nested dictionaries. If you know exactly which subdictionary you have those keys to remove, you can use:
data['full-workflow']['tasks']['t1'].pop('name')
Using the lookup approach (key: data[key]) in a dictionary comprehension is inefficient however, on such a small data amount you won't notice a difference.
If you don't know the exact path to your nested dictionary, you can use a function (posting another answer for your convenience)
def delete_keys_from_dict(d, lst_keys):
for k in lst_keys:
try:
del dict_del[k]
except KeyError:
pass
for v in dict_del.values():
if isinstance(v, dict):
delete_keys_from_dict(v, lst_keys)
return dict_del
Then you could call
delete_keys_from_dict(data, ['name', 'description'])
Needless to say, should you have name key in multiple nested dictionaries, all of them would be deleted, so be careful.

Travers through a nested json object and store values- Python

This is a follow up on this question. Question
Also this question is similar but does not solve my problem Question2
I am trying to parse a nested json to get Check how many children a specific location has, I am trying to check if "children:" = None and increment counter to check how many levels down i need to go in order to get the lowest child, or
A more efficient solution would be:
I need to get all the child values into a list and keep going until "children:" = None.
The Json object can increase in the amount of children so we can have multiple level of children, Which can get messy if I want to nest the list and get the values, How could I do it dynamically?
{
'locationId': 'location1',
'name': 'Name',
'type': 'Ward',
'patientId': None,
'children': [{
'locationId': 'Child_location2',
'name': 'Name',
'type': 'Bed',
'patientId': None,
'children': [{
'locationId': 'Child_Child_location3',
'name': 'Name',
'type': 'HospitalGroup',
'patientId': None,
'children': None
}]
}, {
'locationId': 'location4',
'name': 'Name',
'type': 'Hospital',
'patientId': None,
'children': None
}, {
'locationId': 'location5',
'name': 'Name',
'type': 'Bed',
'patientId': None,
'children': None
}, {
'locationId': 'location6',
'name': 'Name',
'type': 'Bed',
'patientId': None,
'children': None
}, {
'locationId': 'location27',
'name': 'Name',
'type': 'Bed',
'patientId': None,
'children': None
}]
}
I tried to do something like this
import requests
def Get_Child(URL, Name):
headers = {
'accept': 'text/plain',
}
response = requests.get(
URL + Name,
headers=headers)
json_data = response.json()
print (json_data)
list = []
for locationId in json_data['locationId']:
list.append(locationId)
for children in locationId['children']:
list.append(children)
but that give me the following error,
for children in locationId['locationId']: TypeError: string indices must be integers

Your code shows append, but you ask for a count. Here is a recursive way to get the number of children in this JSON if I am understanding you correctly:
def get_children(body, c=1):
if not body.get('children'):
c += 1
elif isinstance(body.get('children'), list):
c += 1
for subchild in body.get('children'):
c += 1
get_children(subchild, c)
return c
counts = get_children(your_json_blob)
print(counts)
>>> 7
Edit: I purposely did not use if/else because I don't know if you can have subchildren that are dict rather than list which would mean you would need extra conditions, but that's up to you if that ends up being the case.

I found a solution fro my problem,
The following code will get all the children and append them to a list
class Children():
def Get_All_Children(self,json_input, lookup_key):
if isinstance(json_input, dict):
for k, v in json_input.items():
if k == lookup_key:
yield v
else:
yield from self.Get_All_Children(v, lookup_key)
elif isinstance(json_input, list):
for item in json_input:
yield from self.Get_All_Children(item, lookup_key)
for locations in self.Get_All_Children(self.json_data, 'locationId'):
self.mylist.append(locations)

How can I recursively add dictionaries in Python from JSON?

Dear Stackoverflow Members,
I have this JSON array, and it consists of the following items (basically):
{
{
'Name': 'x',
'Id': 'y',
'Unsusedstuff' : 'unused',
'Unsusedstuff2' : 'unused2',
'Children': []
},
{ 'Name' : 'xx',
'Id': 'yy',
'Unsusedstuff' : 'unused',
'Unsusedstuff2' : 'unused2',
'Children': [{
'Name': 'xyx',
'Id' : 'yxy',
'Unsusedstuff' : 'unused',
'Unsusedstuff2' : 'unused2',
'Children: []
}
You get the basic idea. I want to emulate this (and just grab the id and the name and the structure) in a Python-list using the following code:
names = []
def parseNames(col):
for x in col:
if(len(x['Children'])> 0):
names.append({'Name' : x['Name'], 'Id' : x['Id'], 'Children' : parseNames(x['Children'])})
else:
return {'Name' : x['Name'], 'Id' : x['Id']}
But, it only seems to return the first 'root' and the first nested folder, but doesn't loop through them all.
How would I be able to fix this?
Greetings,
Mats

The way I read this, you're trying to convert this tree into a tree of nodes which only have Id, Name and Children. In that case, the way I'd think of it is as cleaning nodes.
To clean a node:
Create a node with the Name and Id of the original node.
Set the new node's Children to be the cleaned versions of the original node's children. (This is the recursive call.)
In code, that would be:
def clean_node(node):
return {
'Name': node['Name'],
'Id': node['Id'],
'Children': map(clean_node, node['Children']),
}
>>> print map(clean_node, data)
[{'Name': 'x', 'Children': [], 'Id': 'y'}, {'Name': 'xx', 'Children': [{'Name': 'xyx', 'Children': [], 'Id': 'yxy'}], 'Id': 'yy'}]
I find it's easier to break recursive problems down like this - trying to use global variables turns simple things very confusing very quickly.

Check this
def parseNames(col):
for x in col:
if(len(x['Children'])> 0):
a = [{
'Name' : x['Name'],
'Id' : x['Id'],
'Children' : x['Children'][0]['Children']
}]
parseNames(a)
names.append({'Name' : x['Name'], 'Id' : x['Id']})
return names
Output I get is
[{'Name': 'x', 'Id': 'y'}, {'Name': 'xx', 'Id': 'yy'}, {'Name': 'xx', 'Id': 'yy'}]

You can parse a Json object with this:
import json
response = json.loads(my_string)
Now response is a dictionary with the keys of every Json object.