I want to store a list of Tags of an Elasticsearch domain in a DynamoDB and i'm facing some errors.
I'm getting the list of tags using list_tags() function :
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/es.html#ElasticsearchService.Client.list_tags
response = client.list_tags(
ARN='string'
)
It returns that :
{
'TagList': [
{
'Key': 'string',
'Value': 'string'
},
]
}
Here's what they say in the doc :
Response Structure
(dict) --
The result of a ListTags operation. Contains tags for all requested Elasticsearch domains.
TagList (list) --
List of Tag for the requested Elasticsearch domain.
(dict) --
Specifies a key value pair for a resource tag.
Now i tried to insert the list in DynamoDB using various ways but i'm always getting errors :
':TagList': {
'M': response_list_tags['TagList']
},
Invalid type for parameter ExpressionAttributeValues.:TagList.M, value: [{'Key': 'Automation', 'Value': 'None'}, {'Key': 'Owner', 'Value': 'owner'}, {'Key': 'BU', 'Value': 'DS'}, {'Key': 'Support', 'Value': 'teamA'}, {'Key': 'Note', 'Value': ''}, {'Key': 'Environment', 'Value': 'dev'}, {'Key': 'Creator', 'Value': ''}, {'Key': 'SubProject', 'Value': ''}, {'Key': 'DateTimeTag', 'Value': 'nodef'}, {'Key': 'ApplicationCode', 'Value': ''}, {'Key': 'Criticity', 'Value': '3'}, {'Key': 'Name', 'Value': 'dev'}], type: , valid types: : ParamValidationError
Tried with L instead of M and got this :
Unknown parameter in ExpressionAttributeValues.:TagList.L[11]: "Value", must be one of: S, N, B, SS, NS, BS, M, L, NULL, BOOL: ParamValidationError
The specific error you are getting is because you are using the native DynamoDB document item JSON format which requires that any attribute value (including key-values in a map, nested in a list) to be fully qualified with a type as a key-value.
There are two ways you can do that and from your question I'm not sure if you wanted to store those key-value tag objects as a list, or you wanted to store that as an actual map in Dynamo.
Either way, I recommend you JSON encode you list and just store it in DynamoDB as a string value. There's no really good reason why you would want to go through the trouble of storing that as a map or list.
However, if you really wanted to you could do the conversion to the DynamoDB native JSON and store as a map. You will end up with something like this:
':TagList': {
'M': {
'Automation': { 'S': 'None' },
'Owner': {'S': 'owner'},
'BU': {'S': 'DS'},
'Support': {'S': 'teamA'}
...
}
}
Another possibility would be using a list of maps:
':TagList': {
'L': [
'M': {'Key': {'S': 'Automation'}, 'Value': { 'S': 'None' }},
'M': {'Key': {'S': 'Owner'}, 'Value' : {'S': 'owner'}},
'M': {'Key': {'S': 'BU'}, 'Value': {'S': 'DS'}},
'M': {'Key': {'S': 'Support'}, 'Value': {'S': 'teamA'}}
...
]
}
But in my experience I have never gotten any real value out of storing data like this in Dynamo. Instead, storing those tags as a JSON string is both easier and less error prone. You end up with this:
':TagList': {
'S': '{\'Key\': \'Automation\', \'Value\': \'None\'}, {\'Key\': \'Owner\', \'Value\': \'owner\'}, {\'Key\': \'BU\', \'Value\': \'DS\'}, {\'Key\': \'Support\', \'Value\': \'teamA\'}, ... }'
}
And all you have to do is writhe the equivalent of:
':TagList': {
'S': json.dumps(response_list_tags['TagList'])
}
Thank you Mike, i eneded up with a similar solution. I stored the Tag List as String like that :
':TagList': {
'S': str(response_list_tags['TagList'])
}
Then to convert the string to a list for a later use i did this :
import ast
...
TagList= ast.literal_eval(db_result['Item']['TagList']['S'])
Related
I am currently writing a scraper that reads from an API that contains a JSON. By doing response.json() it would return a dict where we could easily use the e.g response["object"]to get the value we want as I assume that converts it to a dict. The current mock data looks like this:
data = {
'id': 336461,
'thumbnail': '/images/product/123456?trim&h=80',
'variants': None,
'name': 'Testing',
'data': {
'Videoutgång': {
'Typ av gränssnitt': {
'name': 'Typ av gränssnitt',
'value': 'PCI Test'
}
}
},
'stock': {
'web': 0,
'supplier': None,
'displayCap': '50',
'1': 0,
'orders': {
'CL': {
'ordered': -10,
'status': 1
}
}
}
}
What I am looking after is that the API sometimes does contain "orders -> CL" but sometime doesn't . That means that both happy path and unhappy path is what I am looking for which is the fastest way to get a data from a dict.
I have currently done something like this:
data = {
'id': 336461,
'thumbnail': '/images/product/123456?trim&h=80',
'variants': None,
'name': 'Testing',
'data': {
'Videoutgång': {
'Typ av gränssnitt': {
'name': 'Typ av gränssnitt',
'value': 'PCI Test'
}
}
},
'stock': {
'web': 0,
'supplier': None,
'displayCap': '50',
'1': 0,
'orders': {
'CL': {
'ordered': -10,
'status': 1
}
}
}
}
if (
"stock" in data
and "orders" in data["stock"]
and "CL" in data["stock"]["orders"]
and "status" in data["stock"]["orders"]["CL"]
and data["stock"]["orders"]["CL"]["status"]
):
print(f'{data["stock"]["orders"]["CL"]["status"]}: {data["stock"]["orders"]["CL"]["ordered"]}')
1: -10
However my question is that I would like to know which is the fastest way to get the data from a dict if it is in the dict?
Lookups are faster in dictionaries because Python implements them using hash tables.
If we explain the difference by Big O concepts, dictionaries have constant time complexity, O(1). This is another approach using .get() method as well:
data = {
'id': 336461,
'thumbnail': '/images/product/123456?trim&h=80',
'variants': None,
'name': 'Testing',
'data': {
'Videoutgång': {
'Typ av gränssnitt': {
'name': 'Typ av gränssnitt',
'value': 'PCI Test'
}
}
},
'stock': {
'web': 0,
'supplier': None,
'displayCap': '50',
'1': 0,
'orders': {
'CL': {
'ordered': -10,
'status': 1
}
}
}
}
if (data.get('stock', {}).get('orders', {}).get('CL')):
print(f'{data["stock"]["orders"]["CL"]["status"]}: {data["stock"]["orders"]["CL"]["ordered"]}')
Here is a nice writeup on lookups in Python with list and dictionary as example.
I got your point. For this question, since your stock has just 4 values it is hard to say if .get() method will work faster than using a loop or not. If your dictionary would have more items then certainly .get() would have worked much faster but since there are few keys, using loop will not make much difference.
I have read issues here and here using keysrules and valuesrules but I've only seen them validate nested not root. I'd like to valid the top level root dict keys.
schema = {
'any_arbitrary_str': {
'type': 'dict',
'keysrules': {'type': 'string'},
'valuesrules': {'type': 'integer'},
},
}
v = Validator(schema)
v.validate({'test': {'a': 1, 'b': 2}})
print(v.errors)
In this example, I'd like to just validate that schema is dict of str: Dict[str, int] where the keys can be any arbitrary string.
I'm not sure I'm using it right docs, this fails with cerberus.schema.SchemaError: {'any_arbitrary_str': [{'keysrules': ['unknown rule'], 'valuesrules': ['unknown rule']}]} but it's still looking for any_arbitrary_str instead of any string also.
You can just nest it. Not pretty, but works. I have not found a more elegant solution yet.
schema = {
'document': {
'type': 'dict',
'keysrules': {'type': 'string'},
'valuesrules': {
'type': 'dict',
'keysrules': {'type': 'string'},
'valuesrules': {'type': 'integer'},
},
},
}
v = Validator(schema)
document_to_test = {'test': {'a': 1, 'b': 2}}
v.validate({'document': document_to_test})
print(v.errors)
I need to prepare test which will be comparing content of .json file with expected result (we want to check if values in .json are correctly generated by our dev tool).
For test I will use robot framework or unittests but I don't know yet how to parse correctly json file.
Json example:
{
"Customer": [{
"Information": [{
"Country": "",
"Form": ""
}
],
"Id": "110",
"Res": "",
"Role": "Test",
"Limit": ["100"]
}]
}
So after I execute this:
with open('test_json.json') as f:
hd = json.load(f)
I get dict 'hd' where key is:
dict_keys(['Customer'])
and values:
dict_values([[{'Information': [{'Form': '', 'Country': ''}], 'Role': 'Test', 'Id': '110', 'Res': '', 'Limit': ['100']}]])
My problem is that I don't know how to get to only one value from Dict(e.g: Role: Test), because I can get only extract whole value. I can prepare a long string to compare with but it is not best solution for tests.
Any ideas how I can get to only one row from .json file?
Your JSON has single key 'Customer' and it has a value of list type. So when you ppass dict_keys(['Customer']) you are getting list value.
>>> hd['Customer']
[{'Id': '110', 'Role': 'Test', 'Res': '', 'Information': [{'Form': '', 'Country': ''}], 'Limit': ['100']}]
First element in list:
>>> hd['Customer'][0]
{'Id': '110', 'Role': 'Test', 'Res': '', 'Information': [{'Form': '', 'Country': ''}], 'Limit': ['100']}
Now access inside dict structure using:
>>> hd['Customer'][0]['Role']
'Test'
You can compare the dict that you loaded (say hd) to the expected results dict (say expected_dict) by running
hd.items() == expected_dict.items()
I'm trying to convert dict for current view to format that I can use it in AngularJS object:
data = "{'root': {'host': {'hostname1': {'10.0.0.1': {}}, 'hostname2': {'10.0.0.2': {}}}, 'monitor': {'bandwidth': {'hostname1': {'10.0.0.1': {'hostname1': {'10.0.0.1': {'10': {}}}, 'hostname2': {'10.0.0.2': {'10': {}}}}}, 'hostname2': {'10.0.0.2': {'hostname1': {'10.0.0.1': {'10': {}}}, 'hostname2': {'10.0.0.2': {'10': {}}}}}}}}}"
to format with names and children values, like:
[{
name: "Node 1",
children: [{
name: "Node 1.1",
children:[{name:"Node 1.1.1"},{name: "Node 1.1.2"}]
}]},{
name: "Node 2",
children: [{name: "Node 2.1"},{name: "Node 2.2"}]
}]
I tried few different approaches, but always received partial results. For example I tried to use recursion, it went till the depth value and then ignored all other tree.
def modifydict2(data):
for key, value in data.items():
return [{'name': key, 'children':modifydict2(value)}]
As a result I received only part of my dict back. I understood that my loop never worked cause I returned value before next iteration, but not sure how to fix that:
[{'name': 'root', 'children': [{'name': 'host', 'children': [{'name': 'ctest1.prod01.weave.local', 'children': [{'name': '10.32.62.1', 'children': None}]}]}]}]
You need to append those individual values that you're currently returning to a list and then return that list. Or, using a list comprehension:
def modify_dict(d):
return [{'name': key, 'children': modify_dict(value)}
for key, value in d.items()]
Dear Stackoverflow Members,
I have this JSON array, and it consists of the following items (basically):
{
{
'Name': 'x',
'Id': 'y',
'Unsusedstuff' : 'unused',
'Unsusedstuff2' : 'unused2',
'Children': []
},
{ 'Name' : 'xx',
'Id': 'yy',
'Unsusedstuff' : 'unused',
'Unsusedstuff2' : 'unused2',
'Children': [{
'Name': 'xyx',
'Id' : 'yxy',
'Unsusedstuff' : 'unused',
'Unsusedstuff2' : 'unused2',
'Children: []
}
You get the basic idea. I want to emulate this (and just grab the id and the name and the structure) in a Python-list using the following code:
names = []
def parseNames(col):
for x in col:
if(len(x['Children'])> 0):
names.append({'Name' : x['Name'], 'Id' : x['Id'], 'Children' : parseNames(x['Children'])})
else:
return {'Name' : x['Name'], 'Id' : x['Id']}
But, it only seems to return the first 'root' and the first nested folder, but doesn't loop through them all.
How would I be able to fix this?
Greetings,
Mats
The way I read this, you're trying to convert this tree into a tree of nodes which only have Id, Name and Children. In that case, the way I'd think of it is as cleaning nodes.
To clean a node:
Create a node with the Name and Id of the original node.
Set the new node's Children to be the cleaned versions of the original node's children. (This is the recursive call.)
In code, that would be:
def clean_node(node):
return {
'Name': node['Name'],
'Id': node['Id'],
'Children': map(clean_node, node['Children']),
}
>>> print map(clean_node, data)
[{'Name': 'x', 'Children': [], 'Id': 'y'}, {'Name': 'xx', 'Children': [{'Name': 'xyx', 'Children': [], 'Id': 'yxy'}], 'Id': 'yy'}]
I find it's easier to break recursive problems down like this - trying to use global variables turns simple things very confusing very quickly.
Check this
def parseNames(col):
for x in col:
if(len(x['Children'])> 0):
a = [{
'Name' : x['Name'],
'Id' : x['Id'],
'Children' : x['Children'][0]['Children']
}]
parseNames(a)
names.append({'Name' : x['Name'], 'Id' : x['Id']})
return names
Output I get is
[{'Name': 'x', 'Id': 'y'}, {'Name': 'xx', 'Id': 'yy'}, {'Name': 'xx', 'Id': 'yy'}]
You can parse a Json object with this:
import json
response = json.loads(my_string)
Now response is a dictionary with the keys of every Json object.