Develop Reference

customized OrderedDict format and transfer into dictionary

I have an issue about how to customize OrderedDict format and convert them into a json or dictionary format(but be able to reset the key names and the structure). I have the data below:
result= OrderedDict([('index', 'cfs_fsd_00001'),
('host', 'GIISSP707'),
('source', 'D:\\usrLLSS_SS'),
('_time', '2018-11-02 14:43:30.000 EDT'),
('count', '153')])
...However, I want to change the format like this:
{
"servarname": {
"index": "cfs_fsd_00001",
"host": "GIISSP707"
},
"times": '2018-11-02 14:43:30.000 EDT',
"metricTags": {
"source": 'D:\\ddevel.log'"
},
"metricName": "serverice count",
"metricValue": 153,
"metricType": "count"
}
I will be really appreciate your help. Basically the output I got is pretty flat. But I want to customize the structure. The original structure is
OrderedDict([('index', 'cfs_fsd_00001'),('host', 'GIISSP707').....]).
The output I want to achieve is {"servarname"{"index":"cfs_fsd_00001","host":"GIISSP707"},......

You can simply reference the result dict with the respective keys that you want your target data structure to have:
{
"servarname": {
"index": result['index'],
"host": result['host']
},
"times": result['_time'],
"metricTags": {
"source": result['source']
},
"metricName": "serverice count",
"metricValue": result['count'],
"metricType": "count"
}

No sure how flexible you need for your method. I assume you have a few common keys in your OrderedDict and you want to find the metric there, then reformat them into a new dict. Here is a short function which is implemented in python 3 and I hope it could help.
from collections import OrderedDict
import json
def reformat_ordered_dict(dict_result):
"""Reconstruct the OrderedDict result into specific format
This method assumes that your input OrderedDict has the following common keys: 'index',
'host', 'source', '_time', and a potential metric whcih is subject to change (of course
you can support more metrics with minor tweak of the code). The function also re-map the
keys (for example, mapping '_time' to 'times', pack 'index' and 'source' into 'servarname'
).
:param dict_result: the OrderedDict
:return: the reformated OrderedDict
"""
common_keys = ('index', 'host', 'source', '_time')
assert all(common_key in dict_result for common_key in common_keys), (
'You have to provide all the commen keys!')
# write common keys
reformated = OrderedDict()
reformated["servarname"] = OrderedDict([
("index", dict_result['index']),
("host", dict_result['host'])
])
reformated["times"] = dict_result['_time']
reformated["metricTags"] = {"source": dict_result['source']}
# write metric
metric = None
for key in dict_result.keys():
if key not in common_keys:
metric = key
break
assert metric is not None, 'Cannot find metric in the OrderedDict!'
# don't know where you get this value. But you can customize it if needed
# for exampe if the metric name is needed here
reformated['metricName'] = "serverice count"
reformated['metricValue'] = dict_result[metric]
reformated['metricType'] = metric
return reformated
if __name__ == '__main__':
result= OrderedDict([('index', 'cfs_fsd_00001'),
('host', 'GIISSP707'),
('source', 'D:\\usrLLSS_SS'),
('_time', '2018-11-02 14:43:30.000 EDT'),
('count', '153')])
reformated = reformat_ordered_dict(result)
print(json.dumps(reformated))

Attendees in Google Calendar not always in the same order

So I've just started using the google calendar api and I've had good results so far. I add attendees with their name and email in the events dictionary, like so
events = {
# other stuff here and then this
'items': [
# lots of stuff here, followed by
'attendees': [
{
'email': email1,
'displayName': name1
},
{
'email': email2,
'displayName': name2
},
],
###
]
}
Adding them goes fine, but when I access them, I'm never guaranteed of their order. I thought I could just access the emails like this
for event in events['items']:
print "email1 = " + str(event['attendees'][0]['email'])
print "email2 = " + str(event['attendees'][1]['email'])
and I can. And I've learned that lists in python always have their order preserved, which is convenient because I wanted to access the dictionaries inside the list with the index of the list. But what I've learned is that sometimes the 0 index refers to email1 and sometimes it refers to email2. Why the inconsistency? Is it inherent to the google calendar api or is there something about having dictionary objects within a python list that relaxes the order preservation assumption? Or is it something else I'm missing?

So, as #Colonel Thirty Two pointed out, while lists preserve order, how google return data into a list may not be in the same order as it was submitted to them. This order inconsistency with attendees is inconvenient if you are wanting to count on that order for the retrieval of attendees with something like
for event in events['items']:
print "email1 = " + str(event['attendees'][0]['email'])
print "email2 = " + str(event['attendees'][1]['email'])
What's more is that very few fields are writable with the google calendar api. What is writable, however, is comments. So, I added a value to that field to make the attendees identifiable. Like so
'attendees': [
{
'email': agent_email,
'displayName': agent_name,
'comment': 'agent'
},
{
'email': event_attendee_email,
'displayName': event_attendee_name,
'comment': 'client'
},
Using comment as an identifier helped me in retrieving the email and displayName of each attendee with a simple if-statement.
for i in range(len(event['attendees'])):
if event['attendees'][i]['comment'] == 'client':
event['attendees'][i]['displayName'] = event_attendee_name
event['attendees'][i]['email'] = event_attendee_email
Now it doesn't matter that the google calendar api submits my attendees back to me in a different order than the one in which I added them. I can now retrieve the attendees so I can change them. Problem solved.

Example of update_item in dynamodb boto3

Following the documentation, I'm trying to create an update statement that will update or add if not exists only one attribute in a dynamodb table.
I'm trying this
response = table.update_item(
Key={'ReleaseNumber': '1.0.179'},
UpdateExpression='SET',
ConditionExpression='Attr(\'ReleaseNumber\').eq(\'1.0.179\')',
ExpressionAttributeNames={'attr1': 'val1'},
ExpressionAttributeValues={'val1': 'false'}
)
The error I'm getting is:
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the UpdateItem operation: ExpressionAttributeNames contains invalid key: Syntax error; key: "attr1"
If anyone has done anything similar to what I'm trying to achieve please share example.

Found working example here, very important to list as Keys all the indexes of the table, this will require additional query before update, but it works.
response = table.update_item(
Key={
'ReleaseNumber': releaseNumber,
'Timestamp': result[0]['Timestamp']
},
UpdateExpression="set Sanity = :r",
ExpressionAttributeValues={
':r': 'false',
},
ReturnValues="UPDATED_NEW"
)

Details on dynamodb updates using boto3 seem incredibly sparse online, so I'm hoping these alternative solutions are useful.
get / put
import boto3
table = boto3.resource('dynamodb').Table('my_table')
# get item
response = table.get_item(Key={'pkey': 'asdf12345'})
item = response['Item']
# update
item['status'] = 'complete'
# put (idempotent)
table.put_item(Item=item)
actual update
import boto3
table = boto3.resource('dynamodb').Table('my_table')
table.update_item(
Key={'pkey': 'asdf12345'},
AttributeUpdates={
'status': 'complete',
},
)

If you don't want to check parameter by parameter for the update I wrote a cool function that would return the needed parameters to perform a update_item method using boto3.
def get_update_params(body):
"""Given a dictionary we generate an update expression and a dict of values
to update a dynamodb table.
Params:
body (dict): Parameters to use for formatting.
Returns:
update expression, dict of values.
"""
update_expression = ["set "]
update_values = dict()
for key, val in body.items():
update_expression.append(f" {key} = :{key},")
update_values[f":{key}"] = val
return "".join(update_expression)[:-1], update_values
Here is a quick example:
def update(body):
a, v = get_update_params(body)
response = table.update_item(
Key={'uuid':str(uuid)},
UpdateExpression=a,
ExpressionAttributeValues=dict(v)
)
return response

The original code example:
response = table.update_item(
Key={'ReleaseNumber': '1.0.179'},
UpdateExpression='SET',
ConditionExpression='Attr(\'ReleaseNumber\').eq(\'1.0.179\')',
ExpressionAttributeNames={'attr1': 'val1'},
ExpressionAttributeValues={'val1': 'false'}
)
Fixed:
response = table.update_item(
Key={'ReleaseNumber': '1.0.179'},
UpdateExpression='SET #attr1 = :val1',
ConditionExpression=Attr('ReleaseNumber').eq('1.0.179'),
ExpressionAttributeNames={'#attr1': 'val1'},
ExpressionAttributeValues={':val1': 'false'}
)
In the marked answer it was also revealed that there is a Range Key so that should also be included in the Key. The update_item method must seek to the exact record to be updated, there's no batch updates, and you can't update a range of values filtered to a condition to get to a single record. The ConditionExpression is there to be useful to make updates idempotent; i.e. don't update the value if it is already that value. It's not like a sql where clause.
Regarding the specific error seen.
ExpressionAttributeNames is a list of key placeholders for use in the UpdateExpression, useful if the key is a reserved word.
From the docs, "An expression attribute name must begin with a #, and be followed by one or more alphanumeric characters". The error is because the code hasn't used an ExpressionAttributeName that starts with a # and also not used it in the UpdateExpression.
ExpressionAttributeValues are placeholders for the values you want to update to, and they must start with :

Based on the official example, here's a simple and complete solution which could be used to manually update (not something I would recommend) a table used by a terraform S3 backend.
Let's say this is the table data as shown by the AWS CLI:
$ aws dynamodb scan --table-name terraform_lock --region us-east-1
{
"Items": [
{
"Digest": {
"S": "2f58b12ae16dfb5b037560a217ebd752"
},
"LockID": {
"S": "tf-aws.tfstate-md5"
}
}
],
"Count": 1,
"ScannedCount": 1,
"ConsumedCapacity": null
}
You could update it to a new digest (say you rolled back the state) as follows:
import boto3
dynamodb = boto3.resource('dynamodb', 'us-east-1')
try:
table = dynamodb.Table('terraform_lock')
response = table.update_item(
Key={
"LockID": "tf-aws.tfstate-md5"
},
UpdateExpression="set Digest=:newDigest",
ExpressionAttributeValues={
":newDigest": "50a488ee9bac09a50340c02b33beb24b"
},
ReturnValues="UPDATED_NEW"
)
except Exception as msg:
print(f"Oops, could not update: {msg}")
Note the : at the start of ":newDigest": "50a488ee9bac09a50340c02b33beb24b" they're easy to miss or forget.

Small update of Jam M. Hernandez Quiceno's answer, which includes ExpressionAttributeNames to prevent encoutering errors such as:
"errorMessage": "An error occurred (ValidationException) when calling the UpdateItem operation:
Invalid UpdateExpression: Attribute name is a reserved keyword; reserved keyword: timestamp",
def get_update_params(body):
"""
Given a dictionary of key-value pairs to update an item with in DynamoDB,
generate three objects to be passed to UpdateExpression, ExpressionAttributeValues,
and ExpressionAttributeNames respectively.
"""
update_expression = []
attribute_values = dict()
attribute_names = dict()
for key, val in body.items():
update_expression.append(f" #{key.lower()} = :{key.lower()}")
attribute_values[f":{key.lower()}"] = val
attribute_names[f"#{key.lower()}"] = key
return "set " + ", ".join(update_expression), attribute_values, attribute_names
Example use:
update_expression, attribute_values, attribute_names = get_update_params(
{"Status": "declined", "DeclinedBy": "username"}
)
response = table.update_item(
Key={"uuid": "12345"},
UpdateExpression=update_expression,
ExpressionAttributeValues=attribute_values,
ExpressionAttributeNames=attribute_names,
ReturnValues="UPDATED_NEW"
)
print(response)

An example to update any number of attributes given as a dict, and keep track of the number of updates. Works with reserved words (i.e name).
The following attribute names shouldn't be used as we will overwrite the value: _inc, _start.
from typing import Dict
from boto3 import Session
def getDynamoDBSession(region: str = "eu-west-1"):
"""Connect to DynamoDB resource from boto3."""
return Session().resource("dynamodb", region_name=region)
DYNAMODB = getDynamoDBSession()
def updateItemAndCounter(db_table: str, item_key: Dict, attributes: Dict) -> Dict:
"""
Update item or create new. If the item already exists, return the previous value and
increase the counter: update_counter.
"""
table = DYNAMODB.Table(db_table)
# Init update-expression
update_expression = "SET"
# Build expression-attribute-names, expression-attribute-values, and the update-expression
expression_attribute_names = {}
expression_attribute_values = {}
for key, value in attributes.items():
update_expression += f' #{key} = :{key},' # Notice the "#" to solve issue with reserved keywords
expression_attribute_names[f'#{key}'] = key
expression_attribute_values[f':{key}'] = value
# Add counter start and increment attributes
expression_attribute_values[':_start'] = 0
expression_attribute_values[':_inc'] = 1
# Finish update-expression with our counter
update_expression += " update_counter = if_not_exists(update_counter, :_start) + :_inc"
return table.update_item(
Key=item_key,
UpdateExpression=update_expression,
ExpressionAttributeNames=expression_attribute_names,
ExpressionAttributeValues=expression_attribute_values,
ReturnValues="ALL_OLD"
)
Hope it might be useful to someone!

In a simple way you can use below code to update item value with new one:
response = table.update_item(
Key={"my_id_name": "my_id_value"}, # to get record
UpdateExpression="set item_key_name=:item_key_value", # Operation action (set)
ExpressionAttributeValues={":value": "new_value"}, # item that you need to update
ReturnValues="UPDATED_NEW" # optional for declarative message
)

Simple example with multiple fields:
import boto3
dynamodb_client = boto3.client('dynamodb')
dynamodb_client.update_item(
TableName=table_name,
Key={
'PK1': {'S': 'PRIMARY_KEY_VALUE'},
'SK1': {'S': 'SECONDARY_KEY_VALUE'}
}
UpdateExpression='SET #field1 = :field1, #field2 = :field2',
ExpressionAttributeNames={
'#field1': 'FIELD_1_NAME',
'#field2': 'FIELD_2_NAME',
},
ExpressionAttributeValues={
':field1': {'S': 'FIELD_1_VALUE'},
':field2': {'S': 'FIELD_2_VALUE'},
}
)

using previous answer from eltbus , it worked for me , except for minor bug,
You have to delete the extra comma using update_expression[:-1]

How to retrieve key value pair from a cfg file

I am new to python.
I have a config file as shown below in the same order. I need to retrieve key, value pairs from config file and will use those values in my script
# Name and details
(
{ group => 'abc',
host => 'pqr.com',
user => 'anonymous',
src => '/var/tmp',
dest => '/tmp',
},
{ group => 'abc',
host =>'pqr.com',
user => 'anonymous',
src => '/tmp'
dest => '/var/tmp'
},
{ group => 'pqr',
host =>'abc.com',
user => 'xyz',
src => '/home/pp',
dest => '/var/tmp',
},
{ group => 'xyz',
host =>'p.com',
user => 'x',
src => '/home/',
dest => '/tmp',
}
)
Each
{
}
is considerd as one block..Group,user,host are unique as well as repeated.
I have to read and parse the config file and display key and value pair.Pls help.
Key : group,Value : 'abc'(say)
key : host ,Value :'pqr.com'
Key : user, Value :'anonymous'
Key : src,Value :'/var/tmp',
key : dest,Value : '/tmp'
Thank you,
I have written the code which displays keys and values taking cfg file(shown above) as an input.
idx = 0
dictList = []
while True:
try:
start = config.index("{", idx)
end = config.index("}", start+1)
slice = config[start+1:end-1]
sliceList = [s.strip() for s in slice.split(",") if s.strip()]
dd = {}
for item in sliceList:
key, value = [s.strip() for s in item.split("=>")]
print key, value
Output while displaying keys,values
key 'value'
group 'abc'
host 'pqr.com'
user 'ananymous'
src '/use/tmp
Now the problem is ,how to display the value corresponding to a key.
Eg : print group- should display abc
print host should display pqr.com, and so on.

You'll probably need to parse it, here's a small example on how to do this.
import re
def parse(data):
'''Parse data block, return itertator on objects inside'''
for block in re.finditer('{[^}]*}', data, re.M): # Split to objects
obj = {}
for match in re.finditer("([a-z]+) => '([^']*)'", block.group()):
obj[match.group(1)] = match.group(2)
yield obj
Now you have two problems :)

Your data is bit malformed to be directly interpreted by Python. So you would have to per-process the data before interpreting it
Change all Occurrence of => to : : data.replace("=>",":")
Quote all the Keys : re.sub(" (\w+) ",r"'\1'",data.replace("=>",":"))
You can then feed it to ast.literal_eval
import re,ast
ast.literal_eval(re.sub(" (\w+) ",r"'\1'",data.replace("=>",":")))

http://docs.python.org/library/configparser.html
You want to try that out for this.
But your config file format will want to change to a more ini format
[section]
key = value
http://deron.meranda.us/python/demjson/
demjson also is nice for python objects -> strings and back.
I tend to use these in this situation.

List in a dictionary, looping in Python

I have the following code:
TYPES = {'hotmail':{'type':'hotmail', 'lookup':'mixed', 'dkim': 'no', 'signatures':['|S|Return-Path: postmaster#hotmail.com','|R|^Return-Path:\s*[^#]+#(?:hot|msn)','^Received: from .*hotmail.com$']},
'gmail':{'type':'gmail', 'lookup':'mixed', 'dkim': 'yes', 'signatures':['|S|Subject: unsubscribe','','','']}
}
for type_key, type in TYPES.iteritems():
for sub_type_key, sub_type in type.iteritems():
for sig in sub_type['signatures']:
if ("|S|" in sig):
#String based matching
clean_sig = sig[3:len(sig)]
if (clean_sig in file_contents):
sig_match += 1
elif ("|R|" in sig):
clean_sig = sig[3:len(sig)]
#REGMATCH later
if (sig_match == sig.count):
return sub_type['type']
return None
However, it generates the error:
for sig in sub_type['signatures']:
TypeError: string indices must be integers, not str
I assume that it would see the list being pulled from dictionary element, and allow me to loop over that?
Python newbie is a newbie :(

for type_key, type in TYPES.iteritems():
for sub_type_key, sub_type in type.iteritems():
for sig in sub_type['signatures']:
should be:
for type_key, type in TYPES.iteritems():
for sig in type['signatures']:
But 'type' is a poor name choice in this case... you don't want to shadow a builtin.
Essentially, 'type_key' has the name (either 'hotmail' or 'gmail'), and 'type' has the dictionary that is the value associated with that key. So type['signatures'] is what you're wanting.
Also, you may not need to have 'gmail' inside the nested dictionary; just return 'type_key' instead of type['type'].
Bringing it all together, maybe this will work better: (Warning: untested)
providers = {
'hotmail':{
'type':'hotmail',
'lookup':'mixed',
'dkim': 'no',
'signatures':[
'|S|Return-Path: postmaster#hotmail.com',
'|R|^Return-Path:\s*[^#]+#(?:hot|msn)',
'^Received: from .*hotmail.com$']
},
'gmail':{
'type':'gmail',
'lookup':'mixed',
'dkim': 'yes',
'signatures':['|S|Subject: unsubscribe','','','']
}
}
for provider, provider_info in providers.iteritems():
for sig in provicer_info['signatures']:
if ("|S|" in sig):
#String based matching
clean_sig = sig[3:len(sig)]
if (clean_sig in file_contents):
sig_match += 1
elif ("|R|" in sig):
clean_sig = sig[3:len(sig)]
#REGMATCH later
if (sig_match == sig.count):
return provider
return None

[Posted as an answer instead of a comment because retracile beat me to the answer, but the formatting is still a point worth making.]
Laying out the data helps to visualize it:
TYPES = {
'hotmail': {
'type': 'hotmail',
'lookup': 'mixed',
'dkim': 'no',
'signatures': ['|S|Return-Path: postmaster#hotmail.com',
'|R|^Return-Path:\s*[^#]+#(?:hot|msn)',
'^Received: from .*hotmail.com$'],
},
'gmail': {
'type': 'gmail',
'lookup': 'mixed',
'dkim': 'yes',
'signatures': ['|S|Subject: unsubscribe', '', '', ''],
},
}
Note: You can have an ending comma after the last item in a dict, list, or tuple (used above only for the dicts—it's not always more clear), and you don't have to worry about screwing around with that comma, which is a Good Thing™.