Develop Reference

Convert dictionary of lists to dictionary of dictionary

Current output is of format, dictionary of lists
{
"majestic-service-1.324.02070909": [
"/home/robotics/arm-services/FeaturesDir.yaml",
"/home/robotics/arm-service/majestic-service.tar.gz"
],
}
and I'm looking to change that output format to something like below.(dictionary of dictionary)
{
"majestic-service-1.324.02070909": {
"yaml_file": "/home/robotics/arm-services/FeaturesDir.yaml",
"tar_file": "/home/robotics/arm-services/majestic-service-1.324.02070909.tar.gz",
"kind": "FeaturesDir"
}
}
Corresponding code snippet that I've tried,
output_dict = {}
for file in application_files:
match = re.match(regex_pattern, os.path.basename(file))
if match:
if os.path.exists(os.path.join(os.path.dirname(file), "FeaturesDir.yaml")):
output_dict[file_without_extension(match.string)] = {os.path.join(os.path.dirname(file), "FeaturesDir.yaml")}
output_dict[file_without_extension(match.string)].append(file)
output_dict["Kind"] = "FeaturesDir"
elif os.path.exists(os.path.join(os.path.dirname(file), "output_path/Deviations.yaml")):
output_dict[file_without_extension(match.string)] = {os.path.join(os.path.dirname(file), "output_path/Deviations")}
output_dict[file_without_extension(match.string)].append(file)
output_dict["Kind"] = "Deviations"
# where the function file_without_extension, will return - majestic-service-1.324.02070909 from majestic-service-1.324.02070909.tar.gz
It reports the following error
AttributeError: 'set' object has no attribute 'append'
What am I doing wrong ?

The error occurs because in this line output_dict[file_without_extension(match.string)] = {os.path.join(os.path.dirname(file), "output_path/Deviations")} you're using {} brackets which for python are intended as the data type called set. Sets don't have append attribute, they have add attribute instead but it's a completely different data type. That's why it shows you that kind of error!

How to convert dictionary list?

I have a list and I would like to turn it into json. but I get the following error:
trackerror:list indices must be integers or slices, not str
and
The 'operation' tag of the JSON file is not correct or does not exist
What could it be?.
Observation:
list indices must be integers or slices, not str
It is generated in this line: url = variables["url"]
I have:
event=
[[{'operacion': 'generar','url':'xxxxxxxx', 'items': [{'unidad': 'un', codigo'001'}]
}]]
I need (json):
event=
{
"operacion":"generar",
"url":"xxxxxxxx,"
"items":[
{
"unidad_de_medida":"un",
"codigo":"001"
}
]
}
Code:
def lambda_handler(event, context):
payload=json.dumps(event,indent=4)
variables= json.loads(payload)
url =variables["url"] --------->>>>I have an error here too.<<<<<<<-----------
headers={'content-type': "application/json",'authorization': "xxxxxxxx",'cache-`enter code here`control': "no-cache",'postman-token': "xxxxxxxx"}
response = requests.request("POST", url, data=payload, headers=headers)
The webservice response indicates that it does not exist.

This should work:
event=[[{'operacion': 'generar','url':'xxxxxxxx', 'items': [{'unidad': 'un', 'codigo':'001'}]}]]
json.dumps(event[0][0])
# '{"operacion": "generar", "url": "xxxxxxxx", "items": [{"unidad": "un", "codigo": "001"}]}'

PS: Your dictionary key codigo is formatted incorrectly. If this is not simply a typo when making this post, you will need to fix this before any processing can be done. However, since Python would throw an error, I'm guessing it's just a formatting error in your post?
Your event list is one level too deep to make any sense. Based on your desired output description, you could do this:
event = [
[
{'operacion': 'generar','url':'xxxxxxxx', 'items': [
{'unidad': 'un', 'codigo': '001'}
]
}
]
]
extractedEvents = []
for e in event:
extractedEvents.append(e[0])
print(extractedEvents)
It's a bit easier to see what you're dealing with if you take the time to format the original source similar to what I've done.
By traversing into the redundant list index with e[0] we're flattening the object by one level, like you indicated you wanted

How to check if something exist, then create and save it inside an array

I'm running a script to get information from an URL and then create a Json file. With that, I'll read thought the info, save the ones that I need and then insert into the database.
But, I'm having problem with a part of this info.
Info from the URL, saved in JSON
"images": [
{
"type": "PosterPortrait",
"url": "https://ingresso-a.akamaihd.net/img/cinema/cartaz/22455-cartaz.jpg"
},
{
"type": "PosterHorizontal",
"url": "https://ingresso-a.akamaihd.net/img/cinema/cartaz/22455-destaque.jpg"
}
],
"trailers": []
This is the part and with that I have to:
Check if there is something inside trailers
Save it in a array, regarding for their "type" and "url"
Save them in my database
I made a code to do that, the problem is, I'm new to python and I don't think it's working quiet right.
insert-events.py
if(i['trailers'][0]):
a = array.arr(
array.arr('url' = i['images'][0]['url'], 'type' = i['images'][0]['type']),
array.arr('url' = i['images'][1]['url'], 'type' = i['images'][1]['type']),
array.arr('url' = i['trailers'][0]['url'], 'type' = Trailer),
array.arr('url' = i['trailers'][1]['url'], 'type' = Trailer),
)
else:
a = array.arr(
array.arr('url' = i['images'][0]['url'], 'type' = i['images'][0]['type']),
array.arr('url' = i['images'][1]['url'], 'type' = i['images'][1]['type']),
)
This is the part of the code to get the info from the JSON and, then, save it in as an array of arrays.
Error
File "insert-events.py", line 46
array.arr('url' = i['images'][0]['url'], 'type' = i['images'][0]['type']),
SyntaxError: keyword can't be an expression
So, what I'm doing it wrong? It's the array structure or when I try to save the info?
Any help would be appreciate. Thanks!

The error you are seeing is because it looks like you are trying to call a function/class with keyword arguments, but not using the keywords, instead using the string.
def foo(a, b):
pass
foo(a=1, b=2) # Correct
foo('a'=1, 'b'=2) # Incorrect
If you want to save stuff with key/value associations, I would recommend using a dict (or maybe a list of dicts in your case)
if(i['trailers'][0]):
a = [
{'url': i['images'][0]['url'], 'type': i['images'][0]['type']},
{'url': i['images'][1]['url'], 'type': i['images'][1]['type']},
] # etc.
else:
a = [
{'url': i['images'][0]['url'], 'type': i['images'][0]['type']},
{'url': i['images'][1]['url'], 'type': i['images'][1]['type']},
]
Lastly as a semi-unrelated note, checking for i['trailers'][0] will throw an IndexError if i['trailers'] is an empty list ([]). You can check for something in i['trailers'] using just if i['trailers']:.

Example of update_item in dynamodb boto3

Following the documentation, I'm trying to create an update statement that will update or add if not exists only one attribute in a dynamodb table.
I'm trying this
response = table.update_item(
Key={'ReleaseNumber': '1.0.179'},
UpdateExpression='SET',
ConditionExpression='Attr(\'ReleaseNumber\').eq(\'1.0.179\')',
ExpressionAttributeNames={'attr1': 'val1'},
ExpressionAttributeValues={'val1': 'false'}
)
The error I'm getting is:
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the UpdateItem operation: ExpressionAttributeNames contains invalid key: Syntax error; key: "attr1"
If anyone has done anything similar to what I'm trying to achieve please share example.

Found working example here, very important to list as Keys all the indexes of the table, this will require additional query before update, but it works.
response = table.update_item(
Key={
'ReleaseNumber': releaseNumber,
'Timestamp': result[0]['Timestamp']
},
UpdateExpression="set Sanity = :r",
ExpressionAttributeValues={
':r': 'false',
},
ReturnValues="UPDATED_NEW"
)

Details on dynamodb updates using boto3 seem incredibly sparse online, so I'm hoping these alternative solutions are useful.
get / put
import boto3
table = boto3.resource('dynamodb').Table('my_table')
# get item
response = table.get_item(Key={'pkey': 'asdf12345'})
item = response['Item']
# update
item['status'] = 'complete'
# put (idempotent)
table.put_item(Item=item)
actual update
import boto3
table = boto3.resource('dynamodb').Table('my_table')
table.update_item(
Key={'pkey': 'asdf12345'},
AttributeUpdates={
'status': 'complete',
},
)

If you don't want to check parameter by parameter for the update I wrote a cool function that would return the needed parameters to perform a update_item method using boto3.
def get_update_params(body):
"""Given a dictionary we generate an update expression and a dict of values
to update a dynamodb table.
Params:
body (dict): Parameters to use for formatting.
Returns:
update expression, dict of values.
"""
update_expression = ["set "]
update_values = dict()
for key, val in body.items():
update_expression.append(f" {key} = :{key},")
update_values[f":{key}"] = val
return "".join(update_expression)[:-1], update_values
Here is a quick example:
def update(body):
a, v = get_update_params(body)
response = table.update_item(
Key={'uuid':str(uuid)},
UpdateExpression=a,
ExpressionAttributeValues=dict(v)
)
return response

The original code example:
response = table.update_item(
Key={'ReleaseNumber': '1.0.179'},
UpdateExpression='SET',
ConditionExpression='Attr(\'ReleaseNumber\').eq(\'1.0.179\')',
ExpressionAttributeNames={'attr1': 'val1'},
ExpressionAttributeValues={'val1': 'false'}
)
Fixed:
response = table.update_item(
Key={'ReleaseNumber': '1.0.179'},
UpdateExpression='SET #attr1 = :val1',
ConditionExpression=Attr('ReleaseNumber').eq('1.0.179'),
ExpressionAttributeNames={'#attr1': 'val1'},
ExpressionAttributeValues={':val1': 'false'}
)
In the marked answer it was also revealed that there is a Range Key so that should also be included in the Key. The update_item method must seek to the exact record to be updated, there's no batch updates, and you can't update a range of values filtered to a condition to get to a single record. The ConditionExpression is there to be useful to make updates idempotent; i.e. don't update the value if it is already that value. It's not like a sql where clause.
Regarding the specific error seen.
ExpressionAttributeNames is a list of key placeholders for use in the UpdateExpression, useful if the key is a reserved word.
From the docs, "An expression attribute name must begin with a #, and be followed by one or more alphanumeric characters". The error is because the code hasn't used an ExpressionAttributeName that starts with a # and also not used it in the UpdateExpression.
ExpressionAttributeValues are placeholders for the values you want to update to, and they must start with :

Based on the official example, here's a simple and complete solution which could be used to manually update (not something I would recommend) a table used by a terraform S3 backend.
Let's say this is the table data as shown by the AWS CLI:
$ aws dynamodb scan --table-name terraform_lock --region us-east-1
{
"Items": [
{
"Digest": {
"S": "2f58b12ae16dfb5b037560a217ebd752"
},
"LockID": {
"S": "tf-aws.tfstate-md5"
}
}
],
"Count": 1,
"ScannedCount": 1,
"ConsumedCapacity": null
}
You could update it to a new digest (say you rolled back the state) as follows:
import boto3
dynamodb = boto3.resource('dynamodb', 'us-east-1')
try:
table = dynamodb.Table('terraform_lock')
response = table.update_item(
Key={
"LockID": "tf-aws.tfstate-md5"
},
UpdateExpression="set Digest=:newDigest",
ExpressionAttributeValues={
":newDigest": "50a488ee9bac09a50340c02b33beb24b"
},
ReturnValues="UPDATED_NEW"
)
except Exception as msg:
print(f"Oops, could not update: {msg}")
Note the : at the start of ":newDigest": "50a488ee9bac09a50340c02b33beb24b" they're easy to miss or forget.

Small update of Jam M. Hernandez Quiceno's answer, which includes ExpressionAttributeNames to prevent encoutering errors such as:
"errorMessage": "An error occurred (ValidationException) when calling the UpdateItem operation:
Invalid UpdateExpression: Attribute name is a reserved keyword; reserved keyword: timestamp",
def get_update_params(body):
"""
Given a dictionary of key-value pairs to update an item with in DynamoDB,
generate three objects to be passed to UpdateExpression, ExpressionAttributeValues,
and ExpressionAttributeNames respectively.
"""
update_expression = []
attribute_values = dict()
attribute_names = dict()
for key, val in body.items():
update_expression.append(f" #{key.lower()} = :{key.lower()}")
attribute_values[f":{key.lower()}"] = val
attribute_names[f"#{key.lower()}"] = key
return "set " + ", ".join(update_expression), attribute_values, attribute_names
Example use:
update_expression, attribute_values, attribute_names = get_update_params(
{"Status": "declined", "DeclinedBy": "username"}
)
response = table.update_item(
Key={"uuid": "12345"},
UpdateExpression=update_expression,
ExpressionAttributeValues=attribute_values,
ExpressionAttributeNames=attribute_names,
ReturnValues="UPDATED_NEW"
)
print(response)

An example to update any number of attributes given as a dict, and keep track of the number of updates. Works with reserved words (i.e name).
The following attribute names shouldn't be used as we will overwrite the value: _inc, _start.
from typing import Dict
from boto3 import Session
def getDynamoDBSession(region: str = "eu-west-1"):
"""Connect to DynamoDB resource from boto3."""
return Session().resource("dynamodb", region_name=region)
DYNAMODB = getDynamoDBSession()
def updateItemAndCounter(db_table: str, item_key: Dict, attributes: Dict) -> Dict:
"""
Update item or create new. If the item already exists, return the previous value and
increase the counter: update_counter.
"""
table = DYNAMODB.Table(db_table)
# Init update-expression
update_expression = "SET"
# Build expression-attribute-names, expression-attribute-values, and the update-expression
expression_attribute_names = {}
expression_attribute_values = {}
for key, value in attributes.items():
update_expression += f' #{key} = :{key},' # Notice the "#" to solve issue with reserved keywords
expression_attribute_names[f'#{key}'] = key
expression_attribute_values[f':{key}'] = value
# Add counter start and increment attributes
expression_attribute_values[':_start'] = 0
expression_attribute_values[':_inc'] = 1
# Finish update-expression with our counter
update_expression += " update_counter = if_not_exists(update_counter, :_start) + :_inc"
return table.update_item(
Key=item_key,
UpdateExpression=update_expression,
ExpressionAttributeNames=expression_attribute_names,
ExpressionAttributeValues=expression_attribute_values,
ReturnValues="ALL_OLD"
)
Hope it might be useful to someone!

In a simple way you can use below code to update item value with new one:
response = table.update_item(
Key={"my_id_name": "my_id_value"}, # to get record
UpdateExpression="set item_key_name=:item_key_value", # Operation action (set)
ExpressionAttributeValues={":value": "new_value"}, # item that you need to update
ReturnValues="UPDATED_NEW" # optional for declarative message
)

Simple example with multiple fields:
import boto3
dynamodb_client = boto3.client('dynamodb')
dynamodb_client.update_item(
TableName=table_name,
Key={
'PK1': {'S': 'PRIMARY_KEY_VALUE'},
'SK1': {'S': 'SECONDARY_KEY_VALUE'}
}
UpdateExpression='SET #field1 = :field1, #field2 = :field2',
ExpressionAttributeNames={
'#field1': 'FIELD_1_NAME',
'#field2': 'FIELD_2_NAME',
},
ExpressionAttributeValues={
':field1': {'S': 'FIELD_1_VALUE'},
':field2': {'S': 'FIELD_2_VALUE'},
}
)

using previous answer from eltbus , it worked for me , except for minor bug,
You have to delete the extra comma using update_expression[:-1]

MongoDB - Upsert with increment

I am trying to run the following query:
data = {
'user_id':1,
'text':'Lorem ipsum',
'$inc':{'count':1},
'$set':{'updated':datetime.now()},
}
self.db.collection('collection').update({'user_id':1}, data, upsert=True)
but the two '$' queries cause it to fail. Is it possible to do this within one statement?

First of all, when you ask a question like this it's very helpful to add information on why it's failing (e.g. copy the error).
Your query fails because you're mixing $ operators with document overrides. You should use the $set operator for the user_id and text fields as well (although the user_id part in your update is irrelevant at this example).
So convert this to pymongo query:
db.test.update({user_id:1},
{$set:{text:"Lorem ipsum", updated:new Date()}, $inc:{count:1}},
true,
false)
I've removed the user_id in the update because that isn't necessary. If the document exists this value will already be 1. If it doesn't exist the upsert will copy the query part of your update into the new document.

If you're trying to do the following:
If the doc doesn't exist, insert a new doc.
If it exists, then only increment one field.
Then you can use a combo of $setOnInsert and $inc. If the song exists then $setOnInsert won't do anything and $inc will increase the value of "listened". If the song doesn't exist, then it will create a new doc with the fields "songId" and "songName". Then $inc will create the field and set the value to be 1.
let songsSchema = new mongoose.Schema({
songId: String,
songName: String,
listened: Number
})
let Song = mongoose.model('Song', songsSchema);
let saveSong = (song) => {
return Song.updateOne(
{songId: song.songId},
{
$inc: {listened: 1},
$setOnInsert: {
songId: song.songId,
songName: song.songName,
}
},
{upsert: true}
)
.then((savedSong) => {
return savedSong;
})
.catch((err) => {
console.log('ERROR SAVING SONG IN DB', err);
})