I'm trying to list the last modified file in S3 buckets for a report but the report is showing the first modified (ie when the first file was uploaded not the last file).
I'm using this:
top_level_folders[folder]['modified'] = obj.last_modified
and adding to the report here:
report.add_row([folder[1]['name'], folder[1]['objects'],
str(round(folder[1]['size'],2)), status, folder[1]['modified']])
I've tried adding
=obj.last_modified, reverse=True but keep getting invalid syntax errors.
This is what the report looks like:
I'm not exactly sure as to what you're doing when it comes to writing to the report, but the code below will return a list of dictionaries with the name of each bucket and the time the last-modified file was last modified. E.g.,
[
{
'Folder': 'bucket_1',
'Last Modified': '2021-11-30 13:10:32+00:00'
},
{
'Folder': 'bucket_2',
'Last Modified': '2021-09-27 17:18:27+00:00'
}
]
import datetime
import boto3
s3_client = boto3.client('s3',
aws_access_key_id="AKXXXXXXXXXXXXXXXXXX",
aws_secret_access_key="YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY",
region_name="eu-west-2"
)
def find_last_modified_file_in_bucket(bucket_name: str) -> datetime:
last_modified = []
for bucket_object in s3_client.list_objects(Bucket=bucket_name)["Contents"]:
last_modified.append(bucket_object["LastModified"])
return max(last_modified)
def fetch_last_modified() -> [{}]:
last_modified_file_by_bucket: list[{}] = []
for bucket_name in list(map(lambda bucket: bucket["Name"], s3_client.list_buckets()["Buckets"])):
latest_time_of_last_modified_file: datetime = find_last_modified_file_in_bucket(bucket_name)
last_modified_file_by_bucket.append(
{
"Folder": bucket_name,
"Last Modified": str(latest_time_of_last_modified_file)
}
)
return last_modified_file_by_bucket
Without the source code or knowledge of the type of folder, I can't say with certainty how you would use the above code to update the folder, but it will likely come down to iterating over the dict returned by fetch_last_modified(). E.g.,
def update_report(report: Report, folder_with_last_modified: dict):
for folder in folder_with_last_modified:
report.add_row(folder['Folder'], folder['Last Modified'])
folder_with_last_modified = fetch_last_modified()
update_report(report, folder_with_last_modified)
I am using the python API of JDownloader myjdapi
With the device.linkgrabber.query_links() I got the following object:
{'enabled': True, 'name': 'EQJ_X8gUcAMQX13.jpg', 'packageUUID': 1581524887390, 'uuid': 1581524890696, 'url': 'https://pbs.twimg.com/media/x.jpg?name=orig', 'availability': 'ONLINE'}
Now I want to move to the download list with the function:
device.linkgrabber.move_to_downloadlist('1581524890696', '1581524887390')
The move_to_downloadlist function (githubrepo) says:
def move_to_downloadlist(self, link_ids, package_ids):
"""
Moves packages and/or links to download list.
:param package_ids: Package UUID's.
:type: list of strings.
:param link_ids: Link UUID's.
"""
params = [link_ids, package_ids]
resp = self.device.action(self.url + "/moveToDownloadlist", params)
return resp
But I get always json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The official API said its a 200 Error, and the reason can be anything.
How I can fix that?
The parameter names are link_ids and package_ids, that's plural. That would be a good indication that lists are expected here, not single values.
Try this:
device.linkgrabber.move_to_downloadlist(['1581524890696'], ['1581524887390'])
I have a small mongo DB and I am trying to write code that will reverence the value of parameters (keys) imbedded in a document.
I have a mongo DB server on localhost and am able to successfully input the json format data structure into my DB. However, when I try to reference the values of this data within, I get the errors as shown.
import pymongo
import json
import time
from datetime import datetime
server = pymongo.MongoClient('localhost')
database = server['testData']
collection = database['testCollection']
test_entry1 = '{"Level1" : {"level2_1" : {"param1" : "1.6","param2" : "32.3","param3" : "11.0"}, "level2_2" : {"param1" : "2.6","param2" : "9.3","param3" : "112.0"}}}'
mongo_import = json.loads(test_entry1)
collection.insert_one(mongo_import)
The above works fine and when I query the database I get the following response (as expected):
{ "_id" : ObjectId("5d0081e931775cbc28cf7704"), "Level1" : { "level2_1" : { "param1" : "1.6", "param2" : "32.3", "param3" : "11.0" }, "level2_2" : { "param1" : "2.6", "param2" : "9.3", "param3" : "112.0" } } }
Now, I would like to reference the data in these parameters. I would like to get a response for all of what is in "level 2_1". The attempt #I made is below...and the error received is below that.
level_1_param_1 = collection.find("level1" : "level2_1")
error:
File "Desktop/Scripts/StackOverflowSnippit.py", line 29, in <module>
level_1_param_1 = collection.find('"level1" : "level2_1"')
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymongo/collection.py", line 1456, in find
return Cursor(self, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymongo/cursor.py", line 146, in __init__
validate_is_mapping("filter", spec)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pymongo/common.py", line 452, in validate_is_mapping
"collections.Mapping" % (option,))
TypeError: filter must be an instance of dict, bson.son.SON, or other type that inherits from collections.Mapping
I would also like to be able to get the value of a parameter that is one layer lower like "level1 --> level2_1 --> param2" for instance but so far have been unable to do so.
My hope is to be able to reference the data in this structure as needed.
Your filter should have brackets:
level_1_param_1 = collection.find({"level1": "level2_1"})
I'm trying to get all index document using python client but the result show me only the first document
This is my python code :
res = es.search(index="92c603b3-8173-4d7a-9aca-f8c115ff5a18", doc_type="doc", body = {
'size' : 10000,
'query': {
'match_all' : {}
}
})
print("%d documents found" % res['hits']['total'])
data = [doc for doc in res['hits']['hits']]
for doc in data:
print(doc)
return "%s %s %s" % (doc['_id'], doc['_source']['0'], doc['_source']['5'])
try "_doc" instead of "doc"
res = es.search(index="92c603b3-8173-4d7a-9aca-f8c115ff5a18", doc_type="_doc", body = {
'size' : 100,
'query': {
'match_all' : {}
}
})
Elasticsearch by default retrieve only 10 documents. You could change this behaviour - doc here . The best practice for pagination are search after query and scroll query. It depends from your needs. Please read this answer Elastic search not giving data with big number for page size
To show all the results:
for doc in res['hits']['hits']:
print doc['_id'], doc['_source']
You can try the following query. It will return all the documents.
result = es.search(index="index_name", body={"query":{"match_all":{}}})
You can also use elasticsearch_dsl and its Search API which allows you to iterate over all your documents via the scan method.
import elasticsearch
from elasticsearch_dsl import Search
client = elasticsearch.Elasticsearch()
search = Search(using=client, index="92c603b3-8173-4d7a-9aca-f8c115ff5a18")
for hit in search.scan():
print(hit)
I dont see mentioned that the index must be refreshed if you just added data. Use this:
es.indices.refresh(index="index_name")
Following the documentation, I'm trying to create an update statement that will update or add if not exists only one attribute in a dynamodb table.
I'm trying this
response = table.update_item(
Key={'ReleaseNumber': '1.0.179'},
UpdateExpression='SET',
ConditionExpression='Attr(\'ReleaseNumber\').eq(\'1.0.179\')',
ExpressionAttributeNames={'attr1': 'val1'},
ExpressionAttributeValues={'val1': 'false'}
)
The error I'm getting is:
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the UpdateItem operation: ExpressionAttributeNames contains invalid key: Syntax error; key: "attr1"
If anyone has done anything similar to what I'm trying to achieve please share example.
Found working example here, very important to list as Keys all the indexes of the table, this will require additional query before update, but it works.
response = table.update_item(
Key={
'ReleaseNumber': releaseNumber,
'Timestamp': result[0]['Timestamp']
},
UpdateExpression="set Sanity = :r",
ExpressionAttributeValues={
':r': 'false',
},
ReturnValues="UPDATED_NEW"
)
Details on dynamodb updates using boto3 seem incredibly sparse online, so I'm hoping these alternative solutions are useful.
get / put
import boto3
table = boto3.resource('dynamodb').Table('my_table')
# get item
response = table.get_item(Key={'pkey': 'asdf12345'})
item = response['Item']
# update
item['status'] = 'complete'
# put (idempotent)
table.put_item(Item=item)
actual update
import boto3
table = boto3.resource('dynamodb').Table('my_table')
table.update_item(
Key={'pkey': 'asdf12345'},
AttributeUpdates={
'status': 'complete',
},
)
If you don't want to check parameter by parameter for the update I wrote a cool function that would return the needed parameters to perform a update_item method using boto3.
def get_update_params(body):
"""Given a dictionary we generate an update expression and a dict of values
to update a dynamodb table.
Params:
body (dict): Parameters to use for formatting.
Returns:
update expression, dict of values.
"""
update_expression = ["set "]
update_values = dict()
for key, val in body.items():
update_expression.append(f" {key} = :{key},")
update_values[f":{key}"] = val
return "".join(update_expression)[:-1], update_values
Here is a quick example:
def update(body):
a, v = get_update_params(body)
response = table.update_item(
Key={'uuid':str(uuid)},
UpdateExpression=a,
ExpressionAttributeValues=dict(v)
)
return response
The original code example:
response = table.update_item(
Key={'ReleaseNumber': '1.0.179'},
UpdateExpression='SET',
ConditionExpression='Attr(\'ReleaseNumber\').eq(\'1.0.179\')',
ExpressionAttributeNames={'attr1': 'val1'},
ExpressionAttributeValues={'val1': 'false'}
)
Fixed:
response = table.update_item(
Key={'ReleaseNumber': '1.0.179'},
UpdateExpression='SET #attr1 = :val1',
ConditionExpression=Attr('ReleaseNumber').eq('1.0.179'),
ExpressionAttributeNames={'#attr1': 'val1'},
ExpressionAttributeValues={':val1': 'false'}
)
In the marked answer it was also revealed that there is a Range Key so that should also be included in the Key. The update_item method must seek to the exact record to be updated, there's no batch updates, and you can't update a range of values filtered to a condition to get to a single record. The ConditionExpression is there to be useful to make updates idempotent; i.e. don't update the value if it is already that value. It's not like a sql where clause.
Regarding the specific error seen.
ExpressionAttributeNames is a list of key placeholders for use in the UpdateExpression, useful if the key is a reserved word.
From the docs, "An expression attribute name must begin with a #, and be followed by one or more alphanumeric characters". The error is because the code hasn't used an ExpressionAttributeName that starts with a # and also not used it in the UpdateExpression.
ExpressionAttributeValues are placeholders for the values you want to update to, and they must start with :
Based on the official example, here's a simple and complete solution which could be used to manually update (not something I would recommend) a table used by a terraform S3 backend.
Let's say this is the table data as shown by the AWS CLI:
$ aws dynamodb scan --table-name terraform_lock --region us-east-1
{
"Items": [
{
"Digest": {
"S": "2f58b12ae16dfb5b037560a217ebd752"
},
"LockID": {
"S": "tf-aws.tfstate-md5"
}
}
],
"Count": 1,
"ScannedCount": 1,
"ConsumedCapacity": null
}
You could update it to a new digest (say you rolled back the state) as follows:
import boto3
dynamodb = boto3.resource('dynamodb', 'us-east-1')
try:
table = dynamodb.Table('terraform_lock')
response = table.update_item(
Key={
"LockID": "tf-aws.tfstate-md5"
},
UpdateExpression="set Digest=:newDigest",
ExpressionAttributeValues={
":newDigest": "50a488ee9bac09a50340c02b33beb24b"
},
ReturnValues="UPDATED_NEW"
)
except Exception as msg:
print(f"Oops, could not update: {msg}")
Note the : at the start of ":newDigest": "50a488ee9bac09a50340c02b33beb24b" they're easy to miss or forget.
Small update of Jam M. Hernandez Quiceno's answer, which includes ExpressionAttributeNames to prevent encoutering errors such as:
"errorMessage": "An error occurred (ValidationException) when calling the UpdateItem operation:
Invalid UpdateExpression: Attribute name is a reserved keyword; reserved keyword: timestamp",
def get_update_params(body):
"""
Given a dictionary of key-value pairs to update an item with in DynamoDB,
generate three objects to be passed to UpdateExpression, ExpressionAttributeValues,
and ExpressionAttributeNames respectively.
"""
update_expression = []
attribute_values = dict()
attribute_names = dict()
for key, val in body.items():
update_expression.append(f" #{key.lower()} = :{key.lower()}")
attribute_values[f":{key.lower()}"] = val
attribute_names[f"#{key.lower()}"] = key
return "set " + ", ".join(update_expression), attribute_values, attribute_names
Example use:
update_expression, attribute_values, attribute_names = get_update_params(
{"Status": "declined", "DeclinedBy": "username"}
)
response = table.update_item(
Key={"uuid": "12345"},
UpdateExpression=update_expression,
ExpressionAttributeValues=attribute_values,
ExpressionAttributeNames=attribute_names,
ReturnValues="UPDATED_NEW"
)
print(response)
An example to update any number of attributes given as a dict, and keep track of the number of updates. Works with reserved words (i.e name).
The following attribute names shouldn't be used as we will overwrite the value: _inc, _start.
from typing import Dict
from boto3 import Session
def getDynamoDBSession(region: str = "eu-west-1"):
"""Connect to DynamoDB resource from boto3."""
return Session().resource("dynamodb", region_name=region)
DYNAMODB = getDynamoDBSession()
def updateItemAndCounter(db_table: str, item_key: Dict, attributes: Dict) -> Dict:
"""
Update item or create new. If the item already exists, return the previous value and
increase the counter: update_counter.
"""
table = DYNAMODB.Table(db_table)
# Init update-expression
update_expression = "SET"
# Build expression-attribute-names, expression-attribute-values, and the update-expression
expression_attribute_names = {}
expression_attribute_values = {}
for key, value in attributes.items():
update_expression += f' #{key} = :{key},' # Notice the "#" to solve issue with reserved keywords
expression_attribute_names[f'#{key}'] = key
expression_attribute_values[f':{key}'] = value
# Add counter start and increment attributes
expression_attribute_values[':_start'] = 0
expression_attribute_values[':_inc'] = 1
# Finish update-expression with our counter
update_expression += " update_counter = if_not_exists(update_counter, :_start) + :_inc"
return table.update_item(
Key=item_key,
UpdateExpression=update_expression,
ExpressionAttributeNames=expression_attribute_names,
ExpressionAttributeValues=expression_attribute_values,
ReturnValues="ALL_OLD"
)
Hope it might be useful to someone!
In a simple way you can use below code to update item value with new one:
response = table.update_item(
Key={"my_id_name": "my_id_value"}, # to get record
UpdateExpression="set item_key_name=:item_key_value", # Operation action (set)
ExpressionAttributeValues={":value": "new_value"}, # item that you need to update
ReturnValues="UPDATED_NEW" # optional for declarative message
)
Simple example with multiple fields:
import boto3
dynamodb_client = boto3.client('dynamodb')
dynamodb_client.update_item(
TableName=table_name,
Key={
'PK1': {'S': 'PRIMARY_KEY_VALUE'},
'SK1': {'S': 'SECONDARY_KEY_VALUE'}
}
UpdateExpression='SET #field1 = :field1, #field2 = :field2',
ExpressionAttributeNames={
'#field1': 'FIELD_1_NAME',
'#field2': 'FIELD_2_NAME',
},
ExpressionAttributeValues={
':field1': {'S': 'FIELD_1_VALUE'},
':field2': {'S': 'FIELD_2_VALUE'},
}
)
using previous answer from eltbus , it worked for me , except for minor bug,
You have to delete the extra comma using update_expression[:-1]