Develop Reference

Boto3: Lambda Get lastRecordedPullTime in the return

As specified in Boto3 documentation, if I use describe_image I should get a of informations, more importantly, lastRecordedPullTime.
But when I use it in my Lambda, I only get a little 8 out of 11 informations.
When I use aws-cli, I do get all the informations, included lastRecordedPullTime.
Did I miss something ?
dictKwarg = {'nextToken':nextTk, 'maxResults':1000, 'repositoryName':repository}
images = ecrClient.describe_images(**dictKwarg)
for img in images['imageDetails']:
for k,v in img.items():
print(f"{k}")
returns:
registryId
repositoryName
imageDigest
imageTags
imageSizeInBytes
imagePushedAt
imageManifestMediaType
artifactMediaType
Thanks!

boto3 in a lambda function is not a full version and often its not up to date. Thus, it is quite common for some parameters and attributes to be missing. Thus, you should bundle latest boto3 in your function yourself, for example in a lambda layer.
If you do not want to make your own layer, you can try using publicly available ones, such as from here.

Cloudformation with Lambda Custom Resources error - Python

I have been looking at developing some custom resources via the use of Lambda from CloudFormation (CF) and have been looking at using the custom resource helper, but it started off ok then the CF stack took ages to create or delete. When I checked the cloud watch logs I noticed there was an error after running the create or cloud functions in my Lambda.
[7cfecd7b-69df-4408-ab12-a764a8bf674e][2021-02-07 12:41:12,535][ERROR] send(..) failed executing requests.put(..):
Formatting field not found in record: 'requestid'
I noticed some others had this issue but no resolution. I have used the generic code from the link below, my custom code works and completes but it looks like passing an update to CF. I looked through the crhelper.py the only reference I can find for 'requestid' is this :
logfmt = '[%(requestid)s][%(asctime)s][%(levelname)s] %(message)s \n'
mainlogger.handlers[0].setFormatter(logging.Formatter(logfmt))
return logging.LoggerAdapter(mainlogger, {'requestid': event['RequestId']})
Reference

To understand the error that you're having we need to look at a reproducible code example of what you're doing. Take into consideration that every time that you have some kind of error on a custom resource operation it may take ages to finished, as you have noticed.
But, there is a good alternative to the original custom resource helper that you're using, and, in my experience, this works very well while maintaining the code much simpler (thanks to a good level of abstraction) and follows the best practices. This is the
custom resource helper framework, as explained on this AWS blog.
You can find more details about the implementation on github here.
Basically, after downloading the needed dependencies and load them on your lambda (this depends on how you handle custom lambda dependencies), you can manage your custom resources operations like this:
from crhelper import CfnResource
import logging
logger = logging.getLogger(__name__)
# Initialise the helper
helper = CfnResource()
try:
## put here your initial code for every operation
pass
except Exception as e:
helper.init_failure(e)
#helper.create
def create(event, context):
logger.info("Got Create")
print('Here we are creating some cool stuff')
#helper.update
def update(event, context):
logger.info("Got Update")
print('Here you update the things you want')
#helper.delete
def delete(event, context):
logger.info("Got Delete")
print('Here you handle the delete operation')
# this will call the defined function according the
# cloudformation operation that is running (create, update or delete)
def handler(event, context):
helper(event, context)

How to configure CorsRule for CDK using python

I am trying to figure out the proper syntax for setting up cors on an s3 bucket using CDK (python). The class aws_s3.CorsRule requires 3 params (allowed_methods, allowed_origins, max_age=None). I am trying to specify the allowed_methods which takes in a list of methods but the bases is enum.Enum. So how do I create a list of these methods. This is what I have tried but it doesn't pass validation.
s3.Bucket(self, "StaticSiteBucket",
bucket_name="replaceMeWithBucketName",
versioned=True,
removal_policy=core.RemovalPolicy.DESTROY,
website_index_document="index.html",
cors=s3.CorsRule(allowed_methods=[s3.HttpMethods.DELETE],allowed_origins=["*"],max_age=3000)
)
The only thing Im focused on is the cors line:
cors=s3.CorsRule(allowed_methods=[s3.HttpMethods.DELETE],allowed_origins=["*"],max_age=3000)
Trying to read the documentation is like peeling an onion.
https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_s3/HttpMethods.html#aws_cdk.aws_s3.HttpMethods
I tried calling each one individually as you can see using s3.HttpMethods.DELETE but that fails when it tries to synthesize.

looks like you at least forgot to wrap the param you pass to cors as a list. I agree that the docs are a bit of a rabbit hole, but you can see the Bucket docs specifies the cors param as (Optional[List[CorsRule]])
This is mine:
from aws_cdk import core
from aws_cdk import aws_s3
from aws_cdk import aws_apigateway
aws_s3.Bucket(self,
'my_bucket',
bucket_name='my_bucket',
removal_policy=core.RemovalPolicy.DESTROY,
cors=[aws_s3.CorsRule(
allowed_headers=["*"],
allowed_methods=[aws_s3.HttpMethods.PUT],
allowed_origins=["*"])
])
So yours should be:
cors=[s3.CorsRule(
allowed_methods=[s3.HttpMethods.DELETE],
allowed_origins=["*"],
max_age=3000)]

Test a lambda that is using chalice with moto?

I want to test my lambda in local using boto3, moto, pytest. This lambda is using chalice. When I call the function I try to insert a fake event to make it run but it still missing the context object.
If someone know how to test it the cleanest way possible it'll be great.
I tried to add objects in my s3 and retrieve events from it
I tried to simulate fake events
#app.on_s3_event(bucket=s.BUCKET_NAME, events=['s3:ObjectCreated:*'], prefix=s.PREFIX_PREPROCESSED, suffix=s.SUFFIX)
def handle_pre_processed_event(event):
"""
Lambda for preprocessed data
:param event:
:return:
"""
# Retrieve the json that was add to the bucket S3
json_s3 = get_json_file_s3(event.bucket, event.key)
# Send all the records to dynamoDB
insert_records(json_s3)
# Change the path of the file by copying it and delete it
change_path_file(event.key, s.PREFIX_PREPROCESSED)
Here is the lambda I want to test. Thanks for your responses.

If someone get the same problem it's because chalice use a wrapper. Add your notification and a context in your handler.

Cloudwatch event is not triggering my lambda function, even though it's a target

I have a lambda function that takes in a dataset name, and creates a new lambda specifically for that dataset. Here's the code that sets this up:
lambda_response = lambda_client.create_function(
FunctionName=job_name,
Runtime='python3.6',
Role=role_name,
Handler='streaming_data_lambda.lambda_handler',
Code={
'S3Bucket': code_bucket,
'S3Key': 'streaming_data_lambda.py.zip'
},
Timeout=30,
)
This appears to be creating a lambda correctly, and works when I kick it off manually. I want to have this created lambda run once an hour, so the original lambda script creates the following rules and targets:
rule_response = event_client.put_rule(
Name=rule_name,
ScheduleExpression=schedule_expression
)
event_client.put_targets(
Rule=rule_name,
Targets=[
{
'Id': lambda_response['FunctionName'],
'Arn': lambda_response['FunctionArn'],
'Input': json.dumps(input_string)
}
]
)
where input_string is just something like {"datasetName": "name"}. I can see the rule in the CloudWatch Rules UI, and can see it's linked to the correct lambda and the input text is present. It triggers correctly every hour also, but fails to invoke the lambda function. If I look at the lambda in the UI and add the CloudWatch event rule I created as a trigger in the Designer section there, then it correctly kicks off the lambda, but is there some way to set this up in Python so I don't have to do this last step in the UI?

For anyone who might be looking for the answer to this in the future - you need to add add permission for cloudwatch events to invoke your lambda function, like so:
lambda_client.add_permission(
FunctionName=lambda_response['FunctionName'],
StatementId=some_random_number,
Action='lambda:InvokeFunction',
Principal='events.amazonaws.com',
SourceArn=rule_response['RuleArn']
)

You may need to add
event_client.enable_rule(Name=rule_name)
After the put_rule
In this bit, there is possibly some extra config that the UI adds
event_client.put_rule(
Name=rule_name,
ScheduleExpression=schedule_expression)
try using "DescribeRule" on this rule after it is enabled and working and then duplicate any missing fields ( like RoleArn ) in the python

The answer mentioned by #meowseph_furbalin will work but the issue with this answer is that for each rule it'll add new resource policy to the lambda. AWS limit the maximum size of resource policies to be around 2KB, so after hitting that limit, the target will not be triggered.
One way to overcome this issue is to add wildcard matching to the lambda so that it can match every rule from a single resource policy.
aws lambda add-permission --function-name lambda_name\
--action 'lambda:InvokeFunction' --principal events.amazonaws.com \
--statement-id '1' \
--source-arn arn:aws:events:us-west-2:356280712205:rule/*
After adding this command you don't need to add permissions dynamically to the lambda, and you'll lambda will accept all the cloudwatch rules with a single resource policy.
Please keep in mind while creating cloudwatch rule, you have to keep the SID as '1' for all rules. If you want to change the SID then add another rule corresponding to that SID.