Parse SQS message trigger in AWS Lambda - Python - python

I have a notification on an S3 bucket upload to place a message in an SQS queue. The SQS queue triggers a lambda function. I am trying to extract the name of the file that was uploaded from the SQS message which triggers the lambda function. My SQS event record looks like this when printed to the CloudWatch logs:
{
"Records": [
{
"eventVersion": "2.1",
"eventSource": "aws:s3",
"awsRegion": "eu-west-2",
"eventTime": "2020-04-05T13:55:30.970Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "A2RFWU4TTDGK95"
},
"requestParameters": {
"sourceIPAddress": "HIDDEN"
},
"responseElements": {
"x-amz-request-id": "024EF2A2E94BD5CA",
"x-amz-id-2": "P/5p5mDwfIu29SeZcNo3wjJaGAiM4yqBqp4p3gOfLVPeZhf+w5sRjnxsost3BuYub1FVf7tuMFs9KoC98+fwSI9NrT5WbjYq"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "ImageUpload",
"bucket": {
"name": "HIDDEN",
"ownerIdentity": {
"principalId": "A2RFWU4TTDGK95"
},
"arn": "arn:aws:s3:::HIDDEN"
},
"object": {
"key": "activity1.png",
"size": 41762,
"eTag": "9e1645a32c2948139a90e75522deb5ab",
"sequencer": "005E89E354A986B50D"
}
}
}
]
}
Using this code:
import boto3
rek = boto3.client('rekognition')
def test(event, context):
for record in event['Records']:
print ("test")
payload=record["body"]
fullpayload=str(payload)
print(fullpayload)
Using ['s3]['object]['key'] to access the filename 'activity1.png' on the payload string gives me this error:
's3': KeyError
Traceback (most recent call last):
How can i access the file name from the lambda function?

If this is what I think it is: the S3 Object Create events --> SQS <-- lambda polls,
I ran into this also. I was using the s3 put example test and also used my poll from sqs message to make another test. When it came from actually reading the queue not in a test, I had issues.
The output of print(event) is actually json of the entire event like below:
{ 'Records':
[
{
'messageId': '61155c1d-aaaa-aaaa-aaaa-aaaaaaaaaaaa',
'receiptHandle': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaalF5156pb+aSqhRWbEY1XWIAVpingcBgOM8/uv1pIgfVXtfNRwzjtoCcInH6doGo9C38uWG7V48uEzpiAPr6Ao2IkXn5IEQKgxXzgelT5FtW3gpwhsQ3fvsFZdZNkMj2YiBHpdJ9QDgfmjFOWmqEJL+LWHUyksdAHxqVZMFrdaS1Tmno3Xni7DMBg1Ed+HpHkBmAVOWssDfM25lC1RNUivXj8i3iI/gD0yBlCttA4aioAlYNZ0txBrkm8aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaML+jK3JcKXiaslbu+JNZaB7hwevHRNsGIQ2MLuRhX+eHD4BN',
'body':
'{"Records":
[
{
"eventVersion":"2.1",
"eventSource":"aws:s3",
"awsRegion":"us-east-1",
"eventTime":"2021-02-24T00:30:07.549Z",
"eventName":"ObjectCreated:Put",
"userIdentity":{"principalId":"AWS:AROAAAAAAAAAA:Rolehere"},
"requestParameters":{"sourceIPAddress":"x.x.x.x"},"responseElements":{"x-amz-request-id":"860A2aaaaaaaB19","x-amz-id-2":"J8epzX+FGaLsliSYSiJaaaaaaaaaaaaaETviVcrVCD/FsQjVLNBJgcv8v/PIh37Y9waaaaaaaaaaaaaaaaoUkoqhlr"},
"s3":
{"s3SchemaVersion":"1.0",
"configurationId":"New arrival",
"bucket":
{"name":"molly-bucketname","ownerIdentity":{"principalId":"A2aaaaaaFMND3"},"arn":"arn:aws:s3:::molly-bucketname"},
"object":{"key":"dietcokeofevil.mp3","size":420049,"eTag":"bf153e303affbb6e54feb0a233879d4d","versionId":"B2WJZpLLvpWA4nXP5T5QjVZY09qpnHKa","sequencer":"0060359E131BAA52C0"}
}
}
]
}',
'attributes': {
'ApproximateReceiveCount': '1',
'SentTimestamp': '1614126612305',
'SenderId': 'AIDAJHaaaaaaaaaaJEBU',
'ApproximateFirstReceiveTimestamp': '1614126612308'
},
'messageAttributes': {},
'md5OfMessageAttributes': None,
'md5OfBody': 'c752a7082100075786323ff7e5cdfc26',
'eventSource': 'aws:sqs',
'eventSourceARN': 'arn:aws:sqs:us-east-1:#########:queuename',
'awsRegion': 'us-east-1'
}
]
}
When an s3 doesn't deliver to lambda, lambda is reading from the queue - it seems like there's a wrapper around the json you actually see in the put examples. If you tried to add the printed event (above) to your test in lambda, it will json error. We need to parse the initial Records json & for the body
then json.load the "body"- then parse our s3 info out of it.
import json
import boto3
def lambda_handler(event, context):
#Loops through every file uploaded
for record in event['Records']:
#pull the body out & json load it
jsonmaybe=(record["body"])
jsonmaybe=json.loads(jsonmaybe)
#now the normal stuff works
bucket_name = jsonmaybe["Records"][0]["s3"]["bucket"]["name"]
print(bucket_name)
key=jsonmaybe["Records"][0]["s3"]["object"]["key"]
print(key)

What is the output from print(fullpayload)? I would expect payload to be None because there is no attribute named body in the record.
From the example record in your question, you should be doing this:
record['s3']['object']['key']

this code for Lambda function Python v3.7
import json
import boto3
def lambda_handler(event, context):
print("Reciving Message from myQueue SQS")
for record in event["Records"]:
message = record["s3"]
print(message)

Related

SNS always sends default message instead of the specific protocol message that is given

Here is the code
message = {
"default":"Sample fallback message",
"http":{
"data":[
{
"type":"articles",
"id":"1",
"attributes":{
"title":"JSON:API paints my bikeshed!",
"body":"The shortest article. Ever.",
"created":"2015-05-22T14:56:29.000Z",
"updated":"2015-05-22T14:56:28.000Z"
}
}
]
}
}
message_as_json = json.dumps(message)
response = sns_client.publish(TopicArn = "arn:aws:sns:us-east-1:MY-ARN",
Message = message_as_json, MessageStructure = "json")
print(response)
To test, I used ngrok to connect the localhost (which runs my flask app) to the web and created a http subscription.
When I publish the message I can only see default message in that.
Any idea why this happens?
You need to specify the value of the http key as a simple JSON string value according to the AWS boto3 docs:
Keys in the JSON object that correspond to supported transport
protocols must have simple JSON string values.
Non-string values will cause the key to be ignored.
import json
import boto3
sns_client = boto3.client("sns")
message = {
"default": "Sample fallback message",
"http": json.dumps(
{
"data": [
{
"type": "articles",
"id": "1",
"attributes": {
"title": "JSON:API paints my bikeshed!",
"body": "The shortest article. Ever.",
"created": "2015-05-22T14:56:29.000Z",
"updated": "2015-05-22T14:56:28.000Z",
},
}
]
}
),
}
response = sns_client.publish(
TopicArn="arn:aws:sns:us-east-1:MY-ARN", Message=json.dumps(message), MessageStructure="json"
)

SQS, Lambda & SES param undefined

I had this stack setup and working perfectly before, however, all of a sudden I am seeing a strange error in my CloudWatch.
This is my function (Python) for posting a message to SQS (which triggers a lambda function to send an email with SES):
def post_email(data, creatingUser=None):
sqs = boto3.client("sqs", region_name=settings.AWS_REGION)
# Send message to SQS queue
response = sqs.send_message(
QueueUrl=settings.QUEUE_EMAIL,
DelaySeconds=10,
MessageAttributes={
"ToAddress": {"DataType": "String", "StringValue": data.get("ToAddress")},
"Subject": {"DataType": "String", "StringValue": data.get("Subject")},
"Source": {
"DataType": "String",
"StringValue": data.get("Source", "ANS <noreply#ansfire.net"),
},
},
MessageBody=(data.get("BodyText"))
# When SQS pulls this message off, need to ensure that the email was
# actually delivered, if so create a notification
)
I print the params out and it is setting the above attributes correctly, however when I look in my CloudWatch this is the message:
2020-02-03T20:41:59.847Z f483293f-e48b-56e5-bb85-7f8d6341c0bf INFO {
Destination: { ToAddresses: [ undefined ] },
Message: {
Body: { Text: [Object] },
Subject: { Charset: 'UTF-8', Data: undefined }
},
Source: undefined
}
Any idea of what is going on?
I figured out the error, I needed to get the attributes from the data dictionary prior to calling the send_message function.

My slack app seems to fail in interaction

I'm new to slack app development. Following this page: https://api.slack.com/messaging/interactivity/enabling, I'm trying to create a simple interactive slack-app that
is launched by a slash command
can interact with users through a button.
I use AWS API-gateway and Lambda as backends. Lambda functions are written in Python 3.6.
I succeeded in creating a slash command, but failed in updating a message after pushed the button.
Precise structure of my app
When I type the slash command /test in my slack channel, my app makes a post request to an API gateway (https://xxx.execute-api.ap-northeast-1.amazonaws.com/prod/test) and a lambda function returns the following response with a button.
The lambda function is as follows:
import json
import datetime
def lambda_handler(event, context):
try:
response = {
"statusCode": 200,
"response_type": "in_channel",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "This is a section block with a button."
},
},
{
"type": "actions",
"elements": [
{
"type": "button",
"text": {
"type": "plain_text",
"text": "Button",
"emoji": True
},
"action_id": "test"
}
]
}
]
}
return response
except Exception as e:
return {"error": str(e)}
After the button pushed, my app sends a post request to another API gateway (https://xxx.execute-api.ap-northeast-1.amazonaws.com/prod/test2) and another lambda function receives a request like follows
{
'type':'block_actions',
'team':{
'id':'xxx',
'domain':'xxx'
},
'user':{
'id':'xxx',
'username':'xxx',
'name':'xxx',
'team_id':'xxx'
},
'api_app_id':'xxx',
'token':'xxx',
'container':{
'type':'message',
'message_ts':'1564687520.001900',
'channel_id':'xxx',
'is_ephemeral':False
},
'trigger_id':'xxx.yyy.zzz',
'channel':{
'id':'xxx',
'name':'slack_app_test'
},
'message':{
'type':'message',
'subtype':'bot_message',
'text':"This content can't be displayed.",
'ts':'1564687520.001900',
'bot_id':'xxx',
'blocks':[
{
'type':'section',
'block_id':'ygl6',
'text':{
'type':'mrkdwn',
'text':"This is a section block with a button. {'token': 'xxx', 'team_id': 'xxx', 'team_domain': 'xxx', 'channel_id': 'xxx', 'channel_name': 'slack_app_test', 'user_id': 'xxx', 'user_name': 'mail', 'command': '/recommend', 'response_url': '<https://hooks.slack.com/commands/xxx/yyy/zzz>', 'trigger_id': 'xxx.yyy.zzz'}",
'verbatim':False
}
},
{
'type':'actions',
'block_id':'jlzD',
'elements':[
{
'type':'button',
'action_id':'test',
'text':{
'type':'plain_text',
'text':'Button',
'emoji':True
}
}
]
}
]
},
'response_url':'https://hooks.slack.com/actions/xxx/yyy/zzz',
'actions':[
{
'action_id':'test',
'block_id':'jlzD',
'text':{
'type':'plain_text',
'text':'Button',
'emoji':True
},
'type':'button',
'action_ts':'1564687524.157943'
}
]
}
and try to make a response by the following lambda function:
import json
import urllib.request
def lambda_handler(event, context):
try:
response_url = event.get("response_url", None)
if response_url is not None:
headers = {"Content-Type" : "application/json"}
print("From slack. Response url: {}".format(response_url))
response = json.dumps({
"replace_original": True,
"response_type": "in_channel",
"text": "Success!"
})
request = urllib.request.Request(
response_url,
data=response,
method="POST",
headers=headers
)
return None
except Exception as e:
return {"error": str(e)}
When I look at Cloudwatch logs of the lambda function above, the function seems to successfully send a message to my slack app.
Problem
I expect that my app displays a message Success! after pushing the button, but no updates are shown (just showing a loading animation). Any ideas?

azure httptrigger blob storage using Python

I am trying to setup access to blob storage using a python function app but the file name is received from a post request not preset. The http trigger part works but i'm having trouble accessing files in my blob storage. This is my json:
{
"bindings": [
{
"authLevel": "function",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"post",
"get"
]
},
{
"name": "inputblob",
"type": "blob",
"path": "sites/{httpTrigger}",
"connection": "STORAGE",
"direction": "in"
},
{
"type": "http",
"direction": "out",
"name": "res"
}
],
"disabled": false
}
I saw an example (https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob#input---configuration) using a queue trigger but when i do do something similar using http i get 'No value for named parameter 'httpTrigger''. My issue is that i don't know how to reflect a variable that is assigned in my python code in my path. When i do this container/{variable} i get a nullreference exception. This is my python code:
import os
import json
import sys
import logging
import azure.functions as func
_AZURE_FUNCTION_DEFAULT_METHOD = "GET"
_AZURE_FUNCTION_HTTP_INPUT_ENV_NAME = "req"
_AZURE_FUNCTION_HTTP_OUTPUT_ENV_NAME = "res"
_REQ_PREFIX = "REQ_"
def write_http_response(status, response):
output = open(os.environ[_AZURE_FUNCTION_HTTP_OUTPUT_ENV_NAME], 'w')
output.write(json.dumps(response))
env = os.environ
postreqdata = json.loads(open(env['req']).read())
print ('site: ' + postreqdata['site'])
site = postreqdata['site']+'.xlsx'
input_file = open(os.environ['inputBlob'], 'r')
clear_text = input_file.read()
input_file.close()
print("Content in the blob file: '{0}'".format(clear_text))
# Get HTTP METHOD
http_method = env['REQ_METHOD'] if 'REQ_METHOD' in env else
_AZURE_FUNCTION_DEFAULT_METHOD
print("HTTP METHOD => {}".format(http_method))
# Get QUERY STRING
req_url = env['REQ_HEADERS_X-ORIGINAL-URL'] if 'REQ_HEADERS_X-ORIGINAL-URL'
in env else ''
urlparts =req_url.split('?')
query_string = urlparts[1] if len(urlparts) == 2 else ''
print("QUERY STRING => {}".format(query_string))
if http_method.lower() == 'post':
request_body = open(env[_AZURE_FUNCTION_HTTP_INPUT_ENV_NAME], "r").read()
print("REQUEST BODY => {}".format(request_body))
write_http_response(200, site)
note: i have made my connection string successfully ( i think) and i am new to azure and using the portal only
This looks like an older version of function apps. In the new version, you can actually use the request handler to do all this work for you. I just started working in azure functions and if you want to access a file in blob storage, all you have to do is pass in the filename parameters in the form of http query, and use that query param name as the binding variable.
Ex:
def main(req: func.HttpRequest, inputblob: func.InputStream):
input_file_content = input_blob.read()
and in your binding you give
{
"scriptFile": "__init__.py",
"bindings": [
{
"authLevel": "function",
"type": "httpTrigger",
"direction": "in",
"name": "req",
"methods": [
"get",
"post"
]
},
{
"type": "http",
"direction": "out",
"name": "$return"
},
{
"type": "blob",
"direction":"in",
"name": "inputblob",
"path": "upload/{filename}",
"connection": "AzureWebJobsStorage"
}
]
}
and you simply call the api with the query parameters filename
http://localhost:7071/api/HttpTriggerFileUpload?filename=file.ext
You can take a look at this

Azure function blob storage file name

When using azure functions with blob storage output bindings, how do you get the created blob's name, path or URL? I want to save this to a DB once it saved it.
Im using Python, but any example will do:
blob = open(os.environ['outputBlob'], 'wb')
blob.write(attachment.get_payload(decode=True))
print blob.name # this is not the correct name, but actually the temp file name I think
blob.close()
For C# :
As discussed at https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob#input you can use below type of blob bindings
Below is example of binding json file and the code. I am returning outpuBlob.Uri in http return to get the primary location path of Blob.
Bindings:-
{
"bindings": [
{
"authLevel": "function",
"name": "req",
"type": "httpTrigger",
"direction": "in",
"methods": [
"get",
"post"
]
},
{
"name": "$return",
"type": "http",
"direction": "out"
},
{
"type": "blob",
"name": "outputBlob",
"path": "outcontainer/{rand-guid}",
"connection": "AzureWebJobsDashboard",
"direction": "inout"
}
],
"disabled": false
}
Function Code (C#):
#r "Microsoft.WindowsAzure.Storage"
using System.Net;
using Microsoft.WindowsAzure.Storage.Blob;
public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log,CloudBlockBlob outputBlob)
{
log.Info("C# HTTP trigger function processed a request.");
// parse query parameter
string name = req.GetQueryNameValuePairs()
.FirstOrDefault(q => string.Compare(q.Key, "name", true) == 0)
.Value;
if (name == null)
{
// Get request body
dynamic data = await req.Content.ReadAsAsync<object>();
name = data?.name;
}
return name == null
? req.CreateResponse(HttpStatusCode.BadRequest, "Please pass a name on the query string or in the request body")
: req.CreateResponse(HttpStatusCode.OK, "Hello " + outputBlob.Uri);
}
Take a look at the environment variables in your Python script's process space. You should have one that contains the path of the blob binding.

Categories

Resources