Develop Reference

boto3 : SSM Parameter get_parameters()

I am creating lambda function where I am fetching SSM parameter for EKS-Optimized AMI ID, now about EKS-Optimized AMI, it is the default AMI provided by EKS if we are not specifying any AMI explicitly. EKS-Optimized AMI are different per region & K8 version. I am working on upgrading this AMI on node groups & getting this AMI ID here for K8 version 1.21. I want to pass this k8 version ${EKS_VERSION} to get-parameter() as
ami_id = ssm.get_parameters(Names=["/aws/service/eks/optimized-ami/${EKS_VERSION}/amazon-linux-2/recommended/image_id"])
Can you help me if we can do this in boto3, if yes,how ?
Thanks in advance!

Maybe I am missing the point of the question but it is pretty straightforward as you already have the request in your question. If you put the following code into your lambda, it should get you the version you want in that region.
For something like this, you may want to use a lambda env variable with a default, and overwrite it when you want something different.
import boto3
import ssm
# get an ssm client
ssm_client = boto3.client('ssm')
# you need to pass the var somehow, here assuming you are using an environment variable in your lambda. You could use some other system to trigger and pass the information to your lambda, e.g. sns
eks_version = os.getenv('EKS_VERSION')
# set the parameter name you want to receive, note the f-string to pass the variable to it
param_name = f"/aws/service/eks/optimized-ami/{eks_version}/amazon-linux-2/recommended/image_id"
# get_parameter
response = ssm_client.get_parameters(Names=[param_name])
# print / return response
print(response)
For overwriting the param, you could use sns or cloudwatch with lambda if you are building some kind of automation but you would need to parse the input from them.
For example a simple json payload in sns
{
"eks_version": 1.21
}
and in your code, you can change make small adjustment once you parsed the sns payload. e.g.
import json
if 'Sns' in the event:
sns_eks_version = json.loads(event['Records'][0]['Sns']['Message']['eks_version'])
else:
sns_eks_version = None
eks_version = sns_eks_version or os.get_env('EKS_VERSION')

This is how I did :
import json
import os
import boto3
ssm_client = boto3.client('ssm')
eks_client = boto3.client('eks')
eksClusterName='dev-infra2-eks'
def lambda_handler(event, context):
# Get current EKS Version
response = eks_client.describe_cluster(
name = eksClusterName
)
eksVersion = response['cluster']['version']
aws_eks_ami_ssm_param = "/aws/service/eks/optimized-ami/"+eksVersion+"/amazon-linux-2/recommended/image_id"
# Get SSM param for AMI ID
try:
eks_ssm_ami = ssm_client.get_parameter(Name=aws_eks_ami_ssm_param)
latest_ami_id = eks_ssm_ami['Parameter']['Value']
return latest_ami_id
except client.exceptions.ParameterNotFound:
logging.error("Parameter Not Found")

How to upload image to s3 with lambda and API gateway

I want to upload an image to s3 with lambda and Api gateway when i submit form how can i do it in python.
currently i am getting this error while i am trying to upload image through PostMan
Could not parse request body into json: Could not parse payload into json: Unexpected character (\'-\' (code 45))
my code currently is
import json
import boto3
import base64
s3 = boto3.client('s3')
def lambda_handler(event, context):
print(event)
try:
if event['httpMethod'] == 'POST' :
print(event['body'])
data = json.loads(event['body'])
name = data['name']
image = data['file']
image = image[image.find(",")+1:]
dec = base64.b64decode(image + "===")
s3.put_object(Bucket='', Key="", Body=dec)
return {'statusCode': 200, 'body': json.dumps({'message': 'successful lambda function call'}), 'headers': {'Access-Control-Allow-Origin': '*'}}
except Exception as e:
return{
'body':json.dumps(e)
}

Doing an upload through API Gateway and Lambda has its limitations:
You can not handle large files and there is an execution timeout of 30 seconds as I recall.
I would go with creating a presigned url that is requested by the client through API gateway, then use it as the endpoint to put the file.
Something like this will go in your Lambda ,
(This is a NodeJs example)
const uploadUrl = S3.getSignedUrl( 'putObject', {
Bucket: get(aPicture, 'Bucket'),
Key: get( aPicture, 'Key'),
Expires: 600,
})
callback( null, { url })
(NodeJs)
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getSignedUrl-property
(Python)
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.generate_presigned_url

You don't need Lambda for this. You can proxy S3 API with API Gateway
https://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-s3.html

Ooops it's over engineering.
anyways, it's seems like your are getting error from API-gateway, first check the lambda through "test lambda" using aws console, if it's working fine and getting the response back from lambda then please check with API-gateway side.
i doubt you are using some mapping templates in gateway, AWS uses AWS Velocity templates, which looks like JSON, but it's different. those mapping template at integration request causing this issue.

Module object not callable error when using Boto3 with DynamoDB

I'm trying to insert an item into my DynamoDB using the Python SDK (I'm not familiar with Python too much). I'm also using MQTT to retrieve some data. Currently the MQTT part works fine (I have some hard coded values for now so I can focus on getting the AWS functionality working)
I'm also using an AWS Educate account so need to pass in my access keys etc (which I've removed from the code for posting here). The code right now is:
from pprint import pprint
import boto3
import paho.mqtt.client as paho
import json
from types import SimpleNamespace
broker = "35.173.177.9"
#MQTT function
def onMessage(client, userdata, message):
print("Message recieved: ", str(message.payload.decode("utf-8", "ignore")))
movie_resp = putItem(1000, "1.1.1.1", "Pi", "06/01/21")
print("Put item succeeded:")
pprint(movie_resp, sort_dicts=False)
def putItem(id, ip, deviceName, clickDate, dynamodb=None):
session = boto3.session(
aws_access_key_id="",
aws_secret_access_key="",
aws_session_token="",
region_name='us-west-1'
)
if not dynamodb:
dynamodb = session.resource('dynamodb')
table = dynamodb.Table('piData')
response = table.put_item(
Item={
'ip': ip,
'id': id,
'deviceName': deviceName,
'clickDate': clickDate
}
)
return response
#MQTT code below
client = paho.Client("awsUbuntuEC2")
client.on_message = onMessage
client.connect(broker)
client.subscribe("jsonData")
client.loop_forever()
When running the code I get:
"TypeError: 'module' object is not callable"
on line
session = boto3.session(
Any ideas?

Try adding from boto3 import Session at the headers and try running the file again, please.

change from session = boto3.session to :
session = boto3.Session (Capital S in session )

Using Textract for OCR locally

I want to extract text from images using Python. (Tessaract lib does not work for me because it requires installation).
I have found boto3 lib and Textract, but I'm having trouble working with it. I'm still new to this. Can you tell me what I need to do in order to run my script correctly.
This is my code:
import cv2
import boto3
import textract
#img = cv2.imread('slika2.jpg') #this is jpg file
with open('slika2.pdf', 'rb') as document:
img = bytearray(document.read())
textract = boto3.client('textract',region_name='us-west-2')
response = textract.detect_document_text(Document={'Bytes': img}). #gives me error
print(response)
When I run this code, I get:
botocore.exceptions.ClientError: An error occurred (InvalidSignatureException) when calling the DetectDocumentText operation: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
I have also tried this:
# Document
documentName = "slika2.jpg"
# Read document content
with open(documentName, 'rb') as document:
imageBytes = bytearray(document.read())
# Amazon Textract client
textract = boto3.client('textract',region_name='us-west-2')
# Call Amazon Textract
response = textract.detect_document_text(Document={'Bytes': imageBytes}) #ERROR
#print(response)
# Print detected text
for item in response["Blocks"]:
if item["BlockType"] == "LINE":
print ('\033[94m' + item["Text"] + '\033[0m')
But I get this error:
botocore.exceptions.ClientError: An error occurred (InvalidSignatureException) when calling the DetectDocumentText operation: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
Im noob in this, so any help would be good. How can I read text form my image or pdf file?
I have also added this block of code, but the error is still Unable to locate credentials.
session = boto3.Session(
aws_access_key_id='xxxxxxxxxxxx',
aws_secret_access_key='yyyyyyyyyyyyyyyyyyyyy'
)

There is problem in passing credentials to boto3. You have to pass the credentials while creating boto3 client.
import boto3
# boto3 client
client = boto3.client(
'textract',
region_name='us-west-2',
aws_access_key_id='xxxxxxx',
aws_secret_access_key='xxxxxxx'
)
# Read image
with open('slika2.png', 'rb') as document:
img = bytearray(document.read())
# Call Amazon Textract
response = client.detect_document_text(
Document={'Bytes': img}
)
# Print detected text
for item in response["Blocks"]:
if item["BlockType"] == "LINE":
print ('\033[94m' + item["Text"] + '\033[0m')
Do note, it is not recommended to hardcode AWS Keys in code. Please refer following this document
https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/configuration.html

Error Getting Managed Identity Access Token from Azure Function

I'm having an issue retrieving an Azure Managed Identity access token from my Function App. The function gets a token then accesses a Mysql database using that token as the password.
I am getting this response from the function:
9103 (HY000): An error occurred while validating the access token. Please acquire a new token and retry.
Code:
import logging
import mysql.connector
import requests
import azure.functions as func
def main(req: func.HttpRequest) -> func.HttpResponse:
def get_access_token():
URL = "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fossrdbms-aad.database.windows.net&client_id=<client_id>"
headers = {"Metadata":"true"}
try:
req = requests.get(URL, headers=headers)
except Exception as e:
print(str(e))
return str(e)
else:
password = req.json()["access_token"]
return password
def get_mysql_connection(password):
"""
Get a Mysql Connection.
"""
try:
con = mysql.connector.connect(
host='<host>.mysql.database.azure.com',
user='<user>#<db>',
password=password,
database = 'materials_db',
auth_plugin='mysql_clear_password'
)
except Exception as e:
print(str(e))
return str(e)
else:
return "Connected to DB!"
password = get_access_token()
return func.HttpResponse(get_mysql_connection(password))
Running a modified version of this code on a VM with my managed identity works. It seems that the Function App is not allowed to get an access token. Any help would be appreciated.
Note: I have previously logged in as AzureAD Manager to the DB and created this user with all privileges to this DB.
Edit: No longer calling endpoint for VMs.
def get_access_token():
identity_endpoint = os.environ["IDENTITY_ENDPOINT"] # Env var provided by Azure. Local to service doing the requesting.
identity_header = os.environ["IDENTITY_HEADER"] # Env var provided by Azure. Local to service doing the requesting.
api_version = "2019-08-01" # "2018-02-01" #"2019-03-01" #"2019-08-01"
CLIENT_ID = "<client_id>"
resource_requested = "https%3A%2F%2Fossrdbms-aad.database.windows.net"
# resource_requested = "https://ossrdbms-aad.database.windows.net"
URL = f"{identity_endpoint}?api-version={api_version}&resource={resource_requested}&client_id={CLIENT_ID}"
headers = {"X-IDENTITY-HEADER":identity_header}
try:
req = requests.get(URL, headers=headers)
except Exception as e:
print(str(e))
return str(e)
else:
try:
password = req.json()["access_token"]
except:
password = str(req.text)
return password
But now I am getting this Error:
{"error":{"code":"UnsupportedApiVersion","message":"The HTTP resource that matches the request URI 'http://localhost:8081/msi/token?api-version=2019-08-01&resource=https%3A%2F%2Fossrdbms-aad.database.windows.net&client_id=<client_idxxxxx>' does not support the API version '2019-08-01'.","innerError":null}}
Upon inspection this seems to be a general error. This error message is propagated even if it's not the underlying issue. Noted several times in Github.
Is my endpoint correct now?

For this problem, it was caused by the wrong endpoint you request for the access token. We can just use the endpoint http://169.254.169.254/metadata/identity..... in azure VM, but if in azure function we can not use it.
In azure function, we need to get the IDENTITY_ENDPOINT from the environment.
identity_endpoint = os.environ["IDENTITY_ENDPOINT"]
The endpoint is like:
http://127.0.0.1:xxxxx/MSI/token/
You can refer to this tutorial about it, you can also find the python code sample in the tutorial.
In my function code, I also add the client id of the managed identity I created in the token_auth_uri but I'm not sure if the client_id is necessary here (In my case, I use user-assigned identity but not system-assigned identity).
token_auth_uri = f"{identity_endpoint}?resource={resource_uri}&api-version=2019-08-01&client_id={client_id}"
Update:
#r "Newtonsoft.Json"
using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;
public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{
string resource="https://ossrdbms-aad.database.windows.net";
string clientId="xxxxxxxx";
log.LogInformation("C# HTTP trigger function processed a request.");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(String.Format("{0}/?resource={1}&api-version=2019-08-01&client_id={2}", Environment.GetEnvironmentVariable("IDENTITY_ENDPOINT"), resource,clientId));
request.Headers["X-IDENTITY-HEADER"] = Environment.GetEnvironmentVariable("IDENTITY_HEADER");
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader streamResponse = new StreamReader(response.GetResponseStream());
string stringResponse = streamResponse.ReadToEnd();
log.LogInformation("test:"+stringResponse);
string name = req.Query["name"];
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
name = name ?? data?.name;
return name != null
? (ActionResult)new OkObjectResult($"Hello, {name}")
: new BadRequestObjectResult("Please pass a name on the query string or in the request body");
}

For your latest issue, where you are seeing UnsupportedApiVersion, it is probably this issue: https://github.com/MicrosoftDocs/azure-docs/issues/53726
Here are a couple of options that worked for me:
I am assuming you are hosting the Function app on Linux. I noticed that ApiVersion 2017-09-01 works, but you need to make additional changes (instead of "X-IDENTITY-HEADER", use "secret" header). And also use a system-assigned managed identity for your function app, and not a user assigned identity.
When I hosted the function app on Windows, I didn't have the same issues. So if you want to use an user-assigned managed identity, you can try this option instead. (with the api-version=2019-08-01, and X-IDENTITY-HEADER.