I've deployed an endpoint in sagemaker and was trying to invoke it through my python program. I had tested it using postman and it worked perfectly ok. Then I wrote the invocation code as follows
import boto3
import pandas as pd
import io
import numpy as np
def np2csv(arr):
csv = io.BytesIO()
np.savetxt(csv, arr, delimiter=',', fmt='%g')
return csv.getvalue().decode().rstrip()
runtime= boto3.client('runtime.sagemaker')
payload = np2csv(test_X)
runtime.invoke_endpoint(
EndpointName='<my-endpoint-name>',
Body=payload,
ContentType='text/csv',
Accept='Accept'
)
Now whe I run this I get a validation error
ValidationError: An error occurred (ValidationError) when calling the InvokeEndpoint operation: Endpoint <my-endpoint-name> of account <some-unknown-account-number> not found.
While using postman i had given my access key and secret key but I'm not sure how to pass it when using sagemaker apis. I'm not able to find it in the documentation also.
So my question is, how can I use sagemaker api from my local machine to invoke my endpoint?
I also had this issue and it turned out to be my region was wrong.
Silly but worth a check!
When you are using any of the AWS SDK (including the one for Amazon SageMaker), you need to configure the credentials of your AWS account on the machine that you are using to run your code. If you are using your local machine, you can use the AWS CLI flow. You can find detailed instructions on the Python SDK page: https://aws.amazon.com/developers/getting-started/python/
Please note that when you are deploying the code to a different machine, you will have to make sure that you are giving the EC2, ECS, Lambda or any other target a role that will allow the call to this specific endpoint. While in your local machine it can be OK to give you admin rights or other permissive permissions, when you are deploying to a remote instance, you should restrict the permissions as much as possible.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "sagemaker:InvokeEndpoint",
"Resource": "arn:aws:sagemaker:*:1234567890:endpoint/<my-endpoint-name>"
}
]
}
Based on #Jack's answer, I ran aws configure and changed the default region name and it worked.
Related
I'm working on a project that is almost finished except for one thing and I'm in despirate need of help..
The backend of my project that is written in Django/Python is deployed on AWS Lambda using a library called Zappa because my knowledge about AWS is really low and this seemed (for the first version) the best and fastest way to have my project up and running. It uses a AWS RDS PostgreSQL database to handle my database.
The frontend of my project is written in Javascript/ReactJS has been put on a S3 bucket en made publicly available through Cloudfront by following this tutorial.
Now, all of the above already works perfectly, it's just that now, because we have a lot of data that needs to be send from our backend/database to our frontend through API's, I want to make use of elasticsearch.
So, I started using the AWS Opensearch Service (and using elasticsearch version 7.10) which then generated an endpoint for me to connect at.
This all works great on my localhost, I can connect by using the code below, create my indexes using the ElasticsearchDSL library.
Also because the endpoint is publicly availbable, I can just make an GET request with the endpoint AWS opensearch gives me and I can make use of the Kibana url.
import boto3, os
from requests_aws4auth import AWS4Auth
from elasticsearch import RequestsHttpConnection
service = "es"
region = "eu-central-1"
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(
credentials.access_key,
credentials.secret_key,
region,
service,
session_token=credentials.token
)
ELASTICSEARCH_DSL = {
'default': {
'hosts': 'my-opensearch-domain.com/',
"http_auth": awsauth,
"use_ssl": True,
"verify_certs": True,
"connection_class": RequestsHttpConnection,
}
}
The trouble only comes when deploying this on my lambda project. where it just gives me a timeout when trying to save(index) an object manually.
So, with some reading and research, I found out that my Lambda function runs in a VPC.
I tried putting my domain in the same VPC and security-group.
Now when the project is deployed, I can make connection with the endpoint and manually index an object when I save it, but I can't make connection on my localhost, can't use the kibana url and can't make use of their API endpoints.
Any opinions on what to do? am I seeing this wrong?
Thanks in advance
EDIT: I found this guide on how to connect to ES dashboards outside a VPC.
I'm trying to debug an AWS Lambda function that's using a Docker image, as described here. I'm using the stock AWS Python image: public.ecr.aws/lambda/python:3.8
I'm able to follow the steps described in the above link to test my function locally and it works just fine:
docker run -p 9000:8080 hello-world, followed by curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}' in another Terminal window properly performs the function I'm expecting. However once this is running in Lambda, after successfully tagging the image and pushing it to AWS ECR, the function doesn't seem to be working and I'm not able to find any logs to debug the failed/missing executions.
I'm at a bit of a loss in terms of where these logs are stored, and/or what configuration I may be missing to get these logs into CloudWatch or something similar. Where can I expect to find these logs to further debug my lambda function?
So, there are no technical diferences from working with docker images with lambda compated to the code as zip or in s3. As for the logs, according to AWS documentation (and this is the description directly from the docs):
AWS Lambda automatically monitors Lambda functions on your behalf, reporting metrics through Amazon CloudWatch. To help you troubleshoot failures in a function, Lambda logs all requests handled by your function and also automatically stores logs generated by your code through Amazon CloudWatch Logs.
You can insert logging statements into your code to help you validate that your code is working as expected. Lambda automatically integrates with CloudWatch Logs and pushes all logs from your code to a CloudWatch Logs group associated with a Lambda function, which is named /aws/lambda/.
So, the most basic code would have some sort of logging within your lambda. My suggestion in this case to troubleshoot:
1 - Like in the image bellow, go to your lambda function and try access the cloudwatch logs directly from the console. Make sure to confirm the default region in which your function was deployed.
2 - If the logs exists (the group for the lambda function exists), the check if there are any raise exceptions from your code.
3 - If there are any errors indicating that the group log for cloudwatch doesn't exist or that the group log from the function doesnt exist, then check the configurations from your lambda directly in the console or, if you are using a framework like serverless or cloudwatch, the code structure.
4 - Finally, if everything seems ok this could be only related to one simple thing. User permissions from your account or Role permission from you lambda function (which is mostly the case for these situations).
One thing that you should check is the basic role generated from your lambda, which ensures that you can create new log groups
One policy example should be something like this (You can also add manually the CloudWatch Logs policy, the effect should be similar):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "logs:CreateLogGroup",
"Resource": "arn:aws:logs:us-east-1:XXXXXXXXXX:*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:us-east-1:XXXXXXXXXX:log-group:/aws/lambda/<YOUR-LAMBDA-FUNCTION>r:*"
]
}
]
}
More related to this issue can be found here:
https://aws.amazon.com/pt/premiumsupport/knowledge-center/lambda-cloudwatch-log-streams-error/
I say this because but I have used frequently docker for code dependencies with lambda, based on this first tutorial from when this feature was introduced.
https://aws.amazon.com/pt/blogs/aws/new-for-aws-lambda-container-image-support/
Hopefully this was helpfull!
Feel free to leave additional comments.
For a special case when you are using serverless framework, I had to use the following to get the logs in the cloudwatch.
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event: dict, context: dict) -> dict:
logger.info(json.dumps(event))
# ...
return {'statusCode': 200, 'body': json_str}
For my case, the lambda function runs inside ecr docker container.
I've got a Python Google Cloud Function called "artificiellt_osterbotten" (not a Firebase function), and I want to trigger it through Firebase Hosting. This is my firebase.json file:
{
"hosting": {
"public": "public",
"ignore": ["firebase.json", "**/.*", "**/node_modules/**"],
"rewrites": [
{
"source": "/artificiellt_osterbotten",
"function": "artificiellt_osterbotten"
}
]
}
}
The route seems to be working, but all I'm getting is a 404. I'm assuming this has to do with a disconnect between Firebase and GCP. The function does show up in the Firebase console, however.
Anyone got any idea as to what's the issue here? Is it even possible to trigger GCP Cloud Functions from Firebase Hosting?
I have upgraded my Firebase plan to Blaze.
Turns out, I just had to have the function located in us-central1 for it to work. Wish it could have warned me in the CLI, would have saved me a few hours!
For future readers, there is an open issue that firebase is working on about this and it appears either a warning will be issued, or multi region support will happen.
I'm trying to execute a query in Athena, but it fails.
Code:
client.start_query_execution(QueryString="CREATE DATABASE IF NOT EXISTS db;",
QueryExecutionContext={'Database': 'db'},
ResultConfiguration={
'OutputLocation': "s3://my-bucket/",
'EncryptionConfiguration': {
'EncryptionOption': 'SSE-S3'
}
})
But it raises the following exception:
botocore.errorfactory.InvalidRequestException: An error occurred (InvalidRequestException)
when calling the StartQueryExecution operation: The S3 location provided to save your
query results is invalid. Please check your S3 location is correct and is in the same
region and try again. If you continue to see the issue, contact customer support
for further assistance.
However, if I go to the Athena Console, go to Settings and enter the same S3 location (for example):
the query runs fine.
What's wrong with my code? I've used the API of several the other services (eg, S3) successfully, but in this one I believe I'm passing some incorrect parameters. Thanks.
Python: 3.6.1. Boto3: 1.4.4
I had to add a 'athena-' prefix to my bucket to get it to work. For example, in stead of:
"s3://my-bucket/"
Try:
"s3://athena-my-bucket/"
EDIT: As suggested by Justin, AWS later added support for Athena by adding athena prefix to the bucket. Please upvote his answer.
Accepted Answer:
The S3 location provided to save your query results is invalid. Please check your S3 location is correct and is in the same region and try again.
Since it works when you use the console, it is likely the bucket is in a different region than the one you are using in Boto3. Make sure you use the correct region (the one that worked in the console) when constructing the Boto3 client. By default, Boto3 will use the region configured in the credentials file.
Alternatively try boto3.client('athena', region_name = '<region>')
Ran into the same issue and needed to specify the S3 bucket in the client.
In my case, IAM role didn't have all the permissions for the S3 bucket. I gave IAM role following permissions for Athena results bucket.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::athena_results_bucket",
"arn:aws:s3:::athena_results_bucket"
],
"Effect": "Allow"
}
]
}
I received the OP error, attempted Justin's answer, and got the following error
SYNTAX_ERROR: line 1:15: Schema TableName does not exist
Meaning that it was not able to find the tables that I had previously created through the AWS Athena UI.
The simple solution was to use dclaze's answer instead. These two answers cannot be used simultaneously, or you will get back the initial (OP) error.
I wrote a python script to download some files from an s3 bucket. The script works just fine on one machine, but breaks on another.
Here is the exception I get: botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden.
I am pretty sure it's related to some system configurations, or something related to the registry, but don't know what exactly. Both machines are running Windows 7 and python 3.5.
Any suggestions.
The issue was actually being caused by the system time being incorrect. I fixed the system time and the problem is fixed.
So forbidden means you dont have access to perform the operation. Check you have permission to perform read on that specific bucket and also you have supplied valid IAM keys. Below is the sample policy for getting read and list access to bucket.
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"statement1",
"Effect":"Allow",
"Action":[
"s3:List*",
"s3:GetObject"
],
"Resource":[
"arn:aws:s3:::bucketname/*"
]
}
]
}
More info here:
Specifying Permissions in a Policy
Writing IAM Policies: How to Grant Access to an Amazon S3 Bucket