I am running the k-means example in SageMaker:
from sagemaker import KMeans
data_location = 's3://{}/kmeans_highlevel_example/data'.format(bucket)
output_location = 's3://{}/kmeans_example/output'.format(bucket)
kmeans = KMeans(role=role,
train_instance_count=2,
train_instance_type='ml.c4.8xlarge',
output_path=output_location,
k=10,
data_location=data_location)
When I run this line, it appears access denied error.
%%time
kmeans.fit(kmeans.record_set(train_set[0]))
The error returns:
ClientError: An error occurred (AccessDenied) when calling the
PutObject operation: Access Denied
I also read other questions, but their answers do not solve my problem.
Would you please look at my case?
To be able to training a job in SageMaker, you need to pass in an AWS IAM role allowing SageMaker to access your S3 bucket.
The error means that SageMaker does not have permissions to write files in the bucket that you specified.
You can find the permissions that you need to add to your role hereL https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html#sagemaker-roles-createtrainingjob-perms
sagemaker aws-sagemaker role
Another thing to consider, if you are using an encrypted bucket, that requires kms decryption, make sure to also include kms related permissions
I've noticed sometimes the error shown is PutObject operation: Access Denied while failure is actually KMS related.
I faced the same problem. My Sagemaker Notebook Instance wasn't able to read or write files to my S3 bucket. First step of troubleshooting is locating the role for your **Sagemaker Instance **. You can do that by checking this section
Then go to this specific role from IAM and attach another policy to the role
I attached S3 Full access but you can create a custom policy.
I was getting confused because I was logged in using the admin user. However, when you go with a Sagemaker Instance your user policies/roles will not be used to perform actions.
In my case I had just forgotten to rename the s3 bucket name from the default given to something that is unique
Related
Although I have given the function IAM profile complete AdministratorAccess permissions, along with AmazonDynamoDBFullAccess permissions as well, every time that I test the function I am greeted with the same error message:
no identity-based policy allows the dynamodb:PutItem action.
How do I fix this? I literally cannot give the IAM profile more access, so I am very confused. I have given every permission I can give.
2 things I can think on
Check you are assigning the policies to the Lambda Execution Role.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-permissions.html
Are you part of an organization, check that you no not have any SCP policies in place preventing the PutItem as it would take precedence.
https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html
I've deployed a model as SageMaker Endpoint, it worked fine for some time but now when invoking the model through boto3
import boto3
client = boto3.client('sagemaker-runtime')
response = client.invoke_endpoint(
EndpointName="my-sagemaker-endpoint",
ContentType="text/csv",
Body=payload,
)
I got the following error
ServiceUnavailable: An error occurred (ServiceUnavailable) when calling the InvokeEndpoint operation (reached max retries: 4): A transient exception occurred while retrieving variant instances. Please try again later.
Researching about this error in SageMaker Documentation it states the following
The request has failed due to a temporary failure of the server.
I've also checked the Instance Metrics in CW and there's nothing unusual.
I'm not sure why this error is happening, any suggestions will be helpful.
TL; DR The error originates because the Instance is unable to retrieve the SageMaker Model artifact from s3.
Explanation
SageMaker Endpoints implement a /ping route which check if model artifact is able to load within the Instance. The model artifacts is first retrieved from s3 and then loaded into the instance. If model is not available on s3 it shows the following error (image below)
As the model artifact can't be retrieved from s3 because it was accidentally deleted, it can't be loaded which raises the error No such file or directory when calling the /ping route to check if the endpoint is healthy (see image below)
This in turn makes the Load Balancer to assume the instance has some problem, blocking you access to it, so when you try to invoke the endpoint you get a 503: Service Unavailable Error
Solution
I worked this out only by redeploying to a new endpoint but this time considering the following:
At least num_instances=2 to guarantee each instance is at a different AZ, and the LB communicates with at least a healthy instance.
Ensure only specific roles have s3:PutObject permission on the s3 model artifacts route models/model-name/version
How do I set up direct private bucket access for Tensorflow?
After running
from tensorflow.python.lib.io import file_io
and running print file_io.stat('s3://my/private/bucket/file.json') I end up with an error -
NotFoundError: Object s3://my/private/bucket/file.json does not exist
However, the same line on a public object works without an error:
print file_io.stat('s3://ryft-public-sample-data/wikipedia-20150518.bin')
There appears to be an article on support here: https://github.com/tensorflow/examples/blob/master/community/en/docs/deploy/s3.md
However, I end up with the same error after exporting the variables shown.
I have awscli set up with all credentials, and boto3 can view and download the file in question. I am wondering how I can get Tensorflow to have S3 access directly when the bucket is private.
I had the same problem when trying to access files in private S3 bucket from Sagemaker notebook. The mistake I made was to try using credentials I obtained from boto3, which seem not to be valid outside.
The solution was not to specify credentials (in such case it uses the role attached to the machine), but instead just specify the region name (for some reason it didn't read it from ~/.aws/config file) as follows:
import boto3
import os
session = boto3.Session()
os.environ['AWS_REGION']=session.region_name
NOTE: when debugging this error useful was to look at CloudWatch logs, as the logs of S3 client were printed only there and not in the Jupyter notebook.
In there I have first have seen, that:
when I did specify credentials from boto3 the error was: The AWS Access Key Id you provided does not exist in our records.
When accessing without AWS_REGION env variable set I had The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. which apparently is common when you don't specify bucket (see 301 Moved Permanently after S3 uploading)
I am trying to download files from a s3 bucket by using the Access Key ID and Secret Access Key provided by https://db.humanconnectome.org. However, even though I am able to navigate the database and find the files (as I have configured my credentials via aws cli), attempting to download them results in the following error:
"botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden"
With the same credentials, I can browse the same database and download the files manually via a cloud storage browser such as Cyberduck, so how Cyberduck accesses the data does not invoke a 403 Forbidden error.
I have also verified that boto3 is able to access my aws credentials, and also tried by hardcoding them.
How I am attempting to download the data is very straightforward, and replicates the boto3 docs example: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html
s3 = boto3.client('s3',
aws_access_key_id=ACCESS_KEY_ID,
aws_secret_access_key=ACCESS_KEY,)
s3.download_file(Bucket=BUCKET_NAME, Key=FILE_KEY, Filename=FILE_NAME)
This should download the file to the location and file given by FILE_NAME, but instead invokes the 403 Forbidden error.
You'll need to pass the bucket region as well when downloading the file. Try configuring region using the CLI or pass region_name when creating the client.
s3 = boto3.client('s3',
aws_access_key_id=ACCESS_KEY_ID,
aws_secret_access_key=ACCESS_KEY,
region_name=AWS_REGION)
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html
I know this might sound ridiculous, but make sure you don't have a typo in your bucket name or anything like that.
I worked so long trying to fix this, only to realize I added an extra letter in the env variable I had set for my s3 bucket.
It's weird that they give you a forbidden error as a opposed to "not found" error, but they do.
I want to change Git credential for AWS CodeCommit to Active/Inactive using Boto3.
I tried to use update_service_specific_credential but I got this error:
An error occurred (InvalidClientTokenId) when calling the CreateServiceSpecificCredential operation: The security token included in the request is invalid: ClientError
My code:
iamClient = boto3.client('iam')
response=iamClient.update_service_specific_credential(UserName="****",
ServiceSpecificCredentialId="*****",Status="Active")
someone tried to use it?
Any advice?
Thanks!
AWS errors are often purposefully opaque/non-specific so could you give a bit more detail? Specifically, are the user performing the update and the user whose credentials are being updated two different users? There may be a race condition arising if the user being updated IS the user performing the update.