Develop Reference

list_object not working for cross-account with AWS Lambda

I'm trying to download a file from S3 path in another account using AWS Lambda & python. I'm able to download the files if I provide full key to the copy_object. This will not work for me on a day-to-day scenario as there's no notification and my scheduled lambda code has to check for the presence of file using wildcards. But I'm getting Access denied(An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied) error while trying to list the contents of that bucket even though I have get & list permissions added.
I have the following policy added to my lambda role.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::<src S3 bucket>/",
"arn:aws:s3:::<src S3 bucket>/*"
],
"Effect": "Allow"
},
{
"Action": [
"kms:Decrypt",
"kms:Encrypt",
"kms:GenerateDataKey",
"kms:ReEncrypt*",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-east-1:<src bucket account id>:key/<src bucket kms key>"
],
"Effect": "Allow"
}
]
}
In the source bucket, they have added the below permissions for my lambda role
{
"Sid": <>,
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<my account id>:role/LambdaRole"
]
},
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::<src S3 bucket>",
"arn:aws:s3:::<src S3 bucket>/*"
]
},
Python code to pull the file
def get_s3_file(event):
s3 = boto3.client('s3')
bucket_name = event['Records'][0]['s3']['bucket']['name']
file_key = unquote_plus(event['Records'][0]['s3']['object']['key'])
file_name = file_key+'test.txt'
logger.info('Reading {} from {}'.format(file_key, bucket_name))
objects = s3.list_objects_v2(Bucket=bucket_name, Prefix=file_key)
for object in objects:
print(object.key)
if object.key.endswith('.txt'):
print(object.key)
copy_source = {'Bucket': bucket_name,'Key': file_name}
s3_c.copy_object(Bucket='<my bucket>', Key='/data/', CopySource=copy_source, ACL='bucket-owner-full-control')

Probably the issue is due to KMS. If the lambda and bucket are in different accounts, lambda will not have automatically access to kms key in the second account. You have to modify KMS key policy to allow for that. So try adding lambda role arn to the KMS policy in the second account.

python boto3 error: Not authorized to perform assumed role on resource

I am trying to move files from a S3 bucket in one account(source account) to S3 bucket in another account(destination account)
I am using sagemaker notebook so I have a sagemaker role.
I also have a role in my team account which has full s3 access and fullsagemaker access and in the trust relationship i have given the destination account role arn and sagemaker role arn.
The destination account also has my team role arn and sagemaker role arn in its trust policy.
I am trying to assume my team role and then I will assume the destination role to copy files.
import boto3
sts_client = boto3.client('sts')
assumed_teamrole_object = sts_client.assume_role(DurationSeconds=1800,
RoleArn='myteamrole',
RoleSessionName='test1')
assumed_destrole_object = sts_client.assume_role(DurationSeconds=1800,
ExternalId='externalid provided by destination account',
RoleArn='destination account role',
RoleSessionName='test2')
The first three lines execute fine. when I try to assume the destination role i am getting the error
An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::role/AmazonSageMaker-ExecutionRole-/SageMaker is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::destinationrole
Is there something I am missing, what am i doing wrong. Please help.
I dont have any user , it is just roles
Thanks!

The error message indicates that you are missing sts:AssumeRole permissions. Your comments indicate that this is the case, as you have only S3 permission for now.
To rectify this, you can add inline policy to AmazonSageMaker-ExecutionRole role, in the form of:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "*"
}
]
}
You can further limit the Resource to only arn:aws:iam::destinationrole. But for tests you can try with * as Resource.

ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

I am working with python in a jupyter notebook.
I am trying to access several parquet files from an aws s3 bucket and convert them all into one json file. I know I have access to the data, but I am getting:
ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
Here is the code I have:
import boto3
s3_resource = boto3.resource('s3')
s3_client = boto3.client('s3')
bucket = s3_resource.Bucket(name='my-bucket')
all_objects = s3_client.list_objects(Bucket = 'my-bucket', Prefix = 'Files.parquet')
I am not sure where the access denied is coming from. Thank you so much!

You can follow this link, and attach the following policy to the user/role.
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Action":[
"s3:ListBucket"
],
"Resource":"arn:aws:s3:::my-bucket"
}
]
}

I did everything in this answer but it didn't work. Regardless you should do it first.
What worked for me is setting up new key and secret for my account in AWS IAM console since I was using one provided from another user account which didn't have access to S3.
I've also set up policy for accessing S3 directly in S3.
Click on Bucket --> Permissions --> Bucket policy
and create something like this:
{
"Version": "2012-10-17",
"Id": "Policy1675850598142",
"Statement": [
{
"Sid": "Stmt1675850596389",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::918775241671:user/youraccount#your.com"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::bucketName"
}
]
}
Also you need authenticated session created in boto3 in order to connect, here is my full code:
import boto3
import awswrangler as wr
# Define your AWS access key and secret key
aws_access_key_id = "SSSSL22FTPTYGZWDF4"
aws_secret_access_key = "fjsidkljfksjfkjs490340+eVcGaXmcamIOrDDzW"
# Use boto3 to create a session with your AWS credentials
session = boto3.Session(
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key
)
# Use awswrangler to connect to your S3 bucket
bucket_name = "mybucket"
# List the contents of your S3 bucket
s3 = session.client("s3")
result = s3.list_objects(Bucket=bucket_name)
# Print the contents of the bucket
if "Contents" in result:
for obj in result["Contents"]:
print(obj["Key"])
else:
print("The bucket is empty.")

How to Import Amazon S3 bucket without API Key?

From what I've researched, it seems that S3 buckets are typically downloaded from the same AWS account that the API Key originates from.
I have been given access to a new AWS account, and this account contains an S3 bucket that I need to download on a daily basis. The issue is that I have my API key associated with another AWS account. Is there any way to import the new S3 bucket into my existing AWS account, or would it be simpler to try and create a new API key on the AWS account which already has the S3 bucket I need?
I am using python and boto3. Thanks!

The owner of the Amazon S3 bucket can add a Bucket Policy that grants access either publicly or to specific AWS users, including users from another AWS account. This way, you can use credentials from Account A to access a bucket in Account B.
For example, this Bucket Policy grants access to a user in a different account:
{
"Id": "S3Access",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DownloadFiles",
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-bucket/*",
"Principal": {
"AWS": [
"arn:aws:iam::123456789012:user/fred"
]
}
},
{
"Sid": "ListFiles",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-bucket",
"Principal": {
"AWS": [
"arn:aws:iam::123456789012:user/fred"
]
}
}
]
}
The ARN can be obtained by looking at the IAM User within the IAM management console.
Rather than downloading the whole bucket each time, I would recommend using the AWS Command-Line Interface (CLI), which has a convenient aws s3 sync command. This can synchronize files to/from Amazon S3 and will only copy files that have been added or modified since the last sync. You could run it on a regular basis (eg each hour) to have a local copy of the bucket, similar in concept to DropBox.

Specifying IAM roles for permissions in AWS S3

I'm trying to restrict all my AWS Cognito users to their own subdirectory in my S3 bucket.
I don't want them listing, reading, or writing other people's subdirectories/files in my larger bucket, and I only want them to read & write objects in their own directory.
I'm drawing inspiration from this AWS documentation snippet.
Here's my policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-bucket"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"subfolder/"
]
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}",
"arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}/*"
]
}
]
}
And my code to retrieve the file of a certain user with user_id = test#test.com, but actually allows me to retrieve a restricted file:
import boto
# These keys are *not* hardcoded, I'm just leaving out
# the auth flow to get them from Cognito/STS as described
# here: https://mobile.awsblog.com/post/Tx2FL1QAPDE0UAH/Understanding-Amazon-Cognito-Authentication-Part-2-Developer-Authenticated-Ident
conn = boto.s3.connect_to_region('us-east-1',
aws_access_key_id=ACCESS_KEY_FROM_COGNITO,
aws_secret_access_key=SECRET_KEY_FROM_COGNITO,
security_token=SECURITY_KEY_FROM_COGNITO)
# get the bucket
b = conn.get_bucket('my-bucket', validate=False)
# try to get an object we SHOULD be able to get
k = Key(b)
k.key = 'subfolder/us-east-1:xxxx-xxxx-xxxx-xxxxxxxxx/foobar'
print "Contents:", k.get_contents_as_string() # success!
# try to get and object we SHOUDN'T be able to get
k2 = Key(b)
k2.key = 'subfolder/BLAH_BLAH/restricted'
print "Contents:", k2.get_contents_as_string() # should fail, but doesn't
Unfortunately, I can access and read the contents of both files, yet I'm following the exact same pattern in the AWS blog documentation post. I'm also unsure why I need the validate=False in the boto connection, but it seems to work well enough.
What am I missing?
EDIT: In response to the answer below, I've tried updating my role to the following, but it does not make a difference:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-bucket"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"subfolder/${cognito-identity.amazonaws.com:sub}/*"
]
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}/*"
]
}
]
}
I've also confirmed that the access credentials I'm using are from Cognito by using the access/secret/security token triple retrieved from STS using Cognito token to create a boto IAMConnection object and querying for my role name corresponding to the auth'd cognito users for my identity pool. In doing so, I got the following exception when trying to read this role (which is exactly what should happen since I did not grant access):
BotoServerError: BotoServerError: 403 Forbidden
<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
<Error>
<Type>Sender</Type>
<Code>AccessDenied</Code>
<Message>User: arn:aws:sts::MY_AWS_ACCT_ID:assumed-role/my_policy_role_name/session_name_here is not authorized to perform: iam:GetRole on resource: role awsNameFor_Role_Given_123012313</Message>
</Error>
<RequestId>xxx-xxxx-xxxx-xxxx-xxxxx</RequestId>
</ErrorResponse>
So still no clarity on why this isn't working.

5 things:
Make sure you are using credentials issued by the Amazon Cognito Identity service otherwise ${cognito-identity.amazonaws.com:sub} will be empty and grant you access to everything
Make sure the Amazon Cognito Identity credentials you are using were issued after you updated the policy, the policy is embedded in the session portion of the credentials so if you are using old credentials, they may not have the current policy attached.
You cannot use the username of the user, you must use the Amazon Cognito Identity id. So instead of test#test.com it will be the identity id: us-east-1:beef-beef-beef-xxxxxxxxx
Your pool has 2 roles associated with it, an unauthenticated role and an authenticated role. Make sure you are setting your policy on the correct role, in this case it looks like you are using developer authenticated identities and should be modifying the policy on the authenticated role.
Check your S3 bucket policy, if you allow anonymous access to your bucket, the Cognito role policy will not override it. Turn off anonymous access if that is the case. http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-2

It looks like you're code is using a hard-coded access key and secret key and is NOT using Cognito to retrieve credentials. Instead of embedding the same access key and secret key for all users, to leverage Cognito you'll need to follow the Authentication Flow and use GetId (boto reference, AWS reference) to get an identity ID and then GetCredentialsForIdentity (boto reference AWS reference) to get the Cognito-issued credentials for the current ID. Then use those credentials with the boto S3 connection.
Also be sure to cache the ID for each user and reuse it when making additional calls to Cognito.

The answer was rather silly. Apparently buckets themselves in S3 have their own policies (they are rather tucked away), but I had the directive:
Principal: "*"
which caused the bucket to be world readable.
The second revelation was that if you restrict a bucket with the s3:ListBucket with a Condition, that doesn't mean if you list the bucket you'll get only those results - you must call it by name. As an example in boto:
wrong = bucket.list() # will simply 403
right = bucket.list(prefix="base/subdir/<cognito-id>/") # will succeed
In other words S3 is designed such that you must know the prefix-key of the desired folder, which is good practice anyway.
I have to say, I was quite impressed with how helpful the folks at AWS were in diagnosing this issue here and on their forum. Anyway, a much better understanding of S3 now.