I have a permission problem on an S3 bucket when I try to access certain files using BOTO in Python. Here is the bucket policy :
{
"Version": "2008-10-17",
"Id": "Policy1407346649831",
"Statement": [
{
"Sid": "Stmt1407346646598",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::029030651757:user/my_iam_user"
},
"Action": "s3:*",
"Resource": ["arn:aws:s3:::my_bucket/*",
"arn:aws:s3:::my_bucket"]
},
{
"Sid": "2",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity EFUS443HMBYF"
},
"Action": "s3:GetObject",
"Resource": ["arn:aws:s3:::my_bucket/*",
"arn:aws:s3:::my_bucket"]
},
{
"Sid": "3",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity EFUS443HMBYF"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::my_bucket/*"
}
]
}
I have 3 statements. The first one is to authorize the user my_iam_user to access the bucket my_bucket, the second one is to authorize a Cloudfront distribution to read the bucket and the last one is to authorize a Cloudfront distribution to write on the bucket.
Now I have to files on my bucket profile_pictures/15/file1.jpg and profile_pictures/15/file2.jpg. The first one was created using a signed url and CloudFront, the second one was put on the S3 using Boto. Now I try to access the files using Boto. Here is my code:
import boto
from boto.s3.key import Key
s3 = boto.connect_s3(
aws_access_key_id="access_key_of_my_iam_user",
aws_secret_access_key="secret_key_of_my_iam_user"
)
bucket = s3.get_bucket("my_bucket", validate=False)
k1 = Key(bucket)
k1.key = "profile_pictures/15/file1.jpg"
k1.get_contents_as_string()
k2 = Key(bucket)
k2.key = "profile_pictures/15/file2.jpg"
k2.get_contents_as_string()
The problem is that the access to file1 returns an error:
S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>8C5DE910C7B18F9E</RequestId><HostId>XiKU5Q+B0Wme3GpUNmUoD9KpUN63T3bFu/rAb/wh3rhDMkykoRsdQIFgyIp8zfAwMR1apbqUEFY=</HostId></Error>
whereas the second one is a success. What could be wrong?
Note: The time on the client that is running the code is good.
Even though the IAM user has been granted full access to the bucket by the first policy, they still will not automatically have access to file1 because they are not the owner of that object (Cloudfront is) and they have not been granted explicit access to the object. Also, presumably, the IAM user is also not the owner of the bucket.
If you look at Example 1 on this page you will see an almost identical situation and further explanation of the of how the object context is used to determine whether a request is granted or denied.
You will not have access to the bucket "my-bucket"..
Bucket names are required to be unique across the entire S3 eco-system.
If you try to access a unique bucket name e.g "MY_UNIQUE_ORG_NAME-MY_UNIQUE_BUCKET_NAME" you will probably have better luck..
Related
I have a lambda that runs via a http trigger in API Gateway.
My goal is to:
Copy files from input-bucket (where the user uploads his/her files)
Create a UUID as a new location/folder in a separate bucket (output-bucket)
Paste those objects in that new location
Delete all objects from input-bucket and return a 200 http status with the new location name in the body
However, I keep getting this:
[ERROR] ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied
This is my lambda:
LOGGER = logging.getLogger(__name__)
logging.basicConfig(level=logging.ERROR)
logging.getLogger(__name__).setLevel(logging.DEBUG)
session = boto3.Session()
client = session.client('s3')
s3 = session.resource('s3')
src_bucket = s3.Bucket('input-bucket')
def lambda_handler(event,context):
LOGGER.info('Reading files in {}'.format(src_bucket))
location = str(uuid.uuid4())
client.put_object(Bucket='output-bucket', Body='',Key = (location + "/"))
LOGGER.info('Generated folder location ' + "/"+ location + "/ in output-bucket")
for obj in src_bucket.objects.all():
copy_source = {
'Bucket': 'input-bucket',
'Key': obj.key
}
client.copy_object(Bucket='output-bucket',
Key=location + '/' + obj.key,CopySource = copy_source) ### ERROR OCCURS HERE
LOGGER.info('Copied: {} from {} to {} folder in {}'.format(obj.key,'input-bucket',location,'output-bucket'))
src_bucket.objects.all().delete()
LOGGER.info('Deleted all objects from {}'.format(src_bucket))
return { "statusCode": 200,
"body": json.dumps({
'folder_name': location
})
}
As far as I know, I have enabled everything correctly for the s3 bucket policies (this is only for output-bucket, I have identical policies for input-bucket:
resource "aws_s3_bucket" "file_output" {
bucket = "output-bucket"
acl = "private"
policy = <<EOF
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"ModifySrcBucket",
"Effect":"Allow",
"Principal":{
"AWS":[
"<XXXXXX>"
]
},
"Action":[
"s3:PutObject",
"s3:PutObjectTagging",
"s3:GetObject",
"s3:GetObjectTagging",
"s3:DeleteObject"
],
"Resource":["arn:aws:s3:::output-bucket/*","arn:aws:s3:::output-bucket/*/*"]
},
{
"Effect":"Allow",
"Principal":{
"AWS":"<XXXXXXX>"
},
"Action":[
"s3:ListBucket"
],
"Resource":"arn:aws:s3:::output-bucket"
}
]
}
EOF
}
resource "aws_s3_bucket_ownership_controls" "disable_acl_output" {
bucket = aws_s3_bucket.file_output.id
rule {
object_ownership = "BucketOwnerPreferred"
}
}
Since your source and destination S3 buckets are in the same account as the AWS Lambda function:
Do not create a Bucket Policy
Instead, create an IAM Role and assign it to the Lambda function
Add these permissions to the IAM Role:
{
"Version":"2012-10-17",
"Statement":[
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::input-bucket/*"
},
{
"Effect": "Allow",
"Action": "s3:ListBucket"
"Resource": "arn:aws:s3:::input-bucket"
},
{
"Effect": "Allow",
"Action": "s3:PutObject"
"Resource": "arn:aws:s3:::output-bucket/*"
},
]
}
If the buckets were in different AWS Accounts, then you would need to put a Bucket Policy on the bucket in the 'other' account to grant permission for this IAM Role to be able to access it. (There are plenty of Answers on StackOverflow demonstrating how to do this.)
I believe setting the permissions on the S3 bucket policy might not be enough. Your Lambda Execution role should have permissions to invoke some operations as well, as explained in the AWS docs:
on the source bucket: s3:ListBucket and s3:GetObject
on the destination bucket: s3:ListBucket and s3:PutObject
There might be more permissions required based on your exact use case (server-side encryption with KMS, versioned objects, etc.).
I'm trying to download a file from S3 path in another account using AWS Lambda & python. I'm able to download the files if I provide full key to the copy_object. This will not work for me on a day-to-day scenario as there's no notification and my scheduled lambda code has to check for the presence of file using wildcards. But I'm getting Access denied(An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied) error while trying to list the contents of that bucket even though I have get & list permissions added.
I have the following policy added to my lambda role.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::<src S3 bucket>/",
"arn:aws:s3:::<src S3 bucket>/*"
],
"Effect": "Allow"
},
{
"Action": [
"kms:Decrypt",
"kms:Encrypt",
"kms:GenerateDataKey",
"kms:ReEncrypt*",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-east-1:<src bucket account id>:key/<src bucket kms key>"
],
"Effect": "Allow"
}
]
}
In the source bucket, they have added the below permissions for my lambda role
{
"Sid": <>,
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<my account id>:role/LambdaRole"
]
},
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::<src S3 bucket>",
"arn:aws:s3:::<src S3 bucket>/*"
]
},
Python code to pull the file
def get_s3_file(event):
s3 = boto3.client('s3')
bucket_name = event['Records'][0]['s3']['bucket']['name']
file_key = unquote_plus(event['Records'][0]['s3']['object']['key'])
file_name = file_key+'test.txt'
logger.info('Reading {} from {}'.format(file_key, bucket_name))
objects = s3.list_objects_v2(Bucket=bucket_name, Prefix=file_key)
for object in objects:
print(object.key)
if object.key.endswith('.txt'):
print(object.key)
copy_source = {'Bucket': bucket_name,'Key': file_name}
s3_c.copy_object(Bucket='<my bucket>', Key='/data/', CopySource=copy_source, ACL='bucket-owner-full-control')
Probably the issue is due to KMS. If the lambda and bucket are in different accounts, lambda will not have automatically access to kms key in the second account. You have to modify KMS key policy to allow for that. So try adding lambda role arn to the KMS policy in the second account.
I am working on a boto script to collect logs and store it into an S3 bucket, I am getting an error "botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the PutBucketPolicy operation: The specified bucket does not exist" I am craeating the bucket beforehand and then trying to attach the policy to the bucket,the bucket shows up in the console as well. I don't understand what might be causing this problem.
import boto3
import sys
import json
import time
iam = boto3.client('iam')
sts = boto3.client('sts')
ec2 = boto3.resource('ec2')
cloudtrail = boto3.client('cloudtrail')
s3 = boto3.client('s3')
s3.create_bucket(Bucket='goodbucket3')
# Create a bucket policy
bucket_name = 'goodbucket3'
bucket_policy = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {"Service": "cloudtrail.amazonaws.com"},
"Action": "s3:GetBucketAcl",
"Resource": f"arn:aws:s3:::{bucket_name}"
},
{
"Effect": "Allow",
"Principal": {"Service": "cloudtrail.amazonaws.com"},
"Action":
"s3:PutObject",
"s3:PutObjectAcl"
"s3:GetObject"
"s3:GetObjectAcl"
"s3:DeleteObject"
"Resource": f"arn:aws:s3:::{bucket_name}/AWSLogs/XXXXXXXX/*",
"Condition": {"StringEquals": {"s3:x-amz-acl": "bucket-owner-full-control"}}
}
]
}
# Convert the policy from JSON dict to string
bucket_policy = json.dumps(bucket_policy)
# Set the new policy
s3.put_bucket_policy(Bucket='bucket_name', Policy=bucket_policy)
result = s3.get_bucket_policy(Bucket='bucket_name')
logs = cloudtrail.create_trail(
Name='GoodTrail',
S3BucketName='bucket_name',
)
print(logs)
You may need to wait for the bucket creation to full propagate. You can use a waiter to do this. See the low-level clients documentation or How to use waiters in boto3.
Using a client's get_waiter() method, you can obtain a specific waiter
from its list of possible waiters:
# Retrieve waiter instance that will wait till a specified bucket exists
s3_bucket_exists_waiter = s3.get_waiter('bucket_exists')
Then to actually start waiting, you must call the waiter's wait()
method with the method's appropriate parameters passed in:
# Begin waiting for the S3 bucket, mybucket, to exist
s3_bucket_exists_waiter.wait(Bucket='mybucket')
First time boto3 user.
I had a user with ACL S3FullAccess and used the following code to try and upload a file; it uses a pandas DataFrame as the source.
s3_client = boto3.client('s3')
io = StringIO()
df.to_csv(io)
response = s3_client.put_object(
Bucket=self.bucket,
Body=io,
Key=self.filename
)
This lead to this response
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
So I checked that the secret key and access key were being picked up by boto3 from my ~/.aws/credentials file, and they are, on line 604 of client.py in boto3 - request_signer=self._request_signer
So I researched here on SO, and it seemed a lot of people had to add a Policy document, so I did that as follows:
{
"Version": "2012-10-17",
"Id": "Policyxxx",
"Statement": [
{
"Sid": "Stmtx1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<12 digit id>:root"
},
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::<my-bucket>/*"
},
{
"Sid": "Stmtx6",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<12 digit id>:root"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<my-bucket>"
}
]
}
I still get the same error, so I added this to my put_object call since the S3 bucket uses AES-256 encryption, which I thought was server-side only, but running out of ideas, so worth a try.
SSECustomerKey=os.urandom(32),
SSECustomerAlgorithm='AES256',
Next I removed those terms associated with the SSE keys, realising that the AES-256 encryption is server side and should not affect my access.
Then I tried to generate a new pair of Access keys and use those instead, same result.
Does anyone have any idea what I might look at next, what have I missing in the hundreds of pages of AWS documentation?
This was simply a case of when the user was created they were added to a couple of groups. Administrators and EC2MFA. I was unaware of the implications of this, but assume the EC2MFA group prevented API or CLI access. I am assuming the combination of the Policy on the user and S3 side is sufficiently secure.
I have found lot of questions regarding this on stackoverflow but none solved my problem. After lot of googling still i am facing AccessDenied Exception:
<Error>
<Code>AccessDenied</Code>
</Message><RequestId>ADF9C0DE6C86DF4F</RequestId>
<HostId>JwQLkNB0LuJvh0jwrsJe9wazxLsd+hrZ2qwvjCvmXYd2A/ckCrsotRMHm</HostId>
</Error>
Here are my policy docs for user and group:
User Policy:
{
"Statement":[
{
"Sid":"AllowListBucketIfSpecificPrefixIsIncludedInRequest",
"Action":"s3:*",
"Effect":"Allow",
"Resource":["arn:aws:s3::: mybucket", "arn:aws:s3:::mybucket/*"],
"Condition":{
"StringLike":{"s3:prefix":["Development/*"]
}
}
},
{
"Sid":"AllowUserToReadWriteObjectDataInDevelopmentFolder",
"Action":"s3:*",
"Effect":"Allow",
"Resource":["arn:aws:s3::: mybucket/Development/*"]
},
{
"Sid": "ExplicitlyDenyAnyRequestsForAllOtherFoldersExceptDevelopment",
"Action": ["s3:ListBucket"],
"Effect": "Deny",
"Resource": ["arn:aws:s3::: mybucket", "arn:aws:s3::: mybucket/*"],
"Condition":{ "StringNotLike": {"s3:prefix":["Development/*"] },
"Null" : {"s3:prefix":false }
}
}
]
}
Group Policy:
{
"Statement": [
{
"Sid": "AllowGroupToSeeBucketListAndAlsoAllowGetBucketLocationRequiredForListBucket",
"Action": ["s3:ListAllMyBuckets", "s3:GetBucketLocation"],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::*"]
},
{
"Sid": "AllowRootLevelListingOfCompanyBucket",
"Action": ["s3:ListBucket"],
"Effect": "Allow",
"Resource": ["arn:aws:s3::: mybucket", "arn:aws:s3::: mybucket/*"],
"Condition":{
"StringEquals":{"s3:prefix":[""]}
}
},
{
"Sid": "RequireFolderStyleList",
"Action": ["s3:ListBucket"],
"Effect": "Deny",
"Resource": ["arn:aws:s3:::*"],
"Condition":{
"StringNotEquals":{"s3:delimiter":"/"}
}
},
{
"Sid": "ExplictDenyAccessToPrivateFolderToEveryoneInTheGroup",
"Action": ["s3:*"],
"Effect": "Deny",
"Resource":["arn:aws:s3:::mybucket/Private/*"]
},
{
"Sid": "DenyListBucketOnPrivateFolder",
"Action": ["s3:ListBucket"],
"Effect": "Deny",
"Resource": ["arn:aws:s3:::*"],
"Condition":{
"StringLike":{"s3:prefix":["Private/"]}
}
}
]
}
Created a user with username - testuser
then got access_key and secret_access_key for this IAM user.
Now i am able to access mybucket and its subfolder using aws web console and cyberduck.
But whenever i am trying to access using boto , getting AccessDenied Exception (Error 403).
Boto Code:
<!-- language: python -->
from boto.s3.connection import S3Connection
connect = S3Connection('_______________________','_____________________')
# Without Validate
bucket = conn.get_bucket('mybucket', validate=False) #here got bucket object
bucket.get_key('one/two/three.png') # AccessDenied
#With Validate
bucket = conn.get_bucket('mybucket') #AccessDenied
Even i faced same problem when i was trying to use boto-rsync.
Any Suggestions ??
Error 403 means Access Denied so there is a authentication problem. To analyze the API call and the response one can use the following line:
boto.set_stream_logger('boto')
some points that I have noticed:
the Group and User Rules are okay, with removed leading space in front of "mybucket"
the first directory name is "Development" instead of "one"
"Without Validate" means access file directly
The following code works fine:
import boto
conn = boto.connect_s3("id","secret")
bucket = conn.get_bucket('mybucket', validate=False)
bucket.get_key('Development/two/three.png')
# <Key: mybucket,Development/two/three.png>
But i am new to IAM, and it seems "With Validate" first tries to read "/mybucket/" but it is denied via User Policy ExplicitlyDenyAnyRequestsForAllOtherFoldersExceptDevelopment.
edited to comment "to access all keys inside Development" try this::
list = bucket.list("Development/",delimiter="/")
for key in list:
print key.name