I have my company S3 (companys3) bucket with multiple files for example file1, file2 and file3. And client S3 bucket (clients3) with some files that i don't know.
What I want is the solution for opening only file2 from companys3 to clients3.
I found solutions about how to copy/clone whole buckets. But couldn't find any that copy only specific files.
Till this time wi copy files through Kubernetes pods, but files become too large to handle this way (ower 20GB one file), so I am searching to solution that allows us to quit using Kubernetes pods ad transfer clients.
You can use S3 command line (awscli).
aws s3 cp s3://COMPANY_BUCKET/filename s3://CLIENT_BUCKET/filename
Suppose you have SOURCE and DESTINATION bucket. You need to delegate permission properly.
SOURCE S3
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DelegateS3Access",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::DESTINATION_BUCKET_ACCOUNT_NUMBER:root"
},
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::SOURCE_BUCKET_NAME/*",
"arn:aws:s3:::SOURCE_BUCKET_NAME"
]
}
]
}
DESTINATION S3
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::SOURCE_BUCKET_NAME",
"arn:aws:s3:::SOURCE_BUCKET_NAME/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::DESTINATION_BUCKET_NAME",
"arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
]
}
]
}
After that you will use AWS CLI
aws s3 sync s3://SOURCE-BUCKET-NAME s3://DESTINATION-BUCKET-NAME --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME
You also need to attach an IAM policy to the source bucket to add access for the destination account. And then you can copy the bucket content with AWS CLI Check this guide
Related
I'm trying to download a file from S3 path in another account using AWS Lambda & python. I'm able to download the files if I provide full key to the copy_object. This will not work for me on a day-to-day scenario as there's no notification and my scheduled lambda code has to check for the presence of file using wildcards. But I'm getting Access denied(An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied) error while trying to list the contents of that bucket even though I have get & list permissions added.
I have the following policy added to my lambda role.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::<src S3 bucket>/",
"arn:aws:s3:::<src S3 bucket>/*"
],
"Effect": "Allow"
},
{
"Action": [
"kms:Decrypt",
"kms:Encrypt",
"kms:GenerateDataKey",
"kms:ReEncrypt*",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:us-east-1:<src bucket account id>:key/<src bucket kms key>"
],
"Effect": "Allow"
}
]
}
In the source bucket, they have added the below permissions for my lambda role
{
"Sid": <>,
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<my account id>:role/LambdaRole"
]
},
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::<src S3 bucket>",
"arn:aws:s3:::<src S3 bucket>/*"
]
},
Python code to pull the file
def get_s3_file(event):
s3 = boto3.client('s3')
bucket_name = event['Records'][0]['s3']['bucket']['name']
file_key = unquote_plus(event['Records'][0]['s3']['object']['key'])
file_name = file_key+'test.txt'
logger.info('Reading {} from {}'.format(file_key, bucket_name))
objects = s3.list_objects_v2(Bucket=bucket_name, Prefix=file_key)
for object in objects:
print(object.key)
if object.key.endswith('.txt'):
print(object.key)
copy_source = {'Bucket': bucket_name,'Key': file_name}
s3_c.copy_object(Bucket='<my bucket>', Key='/data/', CopySource=copy_source, ACL='bucket-owner-full-control')
Probably the issue is due to KMS. If the lambda and bucket are in different accounts, lambda will not have automatically access to kms key in the second account. You have to modify KMS key policy to allow for that. So try adding lambda role arn to the KMS policy in the second account.
I want to add an String, created by an lambda, to a existing textfile on my s3 bucket.
When I use:
s3.Object('My_bucket', 'textfile.txt').put(Body=missingtagginginfo)
missingtagginginfo = My created variable
textfile.txt = textfile that exists on the s3 bucket
I get the following error:
"errorMessage": "An error occurred (AccessDenied) when calling the PutObject operation: Access Denied",
"errorType": "ClientError",
I alreday gave the lambda the s3fullAccess.
Does anybody know how I can fix this?
greets
Your policy must contain s3:PutObject in order to upload objects.
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::your-bucket/",
"arn:aws:s3:::your-bucket/*"
]
}
Please don't add wildcard permissions (s3:*) or wildcard resources ("Resource": [ "*" ]) as these give huge potential for bugs and vulnerabilities. The AWS Policy Generator can help you with this.
The IAM Policy looks like this:
"Resource": "arn:aws:s3:::MYBUCKET/*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucketMultipartUploads",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::MYBUCKET/",
"arn:aws:s3:::MYBUCKET/*"
]
},
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "*"
}
Write in Your Bucket's Policy
{
"Version": "2008-10-17",
"Id": "PolicyForCloudFrontPrivateContent",
"Statement": [
{
"Sid": "Allow-OAI-Access-To-Bucket",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::Your-Bucket-Name/*"
}
]
}
I am trying to give permissions to my AWS bucket to be able to upload files. I have tried updating the public access to public, and changing my bucket policy. I am sort of stuck and I am not sure how to proceed this issue.
Bucket Policy:
"Version": "2012-10-17",
"Id": "PolicyID",
"Statement": [
{
"Sid": "SID",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::id:user1"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::bucket/*"
},
{
"Sid": "SID",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::id:user2"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::bucket/*"
}
]
}
Update: I am able to upload a file using the python command console, but not in my source code. Here is what I have
s3.boto3.resource('s3')
s3.meta.client.upload_file('file.csv', 'bucket', 'file.csv')
I have also tried:
s3.boto.resource('s3')
s3.Object('bucket').upload_fileobj(object)
To successfully connect with AWS I had to pass credentials to the s3 resource
s3 = boto3.resource('s3', aws_access_key_id='key', aws_secret_access_key='key')
I am attempting to create an S3 Batch (not AWS Batch, this is S3 Batch) job via boto3, and cannot figure out what permissions I need to enable for successful creation. I keep getting an "Access Denied" when I try to create the job, but it works fine when I apply the S3 Full Access policy to the execution role. Not a good long-term solution, obviously...
I am pretty certain that I need to add a specific permission in IAM, but I can't figure out which one. I can't see a "CreateJob" permission anywhere. Possibly I need to add access to some kind of s3 control bucket where the job is written?
I have tried adding permissions to a couple variations of what could be the S3 control bucket, but I haven't been successful yet.
This works fine when full S3 perms policy is applied:
import boto3
s3_control_client = boto3.client('s3control', region_name='us-east-1')
response = s3_control_client.create_job([very long and boring])
This is the output (scrubbed) that I get in the logs when I try to run with what I think are acceptable permissions.
2019-05-23 18:35:37,934 Starting new HTTPS connection (1): [ACCOUNTIDNUMBER].s3-control.us-east-1.amazonaws.com:443
2019-05-23 18:35:38,040 https://[ACCOUNTIDNUMBER].s3-control.us-east-1.amazonaws.com:443 "POST /v20180820/jobs HTTP/1.1" 403 204
2019-05-23 18:35:38,040 Response headers: {'x-amz-id-2': '[SCRUBBED]', 'x-amz-request-id': '[SCRUBBED], [SCRUBBED]', 'Date': 'Thu, 23 May 2019 18:35:38 GMT', 'Content-Type': 'application/xml', 'Content-Length': '204', 'Server': 'AmazonS3'}
2019-05-23 18:35:38,041 Response body:
b'<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>[SCRUBBED]</RequestId>
Any ideas on what permissions I need to enable here for this to complete?
According to this, you need s3:CreateJob, as well as iam:PassRole to the role that will be attached to the batch job.
So, something likes this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": [
"arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME"
]
},
{
"Effect": "Allow",
"Action": [
"s3:CreateJob"
],
"Resource": [
"*"
]
}
]
}
There are no special S3 batch job permissions that you can use. There might be some variations of permissions that you need depending on your use case. In general, you will need these permissions.
Permissions for your destination bucket
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
Permissions for your source bucket
s3:GetObject
Permissions for your manifest bucket
s3:GetObject
s3:GetObjectVersion
s3:GetBucketLocation
Permissions for your report bucket
s3:PutObject
s3:GetBucketLocation
Here is a template that you can use
{
"Version":"2012-10-17",
"Statement":[
{
"Action": [
"s3:PutObject",
"s3:PutObjectAcl",
"s3:PutObjectTagging"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::{{DestinationBucket}}/*"
},
{
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::{{SourceBucket}}/*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::{{ManifestBucket}}/*"
]
},
{
"Effect":"Allow",
"Action":[
"s3:PutObject",
"s3:GetBucketLocation"
],
"Resource":[
"arn:aws:s3:::{{ReportBucket}}/*"
]
}
]
}
You can check this link for more information.
I'm trying to download some files I already uploaded to S3 with some Python code, but I'm getting headaches trying to use a tight policy.
I can list all the files in the bucket, but when I try do download them with what I see as a correct policy, I get botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Then, when I was trying to add a different policy that worked for 2 different buckets, I added part of the bucket's name, then the asterisk, and for some reason, the same exact thing worked.
So can someone tell me why this happens?
This for example, is what works like a charm:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1499955913000",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::THE-BEGINING-OF-THE-NAME*"
}
]
}
But this doesn't:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1499955913000",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::THE-EXACT-COMPLETE-FULL-NAME"
}
]
}
I can add the python code for the download if it's relevant, but this questions seems long enough, and the code is pretty straightforward
Seems I just needed some rubber duck debugging, the answer was I think counter intuitive, but easy:
It seems the ARN it's not only an identifier for the AWS resource itself, but also its content. So, when giving permissions, you need to give permissions to "the bucket" for listing it, and "the content" to download it
Which leads to a policy like this:
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "Stmt1499955913000",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": [
"arn:aws:s3:::THE-EXACT-COMPLETE-FULL-NAME",
"arn:aws:s3:::THE-EXACT-COMPLETE-FULL-NAME/*"
]
}]
}
Which as I said, gives control over the bucket itself, with no asterisks, and whatever goes after the slash bar.