Develop Reference

How to copy specific files between 2 different accounts S3 buckets

I have my company S3 (companys3) bucket with multiple files for example file1, file2 and file3. And client S3 bucket (clients3) with some files that i don't know.
What I want is the solution for opening only file2 from companys3 to clients3.
I found solutions about how to copy/clone whole buckets. But couldn't find any that copy only specific files.
Till this time wi copy files through Kubernetes pods, but files become too large to handle this way (ower 20GB one file), so I am searching to solution that allows us to quit using Kubernetes pods ad transfer clients.

You can use S3 command line (awscli).
aws s3 cp s3://COMPANY_BUCKET/filename s3://CLIENT_BUCKET/filename

Suppose you have SOURCE and DESTINATION bucket. You need to delegate permission properly.
SOURCE S3
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DelegateS3Access",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::DESTINATION_BUCKET_ACCOUNT_NUMBER:root"
},
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::SOURCE_BUCKET_NAME/*",
"arn:aws:s3:::SOURCE_BUCKET_NAME"
]
}
]
}
DESTINATION S3
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::SOURCE_BUCKET_NAME",
"arn:aws:s3:::SOURCE_BUCKET_NAME/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::DESTINATION_BUCKET_NAME",
"arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
]
}
]
}
After that you will use AWS CLI
aws s3 sync s3://SOURCE-BUCKET-NAME s3://DESTINATION-BUCKET-NAME --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME

You also need to attach an IAM policy to the source bucket to add access for the destination account. And then you can copy the bucket content with AWS CLI Check this guide

An error occurred (AccessDenied) when calling the PutObject operation: Access Denied python

I am trying to give permissions to my AWS bucket to be able to upload files. I have tried updating the public access to public, and changing my bucket policy. I am sort of stuck and I am not sure how to proceed this issue.
Bucket Policy:
"Version": "2012-10-17",
"Id": "PolicyID",
"Statement": [
{
"Sid": "SID",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::id:user1"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::bucket/*"
},
{
"Sid": "SID",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::id:user2"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::bucket/*"
}
]
}
Update: I am able to upload a file using the python command console, but not in my source code. Here is what I have
s3.boto3.resource('s3')
s3.meta.client.upload_file('file.csv', 'bucket', 'file.csv')
I have also tried:
s3.boto.resource('s3')
s3.Object('bucket').upload_fileobj(object)

To successfully connect with AWS I had to pass credentials to the s3 resource
s3 = boto3.resource('s3', aws_access_key_id='key', aws_secret_access_key='key')

Can't find required permission to create S3 Batch job in IAM

I am attempting to create an S3 Batch (not AWS Batch, this is S3 Batch) job via boto3, and cannot figure out what permissions I need to enable for successful creation. I keep getting an "Access Denied" when I try to create the job, but it works fine when I apply the S3 Full Access policy to the execution role. Not a good long-term solution, obviously...
I am pretty certain that I need to add a specific permission in IAM, but I can't figure out which one. I can't see a "CreateJob" permission anywhere. Possibly I need to add access to some kind of s3 control bucket where the job is written?
I have tried adding permissions to a couple variations of what could be the S3 control bucket, but I haven't been successful yet.
This works fine when full S3 perms policy is applied:
import boto3
s3_control_client = boto3.client('s3control', region_name='us-east-1')
response = s3_control_client.create_job([very long and boring])
This is the output (scrubbed) that I get in the logs when I try to run with what I think are acceptable permissions.
2019-05-23 18:35:37,934 Starting new HTTPS connection (1): [ACCOUNTIDNUMBER].s3-control.us-east-1.amazonaws.com:443
2019-05-23 18:35:38,040 https://[ACCOUNTIDNUMBER].s3-control.us-east-1.amazonaws.com:443 "POST /v20180820/jobs HTTP/1.1" 403 204
2019-05-23 18:35:38,040 Response headers: {'x-amz-id-2': '[SCRUBBED]', 'x-amz-request-id': '[SCRUBBED], [SCRUBBED]', 'Date': 'Thu, 23 May 2019 18:35:38 GMT', 'Content-Type': 'application/xml', 'Content-Length': '204', 'Server': 'AmazonS3'}
2019-05-23 18:35:38,041 Response body:
b'<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>[SCRUBBED]</RequestId>
Any ideas on what permissions I need to enable here for this to complete?

According to this, you need s3:CreateJob, as well as iam:PassRole to the role that will be attached to the batch job.
So, something likes this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": [
"arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME"
]
},
{
"Effect": "Allow",
"Action": [
"s3:CreateJob"
],
"Resource": [
"*"
]
}
]
}

There are no special S3 batch job permissions that you can use. There might be some variations of permissions that you need depending on your use case. In general, you will need these permissions.
Permissions for your destination bucket
s3:PutObject
s3:PutObjectAcl
s3:PutObjectTagging
Permissions for your source bucket
s3:GetObject
Permissions for your manifest bucket
s3:GetObject
s3:GetObjectVersion
s3:GetBucketLocation
Permissions for your report bucket
s3:PutObject
s3:GetBucketLocation
Here is a template that you can use
{
"Version":"2012-10-17",
"Statement":[
{
"Action": [
"s3:PutObject",
"s3:PutObjectAcl",
"s3:PutObjectTagging"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::{{DestinationBucket}}/*"
},
{
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::{{SourceBucket}}/*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::{{ManifestBucket}}/*"
]
},
{
"Effect":"Allow",
"Action":[
"s3:PutObject",
"s3:GetBucketLocation"
],
"Resource":[
"arn:aws:s3:::{{ReportBucket}}/*"
]
}
]
}
You can check this link for more information.

Why this AWS IAM policy only works with an asterisk on the resource?

I'm trying to download some files I already uploaded to S3 with some Python code, but I'm getting headaches trying to use a tight policy.
I can list all the files in the bucket, but when I try do download them with what I see as a correct policy, I get botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Then, when I was trying to add a different policy that worked for 2 different buckets, I added part of the bucket's name, then the asterisk, and for some reason, the same exact thing worked.
So can someone tell me why this happens?
This for example, is what works like a charm:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1499955913000",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::THE-BEGINING-OF-THE-NAME*"
}
]
}
But this doesn't:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1499955913000",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::THE-EXACT-COMPLETE-FULL-NAME"
}
]
}
I can add the python code for the download if it's relevant, but this questions seems long enough, and the code is pretty straightforward

Seems I just needed some rubber duck debugging, the answer was I think counter intuitive, but easy:
It seems the ARN it's not only an identifier for the AWS resource itself, but also its content. So, when giving permissions, you need to give permissions to "the bucket" for listing it, and "the content" to download it
Which leads to a policy like this:
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "Stmt1499955913000",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": [
"arn:aws:s3:::THE-EXACT-COMPLETE-FULL-NAME",
"arn:aws:s3:::THE-EXACT-COMPLETE-FULL-NAME/*"
]
}]
}
Which as I said, gives control over the bucket itself, with no asterisks, and whatever goes after the slash bar.

Boto 403 AccessDenied Exception with IAM user credentials, Works in Cyberduck and AWS web console

I have found lot of questions regarding this on stackoverflow but none solved my problem. After lot of googling still i am facing AccessDenied Exception:
<Error>
<Code>AccessDenied</Code>
</Message><RequestId>ADF9C0DE6C86DF4F</RequestId>
<HostId>JwQLkNB0LuJvh0jwrsJe9wazxLsd+hrZ2qwvjCvmXYd2A/ckCrsotRMHm</HostId>
</Error>
Here are my policy docs for user and group:
User Policy:
{
"Statement":[
{
"Sid":"AllowListBucketIfSpecificPrefixIsIncludedInRequest",
"Action":"s3:*",
"Effect":"Allow",
"Resource":["arn:aws:s3::: mybucket", "arn:aws:s3:::mybucket/*"],
"Condition":{
"StringLike":{"s3:prefix":["Development/*"]
}
}
},
{
"Sid":"AllowUserToReadWriteObjectDataInDevelopmentFolder",
"Action":"s3:*",
"Effect":"Allow",
"Resource":["arn:aws:s3::: mybucket/Development/*"]
},
{
"Sid": "ExplicitlyDenyAnyRequestsForAllOtherFoldersExceptDevelopment",
"Action": ["s3:ListBucket"],
"Effect": "Deny",
"Resource": ["arn:aws:s3::: mybucket", "arn:aws:s3::: mybucket/*"],
"Condition":{ "StringNotLike": {"s3:prefix":["Development/*"] },
"Null" : {"s3:prefix":false }
}
}
]
}
Group Policy:
{
"Statement": [
{
"Sid": "AllowGroupToSeeBucketListAndAlsoAllowGetBucketLocationRequiredForListBucket",
"Action": ["s3:ListAllMyBuckets", "s3:GetBucketLocation"],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::*"]
},
{
"Sid": "AllowRootLevelListingOfCompanyBucket",
"Action": ["s3:ListBucket"],
"Effect": "Allow",
"Resource": ["arn:aws:s3::: mybucket", "arn:aws:s3::: mybucket/*"],
"Condition":{
"StringEquals":{"s3:prefix":[""]}
}
},
{
"Sid": "RequireFolderStyleList",
"Action": ["s3:ListBucket"],
"Effect": "Deny",
"Resource": ["arn:aws:s3:::*"],
"Condition":{
"StringNotEquals":{"s3:delimiter":"/"}
}
},
{
"Sid": "ExplictDenyAccessToPrivateFolderToEveryoneInTheGroup",
"Action": ["s3:*"],
"Effect": "Deny",
"Resource":["arn:aws:s3:::mybucket/Private/*"]
},
{
"Sid": "DenyListBucketOnPrivateFolder",
"Action": ["s3:ListBucket"],
"Effect": "Deny",
"Resource": ["arn:aws:s3:::*"],
"Condition":{
"StringLike":{"s3:prefix":["Private/"]}
}
}
]
}
Created a user with username - testuser
then got access_key and secret_access_key for this IAM user.
Now i am able to access mybucket and its subfolder using aws web console and cyberduck.
But whenever i am trying to access using boto , getting AccessDenied Exception (Error 403).
Boto Code:
<!-- language: python -->
from boto.s3.connection import S3Connection
connect = S3Connection('_______________________','_____________________')
# Without Validate
bucket = conn.get_bucket('mybucket', validate=False) #here got bucket object
bucket.get_key('one/two/three.png') # AccessDenied
#With Validate
bucket = conn.get_bucket('mybucket') #AccessDenied
Even i faced same problem when i was trying to use boto-rsync.
Any Suggestions ??

Error 403 means Access Denied so there is a authentication problem. To analyze the API call and the response one can use the following line:
boto.set_stream_logger('boto')
some points that I have noticed:
the Group and User Rules are okay, with removed leading space in front of "mybucket"
the first directory name is "Development" instead of "one"
"Without Validate" means access file directly
The following code works fine:
import boto
conn = boto.connect_s3("id","secret")
bucket = conn.get_bucket('mybucket', validate=False)
bucket.get_key('Development/two/three.png')
# <Key: mybucket,Development/two/three.png>
But i am new to IAM, and it seems "With Validate" first tries to read "/mybucket/" but it is denied via User Policy ExplicitlyDenyAnyRequestsForAllOtherFoldersExceptDevelopment.
edited to comment "to access all keys inside Development" try this::
list = bucket.list("Development/",delimiter="/")
for key in list:
print key.name