Develop Reference

How to copy specific files between 2 different accounts S3 buckets

I have my company S3 (companys3) bucket with multiple files for example file1, file2 and file3. And client S3 bucket (clients3) with some files that i don't know.
What I want is the solution for opening only file2 from companys3 to clients3.
I found solutions about how to copy/clone whole buckets. But couldn't find any that copy only specific files.
Till this time wi copy files through Kubernetes pods, but files become too large to handle this way (ower 20GB one file), so I am searching to solution that allows us to quit using Kubernetes pods ad transfer clients.

You can use S3 command line (awscli).
aws s3 cp s3://COMPANY_BUCKET/filename s3://CLIENT_BUCKET/filename

Suppose you have SOURCE and DESTINATION bucket. You need to delegate permission properly.
SOURCE S3
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DelegateS3Access",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::DESTINATION_BUCKET_ACCOUNT_NUMBER:root"
},
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::SOURCE_BUCKET_NAME/*",
"arn:aws:s3:::SOURCE_BUCKET_NAME"
]
}
]
}
DESTINATION S3
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::SOURCE_BUCKET_NAME",
"arn:aws:s3:::SOURCE_BUCKET_NAME/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": [
"arn:aws:s3:::DESTINATION_BUCKET_NAME",
"arn:aws:s3:::DESTINATION_BUCKET_NAME/*"
]
}
]
}
After that you will use AWS CLI
aws s3 sync s3://SOURCE-BUCKET-NAME s3://DESTINATION-BUCKET-NAME --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME

You also need to attach an IAM policy to the source bucket to add access for the destination account. And then you can copy the bucket content with AWS CLI Check this guide

An error occurred (AccessDenied) when calling the PutObject operation: Access Denied python

I am trying to give permissions to my AWS bucket to be able to upload files. I have tried updating the public access to public, and changing my bucket policy. I am sort of stuck and I am not sure how to proceed this issue.
Bucket Policy:
"Version": "2012-10-17",
"Id": "PolicyID",
"Statement": [
{
"Sid": "SID",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::id:user1"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::bucket/*"
},
{
"Sid": "SID",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::id:user2"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::bucket/*"
}
]
}
Update: I am able to upload a file using the python command console, but not in my source code. Here is what I have
s3.boto3.resource('s3')
s3.meta.client.upload_file('file.csv', 'bucket', 'file.csv')
I have also tried:
s3.boto.resource('s3')
s3.Object('bucket').upload_fileobj(object)

To successfully connect with AWS I had to pass credentials to the s3 resource
s3 = boto3.resource('s3', aws_access_key_id='key', aws_secret_access_key='key')

Why this AWS IAM policy only works with an asterisk on the resource?

I'm trying to download some files I already uploaded to S3 with some Python code, but I'm getting headaches trying to use a tight policy.
I can list all the files in the bucket, but when I try do download them with what I see as a correct policy, I get botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Then, when I was trying to add a different policy that worked for 2 different buckets, I added part of the bucket's name, then the asterisk, and for some reason, the same exact thing worked.
So can someone tell me why this happens?
This for example, is what works like a charm:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1499955913000",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::THE-BEGINING-OF-THE-NAME*"
}
]
}
But this doesn't:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1499955913000",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::THE-EXACT-COMPLETE-FULL-NAME"
}
]
}
I can add the python code for the download if it's relevant, but this questions seems long enough, and the code is pretty straightforward

Seems I just needed some rubber duck debugging, the answer was I think counter intuitive, but easy:
It seems the ARN it's not only an identifier for the AWS resource itself, but also its content. So, when giving permissions, you need to give permissions to "the bucket" for listing it, and "the content" to download it
Which leads to a policy like this:
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "Stmt1499955913000",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": [
"arn:aws:s3:::THE-EXACT-COMPLETE-FULL-NAME",
"arn:aws:s3:::THE-EXACT-COMPLETE-FULL-NAME/*"
]
}]
}
Which as I said, gives control over the bucket itself, with no asterisks, and whatever goes after the slash bar.

S3ResponseError: 403 Forbidden using Boto

I have a permission problem on an S3 bucket when I try to access certain files using BOTO in Python. Here is the bucket policy :
{
"Version": "2008-10-17",
"Id": "Policy1407346649831",
"Statement": [
{
"Sid": "Stmt1407346646598",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::029030651757:user/my_iam_user"
},
"Action": "s3:*",
"Resource": ["arn:aws:s3:::my_bucket/*",
"arn:aws:s3:::my_bucket"]
},
{
"Sid": "2",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity EFUS443HMBYF"
},
"Action": "s3:GetObject",
"Resource": ["arn:aws:s3:::my_bucket/*",
"arn:aws:s3:::my_bucket"]
},
{
"Sid": "3",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity EFUS443HMBYF"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::my_bucket/*"
}
]
}
I have 3 statements. The first one is to authorize the user my_iam_user to access the bucket my_bucket, the second one is to authorize a Cloudfront distribution to read the bucket and the last one is to authorize a Cloudfront distribution to write on the bucket.
Now I have to files on my bucket profile_pictures/15/file1.jpg and profile_pictures/15/file2.jpg. The first one was created using a signed url and CloudFront, the second one was put on the S3 using Boto. Now I try to access the files using Boto. Here is my code:
import boto
from boto.s3.key import Key
s3 = boto.connect_s3(
aws_access_key_id="access_key_of_my_iam_user",
aws_secret_access_key="secret_key_of_my_iam_user"
)
bucket = s3.get_bucket("my_bucket", validate=False)
k1 = Key(bucket)
k1.key = "profile_pictures/15/file1.jpg"
k1.get_contents_as_string()
k2 = Key(bucket)
k2.key = "profile_pictures/15/file2.jpg"
k2.get_contents_as_string()
The problem is that the access to file1 returns an error:
S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>8C5DE910C7B18F9E</RequestId><HostId>XiKU5Q+B0Wme3GpUNmUoD9KpUN63T3bFu/rAb/wh3rhDMkykoRsdQIFgyIp8zfAwMR1apbqUEFY=</HostId></Error>
whereas the second one is a success. What could be wrong?
Note: The time on the client that is running the code is good.

Even though the IAM user has been granted full access to the bucket by the first policy, they still will not automatically have access to file1 because they are not the owner of that object (Cloudfront is) and they have not been granted explicit access to the object. Also, presumably, the IAM user is also not the owner of the bucket.
If you look at Example 1 on this page you will see an almost identical situation and further explanation of the of how the object context is used to determine whether a request is granted or denied.

You will not have access to the bucket "my-bucket"..
Bucket names are required to be unique across the entire S3 eco-system.
If you try to access a unique bucket name e.g "MY_UNIQUE_ORG_NAME-MY_UNIQUE_BUCKET_NAME" you will probably have better luck..

Boto 403 AccessDenied Exception with IAM user credentials, Works in Cyberduck and AWS web console

I have found lot of questions regarding this on stackoverflow but none solved my problem. After lot of googling still i am facing AccessDenied Exception:
<Error>
<Code>AccessDenied</Code>
</Message><RequestId>ADF9C0DE6C86DF4F</RequestId>
<HostId>JwQLkNB0LuJvh0jwrsJe9wazxLsd+hrZ2qwvjCvmXYd2A/ckCrsotRMHm</HostId>
</Error>
Here are my policy docs for user and group:
User Policy:
{
"Statement":[
{
"Sid":"AllowListBucketIfSpecificPrefixIsIncludedInRequest",
"Action":"s3:*",
"Effect":"Allow",
"Resource":["arn:aws:s3::: mybucket", "arn:aws:s3:::mybucket/*"],
"Condition":{
"StringLike":{"s3:prefix":["Development/*"]
}
}
},
{
"Sid":"AllowUserToReadWriteObjectDataInDevelopmentFolder",
"Action":"s3:*",
"Effect":"Allow",
"Resource":["arn:aws:s3::: mybucket/Development/*"]
},
{
"Sid": "ExplicitlyDenyAnyRequestsForAllOtherFoldersExceptDevelopment",
"Action": ["s3:ListBucket"],
"Effect": "Deny",
"Resource": ["arn:aws:s3::: mybucket", "arn:aws:s3::: mybucket/*"],
"Condition":{ "StringNotLike": {"s3:prefix":["Development/*"] },
"Null" : {"s3:prefix":false }
}
}
]
}
Group Policy:
{
"Statement": [
{
"Sid": "AllowGroupToSeeBucketListAndAlsoAllowGetBucketLocationRequiredForListBucket",
"Action": ["s3:ListAllMyBuckets", "s3:GetBucketLocation"],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::*"]
},
{
"Sid": "AllowRootLevelListingOfCompanyBucket",
"Action": ["s3:ListBucket"],
"Effect": "Allow",
"Resource": ["arn:aws:s3::: mybucket", "arn:aws:s3::: mybucket/*"],
"Condition":{
"StringEquals":{"s3:prefix":[""]}
}
},
{
"Sid": "RequireFolderStyleList",
"Action": ["s3:ListBucket"],
"Effect": "Deny",
"Resource": ["arn:aws:s3:::*"],
"Condition":{
"StringNotEquals":{"s3:delimiter":"/"}
}
},
{
"Sid": "ExplictDenyAccessToPrivateFolderToEveryoneInTheGroup",
"Action": ["s3:*"],
"Effect": "Deny",
"Resource":["arn:aws:s3:::mybucket/Private/*"]
},
{
"Sid": "DenyListBucketOnPrivateFolder",
"Action": ["s3:ListBucket"],
"Effect": "Deny",
"Resource": ["arn:aws:s3:::*"],
"Condition":{
"StringLike":{"s3:prefix":["Private/"]}
}
}
]
}
Created a user with username - testuser
then got access_key and secret_access_key for this IAM user.
Now i am able to access mybucket and its subfolder using aws web console and cyberduck.
But whenever i am trying to access using boto , getting AccessDenied Exception (Error 403).
Boto Code:
<!-- language: python -->
from boto.s3.connection import S3Connection
connect = S3Connection('_______________________','_____________________')
# Without Validate
bucket = conn.get_bucket('mybucket', validate=False) #here got bucket object
bucket.get_key('one/two/three.png') # AccessDenied
#With Validate
bucket = conn.get_bucket('mybucket') #AccessDenied
Even i faced same problem when i was trying to use boto-rsync.
Any Suggestions ??

Error 403 means Access Denied so there is a authentication problem. To analyze the API call and the response one can use the following line:
boto.set_stream_logger('boto')
some points that I have noticed:
the Group and User Rules are okay, with removed leading space in front of "mybucket"
the first directory name is "Development" instead of "one"
"Without Validate" means access file directly
The following code works fine:
import boto
conn = boto.connect_s3("id","secret")
bucket = conn.get_bucket('mybucket', validate=False)
bucket.get_key('Development/two/three.png')
# <Key: mybucket,Development/two/three.png>
But i am new to IAM, and it seems "With Validate" first tries to read "/mybucket/" but it is denied via User Policy ExplicitlyDenyAnyRequestsForAllOtherFoldersExceptDevelopment.
edited to comment "to access all keys inside Development" try this::
list = bucket.list("Development/",delimiter="/")
for key in list:
print key.name