I'm trying to restrict all my AWS Cognito users to their own subdirectory in my S3 bucket.
I don't want them listing, reading, or writing other people's subdirectories/files in my larger bucket, and I only want them to read & write objects in their own directory.
I'm drawing inspiration from this AWS documentation snippet.
Here's my policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-bucket"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"subfolder/"
]
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}",
"arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}/*"
]
}
]
}
And my code to retrieve the file of a certain user with user_id = test#test.com, but actually allows me to retrieve a restricted file:
import boto
# These keys are *not* hardcoded, I'm just leaving out
# the auth flow to get them from Cognito/STS as described
# here: https://mobile.awsblog.com/post/Tx2FL1QAPDE0UAH/Understanding-Amazon-Cognito-Authentication-Part-2-Developer-Authenticated-Ident
conn = boto.s3.connect_to_region('us-east-1',
aws_access_key_id=ACCESS_KEY_FROM_COGNITO,
aws_secret_access_key=SECRET_KEY_FROM_COGNITO,
security_token=SECURITY_KEY_FROM_COGNITO)
# get the bucket
b = conn.get_bucket('my-bucket', validate=False)
# try to get an object we SHOULD be able to get
k = Key(b)
k.key = 'subfolder/us-east-1:xxxx-xxxx-xxxx-xxxxxxxxx/foobar'
print "Contents:", k.get_contents_as_string() # success!
# try to get and object we SHOUDN'T be able to get
k2 = Key(b)
k2.key = 'subfolder/BLAH_BLAH/restricted'
print "Contents:", k2.get_contents_as_string() # should fail, but doesn't
Unfortunately, I can access and read the contents of both files, yet I'm following the exact same pattern in the AWS blog documentation post. I'm also unsure why I need the validate=False in the boto connection, but it seems to work well enough.
What am I missing?
EDIT: In response to the answer below, I've tried updating my role to the following, but it does not make a difference:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-bucket"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"subfolder/${cognito-identity.amazonaws.com:sub}/*"
]
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}/*"
]
}
]
}
I've also confirmed that the access credentials I'm using are from Cognito by using the access/secret/security token triple retrieved from STS using Cognito token to create a boto IAMConnection object and querying for my role name corresponding to the auth'd cognito users for my identity pool. In doing so, I got the following exception when trying to read this role (which is exactly what should happen since I did not grant access):
BotoServerError: BotoServerError: 403 Forbidden
<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
<Error>
<Type>Sender</Type>
<Code>AccessDenied</Code>
<Message>User: arn:aws:sts::MY_AWS_ACCT_ID:assumed-role/my_policy_role_name/session_name_here is not authorized to perform: iam:GetRole on resource: role awsNameFor_Role_Given_123012313</Message>
</Error>
<RequestId>xxx-xxxx-xxxx-xxxx-xxxxx</RequestId>
</ErrorResponse>
So still no clarity on why this isn't working.
5 things:
Make sure you are using credentials issued by the Amazon Cognito Identity service otherwise ${cognito-identity.amazonaws.com:sub} will be empty and grant you access to everything
Make sure the Amazon Cognito Identity credentials you are using were issued after you updated the policy, the policy is embedded in the session portion of the credentials so if you are using old credentials, they may not have the current policy attached.
You cannot use the username of the user, you must use the Amazon Cognito Identity id. So instead of test#test.com it will be the identity id: us-east-1:beef-beef-beef-xxxxxxxxx
Your pool has 2 roles associated with it, an unauthenticated role and an authenticated role. Make sure you are setting your policy on the correct role, in this case it looks like you are using developer authenticated identities and should be modifying the policy on the authenticated role.
Check your S3 bucket policy, if you allow anonymous access to your bucket, the Cognito role policy will not override it. Turn off anonymous access if that is the case. http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-2
It looks like you're code is using a hard-coded access key and secret key and is NOT using Cognito to retrieve credentials. Instead of embedding the same access key and secret key for all users, to leverage Cognito you'll need to follow the Authentication Flow and use GetId (boto reference, AWS reference) to get an identity ID and then GetCredentialsForIdentity (boto reference AWS reference) to get the Cognito-issued credentials for the current ID. Then use those credentials with the boto S3 connection.
Also be sure to cache the ID for each user and reuse it when making additional calls to Cognito.
The answer was rather silly. Apparently buckets themselves in S3 have their own policies (they are rather tucked away), but I had the directive:
Principal: "*"
which caused the bucket to be world readable.
The second revelation was that if you restrict a bucket with the s3:ListBucket with a Condition, that doesn't mean if you list the bucket you'll get only those results - you must call it by name. As an example in boto:
wrong = bucket.list() # will simply 403
right = bucket.list(prefix="base/subdir/<cognito-id>/") # will succeed
In other words S3 is designed such that you must know the prefix-key of the desired folder, which is good practice anyway.
I have to say, I was quite impressed with how helpful the folks at AWS were in diagnosing this issue here and on their forum. Anyway, a much better understanding of S3 now.
Related
I am trying to move files from a S3 bucket in one account(source account) to S3 bucket in another account(destination account)
I am using sagemaker notebook so I have a sagemaker role.
I also have a role in my team account which has full s3 access and fullsagemaker access and in the trust relationship i have given the destination account role arn and sagemaker role arn.
The destination account also has my team role arn and sagemaker role arn in its trust policy.
I am trying to assume my team role and then I will assume the destination role to copy files.
import boto3
sts_client = boto3.client('sts')
assumed_teamrole_object = sts_client.assume_role(DurationSeconds=1800,
RoleArn='myteamrole',
RoleSessionName='test1')
assumed_destrole_object = sts_client.assume_role(DurationSeconds=1800,
ExternalId='externalid provided by destination account',
RoleArn='destination account role',
RoleSessionName='test2')
The first three lines execute fine. when I try to assume the destination role i am getting the error
An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::role/AmazonSageMaker-ExecutionRole-/SageMaker is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::destinationrole
Is there something I am missing, what am i doing wrong. Please help.
I dont have any user , it is just roles
Thanks!
The error message indicates that you are missing sts:AssumeRole permissions. Your comments indicate that this is the case, as you have only S3 permission for now.
To rectify this, you can add inline policy to AmazonSageMaker-ExecutionRole role, in the form of:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "*"
}
]
}
You can further limit the Resource to only arn:aws:iam::destinationrole. But for tests you can try with * as Resource.
I have the following :
Bucket name for s3 bucket in source account
from where I will grab the files.
IAM role for destination account
and also the s3 bucket name for destination account where I will
drop the files.
I created a user and role in my personal account and
also saved the access key and secret access key.While creating role I used another AWS account option.
I want to transfer files from source s3 bucket to destination s3 bucket.
I created a user on my personal account and gave full s3 access and full sagemaker access and I added the following policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource":"arn:aws:iam::***********:role/destinationrole"
}
]
}
I created role on my personal account for sagemaker.In the Trust relationships I added the user I created above and attached the same policy as above
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": [
"arn:aws:iam::***********:role/destinationrole"
]
}
]
}
I am trying to generate temporary credentials
import boto3
sts_client = boto3.client('sts')
assumed_role_object=sts_client.assume_role(
RoleArn="arn:aws:iam::***********:role/destinationrole,
RoleSessionName="test"
)
Not sure if the steps I am following are right, but when run the above script I am getting the error :An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::my_account:assumed-role//SageMaker is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::destination account:role/destinationrole
I need to move files from source account s3 bucket to destination account s3 bucket. source account has just provided the bucket name and destination account has provided the IAM role and bucket name . Source account has updated the bucket policy on their end allowing the destination account to pull the data. I am lost on what credentials to use and how to assume role and get temp credentials.
I'm creating an API that will ONLY accept requests made from the GitHub Webhook servers by using a Resource Policy with the GitHub IPs. I've successfully done this using the console and manually creating the Resource Policy, but I'm running into a problem when I'm using the CDK.
Here's my code:
delete_trigger_integration = aws_apigateway.LambdaIntegration(
trigger_step_lambda, proxy=False, integration_responses=[])
api_policy_document = aws_iam.PolicyDocument()
api = aws_apigateway.RestApi(
self,
"GithubWebhookApi",
rest_api_name=PROJECT_NAME + "-apigateway-trigger-delete",
default_integration=delete_trigger_integration,
policy=api_policy_document)
delete_execution_resource = api.root.add_resource("execution")
delete_execution_method = delete_execution_resource.add_method(
"POST", delete_trigger_integration)
delete_execution_resource.add_cors_preflight(allow_origins=["*"])
create_repo_lambda.add_environment("API_URL",
delete_execution_resource.url)
api_policy_document.add_statements(
aws_iam.PolicyStatement(
effect=aws_iam.Effect.ALLOW,
principals=[aws_iam.AnyPrincipal()],
actions=["execute-api:Invoke"],
resources=[api.arn_for_execute_api()]))
api_policy_document.add_statements(
aws_iam.PolicyStatement(
effect=aws_iam.Effect.DENY,
actions=["execute-api:Invoke"],
conditions={
"NotIpAddress": {
"aws:SourceIp": [
"192.30.252.0/22", "185.199.108.0/22",
"140.82.112.0/20"
]
}
},
principals=[aws_iam.AnyPrincipal()],
resources=[api.arn_for_execute_api()]))
I feel like I'm very close to having a solution but I can't figure out what it is. The problem with the code above is that I get the ValidationError: Circular dependency between resources error when trying to deploy it - which I can understand, the resource policy is addressing the very resource it's inside. But I can't find a way around it in the CDK.
For other resources it's really easy to ad IAM policies after the creation using aws_iam.add_to_role_policy but I can't find the equivalent for the RestApi class in the CDK. It seems you have to add the PolicyDocument when you declare the RestApi.
Also, here's the resource policy that I'm trying to recreate in the CDK (it's those resource ARNs which are causing the problem):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "arn:aws:execute-api:eu-west-1:000000000000:aaaaaaaaaa/*/*/*"
},
{
"Effect": "Deny",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "arn:aws:execute-api:eu-west-1:000000000000:aaaaaaaaaa/*/*/*",
"Condition": {
"NotIpAddress": {
"aws:SourceIp": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20"
]
}
}
}
]
}
Does anyone know a solution to my problem? Thanks in advance!
Also, you can ignore the empty integration responses - I'm still working on that.
This is related to how the underlying CloudFormation resource works.
As the Policy must be defined within the AWS::ApiGateway::RestApi resource, it cannot reference itself.
From the documentation:
To set the ARN for the policy, use the !Join intrinsic function with "" as delimiter and values of "execute-api:/" and "*".
Which translates into the following in your CDK code:
resources=[core.Fn.join('', ['execute-api:/', '*'])]
First time boto3 user.
I had a user with ACL S3FullAccess and used the following code to try and upload a file; it uses a pandas DataFrame as the source.
s3_client = boto3.client('s3')
io = StringIO()
df.to_csv(io)
response = s3_client.put_object(
Bucket=self.bucket,
Body=io,
Key=self.filename
)
This lead to this response
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
So I checked that the secret key and access key were being picked up by boto3 from my ~/.aws/credentials file, and they are, on line 604 of client.py in boto3 - request_signer=self._request_signer
So I researched here on SO, and it seemed a lot of people had to add a Policy document, so I did that as follows:
{
"Version": "2012-10-17",
"Id": "Policyxxx",
"Statement": [
{
"Sid": "Stmtx1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<12 digit id>:root"
},
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::<my-bucket>/*"
},
{
"Sid": "Stmtx6",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<12 digit id>:root"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<my-bucket>"
}
]
}
I still get the same error, so I added this to my put_object call since the S3 bucket uses AES-256 encryption, which I thought was server-side only, but running out of ideas, so worth a try.
SSECustomerKey=os.urandom(32),
SSECustomerAlgorithm='AES256',
Next I removed those terms associated with the SSE keys, realising that the AES-256 encryption is server side and should not affect my access.
Then I tried to generate a new pair of Access keys and use those instead, same result.
Does anyone have any idea what I might look at next, what have I missing in the hundreds of pages of AWS documentation?
This was simply a case of when the user was created they were added to a couple of groups. Administrators and EC2MFA. I was unaware of the implications of this, but assume the EC2MFA group prevented API or CLI access. I am assuming the combination of the Policy on the user and S3 side is sufficiently secure.
From what I've researched, it seems that S3 buckets are typically downloaded from the same AWS account that the API Key originates from.
I have been given access to a new AWS account, and this account contains an S3 bucket that I need to download on a daily basis. The issue is that I have my API key associated with another AWS account. Is there any way to import the new S3 bucket into my existing AWS account, or would it be simpler to try and create a new API key on the AWS account which already has the S3 bucket I need?
I am using python and boto3. Thanks!
The owner of the Amazon S3 bucket can add a Bucket Policy that grants access either publicly or to specific AWS users, including users from another AWS account. This way, you can use credentials from Account A to access a bucket in Account B.
For example, this Bucket Policy grants access to a user in a different account:
{
"Id": "S3Access",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DownloadFiles",
"Action": [
"s3:GetObject"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-bucket/*",
"Principal": {
"AWS": [
"arn:aws:iam::123456789012:user/fred"
]
}
},
{
"Sid": "ListFiles",
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::my-bucket",
"Principal": {
"AWS": [
"arn:aws:iam::123456789012:user/fred"
]
}
}
]
}
The ARN can be obtained by looking at the IAM User within the IAM management console.
Rather than downloading the whole bucket each time, I would recommend using the AWS Command-Line Interface (CLI), which has a convenient aws s3 sync command. This can synchronize files to/from Amazon S3 and will only copy files that have been added or modified since the last sync. You could run it on a regular basis (eg each hour) to have a local copy of the bucket, similar in concept to DropBox.