First time boto3 user.
I had a user with ACL S3FullAccess and used the following code to try and upload a file; it uses a pandas DataFrame as the source.
s3_client = boto3.client('s3')
io = StringIO()
df.to_csv(io)
response = s3_client.put_object(
Bucket=self.bucket,
Body=io,
Key=self.filename
)
This lead to this response
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
So I checked that the secret key and access key were being picked up by boto3 from my ~/.aws/credentials file, and they are, on line 604 of client.py in boto3 - request_signer=self._request_signer
So I researched here on SO, and it seemed a lot of people had to add a Policy document, so I did that as follows:
{
"Version": "2012-10-17",
"Id": "Policyxxx",
"Statement": [
{
"Sid": "Stmtx1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<12 digit id>:root"
},
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::<my-bucket>/*"
},
{
"Sid": "Stmtx6",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<12 digit id>:root"
},
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::<my-bucket>"
}
]
}
I still get the same error, so I added this to my put_object call since the S3 bucket uses AES-256 encryption, which I thought was server-side only, but running out of ideas, so worth a try.
SSECustomerKey=os.urandom(32),
SSECustomerAlgorithm='AES256',
Next I removed those terms associated with the SSE keys, realising that the AES-256 encryption is server side and should not affect my access.
Then I tried to generate a new pair of Access keys and use those instead, same result.
Does anyone have any idea what I might look at next, what have I missing in the hundreds of pages of AWS documentation?
This was simply a case of when the user was created they were added to a couple of groups. Administrators and EC2MFA. I was unaware of the implications of this, but assume the EC2MFA group prevented API or CLI access. I am assuming the combination of the Policy on the user and S3 side is sufficiently secure.
Related
I have the following permission policy on an IAM Role
statement {
effect = "Allow"
actions = [
"sagemaker:CreateModelPackageGroup",
"sagemaker:ListModelPackageGroups",
]
resources = [
"arn:aws:sagemaker:my_account_region:my_account_id:model-package-group/*",
]
to be able to create a ModelPackageGroup using boto3
import boto3
import logging
logger = logging.getLogger(__name__)
sagemaker_client = boto3.client('sagemaker')
mpg_name = 'my_model_package_name'
matching_mpg = sagemaker_client.list_model_package_groups(NameContains=mpg_name)[
"ModelPackageGroupSummaryList"
]
if matching_mpg:
logger.info(f"Using existing Model Package Group: {mpg_name}")
else:
mpg_input_dict = {
"ModelPackageGroupName": mpg_name,
"ModelPackageGroupDescription": mpg_description,
}
mpg_response = sagemaker_client.create_model_package_group(**mpg_input_dict)
but I got the following error
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the ListModelPackageGroups operation: User: arn:aws:sts::my_account_id:assumed-role/my_role_name/botocore-session-some_session_id is not authorized to perform: sagemaker:ListModelPackageGroups because no identity-based policy allows the sagemaker:ListModelPackageGroups action
After doing some research I've found out that the SageMaker Action ListModelPackageGroups requires an All Resources permission
The actions in your policy do not support resource-level permissions and require you to choose All resources
In fact, the Actions defined by Amazon SageMaker Documentation mentions that some SageMaker actions support resource level permission and others must specify all resources
The Resource types column indicates whether each action supports resource-level permissions. If there is no value for this column, you must specify all resources ("*") in the Resource element of your policy statement. If the column includes a resource type, then you can specify an ARN of that type in a statement with that action.
So, changing to the following policy worked out
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Action": [
"sagemaker:ListModelPackageGroups"
],
"Resource": "*"
},
{
"Sid": "",
"Effect": "Allow",
"Action": [
"sagemaker:CreateModelPackageGroup"
],
"Resource": "arn:aws:sagemaker:my_account_region:my_account_id:model-package-group/*"
}
]
}
I'm creating an API that will ONLY accept requests made from the GitHub Webhook servers by using a Resource Policy with the GitHub IPs. I've successfully done this using the console and manually creating the Resource Policy, but I'm running into a problem when I'm using the CDK.
Here's my code:
delete_trigger_integration = aws_apigateway.LambdaIntegration(
trigger_step_lambda, proxy=False, integration_responses=[])
api_policy_document = aws_iam.PolicyDocument()
api = aws_apigateway.RestApi(
self,
"GithubWebhookApi",
rest_api_name=PROJECT_NAME + "-apigateway-trigger-delete",
default_integration=delete_trigger_integration,
policy=api_policy_document)
delete_execution_resource = api.root.add_resource("execution")
delete_execution_method = delete_execution_resource.add_method(
"POST", delete_trigger_integration)
delete_execution_resource.add_cors_preflight(allow_origins=["*"])
create_repo_lambda.add_environment("API_URL",
delete_execution_resource.url)
api_policy_document.add_statements(
aws_iam.PolicyStatement(
effect=aws_iam.Effect.ALLOW,
principals=[aws_iam.AnyPrincipal()],
actions=["execute-api:Invoke"],
resources=[api.arn_for_execute_api()]))
api_policy_document.add_statements(
aws_iam.PolicyStatement(
effect=aws_iam.Effect.DENY,
actions=["execute-api:Invoke"],
conditions={
"NotIpAddress": {
"aws:SourceIp": [
"192.30.252.0/22", "185.199.108.0/22",
"140.82.112.0/20"
]
}
},
principals=[aws_iam.AnyPrincipal()],
resources=[api.arn_for_execute_api()]))
I feel like I'm very close to having a solution but I can't figure out what it is. The problem with the code above is that I get the ValidationError: Circular dependency between resources error when trying to deploy it - which I can understand, the resource policy is addressing the very resource it's inside. But I can't find a way around it in the CDK.
For other resources it's really easy to ad IAM policies after the creation using aws_iam.add_to_role_policy but I can't find the equivalent for the RestApi class in the CDK. It seems you have to add the PolicyDocument when you declare the RestApi.
Also, here's the resource policy that I'm trying to recreate in the CDK (it's those resource ARNs which are causing the problem):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "arn:aws:execute-api:eu-west-1:000000000000:aaaaaaaaaa/*/*/*"
},
{
"Effect": "Deny",
"Principal": "*",
"Action": "execute-api:Invoke",
"Resource": "arn:aws:execute-api:eu-west-1:000000000000:aaaaaaaaaa/*/*/*",
"Condition": {
"NotIpAddress": {
"aws:SourceIp": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20"
]
}
}
}
]
}
Does anyone know a solution to my problem? Thanks in advance!
Also, you can ignore the empty integration responses - I'm still working on that.
This is related to how the underlying CloudFormation resource works.
As the Policy must be defined within the AWS::ApiGateway::RestApi resource, it cannot reference itself.
From the documentation:
To set the ARN for the policy, use the !Join intrinsic function with "" as delimiter and values of "execute-api:/" and "*".
Which translates into the following in your CDK code:
resources=[core.Fn.join('', ['execute-api:/', '*'])]
I'm trying to download some files I already uploaded to S3 with some Python code, but I'm getting headaches trying to use a tight policy.
I can list all the files in the bucket, but when I try do download them with what I see as a correct policy, I get botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
Then, when I was trying to add a different policy that worked for 2 different buckets, I added part of the bucket's name, then the asterisk, and for some reason, the same exact thing worked.
So can someone tell me why this happens?
This for example, is what works like a charm:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1499955913000",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::THE-BEGINING-OF-THE-NAME*"
}
]
}
But this doesn't:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1499955913000",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::THE-EXACT-COMPLETE-FULL-NAME"
}
]
}
I can add the python code for the download if it's relevant, but this questions seems long enough, and the code is pretty straightforward
Seems I just needed some rubber duck debugging, the answer was I think counter intuitive, but easy:
It seems the ARN it's not only an identifier for the AWS resource itself, but also its content. So, when giving permissions, you need to give permissions to "the bucket" for listing it, and "the content" to download it
Which leads to a policy like this:
{
"Version": "2012-10-17",
"Statement": [{
"Sid": "Stmt1499955913000",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": [
"arn:aws:s3:::THE-EXACT-COMPLETE-FULL-NAME",
"arn:aws:s3:::THE-EXACT-COMPLETE-FULL-NAME/*"
]
}]
}
Which as I said, gives control over the bucket itself, with no asterisks, and whatever goes after the slash bar.
I have a permission problem on an S3 bucket when I try to access certain files using BOTO in Python. Here is the bucket policy :
{
"Version": "2008-10-17",
"Id": "Policy1407346649831",
"Statement": [
{
"Sid": "Stmt1407346646598",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::029030651757:user/my_iam_user"
},
"Action": "s3:*",
"Resource": ["arn:aws:s3:::my_bucket/*",
"arn:aws:s3:::my_bucket"]
},
{
"Sid": "2",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity EFUS443HMBYF"
},
"Action": "s3:GetObject",
"Resource": ["arn:aws:s3:::my_bucket/*",
"arn:aws:s3:::my_bucket"]
},
{
"Sid": "3",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity EFUS443HMBYF"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::my_bucket/*"
}
]
}
I have 3 statements. The first one is to authorize the user my_iam_user to access the bucket my_bucket, the second one is to authorize a Cloudfront distribution to read the bucket and the last one is to authorize a Cloudfront distribution to write on the bucket.
Now I have to files on my bucket profile_pictures/15/file1.jpg and profile_pictures/15/file2.jpg. The first one was created using a signed url and CloudFront, the second one was put on the S3 using Boto. Now I try to access the files using Boto. Here is my code:
import boto
from boto.s3.key import Key
s3 = boto.connect_s3(
aws_access_key_id="access_key_of_my_iam_user",
aws_secret_access_key="secret_key_of_my_iam_user"
)
bucket = s3.get_bucket("my_bucket", validate=False)
k1 = Key(bucket)
k1.key = "profile_pictures/15/file1.jpg"
k1.get_contents_as_string()
k2 = Key(bucket)
k2.key = "profile_pictures/15/file2.jpg"
k2.get_contents_as_string()
The problem is that the access to file1 returns an error:
S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>8C5DE910C7B18F9E</RequestId><HostId>XiKU5Q+B0Wme3GpUNmUoD9KpUN63T3bFu/rAb/wh3rhDMkykoRsdQIFgyIp8zfAwMR1apbqUEFY=</HostId></Error>
whereas the second one is a success. What could be wrong?
Note: The time on the client that is running the code is good.
Even though the IAM user has been granted full access to the bucket by the first policy, they still will not automatically have access to file1 because they are not the owner of that object (Cloudfront is) and they have not been granted explicit access to the object. Also, presumably, the IAM user is also not the owner of the bucket.
If you look at Example 1 on this page you will see an almost identical situation and further explanation of the of how the object context is used to determine whether a request is granted or denied.
You will not have access to the bucket "my-bucket"..
Bucket names are required to be unique across the entire S3 eco-system.
If you try to access a unique bucket name e.g "MY_UNIQUE_ORG_NAME-MY_UNIQUE_BUCKET_NAME" you will probably have better luck..
I'm trying to restrict all my AWS Cognito users to their own subdirectory in my S3 bucket.
I don't want them listing, reading, or writing other people's subdirectories/files in my larger bucket, and I only want them to read & write objects in their own directory.
I'm drawing inspiration from this AWS documentation snippet.
Here's my policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-bucket"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"subfolder/"
]
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}",
"arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}/*"
]
}
]
}
And my code to retrieve the file of a certain user with user_id = test#test.com, but actually allows me to retrieve a restricted file:
import boto
# These keys are *not* hardcoded, I'm just leaving out
# the auth flow to get them from Cognito/STS as described
# here: https://mobile.awsblog.com/post/Tx2FL1QAPDE0UAH/Understanding-Amazon-Cognito-Authentication-Part-2-Developer-Authenticated-Ident
conn = boto.s3.connect_to_region('us-east-1',
aws_access_key_id=ACCESS_KEY_FROM_COGNITO,
aws_secret_access_key=SECRET_KEY_FROM_COGNITO,
security_token=SECURITY_KEY_FROM_COGNITO)
# get the bucket
b = conn.get_bucket('my-bucket', validate=False)
# try to get an object we SHOULD be able to get
k = Key(b)
k.key = 'subfolder/us-east-1:xxxx-xxxx-xxxx-xxxxxxxxx/foobar'
print "Contents:", k.get_contents_as_string() # success!
# try to get and object we SHOUDN'T be able to get
k2 = Key(b)
k2.key = 'subfolder/BLAH_BLAH/restricted'
print "Contents:", k2.get_contents_as_string() # should fail, but doesn't
Unfortunately, I can access and read the contents of both files, yet I'm following the exact same pattern in the AWS blog documentation post. I'm also unsure why I need the validate=False in the boto connection, but it seems to work well enough.
What am I missing?
EDIT: In response to the answer below, I've tried updating my role to the following, but it does not make a difference:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-bucket"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"subfolder/${cognito-identity.amazonaws.com:sub}/*"
]
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}/*"
]
}
]
}
I've also confirmed that the access credentials I'm using are from Cognito by using the access/secret/security token triple retrieved from STS using Cognito token to create a boto IAMConnection object and querying for my role name corresponding to the auth'd cognito users for my identity pool. In doing so, I got the following exception when trying to read this role (which is exactly what should happen since I did not grant access):
BotoServerError: BotoServerError: 403 Forbidden
<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
<Error>
<Type>Sender</Type>
<Code>AccessDenied</Code>
<Message>User: arn:aws:sts::MY_AWS_ACCT_ID:assumed-role/my_policy_role_name/session_name_here is not authorized to perform: iam:GetRole on resource: role awsNameFor_Role_Given_123012313</Message>
</Error>
<RequestId>xxx-xxxx-xxxx-xxxx-xxxxx</RequestId>
</ErrorResponse>
So still no clarity on why this isn't working.
5 things:
Make sure you are using credentials issued by the Amazon Cognito Identity service otherwise ${cognito-identity.amazonaws.com:sub} will be empty and grant you access to everything
Make sure the Amazon Cognito Identity credentials you are using were issued after you updated the policy, the policy is embedded in the session portion of the credentials so if you are using old credentials, they may not have the current policy attached.
You cannot use the username of the user, you must use the Amazon Cognito Identity id. So instead of test#test.com it will be the identity id: us-east-1:beef-beef-beef-xxxxxxxxx
Your pool has 2 roles associated with it, an unauthenticated role and an authenticated role. Make sure you are setting your policy on the correct role, in this case it looks like you are using developer authenticated identities and should be modifying the policy on the authenticated role.
Check your S3 bucket policy, if you allow anonymous access to your bucket, the Cognito role policy will not override it. Turn off anonymous access if that is the case. http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-2
It looks like you're code is using a hard-coded access key and secret key and is NOT using Cognito to retrieve credentials. Instead of embedding the same access key and secret key for all users, to leverage Cognito you'll need to follow the Authentication Flow and use GetId (boto reference, AWS reference) to get an identity ID and then GetCredentialsForIdentity (boto reference AWS reference) to get the Cognito-issued credentials for the current ID. Then use those credentials with the boto S3 connection.
Also be sure to cache the ID for each user and reuse it when making additional calls to Cognito.
The answer was rather silly. Apparently buckets themselves in S3 have their own policies (they are rather tucked away), but I had the directive:
Principal: "*"
which caused the bucket to be world readable.
The second revelation was that if you restrict a bucket with the s3:ListBucket with a Condition, that doesn't mean if you list the bucket you'll get only those results - you must call it by name. As an example in boto:
wrong = bucket.list() # will simply 403
right = bucket.list(prefix="base/subdir/<cognito-id>/") # will succeed
In other words S3 is designed such that you must know the prefix-key of the desired folder, which is good practice anyway.
I have to say, I was quite impressed with how helpful the folks at AWS were in diagnosing this issue here and on their forum. Anyway, a much better understanding of S3 now.