boto3 provides default waiters for some services like EC2, S3, etc. This is not provided by default for all services. Now, I've a case where an EFS volume is created and the lifecycle policy is added to the file system. The EFS creation takes some time and the lifecycle policy isn't in the required efs state. i.e., efs created.
How to wait for EFS to be created in a python boto3 code, so that policies can be added?
Looks like a waiter doesn't exist for this, but you can create your own customer waiters like this:
import boto3
from botocore.waiter import WaiterModel
from botocore.waiter import create_waiter_with_client
client = boto3.client('efs')
waiter_name = "LifeCycleCompleted"
waiter_config = {
"version": 2,
"waiters": {
"LifeCycleCompleted": {
"operation": "DescribeLifecycleConfiguration",
"delay": 60, # Number of seconds to delay
"maxAttempts": 5, # Max attempts before failure
"acceptors": [
{
"matcher": "path",
"expected": lifecycle_policy,
"argument": "length(LifecyclePolicies[]) > `0`",
"state": "success"
}
]
}
}
}
waiter_model = WaiterModel(waiter_config)
efs_lifecycle_waiter = create_waiter_with_client(waiter_name, waiter_model, client)
efs_lifecycle_waiter.wait(FileSystemId='MyFileSystemId')
Or Use something simpler as mentioned by #jordanm
import time
import boto3
efs_client = boto3.client('efs')
filesystem_status=""
while filesystem_status != "available":
response = efs_client.describe_file_systems(
FileSystemId=file_system_id,
)
filesystem_status=response["FileSystems"][0]["LifeCycleState"]
print(f"{file_system_id} {filesystem_status} .... waiting")
time.sleep(20)
Related
TLDR: azurerm_function_app_function will work fine on Terraform Apply, but disappears from Azure Portal afterwards.
I am trying to deploy an Azure Function via Terraform for months now and have not had any luck with it.
The Terraform apply will run fine. I will then go into the Azure Portal and look at the function app functions and this function will be there. However when I refresh the blade the function will disappear. I have made the same function and deployed it via VS Code no issues, but with Terraform there is no luck.
resource "azurerm_linux_function_app" "main" {
name = "tf-linux-app"
location = azurerm_resource_group.main.location
resource_group_name = azurerm_resource_group.main.name
service_plan_id = azurerm_service_plan.main.id
storage_account_name = azurerm_storage_account.main.name
storage_account_access_key = azurerm_storage_account.main.primary_access_key
site_config {
app_scale_limit = 200
elastic_instance_minimum = 0
application_stack {
python_version = "3.9"
}
}
app_settings = {
"${azurerm_storage_account.main.name}_STORAGE" = azurerm_storage_account.main.primary_connection_string
}
client_certificate_mode = "Required"
identity {
type = "SystemAssigned"
}
}
resource "azurerm_function_app_function" "main" {
name = "tf-BlobTrigger"
function_app_id = azurerm_linux_function_app.main.id
language = "Python"
file {
name = "__init__.py"
content = file("__init__.py")
}
test_data = "${azurerm_storage_container.container1.name}/{name}"
config_json = jsonencode({
"scriptFile" : "__init__.py",
"disabled": false,
"bindings" : [
{
"name" : "myblob",
"type" : "blobTrigger",
"direction" : "in",
"path" : "${azurerm_storage_container.container1.name}/{name}",
"connection" : "${azurerm_storage_container.container1.name}_STORAGE"
}
]
})
}
As far as the Python script, I'm literally just trying
the demo found here
that Azure provides.
__init__.py:
import logging
import azure.functions as func
def main(myblob: func.InputStream):
logging.info('Python Blob trigger function processed %s', myblob.name)
I tried running Terraform apply, I expected the function app to appear and stay there, but it appears and then disappears. I also tried deploying a C# function to a Windows app. This worked as expected, but I now need the script in Python.
I am working on a boto script to collect logs and store it into an S3 bucket, I am getting an error "botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the PutBucketPolicy operation: The specified bucket does not exist" I am craeating the bucket beforehand and then trying to attach the policy to the bucket,the bucket shows up in the console as well. I don't understand what might be causing this problem.
import boto3
import sys
import json
import time
iam = boto3.client('iam')
sts = boto3.client('sts')
ec2 = boto3.resource('ec2')
cloudtrail = boto3.client('cloudtrail')
s3 = boto3.client('s3')
s3.create_bucket(Bucket='goodbucket3')
# Create a bucket policy
bucket_name = 'goodbucket3'
bucket_policy = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {"Service": "cloudtrail.amazonaws.com"},
"Action": "s3:GetBucketAcl",
"Resource": f"arn:aws:s3:::{bucket_name}"
},
{
"Effect": "Allow",
"Principal": {"Service": "cloudtrail.amazonaws.com"},
"Action":
"s3:PutObject",
"s3:PutObjectAcl"
"s3:GetObject"
"s3:GetObjectAcl"
"s3:DeleteObject"
"Resource": f"arn:aws:s3:::{bucket_name}/AWSLogs/XXXXXXXX/*",
"Condition": {"StringEquals": {"s3:x-amz-acl": "bucket-owner-full-control"}}
}
]
}
# Convert the policy from JSON dict to string
bucket_policy = json.dumps(bucket_policy)
# Set the new policy
s3.put_bucket_policy(Bucket='bucket_name', Policy=bucket_policy)
result = s3.get_bucket_policy(Bucket='bucket_name')
logs = cloudtrail.create_trail(
Name='GoodTrail',
S3BucketName='bucket_name',
)
print(logs)
You may need to wait for the bucket creation to full propagate. You can use a waiter to do this. See the low-level clients documentation or How to use waiters in boto3.
Using a client's get_waiter() method, you can obtain a specific waiter
from its list of possible waiters:
# Retrieve waiter instance that will wait till a specified bucket exists
s3_bucket_exists_waiter = s3.get_waiter('bucket_exists')
Then to actually start waiting, you must call the waiter's wait()
method with the method's appropriate parameters passed in:
# Begin waiting for the S3 bucket, mybucket, to exist
s3_bucket_exists_waiter.wait(Bucket='mybucket')
I want to create ~267 Cloudwatch alarms, manual process is so pathetic, can someone guide me to use Boto3 script so that I can set up all alarms in a one shot.
import boto3
# Create CloudWatch client
cloudwatch = boto3.client('cloudwatch')
# Create alarm
cloudwatch.put_metric_alarm(
AlarmName='Web_Server_CPU_Utilization',
ComparisonOperator='GreaterThanThreshold',
EvaluationPeriods=1,
MetricName='CPUUtilization',
Namespace='AWS/EC2',
Period=60,
Statistic='Average',
Threshold=70.0,
ActionsEnabled=False,
AlarmDescription='Alarm when server CPU exceeds 70%',
Dimensions=[
{
'Name': 'InstanceId',
'Value': 'i-xxxxxxxxxx'
},
],
Unit='Seconds'
)
Assuming you want to add a CloudWatch alarm for different EC2 instances, you can simply put the instance IDs in a list and iterate over that list to create the alarms. That'd look like:
import boto3
cloudwatch = boto3.client('cloudwatch')
ec2_instances = [
'i-xxxxxxxxx1',
'i-xxxxxxxxx2',
'i-xxxxxxxxx3'
]
for ec2_instance in ec2_instances:
cloudwatch.put_metric_alarm(
AlarmName='Web_Server_CPU_Utilization_%s' % ec2_instance,
ComparisonOperator='GreaterThanThreshold',
EvaluationPeriods=1,
MetricName='CPUUtilization',
Namespace='AWS/EC2',
Period=60,
Statistic='Average',
Threshold=70.0,
ActionsEnabled=False,
AlarmDescription='Alarm when server CPU exceeds 70%',
Dimensions=[
{
'Name': 'InstanceId',
'Value': ec2_instance
},
],
Unit='Seconds'
)
Here is a simple script I use to set up CloudWatch alarms on my running EC2 instances. The aim is to reboot my EC2 instances if StatusCheckFailed_Instance is True.
In case you are getting the "Insufficient Data" message as well, its worthwhile creating the same alarm on the EC2 console and then making sure your put_metric_alarm call matches the source/CloudFormation JSON.
AWS seems to be really fussy about the JSON. Once I matched the EC2 console's JSON exactly it worked like a charm.
Hope this helps someone.
import boto3
# Specify your region here
region = "ap-northeast-1"
ec2_client = boto3.client("ec2", region_name=region)
cloudwatch = boto3.client('cloudwatch')
# Get running EC2 instances
reservations = ec2_client.describe_instances(Filters=[
{
"Name": "instance-state-name",
"Values": ["running"],
}
]).get("Reservations")
# Set up an alarm for each instance
for reservation in reservations:
for instance in reservation["Instances"]:
instance_id = instance['InstanceId']
cloudwatch.put_metric_alarm(
AlarmName=f'Status_Check_{instance_id}',
AlarmDescription=f'Alarm when status check fails on {instance_id}',
ActionsEnabled=True,
OKActions=[],
AlarmActions=[
f"arn:aws:automate:{region}:ec2:reboot"
],
InsufficientDataActions=[],
MetricName='StatusCheckFailed_Instance',
Namespace='AWS/EC2',
Statistic='Maximum',
Dimensions=[
{
'Name': 'InstanceId',
'Value': instance_id
},
],
Period=60,
EvaluationPeriods=2,
DatapointsToAlarm=2,
Threshold=0.99,
ComparisonOperator='GreaterThanOrEqualToThreshold'
)
I currently have a python script that looks like:
import boto3
...
response = dynamodb.get_item(
TableName = dynamodb_table_name,
Key = {
"snippet_id": {
"S": snippet_id
}
}
)
if "Item" in response:
item = response["Item"]
print(json.dumps(item, indent=4, cls=DecimalEncoder))
This prints something akin to:
{
"var_1": {
"BOOL": false
},
"var_2": {
"S": "Text"
},
"snippet_id": {
"S": "3a97e45c-ffed-4c76-8bb4-b2a32f49a5d2"
}
}
Any idea how to do the type detection and return:
{
"var_1": False,
"var_2": "Text",
"snippet_id": "3a97e45c-ffed-4c76-8bb4-b2a32f49a5d2"
}
Also, can this be done for the query as well?
TLDR
Use resource instead of client.
Summary
In essence, you can call boto3.client() or boto3.resource().
Client returns dynamoDB syntax, which looks like this:
'var_1' : {'S':"string"}
Resource returns normal syntax, which looks like this:
'var_1' : "string"
Further Reading
At its core, all that Boto3 does is call AWS APIs on your behalf. For the majority of the AWS services, Boto3 offers two distinct ways of accessing these abstracted APIs:
Client: low-level service access
Resource: higher-level object-oriented service access
Reference: https://realpython.com/lessons/clients-and-resources/
I'm trying to restrict all my AWS Cognito users to their own subdirectory in my S3 bucket.
I don't want them listing, reading, or writing other people's subdirectories/files in my larger bucket, and I only want them to read & write objects in their own directory.
I'm drawing inspiration from this AWS documentation snippet.
Here's my policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-bucket"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"subfolder/"
]
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}",
"arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}/*"
]
}
]
}
And my code to retrieve the file of a certain user with user_id = test#test.com, but actually allows me to retrieve a restricted file:
import boto
# These keys are *not* hardcoded, I'm just leaving out
# the auth flow to get them from Cognito/STS as described
# here: https://mobile.awsblog.com/post/Tx2FL1QAPDE0UAH/Understanding-Amazon-Cognito-Authentication-Part-2-Developer-Authenticated-Ident
conn = boto.s3.connect_to_region('us-east-1',
aws_access_key_id=ACCESS_KEY_FROM_COGNITO,
aws_secret_access_key=SECRET_KEY_FROM_COGNITO,
security_token=SECURITY_KEY_FROM_COGNITO)
# get the bucket
b = conn.get_bucket('my-bucket', validate=False)
# try to get an object we SHOULD be able to get
k = Key(b)
k.key = 'subfolder/us-east-1:xxxx-xxxx-xxxx-xxxxxxxxx/foobar'
print "Contents:", k.get_contents_as_string() # success!
# try to get and object we SHOUDN'T be able to get
k2 = Key(b)
k2.key = 'subfolder/BLAH_BLAH/restricted'
print "Contents:", k2.get_contents_as_string() # should fail, but doesn't
Unfortunately, I can access and read the contents of both files, yet I'm following the exact same pattern in the AWS blog documentation post. I'm also unsure why I need the validate=False in the boto connection, but it seems to work well enough.
What am I missing?
EDIT: In response to the answer below, I've tried updating my role to the following, but it does not make a difference:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-bucket"
],
"Condition": {
"StringLike": {
"s3:prefix": [
"subfolder/${cognito-identity.amazonaws.com:sub}/*"
]
}
}
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::my-bucket/subfolder/${cognito-identity.amazonaws.com:sub}/*"
]
}
]
}
I've also confirmed that the access credentials I'm using are from Cognito by using the access/secret/security token triple retrieved from STS using Cognito token to create a boto IAMConnection object and querying for my role name corresponding to the auth'd cognito users for my identity pool. In doing so, I got the following exception when trying to read this role (which is exactly what should happen since I did not grant access):
BotoServerError: BotoServerError: 403 Forbidden
<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
<Error>
<Type>Sender</Type>
<Code>AccessDenied</Code>
<Message>User: arn:aws:sts::MY_AWS_ACCT_ID:assumed-role/my_policy_role_name/session_name_here is not authorized to perform: iam:GetRole on resource: role awsNameFor_Role_Given_123012313</Message>
</Error>
<RequestId>xxx-xxxx-xxxx-xxxx-xxxxx</RequestId>
</ErrorResponse>
So still no clarity on why this isn't working.
5 things:
Make sure you are using credentials issued by the Amazon Cognito Identity service otherwise ${cognito-identity.amazonaws.com:sub} will be empty and grant you access to everything
Make sure the Amazon Cognito Identity credentials you are using were issued after you updated the policy, the policy is embedded in the session portion of the credentials so if you are using old credentials, they may not have the current policy attached.
You cannot use the username of the user, you must use the Amazon Cognito Identity id. So instead of test#test.com it will be the identity id: us-east-1:beef-beef-beef-xxxxxxxxx
Your pool has 2 roles associated with it, an unauthenticated role and an authenticated role. Make sure you are setting your policy on the correct role, in this case it looks like you are using developer authenticated identities and should be modifying the policy on the authenticated role.
Check your S3 bucket policy, if you allow anonymous access to your bucket, the Cognito role policy will not override it. Turn off anonymous access if that is the case. http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-2
It looks like you're code is using a hard-coded access key and secret key and is NOT using Cognito to retrieve credentials. Instead of embedding the same access key and secret key for all users, to leverage Cognito you'll need to follow the Authentication Flow and use GetId (boto reference, AWS reference) to get an identity ID and then GetCredentialsForIdentity (boto reference AWS reference) to get the Cognito-issued credentials for the current ID. Then use those credentials with the boto S3 connection.
Also be sure to cache the ID for each user and reuse it when making additional calls to Cognito.
The answer was rather silly. Apparently buckets themselves in S3 have their own policies (they are rather tucked away), but I had the directive:
Principal: "*"
which caused the bucket to be world readable.
The second revelation was that if you restrict a bucket with the s3:ListBucket with a Condition, that doesn't mean if you list the bucket you'll get only those results - you must call it by name. As an example in boto:
wrong = bucket.list() # will simply 403
right = bucket.list(prefix="base/subdir/<cognito-id>/") # will succeed
In other words S3 is designed such that you must know the prefix-key of the desired folder, which is good practice anyway.
I have to say, I was quite impressed with how helpful the folks at AWS were in diagnosing this issue here and on their forum. Anyway, a much better understanding of S3 now.