Develop Reference

Accessing RDS Instance subnet for Glue Connection using AWS CDK Python

Using CDK in Python, I've created a VPC with Public and Private subnets in two availability zones and deployed an RDS DB Instance into the Private subnets.
How can I retrieve the Subnet ID that the RDS Instance is in so that I don't have to hardcode it into this glue connection? I've tried several different iterations of CfnOutputs and CDK constructs, but am not able to get the one that I need. Thanks for taking a look!
glue_connection = aws_glue.CfnConnection(
self,
connection_id,
catalog_id = self.account_name,
connection_input = aws_glue.CfnConnection.ConnectionInputProperty(
name = str(self.glue_connection_name),
connection_type = 'JDBC',
physical_connection_requirements = aws_glue.CfnConnection.PhysicalConnectionRequirementsProperty(
subnet_id = 'PrivateSubnet2',
security_group_id_list = [self.rds_SG.security_group_id, self.ec2_SG.security_group_id],
availability_zone = 'us-east-1b',
),
connection_properties = {
'JDBC_CONNECTION_URL': f'jdbc:mysql://{self.DBInstance.db_instance_endpoint_address}:{self.DBInstance.db_instance_endpoint_port}/test',
'PASSWORD': self.DBInstance.secret.secret_value_from_json("password").to_string(),
'USERNAME': self.db_username,
'JDBC_ENFORCE_SSL': 'false',
},
),
)

Use the L2 Connection construct - it's simpler - and get the first subnet from the list of subnets that the instance is in:
glue_connection = aws_glue.Connection(
self,
"my_connection",
type=aws_glue.ConnectionType.JDBC,
properties={
'JDBC_CONNECTION_URL': f'jdbc:mysql://{self.DBInstance.db_instance_endpoint_address}:{self.DBInstance.db_instance_endpoint_port}/test',
'PASSWORD': self.DBInstance.secret.secret_value_from_json("password").to_string(),
'USERNAME': self.db_username,
'JDBC_ENFORCE_SSL': 'false',
},
security_groups=[my_rds_instance.connections.security_groups],
subnet=my_vpc.private_subnets[0]
)

Custom wait condition in boto3 for efs

boto3 provides default waiters for some services like EC2, S3, etc. This is not provided by default for all services. Now, I've a case where an EFS volume is created and the lifecycle policy is added to the file system. The EFS creation takes some time and the lifecycle policy isn't in the required efs state. i.e., efs created.
How to wait for EFS to be created in a python boto3 code, so that policies can be added?

Looks like a waiter doesn't exist for this, but you can create your own customer waiters like this:
import boto3
from botocore.waiter import WaiterModel
from botocore.waiter import create_waiter_with_client
client = boto3.client('efs')
waiter_name = "LifeCycleCompleted"
waiter_config = {
"version": 2,
"waiters": {
"LifeCycleCompleted": {
"operation": "DescribeLifecycleConfiguration",
"delay": 60, # Number of seconds to delay
"maxAttempts": 5, # Max attempts before failure
"acceptors": [
{
"matcher": "path",
"expected": lifecycle_policy,
"argument": "length(LifecyclePolicies[]) > `0`",
"state": "success"
}
]
}
}
}
waiter_model = WaiterModel(waiter_config)
efs_lifecycle_waiter = create_waiter_with_client(waiter_name, waiter_model, client)
efs_lifecycle_waiter.wait(FileSystemId='MyFileSystemId')

Or Use something simpler as mentioned by #jordanm
import time
import boto3
efs_client = boto3.client('efs')
filesystem_status=""
while filesystem_status != "available":
response = efs_client.describe_file_systems(
FileSystemId=file_system_id,
)
filesystem_status=response["FileSystems"][0]["LifeCycleState"]
print(f"{file_system_id} {filesystem_status} .... waiting")
time.sleep(20)

Can I get a list of AWS EC2 instances running for each of my accounts?

I have a list of AWS accounts with account ids. I want to fetch the list of all the EC2 instances running in each of those accounts. Can I fetch that list using boto3?

First, you can access your AWS account programmatically by creating boto3 client.
import boto3
client = boto3.client(
's3',
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY,
aws_session_token=SESSION_TOKEN,
)
See: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#method-parameters
After creating boto3 client with your credentials, You can use boto3's EC2.Client.describe_instances()
response = client.describe_instances(
Filters=[
{
'Name': 'string',
'Values': [
'string',
]
},
],
InstanceIds=[
'string',
],
DryRun=True|False,
MaxResults=123,
NextToken='string'
)
Use 'instance-state-name' : 'running' in Filters, like:
{
'Name': 'instance-state-name',
'Values': [
'running',
]
}
then you can get informations of your running ec2 instances.

To access AWS Account from CLI/Programs, you need to have 'aws_access_key_id_value' and 'aws_secret_access_key_value'. Once you have these details, you can use the following code to list EC2 instances from the AWS Account you specified keys of.
Create config.properties with the following code.
aws_access_key_id_value='YOUR-ACCESS-KEY-OF-THE-AWS-ACCOUNT'
aws_secret_access_key_value='YOUR-SECRETE-KEY-OF-THE-AWS-ACCOUNT'
Create list-ec2-instances.py with the following code.
import boto3
def getVarFromFile(filename):
import imp
f = open(filename)
global data
data = imp.load_source('data', '', f)
f.close()
getVarFromFile('config.properties')
ec2client = boto3.resource(
'ec2',
aws_access_key_id=data.aws_access_key_id_value,
aws_secret_access_key=data.aws_secret_access_key_value
)
for instance in ec2client.instances.all():
print(
"Id: {0}\nPlatform: {1}\nType: {2}\nPublic IPv4: {3}\nAMI: {4}\nState: {5}\n".format(
instance.id, instance.platform, instance.instance_type, instance.public_ip_address, instance.image.id, instance.state
)
)
Execute the following command.
python list-ec2-instances.py

How to create multple cloudwatch alarms using Boto3 in a one shot

I want to create ~267 Cloudwatch alarms, manual process is so pathetic, can someone guide me to use Boto3 script so that I can set up all alarms in a one shot.
import boto3
# Create CloudWatch client
cloudwatch = boto3.client('cloudwatch')
# Create alarm
cloudwatch.put_metric_alarm(
AlarmName='Web_Server_CPU_Utilization',
ComparisonOperator='GreaterThanThreshold',
EvaluationPeriods=1,
MetricName='CPUUtilization',
Namespace='AWS/EC2',
Period=60,
Statistic='Average',
Threshold=70.0,
ActionsEnabled=False,
AlarmDescription='Alarm when server CPU exceeds 70%',
Dimensions=[
{
'Name': 'InstanceId',
'Value': 'i-xxxxxxxxxx'
},
],
Unit='Seconds'
)

Assuming you want to add a CloudWatch alarm for different EC2 instances, you can simply put the instance IDs in a list and iterate over that list to create the alarms. That'd look like:
import boto3
cloudwatch = boto3.client('cloudwatch')
ec2_instances = [
'i-xxxxxxxxx1',
'i-xxxxxxxxx2',
'i-xxxxxxxxx3'
]
for ec2_instance in ec2_instances:
cloudwatch.put_metric_alarm(
AlarmName='Web_Server_CPU_Utilization_%s' % ec2_instance,
ComparisonOperator='GreaterThanThreshold',
EvaluationPeriods=1,
MetricName='CPUUtilization',
Namespace='AWS/EC2',
Period=60,
Statistic='Average',
Threshold=70.0,
ActionsEnabled=False,
AlarmDescription='Alarm when server CPU exceeds 70%',
Dimensions=[
{
'Name': 'InstanceId',
'Value': ec2_instance
},
],
Unit='Seconds'
)

Here is a simple script I use to set up CloudWatch alarms on my running EC2 instances. The aim is to reboot my EC2 instances if StatusCheckFailed_Instance is True.
In case you are getting the "Insufficient Data" message as well, its worthwhile creating the same alarm on the EC2 console and then making sure your put_metric_alarm call matches the source/CloudFormation JSON.
AWS seems to be really fussy about the JSON. Once I matched the EC2 console's JSON exactly it worked like a charm.
Hope this helps someone.
import boto3
# Specify your region here
region = "ap-northeast-1"
ec2_client = boto3.client("ec2", region_name=region)
cloudwatch = boto3.client('cloudwatch')
# Get running EC2 instances
reservations = ec2_client.describe_instances(Filters=[
{
"Name": "instance-state-name",
"Values": ["running"],
}
]).get("Reservations")
# Set up an alarm for each instance
for reservation in reservations:
for instance in reservation["Instances"]:
instance_id = instance['InstanceId']
cloudwatch.put_metric_alarm(
AlarmName=f'Status_Check_{instance_id}',
AlarmDescription=f'Alarm when status check fails on {instance_id}',
ActionsEnabled=True,
OKActions=[],
AlarmActions=[
f"arn:aws:automate:{region}:ec2:reboot"
],
InsufficientDataActions=[],
MetricName='StatusCheckFailed_Instance',
Namespace='AWS/EC2',
Statistic='Maximum',
Dimensions=[
{
'Name': 'InstanceId',
'Value': instance_id
},
],
Period=60,
EvaluationPeriods=2,
DatapointsToAlarm=2,
Threshold=0.99,
ComparisonOperator='GreaterThanOrEqualToThreshold'
)

Loading JSON file in BigQuery using Google BigQuery Client API

Is there a way to load a JSON file from local file system to BigQuery using Google BigQuery Client API?
All the options I found are:
1- Streaming the records one by one.
2- Loading JSON data from GCS.
3- Using raw POST requests to load the JSON (i.e. not through Google Client API).

I'm assuming from the python tag that you want to do this from python. There is a load example here that loads data from a local file (it uses CSV, but it is easy to adapt it to JSON... there is another json example in the same directory).
The basic flow is:
# Load configuration with the destination specified.
load_config = {
'destinationTable': {
'projectId': PROJECT_ID,
'datasetId': DATASET_ID,
'tableId': TABLE_ID
}
}
load_config['schema'] = {
'fields': [
{'name':'string_f', 'type':'STRING'},
{'name':'boolean_f', 'type':'BOOLEAN'},
{'name':'integer_f', 'type':'INTEGER'},
{'name':'float_f', 'type':'FLOAT'},
{'name':'timestamp_f', 'type':'TIMESTAMP'}
]
}
load_config['sourceFormat'] = 'NEWLINE_DELIMITED_JSON'
# This tells it to perform a resumable upload of a local file
# called 'foo.json'
upload = MediaFileUpload('foo.json',
mimetype='application/octet-stream',
# This enables resumable uploads.
resumable=True)
start = time.time()
job_id = 'job_%d' % start
# Create the job.
result = jobs.insert(
projectId=project_id,
body={
'jobReference': {
'jobId': job_id
},
'configuration': {
'load': load
}
},
media_body=upload).execute()
# Then you'd also want to wait for the result and check the status. (check out
# the example at the link for more info).