Develop Reference

AWS Lambda : CodePipeline job never ends, even after retuning sucess

I am working on executing an AWS Lambda code from Code-pipeline. I have given the lambda role full access to EC2 and code-deploy. The commands generally work when I am not triggering them from code-pipeline. When Triggered from code-pipeline, they just keep on running, even though success is sent. What am i doing wrong?
Code :
import boto3
import json
def lambda_handler(event, context):
reservations = boto3.client('ec2').describe_instances()['Reservations']
instances_list = []
process_instance_list = []
command = 'COMMAND TO EXECUTE ON SERVER'
ssm = boto3.client('ssm')
for res in reservations:
instances = res['Instances']
for inst in res['Instances']:
for tag in inst['Tags']:
#print("Tag value is {}".format(tag['Value']))
if tag['Value']=='Ubuntu_Magento':
print("{} {} {}".format(tag['Value'], inst['InstanceId'], inst['LaunchTime']))
instances_list.append(inst)
instances_list.sort(key=lambda x: x['LaunchTime'])
instance_id = instances_list[0]['InstanceId']
ssmresponse = ssm.send_command(InstanceIds=[instance_id], DocumentName='AWS-RunShellScript', Parameters= { 'commands': [command]})
code_pipeline = boto3.client('codepipeline')
job_id = event['CodePipeline.job']['id']
code_pipeline.put_job_success_result(jobId=job_Id)

Any lambda by default has lifespan of 15 mins only, after that it exits no matter what. I think it has something to do with the way you are trigger it.

How to retrieve Public DNS from a created AWS instance by a BOTO3 program?

Have the same problem as in Retrieving public dns of EC2 instance with BOTO3
First I create and run an instance:
r = client.run_instances(
ImageId=aws_config['base_ami'],
MinCount=1,
MaxCount=1,
KeyName=aws_config['key_name'],
InstanceType="t2.small",
NetworkInterfaces=[
{
'DeviceIndex': 0,
'SubnetId' : aws_config['subnet'],
'Groups': aws_config['security_groups'],
'AssociatePublicIpAddress': True
}]
)
instance = r["Instances"][0]
The answer suggests using instance.load() to upload to the instance the new public DNS IP. By the way, I can see the public IP from the AWS console
In my program I have the AttributeError after instance.load()
instance.load()
AttributeError: 'dict' object has no attribute 'load

The problem is that client.run_instances does not return the list of EC2 instances, it returns the list of instance dictionaries.
To convert return to the EC2 instance I did the following after the above code:
ec2 = boto3.resource('ec2')
inst = ec2.Instance(instance['InstanceId'])
Then inst.load() works and uploads the public DNS IP

Error in retrieving public dns name of ec2 instance

I am trying to retrieve public dns name of an ec2 instance.
Here is my python3 script.
import sys
import boto3
from botocore.exceptions import ClientError
instance_id = "i-03e7f6391a0f523ee"
action = 'ON'
ec2 = boto3.client('ec2')
if action == 'ON':
# Do a dryrun first to verify permissions
try:
ec2.start_instances(InstanceIds=[instance_id], DryRun=True)
except ClientError as e:
if 'DryRunOperation' not in str(e):
raise
# Dry run succeeded, run start_instances without dryrun
try:
response = ec2.start_instances(InstanceIds=[instance_id], DryRun=False)
print(response)
except ClientError as e:
print(e)
else:
# Do a dryrun first to verify permissions
try:
ec2.stop_instances(InstanceIds=[instance_id], DryRun=True)
except ClientError as e:
if 'DryRunOperation' not in str(e):
raise
# Dry run succeeded, call stop_instances without dryrun
try:
response = ec2.stop_instances(InstanceIds=[instance_id], DryRun=False)
print(response)
except ClientError as e:
print(e)
instance = ec2.Instance('i-1234567890123456')
while instance.state['Name'] not in ('running', 'stopped'):
sleep(5)
print("the instance is initializing")
#pubdns=instance.PublicDnsName
#print ("public dns name"+pubdns)
def get_name(inst):
client = boto3.client('ec2')
response = client.describe_instances(InstanceIds = [inst[0].instance_id])
foo = response['Reservations'][0]['Instances'][0]['NetworkInterfaces'][0]['Association']['PublicDnsName']
return foo
foo = get_name(instance_id)
print (foo)
If I use
ec2 = boto3.client('ec2')
in the above code, I get the following error:
AttributeError: 'EC2' object has no attribute 'Instance'
and if I use
ec2 = boto3.resource('ec2')
then I get this error instead:
AttributeError: 'ec2.ServiceResource' object has no attribute 'start_instances'
what I want to do is to be able to connect to an ec2 instance and retrieve its publicdns name.
I have changed the code now based on below suggestions
import sys
import boto3
instance_id = "i-03e7f6391a0f523ee"
action = 'ON'
ec2 = boto3.client('ec2')
#instance = ec2.resource('ec2').instance(instance_id)
if action == 'ON':
response = ec2.start_instances(InstanceIds=[instance_id], DryRun=False)
else:
response = ec2.stop_instances(InstanceIds=[instance_id], DryRun=False)
print(response)
def get_name(inst):
client = boto3.client('ec2')
response = client.describe_instances(InstanceIds = [inst[0].instance_id])
foo = response['Reservations'][0]['Instances'][0]['NetworkInterfaces'][0]['Association']['PublicDnsName']
return foo
foo = get_name(instance_id)
print (foo)
but now I get error
in get_name
response = client.describe_instances(InstanceIds = [inst[0].instance_id])
AttributeError: 'str' object has no attribute 'instance_id'

You're conflating two ideas in one.
boto3.client creates an object through which you look up resources like ec2.
Once you have a resource, you can begin to manipulate it.
Use
ec2 = boto3.client('ec2')
and then
instance = ec2.resource('ec2').instance(instance_id)
The second looks up your ec2 instance from the ec2 resource, not the boto3 ec2 client.

Here is a working code in case any one hits here in future I am posting it.This will print public DNS name of all of your instances after switching them on and then shutdown them.
import boto3
from pprint import pprint
ec2=boto3.client('ec2')
response=ec2.describe_instances()
print (response)
instancelist = []
for reservation in (response["Reservations"]):
for instance in reservation["Instances"]:
instancelist.append(instance["InstanceId"])
print (instancelist)
action ='ON'
if action == 'ON':
response = ec2.start_instances(InstanceIds=instancelist, DryRun=False)
ec2client = boto3.resource('ec2')
#response = ec2client.describe_instances()
instances = ec2client.instances.filter(Filters=[{'Name': 'instance-state-name', 'Values': ['running','stopped']}])
ids = []
for instance in instances:
print(instance.id, instance.instance_type)
ids.append(instance.id)
resp=ec2.describe_network_interfaces();
print ("printing pub dns name")
print(resp['NetworkInterfaces'][0]['Association']['PublicDnsName'])
ec2client.instances.filter(InstanceIds=ids).stop()

boto3 check if Athena database exists

Im making a script that creates a database in AWS Athena and then creates tables for that database, today the DB creation was taking ages, so the tables being created referred to a db that doesn't exists, is there a way to check if a DB is already created in Athena using boto3?
This is the part that created the db:
client = boto3.client('athena')
client.start_query_execution(
QueryString='create database {}'.format('db_name'),
ResultConfiguration=config
)

# -*- coding: utf-8 -*-
import logging
import os
from time import sleep
import boto3
import pandas as pd
from backports.tempfile import TemporaryDirectory
logger = logging.getLogger(__name__)
class AthenaQueryFailed(Exception):
pass
class Athena(object):
S3_TEMP_BUCKET = "please-replace-with-your-bucket"
def __init__(self, bucket=S3_TEMP_BUCKET):
self.bucket = bucket
self.client = boto3.Session().client("athena")
def execute_query_in_athena(self, query, output_s3_directory, database="csv_dumps"):
""" Useful when client executes a query in Athena and want result in the given `s3_directory`
:param query: Query to be executed in Athena
:param output_s3_directory: s3 path in which client want results to be stored
:return: s3 path
"""
response = self.client.start_query_execution(
QueryString=query,
QueryExecutionContext={"Database": database},
ResultConfiguration={"OutputLocation": output_s3_directory},
)
query_execution_id = response["QueryExecutionId"]
filename = "{filename}.csv".format(filename=response["QueryExecutionId"])
s3_result_path = os.path.join(output_s3_directory, filename)
logger.info(
"Query query_execution_id <<{query_execution_id}>>, result_s3path <<{s3path}>>".format(
query_execution_id=query_execution_id, s3path=s3_result_path
)
)
self.wait_for_query_to_complete(query_execution_id)
return s3_result_path
def wait_for_query_to_complete(self, query_execution_id):
is_query_running = True
backoff_time = 10
while is_query_running:
response = self.__get_query_status_response(query_execution_id)
status = response["QueryExecution"]["Status"][
"State"
] # possible responses: QUEUED | RUNNING | SUCCEEDED | FAILED | CANCELLED
if status == "SUCCEEDED":
is_query_running = False
elif status in ["CANCELED", "FAILED"]:
raise AthenaQueryFailed(status)
elif status in ["QUEUED", "RUNNING"]:
logger.info("Backing off for {} seconds.".format(backoff_time))
sleep(backoff_time)
else:
raise AthenaQueryFailed(status)
def __get_query_status_response(self, query_execution_id):
response = self.client.get_query_execution(QueryExecutionId=query_execution_id)
return response
As pointed in above answer, Athena Waiter is still not there implemented.
I use this light weighted Athena client to do the query, it returns the s3 path of result when the query is completed.

The waiter functions for Athena are not implemented yet: Athena Waiter
See: Support AWS Athena waiter feature for a possible workaround until it is implemented in Boto3. This is how it is implemented in AWS CLI.
while True:
stats = self.athena.get_query_execution(execution_id)
status = stats['QueryExecution']['Status']['State']
if status in ['SUCCEEDED', 'FAILED', 'CANCELLED']:
break
time.sleep(0.2)

Retrieving public dns of EC2 instance with BOTO3

I'm using ipython to get an understanding of Boto3 and interacting with EC2 instances. Here is the code I'm using to create an instance:
import boto3
ec2 = boto3.resource('ec2')
client = boto3.client('ec2')
new_instance = ec2.create_instances(
ImageId='ami-d05e75b8',
MinCount=1,
MaxCount=1,
InstanceType='t2.micro',
KeyName=<name_of_my_key>,
SecurityGroups=['<security_group_name>'],
DryRun = False
)
This starts an EC2 instance fine, and I can get the public DNS name, ip and other info from the AWS console. But, when I try to get the public DNS using Boto, by doing this:
new_instance[0].public_dns_name
Returns blank quotes. Yet, other instance details, such as:
new_instance[0].instance_type
Returns the correct information.
Any ideas? Thanks.
EDIT:
So if I do:
def get_name(inst):
client = boto3.client('ec2')
response = client.describe_instances(InstanceIds = [inst[0].instance_id])
foo = response['Reservations'][0]['Instances'][0]['NetworkInterfaces'][0]['Association']['PublicDnsName']
return foo
foo = get_name(new_instance)
print foo
Then it will return the public DNS. But it doesn't make sense to me why I need to do all of this.

The Instance object you get back is only hydrated with the response attributes from the create_instances call. Since the DNS name is not available until the instance has reached the running state [1], it will not be immediately present. I imagine the time between you creating the instance and calling describe instances is long enough for the micro instance to start.
import boto3
ec2 = boto3.resource('ec2')
instances = ec2.create_instances(
ImageId='ami-f0091d91',
MinCount=1,
MaxCount=1,
InstanceType='t2.micro',
KeyName='<KEY-NAME>',
SecurityGroups=['<GROUP-NAME>'])
instance = instances[0]
# Wait for the instance to enter the running state
instance.wait_until_running()
# Reload the instance attributes
instance.load()
print(instance.public_dns_name)

Here my wrapper:
import boto3
from boto3.session import Session
def credentials():
"""Credentials:"""
session = Session(aws_access_key_id= 'XXXXXXXXX',
aws_secret_access_key= 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')
ec2 = boto3.resource('ec2', region_name='us-east-2')
return ec2
def get_public_dns(instance_id):
"""having the instance_id, gives you the public DNS"""
ec2 = credentials()
instance = ec2.Instance(instance_id)
instancePublicDNS = instance.public_dns_name
return instancePublicDNS
Then you just need to use your instance_id to get public dns of any of your actives ec2:
dns = get_public_dns(instance_id)
Remember to change "region_name" to your zone and add your "aws_access_key_id" and "aws_secret_access_key"

import boto3
import pandas as pd
session = boto3.Session(profile_name='aws_dev')
dev_ec2_client = session.client('ec2')
response = dev_ec2_client.describe_instances()
df = pd.DataFrame(columns=['InstanceId', 'InstanceType', 'PrivateIpAddress','PublicDnsName'])
i = 0
for res in response['Reservations']:
df.loc[i, 'InstanceId'] = res['Instances'][0]['InstanceId']
df.loc[i, 'InstanceType'] = res['Instances'][0]['InstanceType']
df.loc[i, 'PrivateIpAddress'] = res['Instances'][0]['PrivateIpAddress']
df.loc[i, 'PublicDnsName'] = res['Instances'][0]['PublicDnsName']
i += 1
print df
Note:
Change this profile with your AWS profile name profile_name='aws_dev'
This code is working for Python3