I have an AWS Lambda function which is triggered by an S3 bucket. The Lambda function pings an EC2 instance inside a VPC. The VPC does not have a NAT. For some reason the Lambda is not able to connect to the EC2 and it is timing out. I have tried keeping the lambda both in the VPC and outside the VPC. I have also tried using the public and private IP of the instance but not seems to work. Any suggestions on what I could do next.
The The code for the lambda function is as follows
from __future__ import print_function
import json
import boto3
import urllib2
print('Loading function')
s3 = boto3.client('s3')
def lambda_handler(event, context):
bucket = event['Records'][0]['s3']['bucket']['name']
url = urllib2.urlopen("http://ip-address/API/")
First, with the Lambda function inside the VPC, make sure you use the private IP of the EC2 instance. Then open port 80 in the EC2 instance's security group to allow incoming connections from anything belonging to the security group you assigned to the Lambda function.
Related
I am trying to use the list_objects_v2 function of the Python3 Boto3 S3 API client to list objects from an S3 access point.
Sample Code:
import boto3
import botocore
access_point_arn = "arn:aws:s3:region:account-id:accesspoint/resource"
client = boto3.client('s3')
response = client.list_objects_v2(Bucket=access_point_arn)
Somehow getting the error below:
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "arn:aws:s3:region:account-id:accesspoint/resource": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$"
Based on the documentation: https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html, i should be able to pass an access point to the list_objects_v2 function as the Bucket name. The odd thing is, this function works locally on my Windows 10 laptop. The same Python3.6 code with the same Boto3 and Botocore package versions throws this error in AWS Glue Python Shell job. I also made sure the Glue role has S3 Full Access and Glue Service policies attached.
I would appreciate if someone can shed some lights on this.
I have aws lambda which is not able to connect with mongo through VPC.
import pymongo
def handler(event, context):
try:
client = pymongo.MongoClient(host="xxxxxxx", port=27017, username=x1, password=x2, authsource="x3, authMechanism='SCRAM-SHA-1')
except pymongo.errors.ServerSelectionTimeoutError as err:
print(err)
Not able to found the server.
I have created a security group and new roles have given VPC and lambda full access too but not able to connect.
Taken help from https://blog.shikisoft.com/access-mongodb-instance-from-aws-lambda-python/ as well as https://blog.shikisoft.com/running-aws-lambda-in-vpc-accessing-rds/ links.
Please be helpful.
Trying since yesterday but no luck.
Let me try to help you figured out where the problem is.
1. Are your MongoDB EC2 Instance and your Lambda hosted on the same VPC?
If this the cause of your problem, you should move your services into the same VPC.
2. Is your Security Group that attached to your MongoDB EC2 Instance
and your Lambda has whitelisted/include the default sg?
You have to include the default sg into your Security Group so, services/instances within that VPC can communicate.
3. Is your hostname publicly or privately accessed ?
If Lambda needs to connect over Internet to access your MongoDB instance, you don't need to attach your Lambda into a VPC.
Inside a VPC, Lambda requires a NAT Gateway to communicate to open
world. Try to communicate privately if your MongoDB instance and
Lambda are in the same VPC.
https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/
Hope these answers are helpful to you.
I have a python app running in a Docker container on a EC2 instance managed by ECS (well, that's what I would like...). However, to use services like SSM with boto3, I need to know the region where the instance is running. I dont need any credentials as I use a role for the instance which grants access to the service, so a default Session is ok.
I know that it is possible to fetch the region with a curl to get the dynamic metadata, but is there any more elegant way to instantiate a client with a region name (of credentials) inside an EC2 instance ?
I ran through the boto3 documentation and found
Note that if you've launched an EC2 instance with an IAM role configured, there's no explicit configuration you need to set in boto3 to use these credentials. Boto3 will automatically use IAM role credentials if it does not find credentials in any of the other places listed above.
So why do I need to pass the region name for SSM client for example ? Is there a workaround ?
Region is a required parameter for the SSM client to know which region it should be interacting with. It does not try to assume even if you’re in the AWS cloud.
If you want it to assume in your container the simplest way in which to implement is to use the AWS environment variables.
In your container definition specify the environment attribute specify a variable with name AWS_DEFAULT_REGION and the value of your current region.
By doing this you will not have to specify a region in the SDK within the container.
This example uses the environment attribute for more information.
Here is how to retrieve a parameter from the Parameter Store using the instance profile credentials:
#!/usr/bin/env python3
from ec2_metadata import ec2_metadata
import boto3
session = boto3.Session(region_name=ec2_metadata.region)
ssm = session.client('ssm')
parameter = ssm.get_parameter(Name='/path/to/a/parameter', WithDecryption=True)
print(parameter['Parameter']['Value'])
Replace the client section with the service of your choice and you should be set.
I have S3 bucket named 'files'. Every day new file arrives there. Example:
/files/data-01-23-2017--11-33am.txt
/files/data-01-24-2017--10-28am.txt
How would I make a Lambda function and set a trigger to execute one shell script on EC2 when new file arrives?
Example of new file is:
/files/data-01-25-2017--11-43am.txt
Command that I would want to execute on EC2 is (with parameter as new file name that just arrived):
python /home/ec2-user/jobs/run_job.py data-01-25-2017--11-43am.txt
Amazon S3 can be configured to trigger an AWS Lambda function when a new object is created. However, Lambda functions do not have access to your Amazon EC2 instances. It is not an appropriate architecture to use.
Some alternative options (these are separate options, not multiple steps):
Instead of running a command on an Amazon EC2 instance, put your code in the Lambda function (no EC2 instance required). (Best option!)
Configure Amazon S3 to push a message into an Amazon SQS queue. Have your code on the EC2 instance regularly poll the queue. When it receives a message, process the object in S3.
Configure Amazon S3 to send a message to an Amazon SNS topic. Subscribe an end-point of your application (effectively an API) to the SNS queue, so that it receives a message when a new object has been created.
I have a lambda function which SSH ec2 instance and run some commands. This lambda function is triggered from SNS topic. SNS topic is integrated with a cloudwatch alarm. I am using python 2.7 in lambda function followed this thread https://aws.amazon.com/blogs/compute/scheduling-ssh-jobs-using-aws-lambda/. is it possible to get EC2 public IP address which actually triggered alarm?
It depends on the CloudWatch Alarm you are using to trigger the SNS publish.
My suggestion is to print out the entire event dictionary in your function and check if there is any mention about EC2 instance ID.
In case of CloudWatch EC2 alarm (eg. CPU usage) you'll find the instance ID in the metric Dimension.
# Python example
import json
message = json.loads(event['Records'][0]['Sns']['Message'])
instance_id = message['Trigger']['Dimensions'][0]
If you have the instance ID you can easily retrieve the instance public IP using boto3 as follows:
# Python example
import boto3
instance_id = 'xxxx' # This is the instance ID from the event
ec2 = boto3.client('ec2')
instances = ec2.describe_instances(InstanceIds=[instance_id])
public_ip = instances[0]['Reservations'][0]['Instances'][0]['PublicIpAddress']
Finally, as you are performing SSH from Lambda function to your EC2 instance keep in mind that Lambda Functions out of VPC get a dynamic public IP therefore it is impossible to restrict your EC2 instance security group for SSH. Leaving SSH opened to the entire world is not a good practice from a security perspective.
I suggest to run both EC2 and Lambda function in VPC restricting SSH access to your EC2 instances from Lambda vpc security group only. In that case you'll need to retrieve the private Ip address rather than the public one to be able to ssh your instance (the python logic is the same as above, the only difference is that you use 'PrivateIpAddress' instead of 'PublicIpAddress') . This is way more secure than using public internet.
I hope this helps.
G