I am trying to create an AWS SAM app with multiple AWS Serverless functions.
The app has 1 template.yaml file which have resource of 2 different serverless lambda functions, for instance "Consumer Lambda" and "Worker Lambda". Consumer gets triggered at a rate of 5 minutes. The consumer uses boto3 library to trigger the worker lambda function. This code works when the worker lambda is deployed on AWS.
But I want to test both the functions locally with Sam local invoke "Consumer" which invokes "Worker" also locally.
Here's a screenshot of the YAML file:
I am using Pycharm to run the project. There is an option to run only 1 function at a time which then creates only one folder in the build folder.
I have to test if Consumer is able to invoke worker locally in pycharm before deployment. I think there is some way to do it but not sure how to. I did some extensive search but didn't yield anything.
Any help is appreciated. Thanks in advance
You can start the lambda invoke endpoint in the following way (official docs):
sam local start-lambda
Now you can point your AWS resource client to port 3001 and trigger the functions locally.
For eg. If you are doing this on Python, it can be acheived in the following way with boto3:
boto3
# Create a lambda client
lambda_client = boto3.client('lambda',
region_name="<localhost>",
endpoint_url="<http://127.0.0.1:3001>",
use_ssl=False,
verify=False)
# Invoke the function
lambda_client.invoke(FunctionName=<function_name>,
Payload=<lambda_payload>)
Related
I am looking for an AWS instance that will allow me to run a simple python script made with a flask that runs constantly in an infinite loop.
The instance must be created and configured via an API call from the main web page with its own database in postgres.
Example:
Client1 -> Go to the web page and create his own instance -> The web page calls the AWS API -> AWS creates an instance1 for client1 which is constantly running.
Note: I am looking for a cheaper service for it. It may be something between the EC2 instance and the Lambda function (the Labmda function doesn't work because it needs to be autonomous).
If the Python instance script fails for any reason, it should restart automatically.
Also the client1 has the option to turn the script off and on whenever he wants.
Thank you very much in advance!
Elastic Beanstalk is the equivalent of Heroku.
https://aws.amazon.com/elasticbeanstalk/
I have a lambda function build with python.
The lambda function executes a docker image which is hosted on AWS ECR.
The python code in the docker container:
pulls data from rest api
parses the data
sends an event based on the parsed data via AWS SNS
When I trigger the AWS function via test it just gets executed once.
When I execute the code locally on my machine it also just gets executed once.
But when the lambda function gets triggered by a CloudWatch event (Cloudbridge) lets say every 5 minutes.
The code gets executed multiple times.
I tried the following:
try using python sleep timer
configure the timeout from aws lambda to 5 minutes
deleting the function and pushing the docker image into ecr again
writing the to be send events into a tmp/ file and read from it again
I use the docker image to not use with dependencies in lambda.
To deploy a single function for a single trigger event we can follow the instructions as outlined in the documentation on deploying Google Cloud Functions:
gcloud functions deploy NAME --runtime RUNTIME TRIGGER [FLAGS...]
It takes on average 30s-2m to deploy, which is fine and reasonable.
However, I was wondering if its possible to write a script (e.g. in python) to deploy multiple functions at once?
e.g. :
//somefile.py
gcloud functions deploy function_1 --runtime RUNTIME TRIGGER [FLAGS...]
gcloud functions deploy function_2 --runtime RUNTIME TRIGGER [FLAGS...]
I really like to use the invoke library for problems like this. In particular, it is well suited for running bash commands (e.g. gcloud) within a Python script without mucking about in subprocess.
In your case, you could make a tasks.py file that looks like
from invoke import task
#task
def deploy_cloud_functions(c):
c.run('gcloud functions deploy function_1 --runtime RUNTIME TRIGGER [FLAGS...]')
c.run('gcloud functions deploy function_2 --runtime RUNTIME TRIGGER [FLAGS...]')
and then run it by calling
invoke deploy-cloud-functions
Note that if you name your function deploy_cloud_functions you have to call it using: invoke deploy-cloud-functions (note the -). You can find a list of current available tasks in your directory using invoke --list
You can also parallelize it using the threading library (though I haven't tested using it within invoke myself). It will definitely make for ugly output in the console though. I.e.
from threading import Thread
from invoke import task
#task
def deploy_cloud_functions(c):
Thread(lambda x:
c.run('gcloud functions deploy function_1 --runtime RUNTIME TRIGGER [FLAGS...]')
).start()
Thread(lambda x:
c.run('gcloud functions deploy function_2 --runtime RUNTIME TRIGGER [FLAGS...]')
).start()
If you don't want to just use a python script to do calls to the gcloud command, since it is the same as doing a bash script, you can use the Cloud Functions API Client Library for Python.
What this library does, is create and execute HTTP calls to the Cloud Functions API. You can check Cloud Functions REST reference to see how these calls are structured, and how to build them.
For example, I did a quick example to test this API library, to list the functions running in my project:
import httplib2
import pprint
from googleapiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials
credentials = ServiceAccountCredentials.from_json_keyfile_name(
"key.json",
scopes="https://www.googleapis.com/auth/cloud-platform")
http = httplib2.Http()
http = credentials.authorize(http)
service = build("cloudfunctions", "v1", http=http)
operation = service.projects().locations().functions().list(parent='projects/wave16-joan/locations/europe-west1')
pprint.pprint(operation)
You will have to install the modules oauth2client, google-api-python-client and httplib2. As you can see, you will need to create a service account in order to execute the REST API calls, which needs "https://www.googleapis.com/auth/cloud-platform" scopes to create the CF. I created a service account with project/editor permissions myself, which I believe that are the required roles to create CFs.
Finally, to execute this script, you can just do python <script_name>.py
Now, since you want to create multiple functions (see how this API call needs to be structured here), the service to call should be the following, instead:
operation = service.projects().locations().functions().create(
location='projects/wave16-joan/locations/europe-west1',
body={
"name":"...",
"entryPoint":"..."
"httpsTrigger": {
"url":"..."
}
}
)
You will have to populate the body of the request with the some of the parameters listed here. For example, the "name" key should read:
"name":"projects/YOUR_PROJECT/locations/YOUR_PROJECT_LOCATION/functions/FUNCTION_NAME"
As a side note, most of the body parameters listed in the previous documentation are optional, but you will require the name, entryPoint, source, trigger, etc.
Of course this requires more work than creating a bash script, but the result is more portable and reliable, and it will allow you to create multiple operations to deploy multiple functions in the same way.
I am trying to learn AWS greengrass and so I was following this tutorial https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-gs.html which explains step by step on setting up with greengrass on raspberry pi and publishing some messages using a lambda function.
A simple lambda function is as following :
import greengrasssdk
import platform
from threading import Timer
import time
# Creating a greengrass core sdk client
client = greengrasssdk.client('iot-data')
# Retrieving platform information to send from Greengrass Core
my_platform = platform.platform()
def greengrass_hello_world_run():
if not my_platform:
client.publish(topic='hello/world', payload='hello Sent from Greengrass Core.')
else:
client.publish(topic='hello/world', payload='hello Sent from Greengrass Core running on platform: {}'.format(my_platform))
# Asynchronously schedule this function to be run again in 5 seconds
Timer(5, greengrass_hello_world_run).start()
# Execute the function above
greengrass_hello_world_run()
# This is a dummy handler and will not be invoked
# Instead the code above will be executed in an infinite loop for our example
def function_handler(event, context):
return
Here this works but I am trying to understand it better by having a lambda function to do some extra work for example opening a file and writing to it.
I modified the greengrass_hello_world_run() function as following
def greengrass_hello_world_run():
if not my_platform:
client.publish(topic='hello/world', payload='hello Sent from Greengrass Core.')
else:
stdout = "hello from greengrass\n"
with open('/home/pi/log', 'w') as file:
for line in stdout:
file.write(line)
client.publish(topic='hello/world', payload='hello Sent from Greengrass Core running on platform: {}'.format(my_platform))
I expect upon deploying, the daemon running on my local pi should create that file in the given directory coz I believe greengrass core tries to run this lambda function on local device. However it doesnt create any file nor it publish anything coz I believe this code might be breaking. Not sure how though, I tried looking into cloudwatch but I dont see any events or errors being reported.
Any help on this would be really appreciated,
cheers !
A few thoughts on this...
If you turn on the local logging in your GG group set up it will start to write logs locally on your PI. The settings are:
The logs are located in: /greengrass/ggc/var/log/system
If you tail the python_runtime.log you can see any errors from the lambda execution.
If you want to access local resources you will need to create a resource in your GG group definition. You can then give this access to a volume which is where you can write your file.
You do need to deploy your group once this has been done for the changes to take effect.
I think I found the answer, we have to create the resource in lambda environment and also make sure to give read and write access to lambda for accessing that resource. By default lambda can only access /tmp folder.
Here is the link to the documentation
https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html
I'm using mrjob to run some MapReduce tasks on EMR, and I want to run a job flow in a VPC. I looked at the documentation of mrjob and boto, and none of them seems to support this.
Does anyone know if this is possible to do?
Right now (v 0.3.5) is not possible. I made a pull request on the github project to add support for the 'api_params' parameter of boto, so you can pass parameters directly to the AWS API, and use the 'Instances.Ec2SubnetId' parameter to run a job flow in a VPC subnet.