I have a couple of AWS Lambda functions. All of those functions use some common helper functions. I have placed these helper functions in a separate file called helper_functions.py. I want to import this module in all of my AWS Lambda functions. I am unable to find a place to store this module (helper_functions.py), so when I make a change in this module I don't have to change anything in my Lambda functions.
Some of the options I thought of are:
Uploading the module on AWS S3 and then loading it in each Lambda
function in the start from S3 and using the functions. (if possible)
Writing some script (which I haven't figured out yet) that packages the module along with the Lambda
functions' Python file in a zip and uploads it on AWS Lambda
Please suggest a better solution to manage the module and import it in a much more efficient way.
I struggled with this for a long time. Here's my solution (there might be a better way):
setup your helper function in your file system like this:
pathToSomewhere/my_helper/helper_functions.py
pathToSomewhere/my_helper/__init__.py
pathToSomewhere/setup.py
Where __init__.py is:
from .helper_functions import *
and setup.py is
from setuptools import setup
setup(name='my_helper',
version='0.10000',
description='My helper functions',
url='http://github.com/user/example',
license='Proprietary',
author='Null',
author_email='null#example.com',
packages=['my_helper'],
install_requires=['boto3'],
zip_safe=False)
Now let's package up my_helper. From pathToSomewhere/ run:
python setup.py sdist
I'm assuming you already know how to create and upload a virtual environment for running your lambda function. If not, let me know.
Now let's install my_helper into the virtual env of your lambda function. Let's assume your virtual environment is called worker_env
./worker-env/bin/pip install file://pathToSomewhere/my_helper
Now zip up worker-env and your actual lambda script and upload that.
Related
When creating an AWS Lambda using Python:
Can the Lambda access local imports if the modules are included in the Lambda handler zip; and
What are the implications of including the __pycache__ directories in the zip?
Question 1: Can the runtime access local imports?
The AWS documentation focuses on the Lambda handler Python file containing the handler function itself. This is obviously must be included in the deployment zip. But we don't want one big function, or even several functions or classes in one big file.
If using the usual Python approach of creating sub-directories, containing modules and packages, included in the directory containing the handler itself, and included in the zip that is uploaded to the AWS Lambda handler, will that code be accessible at run-time, and therefore be importable by the Lambda handler?
I'm not referring to the AWS Lambda support for "layers", which is normally used for providing access to packages that are installed in a virtual environment with pip etc. I (think I) understand that support and am not asking about that.
I specifically just want to clarify: can the Lambda handler import from local files, for instance, an adjacent definitions.py being referenced by from definitions import * (please no judgements about don't import star :-) as long as it's also in the zip?
Question 2: Is it good practice to include the __pycache__ directories?
In the AWS Python Lambda deployment documentation the output of a zip command shows the packages included with the Lambda handler Python file, also including a __pycache__ directory. Additional libraries are also shown but it seems intended that these are collected from the layers.
The AWS documentation shows the inclusion of __pycache__ but no mention is made at all.
I believe AWS Lambda run-times are certain specific versions of Python running on AWS Linux images. I'm currently forced to develop on Windows :-(. Will this mismatch cause issues for the run-time? Would other considerations come into play such as ensuring included bytecode is the same version of Python as the runtime?
Doesn't Python expect bytecode for particular packages in a particular relative location? Presumably the top-level (in the zip) __pycache__ directory should only contain handler routine bytecode?
Does the Lambda runtime even use the __pycache__ directories? If this is workable and working, given the Lambda may only run once before being destroyed, does that imply that developers should put effort in to providing the bytecode in the Lambda zip to improve Lambda performance? In this case, is it necessary to run sufficient tests across the code before zipping it for Lambda, to ensure all the bytecode is generated?
Context
I have reviewed various articles on creating an AWS Lambda zip using Python, including the AWS documentation, but the content is shallow and simplistic, failing to clarify the precise "reach" of the runtime. It's not even in the AWS Lambda Handler Cookbook.
I do not (yet) have access to a live AWS environment to test this out, and given this broad omission in the on-line documentation and community commentaries (most articles just parrot the AWS documentation anyway, albeit with worse grammar), I thought it would be good to get a clarification on SO.
I would like to type in my Google Cloud function:
from my_google_cloud_project import another_google_cloud_func
another_google_cloud_func()
I don't want to invoke that function via HTTP request. How can I just import it?
Both functions are Google Cloud functions, but not just python code!
you can create a code.py file and write your code there
read the content of that file
with open('code.py') as f
content = f.read()
then execute those content using
exec(content)
Please read the official documentation regarding :
Packaging local dependencies
You can also package and deploy dependencies alongside your function.
This approach is useful if your dependency is not available via the
pip package manager or if your Cloud Functions environment's internet
access is restricted. For example, you might use a directory structure
such as the following:
You can then use code as usual from the included local dependency, localpackage. You can use this approach to bundle
any Python packages with your deployment.
I need a way to upgrade/downgrade the boto3 lib inside my Python 3.7 env inside Lambda.
Right now, the version is 1.9.42 inside Lambda. I cannot use certain things like Textract (boto3.client('textract'), but I can on my local machine (boto3 version 1.9.138.
So, I decided to install boto3 into a package (pip3 install boto3 -t dir/ --system) then upload it to Lambda after zipping it.
This didn't work because Lambda won't accept a package larger than 3MB (it's around 8MB)
Any other workarounds?
edit: I know I could always just write code that works and keep uploading it to Lambda, but this will become cumbersome as I'd have to include all the packages installed in the package and rebuilding it as I make changes.
The Serverless Application Model is a tool provided by AWS that lets you develop locally as it simulates the lamdba environment inside a docker container. Once you are ready you can deploy your code to a lambda and it will work as expect.
If you really want to keep editing the code in the web platform, there is a workaround by using lambda layers. You create a package with all of your dependencies and upload that to a lambda layer. Then include your layer in the lambda and just modify your own code there. As it has been pointed out in the comments this is not the way to go for real development.
I'm struggling to understand how best to manage python packages to get zipped up for an AWS lambda function.
In my project folder I have a number of .py files. As part of my build process I zip these up and use the AWS APIs to create and publish my lambda function supplying the zip file as part of that call.
Therefore, it is my belief that I need to have all the packages my lambda is dependant on within my project folder.
With that in mind, I call pip as follows:
pip install -t . tzlocal
This seems to fill my project folder with lots of stuff and I'm unsure if all of it needs to get zipped up into my lambda function deployment e.g.
.\pytz
.\pytz-2018.4.dist-info
.\tzlocal
...
...
First question - does all of this stuff need to be zipped up into my lambda?
If not, how do I get a package that gives me just the bits I need to go into my zip file?
Coming from a .Net / Node background - with the former I NuGet my package in and it goes into a nice packages folder containing just the .dll file I need which I then reference into my project.
If I do need all of these files is there a way to "put" them somewhere more tidy - like in a packages folder?
Finally, is there a way to just download the binary that I actually need? I've read the problem here is that the Lambda function will need a different binary to the one I use on my desktop development environment (Windows) so not sure how to solve that problem either.
Binary libraries used for example by numpy should be compiled on AWS Linux to work on lambda. I find this tutorial useful (https://serverlesscode.com/post/deploy-scikitlearn-on-lamba/). There is even newer version of it which uses docker container so you do not need EC2 instance for compilation and you can do everything locally.
As for the packages: the AWS docs says to install everything to the root, but you may install them all in a ./packages directory and append it to path in the beginning if the lambda handler code
import os
import sys
cwd = os.getcwd()
package_path = os.path.join(cwd, 'packages')
sys.path.append(package_path)
How to deploy to python function which has the dependency on external libraries?
For example, I am trying to deploy to a data-analysis python function. When I try to test the python function from the lambda console, I get:
Unable to import module 'lambda_function': No module named pandas
I am totally new to Aws Lambda
Is there a linux box on which Lambda functions run where I can install these libraries?
You need to create a deployment package as detailed here: http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html#deployment-pkg-for-virtualenv
This just means bundling the contents of site-packages for the environment you're developing on into the deployment package together with the lambda python script into a zip that is uploaded.
If you new to Lambda deployment, you might want check this tutorial (I wrote), which covers the most common pitfalls. And gives you a script as well to automate the entire process.