How to deploy to python function which has the dependency on external libraries?
For example, I am trying to deploy to a data-analysis python function. When I try to test the python function from the lambda console, I get:
Unable to import module 'lambda_function': No module named pandas
I am totally new to Aws Lambda
Is there a linux box on which Lambda functions run where I can install these libraries?
You need to create a deployment package as detailed here: http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html#deployment-pkg-for-virtualenv
This just means bundling the contents of site-packages for the environment you're developing on into the deployment package together with the lambda python script into a zip that is uploaded.
If you new to Lambda deployment, you might want check this tutorial (I wrote), which covers the most common pitfalls. And gives you a script as well to automate the entire process.
Related
I've written a Python program that I want to package and upload to the Python package index. After doing this, is there any way for me to import that Python package and use it in Node.js or ReactJS? I basically want to use the functions and classes in the Python package to get some information and show it in some frontend application.
I am not sure if there is any mode to access the packages directly, however, you sure can execute a python script from your Node.js backend by using the child_process package.
Reference:
How to call a Python function from Node.js
I am trying to create a lambda using python directly in the console and am stuck on installing a dependency. Is there a way to install dependencies in the console? I have tried a subprocess call but says I don't have permission to the folders.
I am used to doing lambdas in javascript and have deploy scripts for that. I just don't know how to translate that to python.
I need a way to upgrade/downgrade the boto3 lib inside my Python 3.7 env inside Lambda.
Right now, the version is 1.9.42 inside Lambda. I cannot use certain things like Textract (boto3.client('textract'), but I can on my local machine (boto3 version 1.9.138.
So, I decided to install boto3 into a package (pip3 install boto3 -t dir/ --system) then upload it to Lambda after zipping it.
This didn't work because Lambda won't accept a package larger than 3MB (it's around 8MB)
Any other workarounds?
edit: I know I could always just write code that works and keep uploading it to Lambda, but this will become cumbersome as I'd have to include all the packages installed in the package and rebuilding it as I make changes.
The Serverless Application Model is a tool provided by AWS that lets you develop locally as it simulates the lamdba environment inside a docker container. Once you are ready you can deploy your code to a lambda and it will work as expect.
If you really want to keep editing the code in the web platform, there is a workaround by using lambda layers. You create a package with all of your dependencies and upload that to a lambda layer. Then include your layer in the lambda and just modify your own code there. As it has been pointed out in the comments this is not the way to go for real development.
I'm struggling to understand how best to manage python packages to get zipped up for an AWS lambda function.
In my project folder I have a number of .py files. As part of my build process I zip these up and use the AWS APIs to create and publish my lambda function supplying the zip file as part of that call.
Therefore, it is my belief that I need to have all the packages my lambda is dependant on within my project folder.
With that in mind, I call pip as follows:
pip install -t . tzlocal
This seems to fill my project folder with lots of stuff and I'm unsure if all of it needs to get zipped up into my lambda function deployment e.g.
.\pytz
.\pytz-2018.4.dist-info
.\tzlocal
...
...
First question - does all of this stuff need to be zipped up into my lambda?
If not, how do I get a package that gives me just the bits I need to go into my zip file?
Coming from a .Net / Node background - with the former I NuGet my package in and it goes into a nice packages folder containing just the .dll file I need which I then reference into my project.
If I do need all of these files is there a way to "put" them somewhere more tidy - like in a packages folder?
Finally, is there a way to just download the binary that I actually need? I've read the problem here is that the Lambda function will need a different binary to the one I use on my desktop development environment (Windows) so not sure how to solve that problem either.
Binary libraries used for example by numpy should be compiled on AWS Linux to work on lambda. I find this tutorial useful (https://serverlesscode.com/post/deploy-scikitlearn-on-lamba/). There is even newer version of it which uses docker container so you do not need EC2 instance for compilation and you can do everything locally.
As for the packages: the AWS docs says to install everything to the root, but you may install them all in a ./packages directory and append it to path in the beginning if the lambda handler code
import os
import sys
cwd = os.getcwd()
package_path = os.path.join(cwd, 'packages')
sys.path.append(package_path)
I have a couple of AWS Lambda functions. All of those functions use some common helper functions. I have placed these helper functions in a separate file called helper_functions.py. I want to import this module in all of my AWS Lambda functions. I am unable to find a place to store this module (helper_functions.py), so when I make a change in this module I don't have to change anything in my Lambda functions.
Some of the options I thought of are:
Uploading the module on AWS S3 and then loading it in each Lambda
function in the start from S3 and using the functions. (if possible)
Writing some script (which I haven't figured out yet) that packages the module along with the Lambda
functions' Python file in a zip and uploads it on AWS Lambda
Please suggest a better solution to manage the module and import it in a much more efficient way.
I struggled with this for a long time. Here's my solution (there might be a better way):
setup your helper function in your file system like this:
pathToSomewhere/my_helper/helper_functions.py
pathToSomewhere/my_helper/__init__.py
pathToSomewhere/setup.py
Where __init__.py is:
from .helper_functions import *
and setup.py is
from setuptools import setup
setup(name='my_helper',
version='0.10000',
description='My helper functions',
url='http://github.com/user/example',
license='Proprietary',
author='Null',
author_email='null#example.com',
packages=['my_helper'],
install_requires=['boto3'],
zip_safe=False)
Now let's package up my_helper. From pathToSomewhere/ run:
python setup.py sdist
I'm assuming you already know how to create and upload a virtual environment for running your lambda function. If not, let me know.
Now let's install my_helper into the virtual env of your lambda function. Let's assume your virtual environment is called worker_env
./worker-env/bin/pip install file://pathToSomewhere/my_helper
Now zip up worker-env and your actual lambda script and upload that.