I am trying to execute a python function on AWS Lambda. In my function I am trying to import the mysql.connector module. But an error is throwing up :
errorMessage": "No module named 'mysql.connector'"
I had written my python code initially in my EC2 Instance. I installed mysql-connector there in my python file directory using pip.
pip install mysql-connector -t /path/to/file/dir
I uploaded the zip file of the only the file and not any folder containing the file.
Pattern that I have been using to deploy Python libs to lambda is following
Firstly, prior to packaging lambda function install all of the requirements into $SOURCE_ROOT/lib folder
pip install -r requirements.txt -t ./lib
Secondly, dd automatic import of this folder in your lambda entrypoint (that is lambda handler)
import os
import sys
# if running in lambda
if 'LAMBDA_TASK_ROOT' in os.environ:
sys.path.append(f"{os.environ['LAMBDA_TASK_ROOT']}/lib")
# this will render all of your packages placed as subdirs available
sys.path.append(os.path.dirname(os.path.realpath(__file__)))
Extending sys.path with your own packaged path is the crucial for this to work.
Note on native compiled extensions
Note that if you are packaging any natively compiled extensions, their compilation should be done on linux-compatible O/S, ideally on EC2 instance created from Amazon Linux AMIs Lambda is running from (currently amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2, but up to date information can be always obtained from Amazon Official Docs. According to my experience, extensions built from official Docker python container worked on lambda, without need to compile on actual EC2 instance created from AMIs, but there is no guarantee as offical docs state that
If you are using any native binaries in your code, make sure they are compiled in this environment. Note that only 64-bit binaries are supported on AWS Lambda.
MySQL Connector
Quick look at MySQL connector for Python gives impression that by default package will use native Python implementation, so no C extension is loaded
Lambda is just like an EC2 instance with only python installed. You need to include all the packages that are required to run the python code with the deployment package itself. None of the packages will be pre-installed when you run the code.
There are two ways to pack python dependencies with Lambda.
Pack it with the function itself. To do this you just go to the root folder of your function, and install dependencies in that folder itself, not into any other folder.
$ pip install -t <some-package> .
Then zip that folder from root and upload, your zip should look something like this:
.
├── lambda_function.py
├── pymysql
│ └── ...
└── PyMySQL-0.9.3.dist-info
└── ...
Second method is a standard that I always follow,I never ship libraries or external packages with my lambda function, I always create Layers.
A layer is a ZIP archive that contains libraries, a custom runtime, or
other dependencies. With layers, you can use libraries in your
function without needing to include them in your deployment package.
Learn More about Lambda Layers in the docs
Related
I am trying to package a local python package¹ and use it within an AWS lambda deployed via the Serverless framework. I already use serverless-python-requirements plugin to add pip dependencies to deployed package.
How can I proceed ?
Shall I create a package and zip it? Or generate a whl file and use pip? And then, how to deploy it?
¹: I cannot just add it to "normal codebase" as I want to share it with other bricks (Glue jobs for example)
Here's the solution:
1.
Build a .whl file corresponding to package using
python setup.py bdist_wheel
within a parent directory.
2.
Add the relative path to this .whl file to used pip requirement file (requirements.txt for instance) :
req0==1.0.9
req1==5.5.0
../<relative path to local package>/dist/<package name>-<version>-<details>.whl # generated .whl file's name
3.
serverless-python-requirements will automagically pack this dependency within the deployed archive when doing sls deploy. How cool is that, huh!
Gettting the below error while running the lambda code , I am using the library called
from flatten_json import flatten
I tried to look for a lambda layer , but did not find any online , please let me know if any one used this before or suggest any alternative
flatten_json library is missing.
Use pip install flatten_json to get it
There are four steps you need to do:
Download the dependency.
Package it in a ZIP file.
Create a new layer in AWS.
Associate the layer with your Lambda.
My answer will focus on 1. and 2. as they are what is most important to your problem. Unfortunately, packaging Python dependencies can be a bit more complicated than for other runtimes.
The main issue is that some dependencies use C code under the hood, especially performance critical libraries, for example for Machine Learning etc.
C code needs to be compiled and if you run pip install on your machine the code will be compiled for your computer. AWS Lambdas use a linux kernel and amd64 architecture. So if you are running pip install on a Linux machine with AMD or Intel processor, you can indeed just use pip install. But if you use macOS or Windows, your best bet is Docker.
Without Docker
pip install --target python flatten_json
zip -r layer.zip python
With Docker
The lambci project provides great Docker container for building and running Lambdas. In the following example I am using their build-python3.8 image.
docker run --rm -v $(pwd):/var/task lambci/lambda:build-python3.8 pip install --target python flatten_json
zip -r layer.zip python
Be aware that $(pwd) is meant to be your current directoy. On macOS and WSL this should work, but if it does not work you can just replace it with the absolute path to your current directory.
Explanation
Those commands will install the dependency into a target folder called python. The name is important, because it is one of two folders of a layer where Lambda looks for dependencies.
The python folder is than archived recursively (-r) in a file called layer.zip.
Your next step is to create a new Layer in AWS and associated your function with that layer.
There are two options to choose from
Option 1) You can use a deployment package to deploy your function code to Lambda.
The deployment package (For e.g zip) will contain your function's code and any dependencies used to run the function's code.
Hence, you can package flatten_json as your code to the Lambda.
Check Creating a function with runtime dependencies page in aws documentation, it explains the use-case of having requests library. In your scenario, the library would be flatten_json
Option 2) Create a layer that has the library dependencies you need, in your case just flatten_json. And then attach that layer to your Lambda.
Check creating and sharing lambda layers guide provided by AWS.
How to decide between 1) and 2)?
Use Option 1) when you just need the dependencies in just that one Lambda. No need to create an extra step of creating a layer.
Layers are useful if you have some common code that you want to share across different Lambdas. So if you need the library accessible in other Lambdas as well, then it's good to have a layer[Option 2)] that can be attached to different lambdas.
You can do this is in a Lambda if you don´t want to create the layer. Keep in mind it will run slower since it has to install the library in every run:
import sys
import subprocess
subprocess.call('pip install flatten_json -t /tmp/ --no-cache-dir'.split(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
sys.path.insert(1, '/tmp/')
import flatten_json
I am havin an issue with importing a common function util file in my AWS lambda. It is a python file and the folder structure looks something like this
(functions folder)
common_util.py
(lambda 1 folder)
lambda1
(lambda 2 folder)
lambda2
I need to access the common_util from both these lambdas. When I run my CDK project locally this is easy i use something like .. on the import statement to tell the file it is one directory up
from ..common_util import (...)
When I deploy to AWS as a lambda (I package all of the above) I need to specify the import without the .. because this is the root folder of the lambda
from common_util import(...)
I need an import statement or a solution that will work for both my CDK project and the lambda.
here is the CDK where the lambda is created
const noteIntegrationLambda = new Function(this as any,"my-lambda",
{
functionName:
"my-lambda",
runtime: StackConfiguration.PYTHON_VERSION,
handler:
"my_lambda.execute",
timeout: Duration.seconds(15),
code: Code.fromAsset("functions/"),
role,
layers: [dependencyLayer],
environment: env,
},
}
);
Lambda layers provide an ideal mechanism to include in solving this problem. As mentioned in https://medium.com/#manojf/sharing-code-among-lambdas-using-lambda-layers-ca097c8cd500,
Lambda layers allow us to share code among lambda functions. We just
have to upload the layer once and reference it in any lambda function.
So consider deploying your common code via a layer. That said, to structure your code, I recommend you create a common package that you install locally using pip install, as outlined at Python how to share package between multiple projects. Then you put that package into a layer that both of your lambdas reference. That completely solves the problem of how to structure code when your local file structure is different than the lambda file structure.
Also consider these resources:
Import a python module in multiple AWS Lambdas
What is the proper way to work with shared modules in Python development?
https://realpython.com/absolute-vs-relative-python-imports/
Python: sharing common code among a family of scripts
Sharing code in AWS Lambda
Installing Python packages from local file system folder to virtualenv with pip
As a layer example, suppose you wanted to include a "common_utils" library for your lambdas to reference. To make a layer, you would need to create a directory structure that contains that code, then zip the entire directory. It may be as follows:
/python
/common_utils
__init__.py
common_util.py
...
When zipped, the zip file must have the "python" folder and inside of that you put your code. If you do this and also install your common code as a package, you can import it in your local code and in your lambdas using the same import.
What I do is use pip install to install to a certain file location--the location that I then zip into a layer. For example, if I wanted to make a layer for the pymysql library I might do
pip install --target=c:\myLayers\python pymysql
That will install the library files into the location I specified, which makes it easy to know what to zip up (just create a zip that includes the "python" directory).
I know this question is old, but I ran into a similar issue. My solution was to detect if the current environment is local or lambda using the os package, and then import differently based on the environment (local or cloud). Will leave here as a reference.
if os.environ.get("AWS_EXECUTION_ENV") is not None:
# For use in lambda function
from package_a import class_a
else:
# For local use
from ...package_a import class_a
Credits to: How to check if Python app is running within AWS lambda function?
I have a simple Lambda function which is using the numpy library,
I have set up a virtual environment in my local, and my code is able to fetch and use the library locally.
I tried to use AWS Lambda's layer, and zipped the venv folder and uploaded to the layer,
Then I attached the correct layer and version to my function,
But the function is not able to fetch the library
Following is the code which works fine on local -
import numpy as np
def main(event, context):
a = np.array([1, 2, 3])
print("Your numpy array:")
print(a)
Following is the venv structure which I zipped and uploaded -
I get the following error -
{
"errorMessage": "Unable to import module 'handler': No module named 'numpy'",
"errorType": "Runtime.ImportModuleError"
}
My Lambda deployment looks like this -
I'm trying to refer this -
https://towardsdatascience.com/introduction-to-amazon-lambda-layers-and-boto3-using-python3-39bd390add17
I've seen that a few libraries like numpy and pandas don't work in Lambda when installed using pip. I have had success using the .whl package files for these libraries to create the Lambda layer. Refer to the steps below:
NOTE: These steps set up the libraries specific to the Python 3.7 runtime. If using any other version, you would need to download the .whl files corresponding to that Python version.
Create an EC2 instance using Amazon Linux AMI and SSH into this instance. We should create our layer in Amazon Linux AMI as the Lambda Python 3.7 runtime runs on this operating system (doc).
Make sure this instance has Python3 and "pip" tool installed.
Download the numpy .whl file for the cp37 Python version and the manylinux1_x86_64 OS by executing the below command:
$ wget https://files.pythonhosted.org/packages/d6/c6/58e517e8b1fb192725cfa23c01c2e60e4e6699314ee9684a1c5f5c9b27e1/numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl
Skip to the next step if you're not using pandas. Download the pandas .whl file for the cp37 Python version and the manylinux1_x86_64 OS by executing the below command:
$ wget https://files.pythonhosted.org/packages/a4/5f/1b6e0efab4bfb738478919d40b0e3e1a06e3d9996da45eb62a77e9a090d9/pandas-1.0.4-cp37-cp37m-manylinux1_x86_64.whl
Next, we will create a directory named "python" and unzip these files into that directory:
$ mkdir python
$ unzip pandas-1.0.4-cp37-cp37m-manylinux1_x86_64.whl -d python/
$ unzip numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl -d python/
We also need to download "pytz" library to successfully import numpy and pandas libraries:
$ pip3 install -t python/ pytz
Next, we would remove the “*.dist-info” files from our package directory to reduce the size of the resulting layer.
$ cd python
$ sudo rm -rf *.dist-info
This will install all the required libraries that we need to run pandas and numpy.
Zip the current "python" directory and upload it to your S3 bucket. Ensure that the libraries are present in the hierarchy as given here.
$ cd ..
$ zip -r lambda-layer.zip python/
$ aws s3 cp lambda-layer.zip s3://YOURBUCKETNAME
The "lambda-layer.zip" file can then be used to create a new layer from the Lambda console.
Base on aws lamda layer doc, https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html your zip package for the layer must have this structure.
my_layer.zip
| python/numpy
| python/numpy-***.dist-info
So what you have to do is create a folder python, and put the content of site-packages inside it, then zip up that python folder. I tried this out with a simple package and it seem to work fine.
Also keep in mind, some package require c/c++ compilation, and for that to work you must install and package on a machine with similar architecture to lambda. Usually you would need to do this on an EC2 where you install and package where it have similar architecture to the lambda.
That's bit of misleading question, because you at least did not mention you use serverless. I found it going through the snapshot of you project structure you provided. That means you probably use serverless for deployment of your project within AWS provider.
Actually, there are multiple ways you can arrange lambda layer. Let's have a look at each of them.
Native AWS
Once you will navigate to Add a layer, you will find 3 options:
[AWS Layers, Custom Layers, Specify an ARN;].
Specify an ARN Guys, who did all work for you: KLayers
so, you need numpy, okay. Within lambda function navigate to the layers --> create a new layer --> out of 3 options, choose Specify an ARN and as the value put: arn:aws:lambda:eu-west-1:770693421928:layer:Klayers-python38-numpy:12.
It will solve your problem and you will be able to work with numpy Namespace.
Custom Layers
Choose a layer from a list of layers created by your AWS account or organization.
For custom layers the way of implementing can differ based on your requirements in terms of deployment.
If are allowed to accomplish things manually, you should have a glimpse at following Medium article. I assume it will help you!
AWS Layers
As for AWS pre-build layers, all is simple.
Layers provided by AWS that are compatible with your function's runtime.
Can differentiate between runtimes
For me I have list of: Perl5, SciPy, AppConfig Extension
Serverless
Within serverless things are much easier, because you can define you layers directly with lambda definition in serverless.yml file. Afterwards, HOW to define them can differ as well.
Examples can be found at: How to publish and use AWS Lambda Layers with the Serverless Framework
If you will have any questions, feel free to expand the discussion.
Cheers!
I have followed all the steps in the documentation:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
create a directory.
Save all of your Python source files (the .py files) at the root level of this directory.
Install any libraries using pip at the root level of the directory.
Zip the content of the project-dir directory)
But after I uploaded the zip-file to lambda function, I got the error message when I test the script
my code:
import psycopg2
#my code...
the error:
Unable to import module 'myfilemane': No module named 'psycopg2._psycopg'
I don't know where is the suffix '_psycopg' from...
Any help regarding this?
You are using native libraries with lambda. We had this similar problem and here is how we solved it.
Spin a machine with AWS supported AMI that runs your real lambda.
https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
As this writing, it is,
AMI name: amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2
Full documentation in installing native modules your python lambda.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
Install the required modules required for your lambda,
pip install module-name -t /path/to/project-dir
and prepare your package to upload along with the native modules under lambda ami environment.
Hope this helps.
I believe this is caused because psycopg2 needs to be build an compiled with statically linked libraries for Linux. Please reference Using psycopg2 with Lambda to Update Redshift (Python) for more details on this issue. Another [reference][1] of problems of compiling psycopg2 on OSX.
There are a few solutions, but basically it comes down to installing the library on a Linux machine and using that as the Psycopg2 Library in your upload package.