I have a python script which is running on my local machine and I want to move it to AWS Lambda for periodic execution. I have 3 import statements in the script for which I am adding layers but facing some issues.
from googleapiclient.discovery import build
import pandas as pd
from datetime import date
For googleapiclient api, I downloaded it in a folder and uploaded to AWS layer and lambda is able to find this module. I wanted to use this along with AWS Data wrangler package but running into layer size restriction issues. So I downloaded pandas to the same folder as google api and then uploaded the zip file to layer. But now I am get a numpy dependency error though numpy was downloaded as part of pandas install. Two folders as part of my libraries folder are numpy, numpy-1.22.0.dist-info which is correct version as per error message below. I also tried downloading numpy separately in same package but that's also not working. The Python version that I am using in Lambda is 3.7.
Please let me know if I am missing something and if this is the correct approach for installing python packages for AWS Lambda. There are some solutions available with regards to using a container for this but I am not well versed with containers. So just wanted to check if there is some fix through which I can achieve this or is containers the way to go about it.
Below is the error message:
Response { "errorMessage": "Unable to import module 'lambda_function':
Unable to import required dependencies:\nnumpy: \n\nIMPORTANT: PLEASE
READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the
numpy C-extensions failed. This error can happen for\nmany reasons,
often due to issues with your setup or how NumPy was\ninstalled.\n\nWe
have compiled some common reasons and troubleshooting tips at:\n\n
https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease
note and check the following:\n\n The Python version is: Python3.7
from "/var/lang/bin/python3.7"\n The NumPy version is:
"1.22.0"\n\nand make sure that they are the versions you
expect.\nPlease carefully study the documentation linked above for
further help.\n\nOriginal error was: No module named
'numpy.core._multiarray_umath'\n", "errorType":
"Runtime.ImportModuleError", "stackTrace": [] }
Function Logs START RequestId: d......-........-.....d
Version: $LATEST ERROR Runtime.ImportModuleError: Unable to import
module 'lambda_function': Unable to import required dependencies:
numpy: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS
ISSUE! Importing the numpy C-extensions failed. This error can happen
for many reasons, often due to issues with your setup or how NumPy was
installed. We have compiled some common reasons and troubleshooting
tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html Please
note and check the following:
The Python version is: Python3.7 from "/var/lang/bin/python3.7" The
NumPy version is: "1.22.0" and make sure that they are the versions
you expect. Please carefully study the documentation linked above for
further help. Original error was: No module named
'numpy.core._multiarray_umath' Traceback (most recent call last): END
RequestId: de72............................6d REPORT RequestId:
de72............................6d Duration: 1.68 ms Billed
Duration: 2 ms Memory Size: 128 MB Max Memory Used: 55 MB Init
Duration: 570.09 ms Request ID de72............................6d
Regards, Dbeings
The issue was that I was building the pandas package on windows machine and uploading it to AWS layer. Building the package on EC2 and then putting it on AWS layer fixed the issue.
Related
I regularly use python on my PC (Spyder, Juptyer etc) without issue. I've recently installed VSCode, to take advantage of its debug features.
However, I can't seem to get it to play nice with Numpy. Running a script simply with:
import numpy as np
Exception has occurred: ImportError
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.9 from "C:\ProgramData\Anaconda3\python.exe"
* The NumPy version is: "1.21.5"
Original error was: DLL load failed while importing _multiarray_umath: The specified module could not be found.
Also:
I have these installed and enabled:
This works in Spyder, Jupyter etc. And I'm using the 'base' conda environment, and can see that numpy is installed in the Packages list.
Any help on fixing this (so I can debug without using 1000x prints!) is much appreciated!
Runtime python 3.7 w/ compatible runtime 3.7
I keep getting Import error when trying to test API in lambda function
PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy C-extensions failed. This error can happen for\nmany reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:\n\n https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n * The Python version is: Python3.7 from \"/var/lang/bin/python3.7\"\n * The NumPy version is: \"1.21.5\"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.\n\nOriginal error was: No module named 'numpy.core._multiarray_umath'\n",
"errorType": "Runtime.ImportModuleError",
"stackTrace": []
I have tried adding numpy pip install inside docker, yet the issue still persists...
The dockfile engine used is amazonlinux:2
Any way of solving this weird issue?
Based on the comments.
The solution was to use Numpy layer provided by AWS.
I am working on aws lambda function, i install the package but i got error
Unable to import module 'lambda_function': cannot import name 'cygrpc' from 'grpc._cython' (/var/task/grpc/_cython/init.py)
How to solve this type of error?
Are you trying to import it in code without haveing it installed on the lambda container? If you gonna use none std libs you should have you function and its libs installed locally and when zip it and upload it to Lambda.
Docs:
https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-package.html
In addition to an answer marked correct I wanted to mention that for libraries using C-extensions is important that the layer was built on the same type of system as it runs in lambda. For example, if you're trying to build the layer including grpc on MacOS most likely it won't work since lambda is running on Linux. To make the layer working you have to build it in a Linux system (for example, on the virtual machine or in Docker)
I'm getting error when I import pandas/numpy in lambda function:
I added pandas/numpy as a aws layer also I have added pymysql in same way, and it's working fine.
Working on windows 10,
Using python3.7 same in lambda also
Building zip package using pip install pandas -t
In dir - python\lib\python3.7\site-packages
{
"errorMessage": "Unable to import module 'lambda_function': \n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy C-extensions failed. This error can happen for\nmany reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:\n\n https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n * The Python version is: Python3.7 from \"/var/lang/bin/python3.7\"\n * The NumPy version is: \"1.19.1\"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.\n\nOriginal error was: No module named 'numpy.core._multiarray_umath'\n",
"errorType": "Runtime.ImportModuleError"
}
zip of the package should be done in the build folder, even though you install the packages in the site-packages
https://www.youtube.com/watch?v=zrrH9nbSPhQ - has some good steps to follow through.
Pandas could be a huge zip file, that lambda layers might not support. You might want to check for pandas- example pandas-xlrd.
Hi i deployed an application with lambda aws (using Serverless Framework).
but i receive the error when i run a test
START RequestId: 5bcf988a-2064-11e8-8d1e-c5a0129510f Version: $LATEST
Unable to import module 'handler':
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.
Original error was: cannot import name multiarray
I removed the numpy package folder multiple times and reinstalled but, still not working
So do you have any solution for this problem ?
for the code :
import sys
import os
sys.path.append(os.path.join(os.path.dirname(os.path.realpath(__file__)),'env','Lib','site-packages'))
import numpy as np
import pandas as pd
import boto3
import csv
import psycopg2
def hello(event, context):
s3 = boto3.resource('s3')
return "test"
Thank you in advance
I resolved the problem by changing the environnement.
I didn't know that python is not portable.
I developed the lambda function in my Windows environment, but lambda is usually run in a Linux environment in AWS, and that's why my lambda was not working.
So the solution is to change the environnement, and use Linux instead of windows
I compiled my code on MACOSX environment and was caught in the same problem these days, and even I put my code on ec2 and installed all the dependencies in linux or ubuntu environment, the error is the same.
https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html docs here gave me the hint that the current lambda execution environment is Public Amazon Linux AMI version (AMI name: amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2),please pay attention to the specific version of AMI, even you use Amazon Linux as your ec2 instance, the newer or older version can make problem.
So after I changed the version of my ec2 instance, recompiled my code on it and store the zipped files on s3, than the AWS lambda run successfully.