Develop Reference

ImportError while using AWS Lambda in Python (via zappa serverless); Numpy

I have an ImportError issue with my AWS Lambda function
I can run this code locally without ImportError
I use zappa serverless for uploading my code to AWS Lambda--which requires a virtual environment and is an end-to-end solution for creating and updating Lambda functions
I previously had this code working on AWS Lambda
I'm not sure of the change I made (#3) that made this code go from working to ImportError. The log output is as follows.
[ERROR] Runtime.ImportModuleError: Unable to import module 'dailycore': Unable to import required dependencies:
numpy:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.8 from "/var/lang/bin/python3.8"
* The NumPy version is: "1.23.3"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: No module named 'numpy.core._multiarray_umath'
I'm unclear if this is a Numpy issue, a Python issue, or a Zappa issue. Two changes I made that may have had an impact:
I separated out some of my personal code, uploaded it to github, and installed it as packages within the virtual environment using pip
I uninstalled Python from my machine and reinstalled it--as I had two versions causing conflict. Upon my reinstallation (Python 3.8), I changed the installation directory from its default.
The code runs from VS Code locally and wholly contained from within the virtual environment (i.e. I don't have any packages installed via pip on my machine outside of virtual environments at this point). The suggestions above may or may apply to this error. I've tried adding a system PATH variable to my Python 3.8 directory. I've tried downgrading numpy to 1.23.0. I've tried uninstalling all dependencies within the virtual environment and reinstalling them.
UPDATE: I found this link and am looking into these comments: https://github.com/Miserlou/Zappa/issues/1222
UPDATE2: I deleted the virtual environment and re-created per a suggestion at the link above. This did not work.
UPDATE3: I think my issue relates to the private package dependencies I installed via pip from my personal github repos (#2-2). This link may be related. Something about dependencies not being at the top level. I am unsure how to fix this issue and/or repackage my private repo or install correctly. Any clarification is appreciated.
UPDATE4: I've isolated my issue to the private package dependencies and how they get called. A the solution may be related to Lambda Layers, tutorial here. Looking into this.

The ImportModuleError is related to this python package: alpha_vantage and not either of my initial theories. I am unsure as to why, though. It seems like it has something to do with Alpha Vantage's setup.py file; although I could be mistaken.
I have removed the reference to Alpha Vantage for now, but if anyone knows why installing this package would throw a numpy error, please elaborate. I think the only reference to numpy would be through the pandas package--which is only an optional install (i.e. extras_requires in setup.py)

Python + VSCode not playing nice with Numpy

I regularly use python on my PC (Spyder, Juptyer etc) without issue. I've recently installed VSCode, to take advantage of its debug features.
However, I can't seem to get it to play nice with Numpy. Running a script simply with:
import numpy as np
Exception has occurred: ImportError
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.9 from "C:\ProgramData\Anaconda3\python.exe"
* The NumPy version is: "1.21.5"
Original error was: DLL load failed while importing _multiarray_umath: The specified module could not be found.
Also:
I have these installed and enabled:
This works in Spyder, Jupyter etc. And I'm using the 'base' conda environment, and can see that numpy is installed in the Packages list.
Any help on fixing this (so I can debug without using 1000x prints!) is much appreciated!

NumPy setup / import issue in lambda function

Runtime python 3.7 w/ compatible runtime 3.7
I keep getting Import error when trying to test API in lambda function
PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy C-extensions failed. This error can happen for\nmany reasons, often due to issues with your setup or how NumPy was\ninstalled.\n\nWe have compiled some common reasons and troubleshooting tips at:\n\n https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease note and check the following:\n\n * The Python version is: Python3.7 from \"/var/lang/bin/python3.7\"\n * The NumPy version is: \"1.21.5\"\n\nand make sure that they are the versions you expect.\nPlease carefully study the documentation linked above for further help.\n\nOriginal error was: No module named 'numpy.core._multiarray_umath'\n",
"errorType": "Runtime.ImportModuleError",
"stackTrace": []
I have tried adding numpy pip install inside docker, yet the issue still persists...
The dockfile engine used is amazonlinux:2
Any way of solving this weird issue?

Based on the comments.
The solution was to use Numpy layer provided by AWS.

AWS Lambda python Error: Runtime.ImportModuleError: Unable to import module 'app': cannot import name 'operatorPrecedence' from 'pyparsing'

I just redeploy my AWS Lambda & accidentally face an issue:
[ERROR]
Runtime.ImportModuleError: Unable to import module 'app': cannot import name 'operatorPrecedence' from 'pyparsing'
(/opt/python/pyparsing/__init__.py)
(/var/task/pyparsing/__init__.py)
How do I solve this?

I saw a release note that this feature was discontinued in version 3 of pyparsing:
operatorPrecedence synonym for infixNotation - convert to calling
infix_notation
So you may be including a different version of pyparsing than you have been using previously--one that does not include the operatorPrecedence functionality, so double check what version of pyparsing you are including.
This Knowledge Center article outlines the most common issues:
You typically receive this error when your Lambda environment can't
find the specified library in the Python code. This is because Lambda
isn't prepackaged with all Python libraries.
To resolve this error, create a deployment package or Lambda layer
that includes the libraries that you want to use in your Python code
for Lambda.
Here's a couple other examples that don't use layers if you want to include the dependency in the code package directly instead:
https://alexharv074.github.io/2018/08/18/creating-a-zip-file-for-an-aws-lambda-python-function.html
https://dev.to/razcodes/how-to-create-a-lambda-using-python-with-dependencies-4846

Basically, I fix this issue by following steps:
make virtualenv
manually add needed packages/library
prepare requirements.txt by python3 -m pip freeze > requirements.txt
deploy your code
Note:
After doing the above step, If you still facing the same issue,
Then used pyparsing==2.4.7 instead of pyparsing==3.0.6 OR updated version

Installing Python Package in AWS Lambda Layer

I have a python script which is running on my local machine and I want to move it to AWS Lambda for periodic execution. I have 3 import statements in the script for which I am adding layers but facing some issues.
from googleapiclient.discovery import build
import pandas as pd
from datetime import date
For googleapiclient api, I downloaded it in a folder and uploaded to AWS layer and lambda is able to find this module. I wanted to use this along with AWS Data wrangler package but running into layer size restriction issues. So I downloaded pandas to the same folder as google api and then uploaded the zip file to layer. But now I am get a numpy dependency error though numpy was downloaded as part of pandas install. Two folders as part of my libraries folder are numpy, numpy-1.22.0.dist-info which is correct version as per error message below. I also tried downloading numpy separately in same package but that's also not working. The Python version that I am using in Lambda is 3.7.
Please let me know if I am missing something and if this is the correct approach for installing python packages for AWS Lambda. There are some solutions available with regards to using a container for this but I am not well versed with containers. So just wanted to check if there is some fix through which I can achieve this or is containers the way to go about it.
Below is the error message:
Response { "errorMessage": "Unable to import module 'lambda_function':
Unable to import required dependencies:\nnumpy: \n\nIMPORTANT: PLEASE
READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the
numpy C-extensions failed. This error can happen for\nmany reasons,
often due to issues with your setup or how NumPy was\ninstalled.\n\nWe
have compiled some common reasons and troubleshooting tips at:\n\n
https://numpy.org/devdocs/user/troubleshooting-importerror.html\n\nPlease
note and check the following:\n\n The Python version is: Python3.7
from "/var/lang/bin/python3.7"\n The NumPy version is:
"1.22.0"\n\nand make sure that they are the versions you
expect.\nPlease carefully study the documentation linked above for
further help.\n\nOriginal error was: No module named
'numpy.core._multiarray_umath'\n", "errorType":
"Runtime.ImportModuleError", "stackTrace": [] }
Function Logs START RequestId: d......-........-.....d
Version: $LATEST ERROR Runtime.ImportModuleError: Unable to import
module 'lambda_function': Unable to import required dependencies:
numpy: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS
ISSUE! Importing the numpy C-extensions failed. This error can happen
for many reasons, often due to issues with your setup or how NumPy was
installed. We have compiled some common reasons and troubleshooting
tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html Please
note and check the following:
The Python version is: Python3.7 from "/var/lang/bin/python3.7" The
NumPy version is: "1.22.0" and make sure that they are the versions
you expect. Please carefully study the documentation linked above for
further help. Original error was: No module named
'numpy.core._multiarray_umath' Traceback (most recent call last): END
RequestId: de72............................6d REPORT RequestId:
de72............................6d Duration: 1.68 ms Billed
Duration: 2 ms Memory Size: 128 MB Max Memory Used: 55 MB Init
Duration: 570.09 ms Request ID de72............................6d
Regards, Dbeings

The issue was that I was building the pandas package on windows machine and uploading it to AWS layer. Building the package on EC2 and then putting it on AWS layer fixed the issue.