Develop Reference

Python Azure Functions: arcgis package has egg-info instead of dist-info in Lib\site-packages

I am testing an Azure Durable Function with the following requirements.txt:
azure-functions
azure-functions-durable
datetime
requests==2.23.0
arcgis==1.8.0.post1
openpyxl==3.0.3
aiohttp==3.7.3
numpy
When I debug it using VS Code, the venv is created adding all the relevant packages, but the Terminal complains saying Exception: ModuleNotFoundError: No module named 'arcgis' despite this module being in the requirements file.
When I check the Troubleshooting Guide at https://aka.ms/functions-modulenotfound, one of the possible reasons they mention is:
However, when I check my venv, there is no dist-info folder, but only:
arcgis-1.8.0.post1-py3.9.egg-info.
Is there a way I can install the wheel instead?

Cannot import name 'cygrpc' from 'grpc._cython' - Google Ads API

I want to deploy working python project in pycharm to aws lambda. The project is using google-ads library to get some report data from google ads.
I tried deploying lambda by importing complete project as a zip file by zipping all the folders/files inside the project and not the project folder itself. But i got the following error:
{
"errorMessage": "Unable to import module 'main': cannot import name 'cygrpc' from 'grpc._cython' (/var/task/grpc/_cython/__init__.py)",
"errorType": "Runtime.ImportModuleError",
"stackTrace": []
}
Assuming that google-ads library is working and that something is wrong with grpc(btw google-ads includes grpcio and stuff on its own), i tried to create a layer for grpcio, cython, cygrpc but the error remains same.
I create projects/layers in aws lambda and they work. I dont know what i am doing wrong here.
Any help would be really appreciated!
versions: google-ads-14.1.0, python-3.9, grpcio-1.43.0

Answering my own question after a lot of workaround. I have made it generic so anyone can use it.
I believe you can fix any type of ImportModuleError as long as your deployment package's file structure, your code and architecture is ok, only then you can deploy and run your code successfully. To fix your structure and architecture, follow steps below:
1- Install "ubuntu 18.04 LTS" from microsoft store (Windows 10).
2- Open CMD and run following commands:
ubuntu1804
Enter password or create user if asked.
cd /mnt/c You can choose any of your drive. I chose C.
mkdir my-lambda-folder Create project folder.
cd my-lambda-folder Enter into project folder.
touch lambda_function.py Create file called lambda_function.py
Now copy and paste your code into file you just created i.e lambda_function.py
pip install --target ./package your-module-name
For Example: pip install --target ./package google-ads will install
google-ads module inside folder 'package'. The folder 'package' will be
created automatically if not found.
cd package
zip -r ../my-deployment-package.zip . This will create deployment package with the installed library at the root of your project folder i.e my-lambda-folder.
cd .. go back to the root of your project folder.
zip -g my-deployment-package.zip lambda_function.py Add your lambda function to the deployment package you just created i.e my-deployment-package.zip.
(Optional) In my case i was using google-ads and to run my code i needed google-ads.yaml file too in my deployment package. So i ran additional command zip -g my-deployment-package.zip google-ads-yaml (i already pasted this file in my project folder).
3- Upload my-deployment-package.zip to your lambda function in AWS console and you are good to go.

For me, it worked just by downloading the packages with pip on ubuntu on docker and packing and uploading them on AWS.

How to import lxml from precompiled binary on AWS Lambda?

I'm trying to import the lxml library in Python to execute an AWS Lambda function but I'm getting the following error: [ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'lxml'. To solve this, I followed the recommendation from this SO answer and used precompiled binaries from the following repo.
I used the lxml_amazon_binaries.zip file from that repo, which has this structure:
lxml_amazon_binaries
├── lxml
└── usr
I uploaded the entire zip file to an AWS Lambda layer, created a new Lambda function, and tested with a simple from lxml import etree, which led to the above error.
Am I uploading/using these binaries correctly? I'm not sure what caused the error. Using different Python runtimes didn't help.

The most reliable way to create lxml layer is using Docker as explain in the AWS blog. Specifically, the verified steps are (executed on Linux, but windows should also work as long as you have Docker):
Create empty folder, e.g. mylayer.
Go to the folder and create requirements.txt file with the content of
lxml
Run the following docker command:
The command will create layer for python3.8:
docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"
Archive the layer as zip:
zip -9 -r mylayer.zip python
Create lambda layer based on mylayer.zip in the AWS Console. Don't forget to specify Compatible runtime to python3.8.
Add the the layer created in step 5 to your function.
I tested the layer using your code:
from lxml import etree
def lambda_handler(event, context):
root = etree.Element("root")
root.append( etree.Element("child1") )
print(etree.tostring(root, pretty_print=True))
It works correctly:
b'<root>\n <child1/>\n</root>\n'

Unable to import module 'lambda_function': No module named *

I am trying to run a python lambda function that uses additional packages. However whenever I upload the .zip file to the lambda console I get the error:
{
"errorMessage": "Unable to import module 'lambda_function': No module named '*'",
"errorType": "Runtime.ImportModuleError"
}
I followed these instructions: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-dependencies which told me to make sure my packages were in a directory local to my lambda function:
~/my-function$ pip install --target ./package Pillow
I am not using Pillow. This is sample code from their site. Nor am I using a package that you can access on Lambda already. It is one that I have got from github and need to attach to my app.
At first this didnt work so I created a setup.cfg file and added in:
[install]
prefix=
Now, when I use the pip command to install to the target, it works (and also adds loads of other folders other than my package but I assumed they were needed so I left them there.
When I go into the directory, the package is there.
I then found this answer: https://stackoverflow.com/a/12493244/5675125 which suggested perhaps some hidden files were not being included and this is how I should zip them.
Again the same error.
How do I get lambda to recognise that my package is there.

If you require Pillow, the easiest way to use it in your function is through a popular repository with a public layers (including pillow) such as keithrozario /
Klayers on github. To use it, you would locate ARN of the layer based on your region. The list of the ARNs for python 3.8 is here.
For example, for us-east-1 the layer added for Python 3.7 would be:
Update
I just created the custom layer with instabot and can confirm that it works.
The technique used includes docker tool described in the recent AWS blog:
How do I create a Lambda layer using a simulated Lambda environment with Docker?
Thus for this question, I verified it as follows:
Create empty folder, e.g. mylayer.
Go to the folder and create requirements.txt file with the content of
instabot
Run the following docker command:
docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"
Remove numpy
The instabot requires numpy which is very large. So we remove it manually,
before
creating a layer. We are going to use numpy layer provided by AWS instead.
sudo rm -rvf ./python/lib/python3.8/site-packages/numpy*
If we don't remove numpy, the layer will be >50MB.
Create layer as zip:
zip -9 -r mylayer.zip python
Create lambda layer based on mylayer.zip in the AWS Console.
Don't forget to specify Compatible runtimes to python3.8.
Add two layers to your function:
The first one is AWSLambda-Python38-SciPy1x provided by AWS with numpy,
while the second one is the one we created above. So
your function will use two layers.
Test the layer in lambda using the following lambda function:
import json
from instabot import Bot
def lambda_handler(event, context):
# TODO implement
bot = Bot(base_path='/tmp')
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}

How to package Scrapy dependency to lambda?

I am writing a python application which dependents on Scrapy module. It works fine locally but failed when I run it from aws lambda test console. My python project has a requirements.txt file with below dependency:
scrapy==1.6.0
I packaged all dependencies by following this link: https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html. And also, I put my source code *.py at the root level of in the zip file. My package script can be found https://github.com/zhaoyi0113/quote-datalake/blob/master/bin/deploy.sh.
It basically does two things, first run command pip install -r requirements.txt -t dist to download all dependencies to dist directory. second, copy app python source code to dist directory.
The deployment is done via terraform and below is the configuration file.
provider "aws" {
profile = "default"
region = "ap-southeast-2"
}
variable "runtime" {
default = "python3.6"
}
data "archive_file" "zipit" {
type = "zip"
source_dir = "crawler/dist"
output_path = "crawler/dist/deploy.zip"
}
resource "aws_lambda_function" "test_lambda" {
filename = "crawler/dist/deploy.zip"
function_name = "quote-crawler"
role = "arn:aws:iam::773592622512:role/LambdaRole"
handler = "handler.handler"
source_code_hash = "${data.archive_file.zipit.output_base64sha256}"
runtime = "${var.runtime}"
}
It zip the directory and upload the file to lambda.
I found I get the runtime error in lambda Unable to import module 'handler': cannot import name 'etree' when there is a statement import scrapy. I didn't use etree in my code so I believe there is something used by scrapy.
My source code can be found at https://github.com/zhaoyi0113/quote-datalake/tree/master/crawler. There are only two simple python files.
It works fine if I run them locally. The error only appears in lambda. Is there a different way to package scrapy to lambda?

Based on the communication with Tim, the issue is caused by incompatible library versions between local and lambda.
The easiest way to resolve this issue is to use the docker image lambci/lambda to build a package with the command:
$ docker run -v $(pwd):/outputs -it --rm lambci/lambda:build-python3.6 pip install scrapy -t /outputs/

You need to provide the entire dependency tree, scrapy also has a set of dependencies (and they may also have dependencies).
The easiest way to download all the required dependencies is to use pip
$ pip -t packages/ install scrapy
This will download scrapy and all its dependencies into the folder packages.
Scrapy has lxml and pyOpenSSL as dependencies that include compiled components. Unless they are statically compiled they will likely require that the c-libraries they require are also installed on the lambda VM.
From the lxml documentation it requires:
libxml2 version 2.9.2 or later.
libxslt version 1.1.27 or later.
We recommend libxslt 1.1.28 or later.
Maybe try adding installation of these to your deploy script. You should be able to use (I'm making a guess at the package names) yum -y install libxml2 libxslt
Another good idea is to test your scripts on an Amazon Linux EC2 instance as this is close to the environment that Lambda executes in.