I've built a web scraper in python using seleniumwire that runs fine on my local machine. I've adjusted it for AWS Lambda, but when I went to deploy I received lots of various errors with components that seleniumwire needs to run. After a lot of troubleshooting and creating lambda packages I've gotten stuck at this error:
[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named '_cffi_backend'
Traceback (most recent call last):
I know that Lambda cannot support the arm architecture and I've been using docker to generate the required packages for this, however am stuck with this, as creating and using the package that contains the cffi module, still doesn't get me through.
So we come to my ask:
I'd like to create an environment on either my machine, or on an EC2 machine, or any other suggestions are welcome. Where I can have the same OS/environment that Lambda uses, that I can ssh/connect into, and 1. run my code to ensure it's working. 2. create a deployment package from that machine.
I really appreciate you reading this, and am grateful for your feedback.
I ended up creating an EC2 instance with AWS-Linux.
I then followed the instructions found here to build the layers needed for my code within the Linux environment. These instructions were amazing. Ultimately the issue for the layers was
Folder structure
Compiling on ARM architecture.
Experiment adding your dependencies as layers.
Here is a video of how you can create layers to add libraries and other dependencies on your lambda function.
Create layer lambda function
Related
I am working on aws lambda function, i install the package but i got error
Unable to import module 'lambda_function': cannot import name 'cygrpc' from 'grpc._cython' (/var/task/grpc/_cython/init.py)
How to solve this type of error?
Are you trying to import it in code without haveing it installed on the lambda container? If you gonna use none std libs you should have you function and its libs installed locally and when zip it and upload it to Lambda.
Docs:
https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-package.html
In addition to an answer marked correct I wanted to mention that for libraries using C-extensions is important that the layer was built on the same type of system as it runs in lambda. For example, if you're trying to build the layer including grpc on MacOS most likely it won't work since lambda is running on Linux. To make the layer working you have to build it in a Linux system (for example, on the virtual machine or in Docker)
I have followed all the steps in the documentation:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
create a directory.
Save all of your Python source files (the .py files) at the root level of this directory.
Install any libraries using pip at the root level of the directory.
Zip the content of the project-dir directory)
But after I uploaded the zip-file to lambda function, I got the error message when I test the script
my code:
import psycopg2
#my code...
the error:
Unable to import module 'myfilemane': No module named 'psycopg2._psycopg'
I don't know where is the suffix '_psycopg' from...
Any help regarding this?
You are using native libraries with lambda. We had this similar problem and here is how we solved it.
Spin a machine with AWS supported AMI that runs your real lambda.
https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
As this writing, it is,
AMI name: amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2
Full documentation in installing native modules your python lambda.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
Install the required modules required for your lambda,
pip install module-name -t /path/to/project-dir
and prepare your package to upload along with the native modules under lambda ami environment.
Hope this helps.
I believe this is caused because psycopg2 needs to be build an compiled with statically linked libraries for Linux. Please reference Using psycopg2 with Lambda to Update Redshift (Python) for more details on this issue. Another [reference][1] of problems of compiling psycopg2 on OSX.
There are a few solutions, but basically it comes down to installing the library on a Linux machine and using that as the Psycopg2 Library in your upload package.
So my team at work had put together a simple python service and had a dockerfile that performs some small tasks like pip installing dependencies, creating a user, rewiring some directories etc.
Everything worked fine locally -- we were able to build and run a docker image locally on my laptop (I have a MacBook Pro, running High Sierra if that is relevant).
We attempted to build the project on Openshift and kept getting a "Generic Build error" which told us to check the logs.
The log line it was failing on was
RUN pip3 install pandas flask-restful xgboost scikit-learn nltk scipy
and there was no related error listed with it. It literally just stopped at that point.
Specifically, it was breaking out when it got to installing xgboost. We removed the xgboost part and the entire Dockerfile ran fine and the build completed successfully so we know it was definitely just that one install causing the issue. Has anyone else encountered this and know why it happens?
We are pretty certain it wasn't any kind of memory or storage issue as the container had plenty of room for how small the service was.
Note: We were able to eventually use a coworker's template image to get our project up with the needed dependencies, just curious why this was happening in case we run into a similar issue in the future
Not sure if this is an option for you, but when I need to build from a Dockerfile, I usually use a hosted service like Quay.io or DockerHub.
I've been working recently on deploying a machine learning model as a web service. I used Azure Machine Learning Studio for creating my own Workspace ID and Authorization Token. Then, I trained LogisticRegressionCV model from sklearn.linear_model locally on my machine (using python 2.7.13) and with the usage of below code snippet I wanted to publish my model as web service:
from azureml import services
#services.publish('workspaceID','authorization_token')
#services.types(var_1= float, var_2= float)
#services.returns(int)
def predicting(var_1, var_2):
input = np.array([var_1, var_2].reshape(1,-1)
return model.predict_proba(input)[0][1]
where input variable is a list with data to be scored and model variable contains trained classifier. Then after defining above function I want to make a prediction on sample input vector:
predicting.service(1.21, 1.34)
However following error occurs:
RuntimeError: Error 0085: The following error occurred during script
evaluation, please view the output log for more information:
And the most important message in log is:
AttributeError: 'module' object has no attribute 'LogisticRegressionCV'
The error is strange to me because when I was using normal sklearn.linear_model.LogisticRegression everything was fine. I was able to make predictions sending POST requests to created endpoint, so I guess sklearn worked correctly.
After changing to LogisticRegressionCV it does not.
Therefore I wanted to update sklearn on my workspace.
Do you have any ideas how to do it? Or even more general question: how to install any python module on azure machine learning studio in a way to use predict functions of any model I develpoed locally?
Thanks
For anyone who came across this question like I did in hopes of installing modules in AzureML notebooks; it seems the current environments sit on Conda on the compute so it's now as simple as executing
!conda env list
# conda environments:
#
base * /anaconda
azureml_py36 /anaconda/envs/azureml_py36
!conda -n azureml_py36 -y <packages>
from within the notebook environment or doing pretty much the same without the ! in the terminal environment
For installing python module on Azure ML Studio, there is a section Technical Notes of the offical document Execute Python Script which introduces it.
The general steps as below.
Create a Python project via virtualenv and active it.
Install all packages you want via pip on the virtual Python environment, and then
Package all files and directorys under the path Lib\site-packages of your project as a zip file.
Upload the zip package into your Azure ML WorkSpace as a dataSet.
Follow the offical document to import Python Module for your Execute Python Script.
For more details, you can refer to the other similar SO thread Updating pandas to version 0.19 in Azure ML Studio, it even introduced how to update the version of Python packages installed by Azure.
Hope it helps.
I struggled with the same issue: error 0085
I was able to resolve it by using Azure ML code example available from their library:
Deployment of AzureML Web Services from Python Notebooks
can be found at https://gallery.cortanaintelligence.com/Notebook/Deployment-of-AzureML-Web-Services-from-Python-Notebooks-4
I won't copy the whole code here, but I used it exactly as is and it worked with Boston dataset. Then I used it with my dataset, and I no longer got error 0085. I haven't tracked down the error yet but it's most likely due to some misbehaving character or indent. Hope this helps.
I am having a hard time building webrtc-native on ubuntu. The Build instructions for webrtc are wayward and poor. I am following https://webrtc.org/native-code/development/.
Figured that I need to install the depot tools and run fetch v8. Then I did gclient sync. Did apt-get install ninja. Now I am stuck at python webrtc/build/gyp_webrtc. I get the following error
Traceback (most recent call last): File "webrtc/build/gyp_webrtc",
line 23, in
import gyp_chromium ImportError: No module named gyp_chromium
I badly need help here. Also, it seems that the community doesn't care much about helping to build native apps using webrtc. The focus is on upstream/Web APIs.
Please check https://github.com/HefnySco/node-webrtc-native
I hade very hard time. I used build-webrtc with some tweaks all listed on the above link.
The linked project goes one step forward and build a node component after building the webrtc lib.
WebRTC recently moved from gyp to gn as the default method to generate ninja files.
You should find the process to get started easier here: https://webrtc.org/native-code/development/