I have been trying hard to package a piece of Python code to be used on AWS Lambda. The problem is that I need the Python script developed in Notebook to be exported along with its dependencies, otherwise it causes various errors after uploading to AWS Lambda. What is the best way to go about this?
I want a .zip file with the .py file and all its dependencies
As described in your question and in comment, Apex will solve your problem. Apex lets you build, deploy, and manage AWS Lambda functions with ease.
You can read about it here and here and also watch video tutorial.
Hope it helps.
Related
Sorry in advance for my lack of proper programming lingo.
I currently have a Python 3.10 script in Jupyter Notebooks that I want to convert into some sort of packageable application for distribution to students as an educational tool. I was thinking to turn it into a container and run it through Docker Desktop, having students install Docker Desktop and distributing the container, but I am unsure of exactly how to do this, especially with specific libraries and environmental variables. Or, if there's another easier way to accomplish this, I am all ears. Thanks in advance!
Jupyter Notebooks says that it can export to HTML, which may be useful, but I'm unsure how to use this to get it to do what I want it to do.
Lambda Layer Docs for Reference: https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html
The project was written in Python 2.7, which is appears AWS Lambda has deprecated. So I've attempted to update the code to the latest release Py 3.9 and most of my errors are resolved.
Due to very poor documentation & lack of comments, I've been stuck trying to figure out which parts of this codebase go to which AWS services.
For example: this project had 3 different init.py files & one of them is even empty.
The code apparently was working before I received it.
I've no idea which pieces of this project go to which services. Some of them are obvious. Certainly the Lambda.py is supposed to go to a lambda function.
In short the project does this:
Get JSON from Webhook, perform calculation & alter JSON, convert to PDF, email PDF to customer, store PDF in S3 bucket.
The project uses a library called pypdftk or Python PDK Toolkit. Hopefully this is compatible with Py 3.9, because apparently Py 2.7 is deprecated on AWS Lambda.
It appears this project was run almost entirely as a Lambda Function, using Lambda.py as an entry point, and importing the rest the .py code files into that entry point.
Will I need to include this Python PDF Toolkit as a Lambda Layer? To my knowledge, all libraries not included in the Standard Python 3.9 library must be added as a Lambda Layer.
Additionally, is this python pdf toolkit only compatible with Python 2.7? 2.7 is no longer supported by AWS Lambda, so I have updated the code to be compatible with Py 3.9
EDIT:
These are the project files & directories.
Do I upload the entire project zipped into Lambda?
Does a portion of this need to be uploaded to a Lambda Layer?
Will I need to set-up & configure an EC2? S3? SES?
I am assuming that you need https://github.com/revolunet/pypdftk, which should be compatible with Python 3.9. This is a wrapper for the pdftk binary, which must be installed separately in your environment. Lambda functions execute in a container with Amazon Linux. The recommended way to use a native binary with Lambda is to create a layer. There are some projects in Github that generate the layer:
https://github.com/lob/lambda-pdftk-example
https://github.com/inetsys/pdftk-aws-lambda
Both of them are old, so use with caution.
I want to run a python script as part of a jenkins pipline triggered from a github repo. If I store the script directly in the repo itself, I can just do sh 'python path/to/my_package/script.py' which work perfectly. However since I want to use this from multiple pipelines from multiple repos, I want to put this in a jenkins shared library.
I found this question which suggested storing the python file in the resources directory and copying it to a temp file before use. That only works if the script is one standalone file. Unfortunately, mine is a package with multiple python files and imports between them, so thats a no go. I also tried to copy the entire folder containing the python package from the answer to this question, which suggests getting the location of the library with
import groovy.transform.SourceURI
import java.nio.file.Path
import java.nio.file.Paths
class ScriptSourceUri {
#SourceURI
static URI uri
}
but its gives me the following error:
Scripts not permitted to use staticMethod java.net.URI create java.lang.String. Administrators can decide whether to approve or reject this signature.
It seems that some additional permissions are required, which I don't think I'll be able to acquire (its a shared machine).
So? Does anyone know how I can run a python package from jenkins shared library? Right now the only solution I can think of is to manually recreate the directory structure of the python package, which is obviously very messy and non-generic.
PS: There is no particular reason for using the python script over writing the same script in groovy. Its just that the python script is well tested, well understood and well supported. Rewriting the whole thing in groovy just isn't feasible right now.
You can go to http://host:8080/jenkins/scriptApproval/ page of your Jenkins installation and approve the request for your scripts, please see below:-
And follow the link for more information.
I need a way to upgrade/downgrade the boto3 lib inside my Python 3.7 env inside Lambda.
Right now, the version is 1.9.42 inside Lambda. I cannot use certain things like Textract (boto3.client('textract'), but I can on my local machine (boto3 version 1.9.138.
So, I decided to install boto3 into a package (pip3 install boto3 -t dir/ --system) then upload it to Lambda after zipping it.
This didn't work because Lambda won't accept a package larger than 3MB (it's around 8MB)
Any other workarounds?
edit: I know I could always just write code that works and keep uploading it to Lambda, but this will become cumbersome as I'd have to include all the packages installed in the package and rebuilding it as I make changes.
The Serverless Application Model is a tool provided by AWS that lets you develop locally as it simulates the lamdba environment inside a docker container. Once you are ready you can deploy your code to a lambda and it will work as expect.
If you really want to keep editing the code in the web platform, there is a workaround by using lambda layers. You create a package with all of your dependencies and upload that to a lambda layer. Then include your layer in the lambda and just modify your own code there. As it has been pointed out in the comments this is not the way to go for real development.
I have a python project with a lot of dependencies (around 30 or so python packages) that i want to deploy on aws using lambda function. Right now i have about 30 custom python packages in my VS solution that i import into the main function - there is a lot of code. What is the best way to build a deployment package and how would i go about doing this?
I watched a few tutorials but i am new to this, so im not sure exactly what concrete steps to take. If i use something like zappa and create a virtual environment how would i then get my project there and install all the dependencies and then zip the file?
Thanks so much, sorry for the stupid questions, i couldn't find a stackoverflow post that covered this
Just go to your python environment folder and found site-package folder (usually in /lib), choose all the dependencies you need and zip them with your code.
I guess it's the easiest way.
For example, I may need beautifulsoup and urllib for dependencies, just zip them (and their dependencies, if needed) with my code, then upload to AWS Lambda, that's all.
BTW, you can also see this gist to know whether the module you need can be directly import to AWS Lambda or not.