I´ve created a set up using python 3.4 code (as I need the library pyrebase) as well as using Cronjobs.
AWS Lambda would do the job perfect if it wasn´t for the lack of Python 3-support.
Any ideas of other AWS computer services supporting Python and Cronjobs? (or other cloud computing services that would support the project?)
The environment that Lambda runs your scripts in has Python 3 available to it, so it is possible to run Python 3 code via Lambda, but it takes a bit of work (I've not done it myself). Here's a similar question: using Python 3 with AWS lamba
And the a link from the answer in that question with some good info: http://www.cloudtrek.com.au/blog/running-python-3-on-aws-lambda/
Basically you'd have a small chunk of Python 2.7 code with the lambda handler function, which then creates a Python 3 virtualenv in which to run your Python 3 code.
As for cronjobs, you could probably put something together with timers in SWS: http://docs.aws.amazon.com/amazonswf/latest/developerguide/swf-dg-timers.html . If it has to be literal cronjobs (vs. running something on a schedule, regardless of method), you'd need an EC2 instance running cron and calling out to other services.
Related
I got a question about python, maven and aws lambda. Basically I am trying to build dependency trees for my repos using the terminal command
mvn dependency:tree
This command is being ran via python using the os library, i.e.
import os
os.system('mvn dependency:tree')
Now comes the issue - I need to run this on AWS Lambda.
Being aware that AWS Lambda is serverless and that the layers of each lambda can only be 250mb, 1) is it possible to run terminal commands via lambda without spinning up any sort of server? and 2) maven usually needs to be installed on a system, thus is it possible, or even viable, to run maven on AWS Lambda?
Any input will be appreciated.
Thanks
is it possible to run terminal commands via lambda without spinning up any sort of server?
Yes, you can run terminal commands in a Lambda function.
maven usually needs to be installed on a system, thus is it possible, or even viable, to run maven on AWS Lambda?
You can create a custom Lambda container image that includes dependencies.
Additional AWS Blog Post: https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/
I have some python libraries that I need to invoke in NodeJS on a Lambda function. I need to do this because some of the python functions are doing external API calls and may take a while to finish, and using NodeJS speeds this up quite a lot thanks to promises.
I have read it is possible to crate custom runtimes as Layers but canont find some samples on NodeJS 12 + Python 3.7 for example, so how to do that? is there a list of already published and available runtimes somewhere?
I think the newly announced option to run Docker images in AWS Lambda might be the best solution here.
You can use either the Python or the NodeJS base image provided by Amazon and then install the rest of the required dependencies. Put it into AWS ECR and then run your Lambda using the Docker image.
Check the news article.
I have a python etl project that I was to run on a schedule. I am a bit lost on where to start. I have seen some tutorials using Heroku and AWS Lambda, but these were all single script files. My main script references multiple packages that are all in the same project directory. Am I able to deploy the entire project and have it run the main script on a schedule? If so, what tools/services should I be looking at?
See Lambda Scheduled Events. You can create a Lambda function and direct AWS Lambda to execute it on a regular schedule. You can specify a fixed rate (for example, execute a Lambda function every hour or 15 minutes), or you can specify a Cron expression.
Be aware of the Lambda package size limits.
I have 3 python scripts perfectly working individually on my EC2 machine, now i want all 3 of them to run one after the other automatically using AWS Step Functions. How can i possibly do that? I have done my part on the research going through almost all the official AWS Documentations, but couldn't find a thing that could help me out.
The scripts must be adapted to work with AWS Step functions, as describe in FAQ:
Q: How does AWS Step Functions work with Amazon EC2 and other compute
resources?
All work in your state machine is done by tasks. A task may be an
Activity, which can consist of any code in any language. Activities
can be hosted on Amazon EC2, Amazon ECS, mobile devices—basically any
computer that can communicate with the AWS Step Functions API.
Activities long-poll Step Functions using API calls to request work,
receive input data, do the work, and return a result.
An example in Ruby is on this page: Example Activity Worker in Ruby:
For Python you can try to use boto3
Can Google Cloud Functions handle python with packages like sklearn, pandas, etc? If so, can someone point me in the direction of resources on how to do so.
I've been searching a while and it seems like this is impossible, all I've found are resources to deploy the base python language to google cloud.
Python 3.7 is supported now.
Steps to create one via the google cloud console:
go to google cloud functions in the google cloud console and click on create function
2.specify the function's properties
select trigger
4.change runtime to python 3.7
enter your cloud function logic and entry point
enter python dependencies in requirements.txt
EDIT: As of July 2018 there is now a Python runtime (3.7) available for Google Cloud Functions!
OLD ANSWER: Google Cloud Functions (GCF) are written in JavaScript (executed in a Node.js runtime), so there is no way for them to actually handle Python at this moment. There is a Python module at GitHub that you might have come across and it can be used to write and deploy GCF with one of three trigger types: http, Pub/Sub and bucket. The module takes care of translating your Python logic to a JavaScript code that is later run inside Google Cloud Platform.
When it comes to other packages like pandas, the ‘translation’ into JavaScript was not prepared for them by anyone AFAIK. If you really don’t like the idea of jumping into JavaScript and writing the Cloud Function code on your own (with the logic you intended to use in a Python script), you have a possible workaround. You can evoke your Python script from inside of the Cloud Function written in JS - the idea was discussed in this topic. Another way is using Object Change Notifications or Pub/Sub Notifications as explained here.
As of 19th July 2018, Google Cloud Functions supports Python 3.7.
Kindly check the Runtime environment to find the Python 3.7 runtime and sample script (based on Flask) .
--UPDATED--
Official Documentation for the Google Cloud Functions - Python 3.7 support Beta Release.
This is a beta release of the Python runtime for Google Cloud
Functions. This feature might be changed in backward-incompatible ways
and is not subject to any SLA or deprecation policy.
SkLearn, Numpy is supported in Google Cloud function. Also I've run a sample test to confirm the availability of Pandas as well and its working fine.
https://github.com/mkanchwala/google-functions-python-example
Hope this helps to all the "Py" lovers.
You can use AWS lambda as well if you want to work around and still use Python as your main language. Some modules/packages will need to be imported via zip file with AWS Lambda but it has a broader range of usable languages than GCF