I am trying to deploy python lambda function with serverless framework. This function need to run for 15 min (AWS Lambda Timeout). I want to simulate 100 IoT devices using AWS Lambda.
I have following code device_status.py
import os
import time
from uptime import uptime
import requests
from random import randrange
from configparser import ConfigParser, ExtendedInterpolation
class DeviceStatus:
def __init__(self):
self.config_file = 'config.ini'
self.config_dict = None
self.read_device_config()
self.dr_ins = DeviceRegistration(self.config_dict)
....
if __name__ == '__main__':
init_ds = DeviceStatus()
status_interval = init_ds.config_dict['status']['interval']
while True:
init_ds.send_device_status()
time.sleep(int(status_interval))
and serverless.yml
service: lambda-device
plugins:
- serverless-python-requirements
provider:
name: aws
runtime: python3.6
region: us-east-1
functions:
lambda-device:
handler: main.device_status
when I try to invoke it I get "errorMessage": "Unable to import module 'main'"
How to refer to main function in serverless.yml ?
The error message that you are receiving is saying that there is no main.py file in your serverless structure.
Referring to your serverless.yml:
functions:
lambda-device:
handler: main.device_status
The explanation from the above section is that you have a serverless-function that is named lambda-device which is having a structure with a filename main.py that in its definition would require to have a method:
def device_status(event, context):
# TODO
pass
So make sure you have main.py file with a method device_status(event, context)
Related
I am using the below azure function which takes http trigger as input.
import logging
import azure.functions as func
from . import test
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
name = req.params.get('name')
strng = req.params.get('strng')
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
return func.HttpResponse(f"Hello, {name}. This HTTP triggered function executed successfully.sum = {test.testfunc(strng)}")
else:
return func.HttpResponse(
"This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.",
status_code=200
)
Below is the test.py file which I imported in the init.py.
import json
import pandas as pd
from pandas import DataFrame
from azure.storage.blob import AppendBlobService
from datetime import datetime
def testfunc(strng):
# return strng
API = json.loads(strng)
test = pd.json_normalize(parse, record_path='Vol', meta=['studyDate'])
df = pd.DataFrame(test)
df["x"] = df["Vol"] * 2
df["y"] = df["Vol"] * 50
df1 = df[['Date', 'x', 'y']]
df2 = df1.to_json(orient='records')
append_blob_service = AppendBlobService(account_name='actname',
account_key='key')
date = datetime.now()
blobname = f"test_cal{date}.json"
append_blob_service.create_blob('container', blobname, if_none_match="*")
append_blob_service.append_blob_from_text('container', blobname, text=df2)
return df2
The above function works when I run it in pycharm and databricks . But when I run the Azure function via visuals studio code, I get the below error.
Exception: ImportError: cannot import name 'AppendBlobService' from 'azure.storage.blob'
I have tried below and still get the same error.
pip install azure-storage --upgrade
pip install azure-storage-blob
Kindly provide assistance with the error.
Is there any other way I can save the df2 variable to Azure storage.
Thank you.
According to Document it says,
If the module (for example, azure-storage-blob) cannot be found, Python throws the ModuleNotFoundError. If it is found, there may be an issue with loading the module or some of its files. Python would throw a ImportError in those cases.
Try setting python to environment variable FUNCTIONS_WORKER_RUNTIME or
Try adding azure.core to your requirements.txt file.
Taken References from:
ImportError: cannot import name 'BlockBlobService' from 'azure.storage.blob'
Azure Functions fails running when deployed
The current version library contains the blobserviceclient instead of the blockblockblockservice. This works for me by using version 2.1.0 can solve it:
pip install azure-storage-blob==2.1.0
I came across this python class ResourcesMoveInfo for moving resources(Azure Images) from one subscription to another with Azure python SDK.
But it's failing when I use it like below:
Pattern 1
reference from https://buildmedia.readthedocs.org/media/pdf/azure-sdk-for-python/v1.0.3/azure-sdk-for-python.pdf
Usage:
metadata = azure.mgmt.resource.resourcemanagement.ResourcesMoveInfo(resources=rid,target_resource_group='/subscriptions/{0}/resourceGroups/{1}'.format(self.prod_subscription_id,self.resource_group))
Error:
AttributeError: module 'azure.mgmt.resource' has no attribute 'resourcemanagement'
Pattern 2
reference from - https://learn.microsoft.com/en-us/python/api/azure-mgmt-resource/azure.mgmt.resource.resources.v2019_07_01.models.resourcesmoveinfo?view=azure-python
Usage:
metadata = azure.mgmt.resource.resources.v2020_06_01.models.ResourcesMoveInfo(resources=rid,target_resource_group='/subscriptions/{0}/resourceGroups/{1}'.format(self.prod_subscription_id,self.resource_group))
Error:
AttributeError: module 'azure.mgmt.resource.resources' has no attribute 'v2020_06_01'
Any help on this requirement/issue would be helpful. Thanks!
Adding code snippet here:
import sys
import os
import time
from azure.common.credentials import ServicePrincipalCredentials
from azure.mgmt.resource import ResourceManagementClient
import azure.mgmt.resource
#from azure.mgmt.resource.resources.v2020_06_01.models import ResourcesMoveInfo
from azure.identity import ClientSecretCredential
from cred_wrapper import CredentialWrapper
class Move():
def __init__(self):
self.nonprod_subscription_id = "abc"
self.prod_subscription_id = "def"
self.credential = ClientSecretCredential(
client_id= os.environ["ARM_CLIENT_ID"],
client_secret= os.environ["ARM_CLIENT_SECRET"],
tenant_id= os.environ["ARM_TENANT_ID"]
)
#resource client for nonprod
self.sp = CredentialWrapper(self.credential)
self.resource_client = ResourceManagementClient(self.sp,self.nonprod_subscription_id)
self.resource_group = "imgs-rg"
def getresourceids(self):
resource_ids = list(resource.id for resource in self.resource_client.resources.list_by_resource_group("{0}".format(self.resource_group)) if resource.id.find("latest")>=0)
return resource_ids
def getresourcenames(self):
resource_names = list(resource.name for resource in self.resource_client.resources.list_by_resource_group("{0}".format(self.resource_group)) if resource.id.find("latest")>=0)
return resource_names
def deleteoldimages(self,name):
#resource client id for prod
rc = ResourceManagementClient(self.sp,self.prod_subscription_id)
for resource in rc.resources.list_by_resource_group("{0}".format(self.resource_group)):
if resource.name == name:
#2019-12-01 is the latest api_version supported for deleting the resource type "image"
rc.resources.begin_delete_by_id(resource.id,"2020-06-01")
print("deleted {0}".format(resource.name))
def moveimages(self):
rnames = self.getresourcenames()
for rname in rnames:
print(rname)
#self.deleteoldimages(rname)
time.sleep(10)
rids = self.getresourceids()
rid = list(rids[0:])
print(rid)
metadata = azure.mgmt.resource.resources.v2020_06_01.models.ResourcesMoveInfo(resources=rid,target_resource_group='/subscriptions/{0}/resourceGroups/{1}'.format(self.prod_subscription_id,self.resource_group))
#moving resources in the rid from nonprod subscription to prod subscription under the resource group avrc-imgs-rg
if rid != []:
print("moving {0}".format(rid))
print(self.resource_client.resources.move_resources(source_resource_group_name="{0}".format(self.resource_group),parameters=metadata))
#self.resource_client.resources.move_resources(source_resource_group_name="{0}".format(self.resource_group),resources=rid,target_resource_group='/subscriptions/{0}/resourceGroups/{1}'.format(self.prod_subscription_id,self.resource_group))
#self.resource_client.resources.begin_move_resources(source_resource_group_name="{0}".format(self.resource_group),parameters=metadata)
if __name__ == "__main__":
Move().moveimages()
From your inputs we can see that the code looks fine. From your error messages, the problem is with importing the modules.
Basically when we import a module few submodules will get installed along with and few will not. This will depend on the version of the package, to understand which modules are involved in a specific version we need to check for version-releases in official documentation.
In your case, looks like some resource modules were missing, if you could see the entire error-trace, there will be a path with sitepackages in our local. Try to find that package and its subfolder(Modules) and compare them with Azure SDK for Python under Resource module, you can find this here.
In such situation we need to explicitly add those sub modules under our package. In your case you can simple download the packaged code from Git link which I gave and can merge in your local folder.
I am new to AWS lambda function and i am trying to add my existing code to AWS lambda. My existing code looks like :
import boto3
import slack
import slack.chat
import time
import itertools
from slacker import Slacker
ACCESS_KEY = ""
SECRET_KEY = ""
slack.api_token = ""
slack_channel = "#my_test_channel"
def gather_info_ansible():
.
.
def call_snapshot_creater(data):
.
.
def call_snapshot_destroyer(data):
.
.
if __name__ == '__main__':
print "Calling Ansible Box Gather detail Method first!"
ansible_box_info = gather_info_ansible()
print "Now Calling the Destroyer of SNAPSHOT!! BEHOLD THIS IS HELL!!"
call_snapshot_destroyer(ansible_box_info)
#mapping = {i[0]: [i[1], i[2]] for i in data}
print "Now Calling the Snapshot Creater!"
call_snapshot_creater(ansible_box_info)
Now i try to create a lambda function from scratch on AWS Console as follows (a hello world)
from __future__ import print_function
import json
print('Loading function')
def lambda_handler(event, context):
#print("Received event: " + json.dumps(event, indent=2))
print("value1 = " + event['key1'])
print("value2 = " + event['key2'])
print("value3 = " + event['key3'])
print("test")
return event['key1'] # Echo back the first key value
#raise Exception('Something went wrong')
and the sample test event on AWS console is :
{
"key3": "value3",
"key2": "value2",
"key1": "value1"
}
I am really not sure how to put my code in AWS lambda coz if i even add the modules in lambda console and run it it throws me error :
Unable to import module 'lambda_function': No module named slack
How to solve this and import my code in lambda?
You have to make a zipped package consisting of your python script containing the lambda function and all the modules that you are importing in the python script. Upload the zipped package on aws.
Whatever module you want to import, you have to include that module in the zip package. Only then the import statements will work.
For example your zip package should consist of
test_package.zip
|-test.py (script containing the lambda_handler function)
|-boto3(module folder)
|-slack(module folder)
|-slacker(module folder)
You receive an error because AWS lambda does not have any information about a module called slack.
A module is a set of .py files that are stored somewhere on a computer.
In case of lambda, you should import all your libraries by creating a deployment package.
Here is an another question that describes similar case and provides several solutions:
AWS Lambda questions
In my main.py have the below code:
app.config.from_object('config.DevelopmentConfig')
In another module I used import main and then used main.app.config['KEY'] to get a parameter, but Python interpreter says that it couldn't load the module in main.py because of the import part. How can I access config parameters in another module in Flask?
Your structure is not really clear but by what I can get, import your configuration object and just pass it to app.config.from_object():
from flask import Flask
from <path_to_config_module>.config import DevelopmentConfig
app = Flask('Project')
app.config.from_object(DevelopmentConfig)
if __name__ == "__main__":
application.run(host="0.0.0.0")
if your your config module is in the same directory where your application module is, you can just use :
from .config import DevelopmentConfig
The solution was to put app initialization in another file (e.g: myapp_init_file.py) in the root:
from flask import Flask
app = Flask(__name__)
# Change this on production environment to: config.ProductionConfig
app.config.from_object('config.DevelopmentConfig')
Now to access config parameters I just need to import this module in different files:
from myapp_init_file import app
Now I have access to my config parameters as below:
app.config['url']
The problem was that I had an import loop an could not run my python app. With this solution everything works like a charm. ;-)
We have an ETL data API repo. We do all etl processing inside of it, then spit the data out in API's. These API's are run one at a time from a single command passing the resource class through to the server to build an API. the resource class is in a web directory in an __init__.py.
This is a wonderful convention and quite simple to use, but the problem I am having is coming from trying to get one of the 3 API's available spun up for testing. Our directory stucture is like this (calling the project 'tomato')
tomato
- category_api
- web
- etl
- test
- here is where we are writing some tests (test_category_api.py)
- data
- article_api
- web
- etl
- test
- data
- recommendation_api
- web
- etl
- test
- data
- common
- common shit
Inside this test, I have the following test class. On the seventh line up from the bottom,
you will see a comment on where it breaks. It is the import_module method.
import unittest
import sys
import os
import sys
import json
from importlib import import_module
from flask import Flask
from flask_restful import Api, abort, wraps
from flask_restful.utils import cors
from flask.ext.testing import TestCase
#dir_above_top_level = os.path.join(os.path.abspath(__file__), '../../.. /')
#sys.path.append(os.path.abspath(dir_above_top_level))
_CATEGORY_ENDPOINT = '/v1/category/'
_PACKAGE_NAME = os.environ['RECO_API_PACKAGE_NAME']
_CORS = cors.crossdomain(origin='*',
headers=['Origin', 'X-Requested-With',
'Content-Type', 'Accept'],
methods=['GET', 'HEAD', 'OPTIONS'],
max_age=3600)
class CategoryTests(TestCase):
def __init__(self):
self.app = Flask(__name__)
self._configure_app()
for resource in self.resource_classes:
self.api.add_resource(self.resource,
self.resource.URI_TEMPLATE)
def test_status_code(self):
self.response = self.client.post(_CATEGORY_ENDPOINT,
data=json.dumps(
{'title': 'Enjoy this delicious food'}),
headers=json.dumps(
{'content-type':'application/json'}))
self.assertEqual(self.response.status_code, 200)
def test_version(self):
self.response = self.client.post(_CATEGORY_ENDPOINT,
data=json.dumps(
{"title": "eat some delicious stuff"}),
headers=json.dumps(
{'content-type':'application/json'}))
self.assertEqual(json.dumps(self.response['version']), '1')
def _configure_app(self):
self.app = Flask(__name__)
self.app.config['TESTING'] = True
self.app.debug = True
self.decorators = [_CORS]
self.app.Threaded = True
self.web_package = 'tomato.category.web'
self.package = import_module('.__init__', self.web_package) # WE BREAK HERE
self.resources = package.RESOURCE_NAMES
self.resource_classes = [ getattr(package, resource) for resource in resources ]
self.api = Api(self.app, catch_all_404s=True, decorators=self.decorators)
if __name__ == '__main__':
unittest.main()
we are given an exception when running these tests:
ImportError: No module named tomato.category.web.__init__
yet cd into the main top dir, and ls tomato/category/web gets us __init__.py and its right there with the resource class.
How do I import this class so that I can instantiate the API to run the tests in this class?
Or if I'm completely on the wrong track what should I be doing instead?
You don't need to import __init__, just like you probably wouldn't do from tomato.category.web import __init__. You should be able to import the web package directly.
self.web_package = 'tomato.category.web'
self.package = import_module(self.web_package)
The problem here lies in the directory structure. In the current path, I am not at the top level. It is a module. So What was needed was to uncomment the line two lines at the top, and change the structure to append the path like this.
dir_above_top_level = os.path.join(os.path.abspath(__file__), '../../../..')
sys.path.append(os.path.abspath(dir_above_top_level))
and now, I can import it using
self.web_package = 'tomato.category.web'
self.package = import_module('.__init__', self.web_package)
and now it will import fine and I can grab the resource class to set up the testing API