Develop Reference

How to error handle in lambda function python

I am new to python. Is there anyway I can raise error in Lambda function? I have a pyqt5 button that connect to a lambda function. I would like to catch the error in lambda function and show it to user. Is there anyway I can bring the message(string) out of the lambda function and then show it to user or is there anyway I can raise an error inside lambda function? I have a try/except in the place where lambda was apply. What I hope is when the lambda function have error and the error can be catch outside of the lambda function.
example:
def test1:
*****
def test2(a):
****
try:
x=lambda(test2(test1))
except Exception as e:
print(e) <<< want it to go here.

In your code you only assign a lambda function to a variable without any calling. In addition, the syntax for lambda seems to be wrong. You could change your code something like as follows:
def test1:
*****
def test2(a):
****
a_func = lambda x: test2(test1(x))
try:
a_func(some_input)
except Exception as e:
print(e)

import os
import boto3
import json
from botocore.exceptions import ClientError
def lambda_handler(event, context):
# Fetch Datebase Identifier Environment variable
DBinstance = os.environ['DBInstanceName']
#create an Rds client
context.rds_client = boto3.client('rds')
# Describe instance
response = context.rds_client.describe_db_instances(DBInstanceIdentifier=DBinstance )
#Check the status of rds instance
status = response['DBInstances'][0]['DBInstanceStatus']
# If Rds Instance is in stopped stop, It will be started
if status == "stopped":
try:
#Stopping the Rds instance
response = context.rds_client.start_db_instance( DBInstanceIdentifier=DBinstance )
#send logs to cloudwatch
print ('Success :: Starting Rds instance:', DBinstance)
#Logs to Lambda function Execution results
return {
'statusCode': 200,
'message': "Starting Rds instance",
'body': json.dumps(response, default=str)
}
except ClientError as e:
#send logs to cloudwatch
print(e)
message = e
#Logs to Lambda function Execution results
return {
'message': "Script execution completed. See Cloudwatch logs for complete output, but instance starting failed",
'body': json.dumps(message, default=str)
}
# if Rds instance is in running state, It will be stopped
elif status == "available":
try:
#Stopping the Rds instance
response = context.rds_client.stop_db_instance( DBInstanceIdentifier=DBinstance )
#send logs to cloudwatch
print ('Success :: Stopping Rds instance:', DBinstance)
#Logs to Lambda function Execution results
return {
'statusCode': 200,
'message': "Stopping Rds instance",
'body': json.dumps(response, default=str)
}
except ClientError as e:
#send logs to cloudwatch
print(e)
message = e
#Logs to Lambda function Execution results
return {
'message': "Script execution completed. See Cloudwatch logs for complete output, but instance stopping failed",
'body': json.dumps(message, default=str)
}

Mocking a lambda response with moto

Somewhere in my code, a lambda is called to return a true/false response. I am trying to mock this lambda in my unit tests with no success.
This is my code:
def _test_update_allowed():
old = ...
new = ...
assert(is_update_allowed(old, new) == True)
Internally, is_update_allowed calls the lambda, which is what I want to mock.
I tried adding the following code above my test:
import zipfile
import io
import boto3
import os
#pytest.fixture(scope='function')
def aws_credentials():
"""Mocked AWS Credentials for moto."""
os.environ['AWS_ACCESS_KEY_ID'] = 'testing'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'testing'
os.environ['AWS_SECURITY_TOKEN'] = 'testing'
os.environ['AWS_SESSION_TOKEN'] = 'testing'
CLIENT = boto3.client('lambda', region_name='us-east-1')
# Expected response setup and zip file for lambda mock creation
def lambda_event():
code = '''
def lambda_handler(event, context):
return event
'''
zip_output = io.BytesIO()
zip_file = zipfile.ZipFile(zip_output, 'w', zipfile.ZIP_DEFLATED)
zip_file.writestr('lambda_function.py', code)
zip_file.close()
zip_output.seek(0)
return zip_output.read()
# create mocked lambda with zip file
def mock_some_lambda(lambda_name, return_event):
return CLIENT.create_function(
FunctionName=lambda_name,
Runtime='python2.7',
Role='arn:aws:iam::123456789:role/does-not-exist',
Handler='lambda_function.lambda_handler',
Code={
'ZipFile': return_event,
},
Publish=True,
Timeout=30,
MemorySize=128
)
and then updated my test to:
#mock_lambda
def _test_update_allowed():
mock_some_lambda('hello-world-lambda', lambda_event())
old = ...
new = ...
assert(is_update_allowed(old, new) == True)
But I'm getting the following error, which makes me think it's actually trying to talk to AWS
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the CreateFunction operation: The security token included in the request is invalid.

From the error message, I can confirm it definitely not an AWS issue. It is clearly stating that it is trying to use some credentials which are not valid. So that boils down to the code.
I am assuming you already have import statements for necessary libs because those are also not visible in the shared code
import pytest
import moto
from mock import mock, patch
from moto import mock_lambda
So you need to use the
def aws_credentials():
.....
while creating the client because from the code I dont see that you are using the same.
#pytest.fixture(scope='function')
def lambda_mock(aws_credentials):
with mock_lambda():
yield boto3.client('lambda', region_name='us-east-1')
and eventually your mock
#pytest.fixture(scope='function')
def mock_some_lambda(lambda_mock):
lambda_mock.create_function(
FunctionName=lambda_name,
Runtime='python2.7',
Role='arn:aws:iam::123456789:role/does-not-exist',
Handler='lambda_function.lambda_handler',
Code={
'ZipFile': return_event,
},
Publish=True,
Timeout=30,
MemorySize=128
)
yield
then test function
def _test_update_allowed(lambda_mock,mock_some_lambda):
lambda_mock.invoke(...)
.....
Cant give a working example, because not sure what the full logic is. Between take a look this post.

The problems seems due to unexisting arn role. Try mocking it like in moto library tests
def get_role_name():
with mock_iam():
iam = boto3.client("iam", region_name=_lambda_region)
try:
return iam.get_role(RoleName="my-role")["Role"]["Arn"]
except ClientError:
return iam.create_role(
RoleName="my-role",
AssumeRolePolicyDocument="some policy",
Path="/my-path/",
)["Role"]["Arn"]

Error Getting Managed Identity Access Token from Azure Function

I'm having an issue retrieving an Azure Managed Identity access token from my Function App. The function gets a token then accesses a Mysql database using that token as the password.
I am getting this response from the function:
9103 (HY000): An error occurred while validating the access token. Please acquire a new token and retry.
Code:
import logging
import mysql.connector
import requests
import azure.functions as func
def main(req: func.HttpRequest) -> func.HttpResponse:
def get_access_token():
URL = "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fossrdbms-aad.database.windows.net&client_id=<client_id>"
headers = {"Metadata":"true"}
try:
req = requests.get(URL, headers=headers)
except Exception as e:
print(str(e))
return str(e)
else:
password = req.json()["access_token"]
return password
def get_mysql_connection(password):
"""
Get a Mysql Connection.
"""
try:
con = mysql.connector.connect(
host='<host>.mysql.database.azure.com',
user='<user>#<db>',
password=password,
database = 'materials_db',
auth_plugin='mysql_clear_password'
)
except Exception as e:
print(str(e))
return str(e)
else:
return "Connected to DB!"
password = get_access_token()
return func.HttpResponse(get_mysql_connection(password))
Running a modified version of this code on a VM with my managed identity works. It seems that the Function App is not allowed to get an access token. Any help would be appreciated.
Note: I have previously logged in as AzureAD Manager to the DB and created this user with all privileges to this DB.
Edit: No longer calling endpoint for VMs.
def get_access_token():
identity_endpoint = os.environ["IDENTITY_ENDPOINT"] # Env var provided by Azure. Local to service doing the requesting.
identity_header = os.environ["IDENTITY_HEADER"] # Env var provided by Azure. Local to service doing the requesting.
api_version = "2019-08-01" # "2018-02-01" #"2019-03-01" #"2019-08-01"
CLIENT_ID = "<client_id>"
resource_requested = "https%3A%2F%2Fossrdbms-aad.database.windows.net"
# resource_requested = "https://ossrdbms-aad.database.windows.net"
URL = f"{identity_endpoint}?api-version={api_version}&resource={resource_requested}&client_id={CLIENT_ID}"
headers = {"X-IDENTITY-HEADER":identity_header}
try:
req = requests.get(URL, headers=headers)
except Exception as e:
print(str(e))
return str(e)
else:
try:
password = req.json()["access_token"]
except:
password = str(req.text)
return password
But now I am getting this Error:
{"error":{"code":"UnsupportedApiVersion","message":"The HTTP resource that matches the request URI 'http://localhost:8081/msi/token?api-version=2019-08-01&resource=https%3A%2F%2Fossrdbms-aad.database.windows.net&client_id=<client_idxxxxx>' does not support the API version '2019-08-01'.","innerError":null}}
Upon inspection this seems to be a general error. This error message is propagated even if it's not the underlying issue. Noted several times in Github.
Is my endpoint correct now?

For this problem, it was caused by the wrong endpoint you request for the access token. We can just use the endpoint http://169.254.169.254/metadata/identity..... in azure VM, but if in azure function we can not use it.
In azure function, we need to get the IDENTITY_ENDPOINT from the environment.
identity_endpoint = os.environ["IDENTITY_ENDPOINT"]
The endpoint is like:
http://127.0.0.1:xxxxx/MSI/token/
You can refer to this tutorial about it, you can also find the python code sample in the tutorial.
In my function code, I also add the client id of the managed identity I created in the token_auth_uri but I'm not sure if the client_id is necessary here (In my case, I use user-assigned identity but not system-assigned identity).
token_auth_uri = f"{identity_endpoint}?resource={resource_uri}&api-version=2019-08-01&client_id={client_id}"
Update:
#r "Newtonsoft.Json"
using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;
public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{
string resource="https://ossrdbms-aad.database.windows.net";
string clientId="xxxxxxxx";
log.LogInformation("C# HTTP trigger function processed a request.");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(String.Format("{0}/?resource={1}&api-version=2019-08-01&client_id={2}", Environment.GetEnvironmentVariable("IDENTITY_ENDPOINT"), resource,clientId));
request.Headers["X-IDENTITY-HEADER"] = Environment.GetEnvironmentVariable("IDENTITY_HEADER");
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader streamResponse = new StreamReader(response.GetResponseStream());
string stringResponse = streamResponse.ReadToEnd();
log.LogInformation("test:"+stringResponse);
string name = req.Query["name"];
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
name = name ?? data?.name;
return name != null
? (ActionResult)new OkObjectResult($"Hello, {name}")
: new BadRequestObjectResult("Please pass a name on the query string or in the request body");
}

For your latest issue, where you are seeing UnsupportedApiVersion, it is probably this issue: https://github.com/MicrosoftDocs/azure-docs/issues/53726
Here are a couple of options that worked for me:
I am assuming you are hosting the Function app on Linux. I noticed that ApiVersion 2017-09-01 works, but you need to make additional changes (instead of "X-IDENTITY-HEADER", use "secret" header). And also use a system-assigned managed identity for your function app, and not a user assigned identity.
When I hosted the function app on Windows, I didn't have the same issues. So if you want to use an user-assigned managed identity, you can try this option instead. (with the api-version=2019-08-01, and X-IDENTITY-HEADER.

How can I fix coroutine was never awaited?

I have a RESTFUL Flask API I am serving with gunicorn and I'm trying to continue running parse_request() after sending a response to whoever made a POST request so they're not left waiting for it to finish
I'm not too sure if this will even achieve what I want but this is the code I have so far.
from threading import Thread
import subprocess
from flask import Flask
import asyncio
application = Flask(__name__)
async def parse_request(data):
try:
command = './webscraper.py -us "{user}" -p "{password}" -url "{url}"'.format(**data)
output = subprocess.check_output(['bash','-c', command])
except Exception as e:
print(e)
#application.route('/scraper/run', methods=['POST'])
def init_scrape():
try:
thread = Thread(target=parse_request, kwargs={'data': request.json})
thread.start()
return jsonify({'Scraping this site: ': request.json["url"]}), 201
except Exception as e:
print(e)
if __name__ == '__main__':
try:
application.run(host="0.0.0.0", port="8080")
except Exception as e:
print(e)
I am sending a POST request similar to this.
localhost:8080/scraper/run
data = {
"user": "username",
"password": "password",
"url": "www.mysite.com"
}
The error I get when sending a POST request is this.
/usr/lib/python3.6/threading.py:864: RuntimeWarning: coroutine 'parse_request' was never awaited
self._target(*self._args, **self._kwargs)

So first things first, why are you calling webscraper.py with subprocess? This is completely pointless. Because webscraper.py is a python script you should be importing the needed functions/classes from webscraper.py and using them directly. Calling it this way is totally defeating what you are wanting to do.
Next, your actual question you have got mixed up between async and threading. I suggest you learn more about it but essentially you want something like the following using Quart which is an async version of Flask, it would suit your situation well.
from quart import Quart, response, jsonify
import asyncio
from webscraper import <Class>, <webscraper_func> # Import what you need or
import webscraper # whatever suits your needs
app = Quart(__name__)
async def parse_request(user, password, url):
webscraper_func(user, password, url)
return 'Success'
#app.route('/scraper/run', methods=['POST'])
async def init_scrape():
user = request.args.get('user')
password = request.args.get('password')
url = request.args.get('url')
asyncio.get_running_loop().run_in_executor(
None,
parse_request(user, password, url)
)
return 'Success'
if __name__ == '__main__':
app.run(host='0.0.0.0', port='8080')

Streaming Twitter direct messages

I am using the following code to stream direct messages received by my Twitter account -:
from tweepy import Stream
from tweepy import OAuthHandler
from tweepy import API
from tweepy.streaming import StreamListener
# These values are appropriately filled in the code
consumer_key = ""
consumer_secret = ""
access_token = ""
access_token_secret = ""
class StdOutListener( StreamListener ):
def __init__( self ):
self.tweetCount = 0
def on_connect( self ):
print("Connection established!!")
def on_disconnect( self, notice ):
print("Connection lost!! : ", notice)
def on_data( self, status ):
print("Entered on_data()")
print(status, flush = True)
return True
def on_direct_message( self, status ):
print("Entered on_direct_message()")
try:
print(status, flush = True)
return True
except BaseException as e:
print("Failed on_direct_message()", str(e))
def on_error( self, status ):
print(status)
def main():
try:
auth = OAuthHandler(consumer_key, consumer_secret)
auth.secure = True
auth.set_access_token(access_token, access_token_secret)
api = API(auth)
# If the authentication was successful, you should
# see the name of the account print out
print(api.me().name)
stream = Stream(auth, StdOutListener())
stream.userstream()
except BaseException as e:
print("Error in main()", e)
if __name__ == '__main__':
main()
I am able to get my name printed as well as the "Connection established!" message.
But whenever I send a direct message to my own profile from my friend's profile, no method is called.
Although, whenever I tweet something from my profile, it is printed correctly by the program.
Me and my friend both follow each other on Twitter, so there shouldn't be any problems with the direct message permissions.
What is the proper way to do it ?
Also, if it cannot be done in Tweepy at all, I am ready to use any other Python library.
I am using Tweepy 3.3.0 and Python 3.4 on Windows 7.

The code provided in the question is indeed correct.
The problem was that I had forgotten to regenerate the access token and secret after changing my application permissions to "Read, write and direct messages".
Note: The direct messages arrive in the on_data() method rather than the on_direct_message() method.

Another solution is to not override the on_data method. That is how I got your code example to work in my case.
The on_data method of tweepy.streaming.StreamListener object contains all the json data handling and calling of other methods, like on_direct_message, that you and I were expecting.

Here is a working version of the code in 2017
from twitter import *
import os
import oauth
#Create a new Twitter app first: https://apps.twitter.com/app/new
APP_KEY,APP_SECRET = 'H3kQtN5PQgRiA0ocRCCjqjt2P', '51UaJFdEally81B7ZXjGHkDoDKTYy430yd1Cb0St5Hb1BVcDfE'
# OAUTH_TOKEN, OAUTH_TOKEN_SECRET = '149655407-TyUPMYjQ8VyLNY5p7jq0aMy8PjtFtd7zkIpDh3ZA', 'IUVpiDpoVmdO75UaHOTinAv5TOsAQmddttNENh9ofYuWO'
MY_TWITTER_CREDS = os.path.expanduser('my_app_credentials')
if not os.path.exists(MY_TWITTER_CREDS):
oauth_dance("crypto sentiments", APP_KEY, APP_SECRET,
MY_TWITTER_CREDS)
oauth_token, oauth_secret = read_token_file(MY_TWITTER_CREDS)
auth = OAuth(
consumer_key=APP_KEY,
consumer_secret=APP_SECRET,
token=oauth_token,
token_secret=oauth_secret
)
twitter_userstream = TwitterStream(auth=auth, domain='userstream.twitter.com')
for msg in twitter_userstream.user():
if 'direct_message' in msg:
print (msg['direct_message']['text'])