I am just getting into aws lambda functions and have written a function that fetches some data from a dynamodb table.
This is the function:
import boto3
from boto3.dynamodb.conditions import Key, Attr
import botocore.exceptions
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
dynamodb=boto3.resource('dynamodb')
appointmentsTable = dynamodb.Table('Appointments')
class NotFoundError(Exception):
pass
def lambda_handler(event, context):
try:
logger.info(f'event: {event}')
bookedAppointments = fetchAppointments(event)
logger.info(f'Response: {bookedAppointments}')
return sendResponse(True , 200 , 'Appointments found' , bookedAppointments)
except NotFoundError:
return sendResponse(True , 400, 'No booked appointments Found' , [])
except Exception as error:
return sendResponse(False , 500 , 'Error in fetch booked appointments' , str(error))
def sendResponse(success , statusCode , message , responseData):
return {
'success' : success,
'statusCode' : statusCode,
'message': message,
'responseData' : responseData
}
def fetchAppointments(event):
consId = event.get('consId')
try:
bookedAppointments = appointmentsTable.query(
IndexName = 'consId-index',
KeyConditionExpression = Key('consId').eq(consId),
FilterExpression = 'booked=:b',
ExpressionAttributeValues = {
':b' : True
}
)
except botocore.exceptions.ClientError as error:
logger.exception(f'Error in fetchAppointments function: {error}')
raise error
if bookedAppointments.get('Items') == []:
raise NotFoundError
sortedResult = sortResult(bookedAppointments.get('Items'))
return sortedResult
def sortResult(listTobeSorted):
return sorted(listTobeSorted , key=lambda k: (k['appointmentDate'] , k['appointmentSlot']))
I know wrapping all of your code logic in a try-catch block is bad practise , so is there a better way to handle any exceptions that may occur in the fetchAppointments function?
There are different ways to solve issues like this, but the underlying truth is: it always depends on what you want to do. So the following advice might be ok for this specific case, but might not be ok for other uses cases. So take the following with a grain of salt.
That said, the first thing to analyse is the different outputs of your fetchAppointments() function:
One or more appointments found
No appointment found
Some application error/exception
Coming to your question: should you use exception handling for this? Yes, but I would not use it as much as you did.
I would only use it for the "proper" exception in this case and the rest should be covered by "normal" application logic.
Consider the following simplified version of your handler:
def lambda_handler(event, context):
try:
appointments = fetchAppointments(event)
except Exception as error:
return sendResponse(False, 500, 'Error in fetch booked appointments', str(error))
if not appointments:
return sendResponse(True, 400, 'No booked appointments Found', [])
return sendResponse(True, 200, 'Appointments found', appointments)
As you can see, only the "real" application exception is handled with exception handling and the rest just uses "normal" logic to figure out, if there are appointments or not.
Code quality is certainly a good thing we don't have to start from scratch IMO, In my experience what I learned is
it should work first and then it should look beautiful
But in Cloud, there is more to it, it should be optimized in terms of cost as well.
So there you already started on the right path already by handling the exception
If you don't handle the exception and let the function fail, by default lambda would try to execute it 3 times
So you would be charged for those executions. You can customize this behaviour
As far as handling the exception goes #Jens explained it real good.
Related
I'm using kafka-python library for my fastapi consumer app and I'm consuming messages in batch with maximum of 100 records. Since the topic has huge traffic and have only one partition, consuming, processing and committing should be as quick as possible hence I want to use commit_async(), instead of synchronous commit().
But I'm not able to find a good example of commit_async(). I'm looking for an example for commit_async() with callback so that I can log in case of commit failure. But I'm not sure what does that callback function takes as argument and what field those arguments contain.
However the docs related to commit_async mentions the arguments, I'm not completely sure how to use them.
I need help in completing my callback function on_commit(), someone please help here
Code
import logging as log
from kafka import KafkaConsumer
from message_handler_impl import MessageHandlerImpl
def on_commit():
pass
class KafkaMessageConsumer:
def __init__(self, bootstrap_servers: str, topic: str, group_id: str):
self.bootstrap_servers = bootstrap_servers
self.topic = topic
self.group_id = group_id
self.consumer = KafkaConsumer(topic, bootstrap_servers=bootstrap_servers, group_id=group_id, enable_auto_commit=False, auto_offset_reset='latest')
def consume_messages(self, max_poll_records: int,
message_handler: MessageHandlerImpl = MessageHandlerImpl()):
try:
while True:
try:
msg_pack = self.consumer.poll(max_records=max_poll_records)
for topic_partition, messages in msg_pack.items():
message_handler.process_messages(messages)
self.consumer.commit_async(callback=on_commit)
except Exception as e:
log.error("Error while consuming message due to: %s", e, exc_info=True)
finally:
log.error("Something went wrong, closing consumer...........")
self.consumer.close()
if __name__ == "__main__":
kafka_consumer = KafkaMessageConsumer("localhost:9092", "test-topic", "test-group")
kafka_consumer.consume_messages(100)
The docs are fairly clear.
Called as callback(offsets, response) with response as either an Exception or an OffsetCommitResponse struct.
def on_commit(offsets, response):
# or maybe try checking type(response)
if hasattr(response, '<some attribute unique to OffsetCommitResponse>'):
print('committed ' + str(offsets))
else:
print(response) # exception
I'm sure you could look at the source code an maybe find a unit test that covers a full example
according to werkzeug docs, InternalServerError has original_exception parameter. That looks cool and I'would like to use it in following way to handle exceptions:
#app.errorhandler(Exception)
def handle_errors(err):
return err if isinstance(err, HTTPException) else InternalServerError(original_exception=err)
and log it in #app.after_request method to pack all additional info from request and response, where standard request logging occurs. Unfortunatelly I'm not able to find out how to use stored original_exception. Where is it stored?
When I check sys.exc_info() in #app.after_request, it's also empty (or (None, None, None)).
One way is probably log exceptions directly in #app.errorhandler, request info should be accessible there and response is not ready (because of error) anyways, but what happens with original_exception after storing in InternalServerError???
Thanks for answer, Michal
I found myself here looking for the same thing - how to access the original_exception, but I eventually gave up and settled on traceback to give me the info I was seeking.
This is the key component, capturing the traceback stack as a list:
formatted_lines = traceback.format_exc().splitlines()
Here's my final code in which I extracted the line number of the error and the specific trigger, then passed them to the page:
from werkzeug.exceptions import InternalServerError
import traceback
#app.errorhandler(InternalServerError)
def handle_500(e):
templateData = {
'errorAt' : 'Internal Server Error',
'errorIs' : 'Check ~/www/gunicorn.error for details'
}
try:
formatted_lines = traceback.format_exc().splitlines()
for error_location in formatted_lines:
if 'intvlm8r.py' in error_location:
templateData['errorAt'] = error_location
templateData['errorIs'] = formatted_lines[-1]
except Exception as e:
app.logger.debug(f'handle_500: Unhandled exception: {e}')
return render_template('500_generic.html', **templateData)
There are three situations when I need to handle exceptions.
When data validation raised exception
When library/module functions raised exceptions (e.g. database connection abort)
When business logic raises exception such as 500, 503, 401, 403 and 404
def library_func():
try:
...
except HTTPException:
raise TwitterServiceException("Twitter is down!")
#view_config(route_name="home", renderer="json")
#validator
#authorization
def home_view(request):
try:
tweets = library_func()
return {"tweets": tweets}
except TwitterServiceException as e:
LOG.critical(e.msg)
raise ParnterServcieError(e.msg) # this is probably a 503 error
def validator(args):
# I will show the high level of this decorator
try:
decode input as JSON
verify data format
except ValueError as err:
error = {'error': "Missing required parameters."}
except json.JSONDecodeError as err:
error = {'error': "Failed to decode the incoming JSON payload."}
if error is not None:
return HTTPBadRequest(body=json.dumps(error),
content_type='application/json')
def authorization(args):
# very similar to validator except it performs authorization and if failed
# 401 is raised with some helpful message.
The doc suggests Custom Exception Views. In my PoC above, I will tie ParnterServcieError as one. I can even generalize HTTPBadRequest and all praymid.httpexceptions using custom exception so that I no longer need to repeat json.dumps and content_type. I can set a boilerplate error body before I return request.response object.
Idea:
#view_config(context=ParnterServcieError)
def 503_service_error_view(e, request):
request.response.status = 503
request.response.json_body = {"error": e.msg}
return request.response
I can generalize one for all uncaught, unspecified exceptions (which results in 500 Internal Server Error) called 500_internal_server_error_view.
Does this seem sane and clean to people? Is my way of handling high and low level of exceptions proper and Pythonic?
I applied this strategy to ToDoPyramid and could encapsulate error handling in a single custom exception view that was repeated multiple times in the application before. Until you could even improve it, you got a great idea. Pyramid rocks.
References
Catching database connection error in ToDoPyramid
When I have two Python exceptions that are the same exception class but a different error message, how do I catch them separately?
For specific use-case:
I'm using the Facepy library to hit the Facebook Graph API. When the API returns an error that isn't Oauth related, Facepy raises a facepy.exceptions.FacebookError and passes the error message given by the Facebook API.
I'm consistently hitting two different errors that I'd like to treat differently and the only way to parse them is the error message, but I can't figure out how to write my except clause--here it is in pseudo-code:
try:
#api query
except facepy.exceptions.OAuthError and error_message = 'object does not exist':
# do something
except facepy.exceptions.OAuthError and error_message = 'Hit API rate limit':
# do something else
How do I write these except clauses to trigger off both the exception and the error message?
Assuming the Exception's error message is in the error_message attribute (it may be something else — look at the Exception's __dict__ or source to find out):
try:
#api query
except facepy.exceptions.OAuthError as e:
if e.error_message == "object does not exist":
print "Do X"
elif e.error_message == "Hit API rate limit":
print "Do Y"
else:
raise
facepy's OAuthError derives from FacebookError and that has message attribute. https://github.com/jgorset/facepy/blob/master/facepy/exceptions.py#L8. So, you can use if condition with the message like this
try:
#api query
except facepy.exceptions.OAuthError as error:
if 'object does not exist' == error.message:
# do something
elif 'Hit API rate limit' == error.message:
# do something else
else:
raise
I am trying to handle jira-python exception but my try, except does not seem to catch it. I also need to add more lines in order to be able to post this. So there they are, the lines.
try:
new_issue = jira.create_issue(fields=issue_dict)
stdout.write(str(new_issue.id))
except jira.exceptions.JIRAError:
stdout.write("JIRAError")
exit(1)
Here is the code that raises the exception:
import json
class JIRAError(Exception):
"""General error raised for all problems in operation of the client."""
def __init__(self, status_code=None, text=None, url=None):
self.status_code = status_code
self.text = text
self.url = url
def __str__(self):
if self.text:
return 'HTTP {0}: "{1}"\n{2}'.format(self.status_code, self.text, self.url)
else:
return 'HTTP {0}: {1}'.format(self.status_code, self.url)
def raise_on_error(r):
if r.status_code >= 400:
error = ''
if r.text:
try:
response = json.loads(r.text)
if 'message' in response:
# JIRA 5.1 errors
error = response['message']
elif 'errorMessages' in response and len(response['errorMessages']) > 0:
# JIRA 5.0.x error messages sometimes come wrapped in this array
# Sometimes this is present but empty
errorMessages = response['errorMessages']
if isinstance(errorMessages, (list, tuple)):
error = errorMessages[0]
else:
error = errorMessages
elif 'errors' in response and len(response['errors']) > 0:
# JIRA 6.x error messages are found in this array.
error = response['errors']
else:
error = r.text
except ValueError:
error = r.text
raise JIRAError(r.status_code, error, r.url)
I know I'm not answering the question, but I feel I need to warn people that could get confused by that code (as I did)...
Maybe you are trying to write your own version of jira-python or it's an old version?
In any case, here the link to the jira-python code for the JIRAError class
and here the list of codes
to catch the exception from that package I use the below code
from jira import JIRA, JIRAError
try:
...
except JIRAError as e:
print e.status_code, e.text
Maybe it is obvious and that's why you don't have it in your code paste, just in case, you do have
from jira.exceptions import JIRAError
somewhere in your code right?
Don't have enough reputation for comments so I'll add it answer for #arynhard:
I found the docs very light in particular in terms of example, you might find the scripts in this repo useful since they're all leveraging jira-python in someway or another.
https://github.com/eucalyptus/jira-scripts/
I might be wrong, but looks like you're catching jira.exceptions.JIRAError, while raising JIRAError - those are different types. You need to either remove "jira.exceptions." part from your except statement, or raise jira.exceptions.JIRAError instead.
Ok it's a very old one but I faced the same problem and this page is still showing up.
Here is how I trap the Exception, I used the Exception object.
try:
issue = jira.issue('jira-1')
except Exception as e:
if 'EXIST' in e.text:
print 'The issue does not exist'
exit(1)
regards