Exception handling in AWS - Explicitly ignore some errors? - python

I have a Lambda function that's used to verify the integrity of a file, but for simplicity sake, let's just assume that it copies a file from a bucket into a bucket upon trigger (lambda gets triggered when it detects file ingestion).
The problem that I am currently facing is that whenever I ingest a lot of files, the function gets triggered as many times as the number of files and that leads to unnecessary invocations. One invocation can handle multiple files so the earlier invocations typically process more than 1 file and the latter realising that there are no more files to process, yield a NoSuchKey error.
So I added in the try-except logic and indicate it to log NoSuchKey error as an info log so that it does not trigger another function that catches ERROR keyword and alert prod support team.
To do that, i used this code which is taken from AWS documentation:
import botocore
import boto3
import logging
# Set up our logger
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
client = boto3.client('s3')
try:
logger.info('Executing copy statement')
# main logic
except botocore.exceptions.ClientError as error:
if error.response['Error']['Code'] == 'NoSuchKey':
logger.info('No object found - File has been moved')
else:
raise error
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/error-handling.html?force_isolation=true
Despite that, I still get the NoSuchKey error. From CloudWatch, it seems that the logic in the "try" section gets executed, followed by an ERROR statement and then the info log 'No object found - File has been moved' that's written in my "except" section.
I thought either "try" or "except" gets executed, but in this case both seemed to run.
Any pros willing to shed some lights?

Related

Python exception handling of logging itself

While I use logging to record exceptions, it occurred to me the methods of the logging library itself can throw exceptions.
For example, if a "bad" path is set for the log file, like
import logging
logging.basicConfig(filename='/bad/path/foo.log')
a FileNotFoundError will be thrown.
Suppose my goal of handling these logging exceptions is to keep the program running (and not exit, which the program would otherwise do). The first thing that comes to mind is
try:
logging.basicConfig(filename='/bad/path/foo.log')
except Exception:
pass
But this is considered by some to be an antipattern. It's also quite ugly to wrap every logging.error with try and except blocks.
What's a good pattern to handle exceptions thrown by logging methods like basicConfig(), and possibly by debug(), error(), etc.?
Unless you re-initialise your logger mid-way through your code, why not just check whether the file exists during logger initialisation:
import os
if os.path.isfile(filename):
# Initialise the logger
This should work normally, unless of course some part of the later code will attemp to delete the file, but I hope that it's not the case.

How to log full call-stack with context in raven/sentry?

When raised exception is caught on the root of call-stack I can see the whole context at every level of call-stack in Sentry.
But, when I use captureMessage() I can't see any context in Sentry.
If I use captureException() as in the code below I can see only the top of call-stack.
try:
raise Exception('Breakpoint!')
except:
raven_client.captureException()
In other words I want to see in Sentry a logged message with full stacktrace and context.
The Python SDK has the ability to capture arbitrary stacktraces by passing stack=True to captureMessage:
raven_client.captureMessage('hello world', stack=True)
There is additionally an auto_log_stacks value that can be turned on when configuring the Client:
raven_client = Client(..., auto_log_stacks=True)
Caveat: Automatically logging stacks is useful, but it's not guaranteed accurate in some common situations. It's also a performance hit, albeit a minor one, as it has to constantly call out to inspect.stack().

Send email after Exception raised continuosly for more than one hour

I have a Python program that deals with a webpage with Selenium every 5 minutes from a cron job. cron sends me an email for every exception raised (cron just catches all stderr). Sometimes the website has some persistent internal errors, and I end up with hundreds of system emails to check. I then started ignoring exceptions, but I found it would be more sane if the program would send me an email if the problem persists for one hour.
I have written the following code:
#!/usr/bin/env python3
import sys, os, time
name = os.path.basename(sys.argv[0])
def send_email(server, message):
# just print for testing
print('Sending email...')
def handle_exception(exception, function):
# save a state file in /dev/shm with the name of the exception
warn_state = '/dev/shm/{}.{}'.format(name, exception.__name__)
try:
function
except(exception) as exception:
if os.path.exists(warn_state):
if time.time() - os.path.getmtime(warn_state) > 3600:
send_email('localhost', exception.args[1])
else:
open(warn_state, 'w').close()
else:
if os.path.exists(warn_state):
os.remove(warn_state)
def my_function():
# try raising a NameError
print(undefined_variable)
# run my_function() catching exceptions
handle_exception(NameError, my_function)
When I run the code above, I have checked that the else: part is being executed, indicating that function is not failing at all. I'm new to programming, so I don't really know if this would be the proper way to do this, but I created this function because I have to deal with at least five different types of exceptions on this script, that are raised randomly by Selenium due to server or network problems.
you are not actually calling function
function
does nothing. you need to do
function()

finally versus atexit

I end up having to write and support short python wrapper scripts with the following high-level structure:
try:
code
...
...
except:
raise
finally:
file_handle.close()
db_conn.close()
Notice that all I do in the except block is re-raise the exception to the script caller sans window-dressing; this is not a problem in my particular context. The idea here is that cleanup code should always be executed by means of the finally block, exception or not.
Am I better off using an atexit handler for this purpose? I could do without the extra level of indentation introduced by try.
The atexit module provides a simple interface to register functions to be called when a program closes down normally. Functions registered are automatically executed upon normal interpreter termination.
import atexit
def cleanup():
print 'performimg cleanup'
# multiple functions can be registered here...
atexit.register(cleanup)
The sys module also provides a hook, sys.exitfunc, but only one function can be registered there.
Finally is accompanied by try except block, functionality of finally can also be used for something similar like cleanup, however at finally block sys.exc_info is all-None.
If the finally clause raises another exception, the saved exception is discarded however you can put try except in the function registered with atexit to handle them.
Another pro-con is atexit functions are only executes when program terminates, however you can use finally (with try-except) anywhere in the code and perform the cleanup
In you scenario, where you want to raise an exception from cleanup content, usage of atexit would be helpful, if you are ok for cleanup to happen at the end of the program
Just use contextlib.closing
with closing(resource1) as f1, closing(resource2) as f2:
f1.something()
f2.something()
And they will be automatically closed. Files objects can be used directly as contexts so you don't need the closing call.
If close is not the only method used by your resources, you can create custom functions with the contextlib.contextmanager decorator.
atexit is called upon program termination, so this is not what you are looking for.

python - Selective handling of exception traceback

I'm trying to have an exception handling mechanism with several layers of information to display to the user for my application, using python's logging module.
In the application, the logging module has 2 handlers: a file handler for keeping DEBUG information and a stream handler for keeping INFO information. By default, the logging level is set to INFO. What I'm trying to achieve is a setup where if any exception occurs, the user gets shown a simple error message without any tracebacks by default. If the logging level is set to DEBUG, the user should still get the simple message only, but this time the exception traceback is logged into a log file through the file handler.
Is it possible to achieve this?
I tried using logger.exception(e), but it always prints the traceback onto the console.
The traceback module may help you. At the top level of your application, you should put a catch all statement:
setup_log_and_other_basic_services()
try:
run_your_app()
except Exception as e:
if is_debug():
traceback.print_stack()
else:
traceback.print_stack(get_log_file())
print e
the code outside the try/catch block should not be allowed to crash.
Write your custom exception handling function, and use it every time you write catch.
In this function you should check which mode is on (INFO or DEBUG) and then extract info about exception and feed it to logger manually when needed.

Categories

Resources