I'd like to keep a solid logging system going, but it's also necessary to raise exceptions. This code accomplishes what I'm going for, but it looks clunky and not very Pythonic. What's a better option?
import logging
if not condition_met:
missing = set_one - set_two
logging.error('Missing keys: {}'.format(missing))
raise ValueError('Missing keys: {}'.format(missing))
you could catch the exception and log the error at this time, so if another exception occurs you can log it as well, and propagate the exception upstream.
try:
# some code
if not condition_met:
missing = set_one - set_two
raise ValueError('Missing keys: {}'.format(missing))
except Exception as e: # or ValueError to narrow it down
logging.error(str(e))
raise # propagate the exception again
note than logging an exception without logging the traceback leaves something unfinished, specially if the exception is caught and handled upstream. It's very likely that you're never going to fix that particular error.
You can use logger's exception() function:
from logger import exception
try:
. . .
except Exception as error:
exception(msg="Your message")
so that all of the stack will be logged.
You can read an interesting article about this here.
Another elegant approach is to define custom exceptions for your application that serve the purpose of clarifying lower-level exceptions such as KeyError as well as centralizing error logic. These custom exceptions can be defined on a separate file to make maintenance and updates easier. custom exceptions are derived from a base Error class to inherit global settings which itself is derived from the built-in Exception class.
exceptions.py
from utils import log
class Error(Exception):
"""base class for errors"""
class EnvironmentAttributeError(Error):
"""
Exception raised when environment variables are empty strings
Attributes:
key_attribute -- environment variable
"""
def __init__(self, environment_variable):
self.environment_variable = environment_variable
self.message = f"Environment variable value for key {environment_variable} was not assigned."
self.log = log.logger.error(f"Environment variable value for key {environment_variable} was not assigned.")
super().__init__(self.message)
class EnvironmentKeyError(Error):
"""
Exception raised when the environment variables dict does not have required keys
Attributes:
key_attribute -- environment variable
"""
def __init__(self, vars):
self.environment_variable = vars
self.message = f"Environment variable {vars} was not declared."
self.log = log.logger.error(f"Environment variable {vars} was not declared.")
super().__init__(self.message)
Notice that the exceptions.py file imports a log utility. That way all you need to do elsewhere in your code is raise the right custom code errors and everything gets logged for you. You can then update these errors in a single place for your entire project.
log.py
import logging
# instantiate a logger object writing to connected-apps.log
logging.basicConfig(
format='%(asctime)s,%(msecs)d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s',
datefmt='%Y-%m-%d:%H:%M:%S',
level=logging.DEBUG,
filename='logs/connected_apps.log'
)
# logger object named after module: https://docs.python.org/3/howto/logging.html#advanced-logging-tutorial
logger = logging.getLogger(__name__)
The logger in the log.py file has been formatted in such a way that logs are both descriptive and readable. You can even define different loggers with different formats and levels.
Here is a simple use of the custom exceptions defined above. Environment variables obtained from .env are sent to this validate() function to verify that the right keys and attributes are available. Notice that we just needed to import exceptions and not logs:
environment.py
from utils import exceptions
def validate(env_dict, env_vars):
# check that each environment variable has been declared and assigned
for vars in env_vars:
try:
# check the local dictionary pulled from os.environ
env_dict[vars]
# check that key value length is non-zero
if len(env_dict[vars]) == 0:
raise exceptions.EnvironmentAttributeError(vars)
except KeyError as error:
# raises error if an environment variable has not been declared
raise exceptions.EnvironmentKeyError(vars)
Related
When I want to log some specific Exception, but otherwise ignore it, I can do that like so:
import logging
logger = logging.getLogger(__name__)
try:
something_that_may_fail()
except NameError:
logger.error("It failed with:", exc_info=True)
(This is in fact an MRE, as something_that_may_fail hasn't been defined, so the try block will raise NameError with message name 'something_that_may_fail' is not defined. 😉)
This however will also log the stack trace:
It failed with:
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
NameError: name 'something_that_may_fail' is not defined
Sometimes that isn't what I want: In some cases, I already know that exception type and exception message (together with my custom log message) will suffice, and don't want to expand the log with stack traces that don't tell me anything new. So I'd want a log entry that simply is
It failed with:
NameError: name 'something_that_may_fail' is not defined
I can achieve that by passing a 3-tuple as the exc_info, with the stack trace component replaced by None:
import logging
import sys
logger = logging.getLogger(__name__)
try:
something_that_may_fail()
except NameError:
exc_type, exc_value, _trace = sys.exc_info()
logger.error("It failed with:", exc_info=(exc_type, exc_value, None))
But I'm not sure how reliable that is. (The documentation doesn't mention how the tuple may or may not deviate from one returned by sys.exc_info().)
Examining the exception myself with
...
except NameError as e:
...
comes with its own problems:
f"{type(e)}" gives string <class 'NameError'> instead of just string NameError
The proper solution to get the fully qualified type name, including packages/modules but without builtin. is rather unwieldy and not something I'd want in exception handling code. See the currently accepted answer to Get fully qualified class name of an object in Python.
Can I rely on the message always being e.args[0]? (I might have uses for other exceptions (with more sub-types) than just NameError, which I've used here only as an example.)
So what is the proper way to log exception type and message without the stack trace? Is there a cleaner way than my make-the-trace-None hack above?
traceback.format_exception_only can be used for that:
import logging
import sys
import traceback
logger = logging.getLogger(__name__)
try:
something_that_may_fail()
except NameError:
exc_type, exc_value, _trace = sys.exc_info()
exc_desc_lines = traceback.format_exception_only(exc_type, exc_value)
exc_desc = ''.join(exc_desc_lines).rstrip()
logger.error(f"It failed with:\n{exc_desc}")
or without sys:
import logging
import traceback
logger = logging.getLogger(__name__)
try:
something_that_may_fail()
except NameError as e:
exc_desc_lines = traceback.format_exception_only(type(e), e)
exc_desc = ''.join(exc_desc_lines).rstrip()
logger.error(f"It failed with:\n{exc_desc}")
(Found this by looking how the logging module actually extracts and formats information from exc_info. There traceback.print_exception is being used, so I looked what else is available in the traceback module.)
I have a script that processes csvs and load them to database. My intern mentor wanted us to use log file to capture what's going on and he wanted it to be flexible so one can use a config.ini file to edit where they want the log file to be created. As a result I did just that, using a config file that use key value pairs in a dict that i can extract the path to the log file from. These are excepts from my code where log file is created and used:
dirconfig_file = r"C:\Users\sys_nsgprobeingestio\Documents\dozie\odfs\venv\odfs_tester_history_dirs.ini"
start_time = datetime.now()
def process_dirconfig_file(config_file_from_sysarg):
try:
if Path.is_file(dirconfig_file_Pobj):
parseddict = {}
configsects_set = set()
for sect in config.sections():
configsects_set.add(sect)
for k, v in config.items(sect):
# print('{} = {}'.format(k, v))
parseddict[k] = v
print(parseddict)
try:
if ("log_dir" not in parseddict or parseddict["log_dir"] == "" or "log_dir" not in configsects_set):
raise Exception(f"Error: Your config file is missing 'logfile path' or properly formatted [log_file] section for this script to run. Please edit config file to include logfile path to capture errors")
except Exception as e:
#raise Exception(e)
logging.exception(e)
print(e)
parse_dict = process_dirconfig_file(dirconfig_file)
logfilepath = parse_dict["log_dir"]
log_file_name = start_time.strftime(logfilepath)
print(log_file_name)
logging.basicConfig(
filename=log_file_name,
level=logging.DEBUG,
format='[Probe Data Quality] %(asctime)s - %(name)s %(levelname)-7.7s %(message)s'
# can you explain this Tenzin?
)
if __name__ == '__main__':
try:
startTime = datetime.now()
db_instance = dbhandler(parse_dict["db_string"])
odfs_tabletest_dict = db_instance['odfs_tester_history_files']
odf_history_from_csv_to_dbtable(db_instance)
#print("test exception")
print(datetime.now() - startTime)
except Exception as e:
logging.exception(e)
print(e)
Doing this, no file is created. The script runs with no errors but no log file is created. I've tried several things including using a hardcoded log file name, instead of calling it from the config file but it didn't work
The only thing that works is when the log file is created up top before any method. Why is this?
When you are calling your process_dirconfig_file function, the logging configuration has not been set yet, so no file could have been created. The script executes top to bottom. It would be similar to doing something like this:
import sys
# default logging points to stdout/stderr kind of like this
my_logger = sys.stdout
my_logger.write("Something")
# Then you've pointed logging to a file
my_logger = open("some_file.log", 'w')
my_logger.write("Something else")
Only Something else would be written to our some_file.log, because my_logger pointed somewhere else beforehand.
Much the same is happening here. By default, the logging.<debug/info> functions do nothing because logging won't do anything with them without additional configuration. logging.error, logging.warning, and logging.exception will always at least write to stdout out of the box.
Also, I don't think the inner try is valid Python, you need a matching except. And I wouldn't just print an exception raised by that function, I'd probably raise and have the program crash:
def process_dirconfig_file(config_file_from_sysarg):
try:
# Don't use logging.<anything> yet
~snip~
except Exception as e:
# Just raise or don't use try/except at all until
# you have a better idea of what you want to do in this circumstance
raise
Especially since you are trying to use the logger while validating that its configuration is correct.
The fix? Don't use the logger until after you've determined it's ready.
I'm a newbie in Flask and I am trying to display the Built-In Exceptions in python but I can't seem to have them display on my end.
NOTE:
set FLASK_DEBUG = 0
CODE:
def do_something:
try:
doing_something()
except Exception as err:
return f"{err}"
Expectation:
It will display one of the built-in exceptions:
KeyError
IndexError
NameError
Etc.
Reality:
It will return the line of code that didn't worked which is more ambiguous to the end user.
Also:
I have no problem seeing the errors when the debug mode is ON but that's not something that I want to do if I open them in public
Flask supplies you with a function that enables you to register an error handler throughout your entire app; you can do something as shown below:
def handle_exceptions(e):
# Log exception in your logs
# get traceback and sys exception info and log as required
# app.logger.error(getattr(e, 'description', str(e)))
# Print traceback
# return your response using getattr(e, 'code', 500) etc.
# Exception is used to catch all exceptions
app.register_error_handler(Exception, handle_exceptions)
In my honest opinion, this is the way to go. - Following the structure found in werkzeug.exceptions.HTTPException as an example is a solid foundation.
Having a unified exception handler that will standardise your Exception handling, visualisation and logging will make your life a tad better. :)
Try with this:
def do_something:
try:
doing_something()
except Exception as err:
return f"{err.__class__.__name__}: {err}"
I want to detect errors in a standalone Python script with Sentry+Raven.
I tried to configure it and raven test ... is workging.
Then I place this on top of the script:
from raven import Client
client = Client('http://...#.../1')
client.captureException()
the exception is generated later on this:
import django
django.setup()
from django.conf import settings
And I want to see the actual stack for this error:
ImportError: Could not import settings 'settings' (Is it on sys.path? Is there an import error in the settings file?): No module named 'settings'
But all I see in Sentry is
which is completely useless.
How can I change this to have a normal traceback?
You misunderstand how client.captureException() works, its not a configuration parameter. You use it when you are catching an exception and it will capture the exception type and message:
try:
f = open('oogah-boogah.txt')
except IOError:
client.captureException()
# do something here
To capture any exceptions that could be generated in a block of code, you can use capture_exceptions:
#client.capture_exceptions
def load_django():
import django
django.setup()
from django.conf import settings
Yes you're right, but is there a way to catch an exception not
wrapping a block of code in a try-except. I can see the error in a
terminal, can I see it in Sentry?
There is a default exception handler - and when an exception is not caught, this default handler catches it and then displays the exception. This is what you see in the terminal.
The function that generates this output is sys.excepthook and it will output to stderr by default.
So, in order for you to catch all exception globally, you'll have to create a global exception handler or map your own function to sys.excepthook.
I would strongly recommend against this, though as you don't know what other side effects it may have.
I am customizing exceptions in my python code. I have inherited exception class in to other and now defining some custom errors as classes derived from my custom exception class like this:
class DataCollectorError(Exception): pass
class ParamNullError(DataCollectorError) : pass
class ParamInvalidTypeError(DataCollectorError) : pass
I am raising these exceptions in my python function like:
def READ_METER_DATA (regIndex, numRegisters, slaveUnit):
try:
if not regIndex:
raise ParamNullError, "register index is null"
if not numRegisters:
raise ParamNullError, "number of registers should not be null"
if not slaveUnit:
raise ParamNullError, "Meter Id should not be null"
and logging error like :
except DataCollectorError as d:
lgr.error('DataCollector Error(READ_METER_DATA): '+d.args[0])
print 'DataCollector Error:(READ_METER_DATA)', d.args[0]
except:
lgr.error('Unexpected Error: ', sys.exc_info())
print 'Unexpected Error: ', sys.exc_info()
pass
but this defeats purpose of unit testing script as it doesn't whether exception raised bcz it is being catched by my catch block before my unit test script knows it. so i wanted to log these errors in base class itself -
Class ParamNullError(DataCollectorError):
<----here----------->
pass
can anybody tell me how to fetch that string passed while raising exception?
Simply extend your error-class with an __init__ and an __str__ method.
Example:
class DataCollectorError(Exception):
def __init__(self, msg=''):
self.msg = msg
log(msg) # use your logging things here
def __str__(self):
return self.msg
Use msg='' because then you don't need to always specify a message.
Don't.
Factor out the calls you need to unit test, and move your exception handler out:
try:
testableFunctionCall()
except:
lgr.exception('Unexpected Error')
and test testableFunctionCall().
Alternatively, use the testfixtures library to test the logging itself:
from testfixtures import LogCapture
with LogCapture() as l:
callFunctionUnderTest()
l.check(
('packagename', 'ERROR', 'DataCollector Error(READ_METER_DATA): foobar'),
)