Develop Reference

Logging and exceptions in Python

I'm currently coding my first larger script which is a console based GUI where the user selects options by entering numbers to start several tasks:
I've recently implemented a lot of error handling to prevent the window from closing if something goes wrong in the background. I'm a little bit confused whether or not my approach is correct.
The basic structure of my code is as following:
There is a function read_excel() which loads some Excel files with pandas:
def read_excel(excel_path):
try:
df = pd.read_excel(excel_path, encoding="utf-8")
except FileNotFoundError:
raise FileNotFoundError('Unable to load assignment file, maybe choose custom')
return
# do stuff....
if not all(len(x) == len(signalnames) for x in [frequencies, cans]):
raise ValueError('Frequency and can number must be given for every signal in assignment file!')
else:
logging.info("Successfully loaded assignment file")
return signalnames, frequencies, group_names, cans, can_paths
This function is then used together with others in the function ft_14() which is called by the GUI:
def ft_14(files, draft assignement_path):
try:
signalnames, frequencies, group_names, cans, can_paths = read_excel(assignement_path)
except (ValueError, FileNotFoundError) as e:
logging.error(e)
return
# do stuff..
try:
wb.save(os.path.join(os.path.dirname(files[0]), "FT.14_results.xlsx"))
wb.close()
except Exception:
logging.error('Unable to save report excel')
So my attempt is to raise exceptions in the backend and then except them in functions which are called by the GUI and use logging to display them for the user. So my question is if this approach is the correct way to use exceptions and logging together or if there is a smarter way, because calling:
try:
# some function()
except Exection as e:
logging.error(e)
doesn't feel right to me.

The code you posted at the end, which makes you uncomfortable, is correct in that it does what it says it will do. If there is an exception, it logs the error.
The concern is that it also handles the exception. In many cases, you don't want to change anything else in the exception handling process -- you just want to log, and let the exception propagate normally.
https://docs.python.org/3/tutorial/errors.html#raising-exceptions
import logging
logger = logging.getLogger(__name__)
def fn(x):
try:
return x / 0
except Exception as e:
logger.error(str(e))
raise e
print ("let's do something risky")
try:
fn(20)
except Exception as e:
pass
print ("it has been done")
Notice how fn(x) can detect and handle exceptions, then re-raise them as if it had not done anything to the exception at all? You do that with "raise" and no arguments.
In your first code example, you catch an exception, and then raise a completely different exception, which happens to be of the same type:
except FileNotFoundError:
raise FileNotFoundError('Unable to load assignment file, maybe choose custom')
This can be fair game if you want to hide info, for example if the exception has internal details you do not want to expose to the outside. But you don't know what info you lost by replacing the exception entirely with a new instance. But none of those is "right" or "wrong," they're just choices you make with how to handle exceptions. (edit: it is good practice to handle them eventually or document what it can throw, but we're off topic)
None of that matters to the logger. You could have logged it right there and re-raised. You could log the original exception, then raise the sanitized version. You could handle it there, log it, and not raise (as the example you posted). The logger doesn't care what you do with the exception. If you want it logged, log it immediately.

Handling Exceptions in Python 3.6

I am trying to handle exceptions in Python 3.6. I want to handle every possible exception and print the exception. When i do
try:
raise RuntimeError("Test")
except:
e = sys.exc_info()[0]
print(e)
it just prints
class '_mysql_exceptions.OperationalError'
How do i get the message of the Exception? In this case i would like the output to be "Test".

You can catch and print the Exception as follows:
try:
raise RuntimeError("Test")
except Exception as e:
print(e)
# Test
I'm not quite sure why you're trying to catch every Exception though, it would seem more prudent to let Python handle and raise these for you in general. Normally you would only catch specific Exceptions.
This behavior is not specific to Python 3.6.

Python: Make exceptions 'exiting'

In Python, is there any (proper) way to change the the default exception handling behaviour so that any uncaught exception will terminate/exit the program?
I don't want to wrap the entire program in a generic try-except block:
try:
// write code here
except Exception:
sys.exit(1)
For those asking for more specificity and/or claiming this is already the case, it's my understanding that not all Python exceptions are system-exiting: docs
Edit: It looks like I have forked processes complicating matters so won't be posting any specific details about my own mess.
If you're looking for an answer to the original question, Dmitry's comment is interesting and useful, references the 2nd answer to this question

You can use Specific exception instead of Exception because Exception is a Base class for all exceptions. For more details refer Exception tutorial
You can write your script like this-
try:
# write code here
except OverflowError:
raise SystemExit
except ArithmeticError:
sys.exit()
except IOError:
quit()
Try this different approaches to find what is exactly you are missing.
Edit 1 - Maintain Program Execution
In order to maintain your program execution try this one-
consider error_handler function is raising SystemExit exception then In your main method you need to add below code so you can maintain your program execution.
try:
error_handler()
except SystemExit:
print "sys.exit was called but I'm proceeding anyway (so there!-)."

Which exceptions i should catch and which should not in Python

For example i have a program with this structure:
Domain logic module -> Settings module -> Settings store backend
Next is a part of Settings module.
def load_from_json(self, json_str):
try:
self.load_from_dict(json.loads(json_str))
except ValueError as e:
raise SettingsLoadDataException('Error loading json')
Need I a custom exception SettingsLoadDataException here, or I could just skip catching json.loads errors?
def load_from_json(self, json_str):
self.load_from_dict(json.loads(json_str))
Update.
Also good variant is:
def load_from_json(self, json_str):
try:
self.load_from_dict(json.loads(json_str))
except ValueError as e:
raise ValueError('Error loading json')

That is a problem only you can answer. You could catch all exceptions, or you could let the program crash if it throws an exception you don't handle. If it is vital that the program doesn't crash, catch the exception. However, you should implement a recovery method then. If the Json doesn't load properly, can your program do anything useful without it ? If it can, I would catch the exception, otherwise you could just display an error and terminate.

You should work with exceptions in such a way, that seeing a stack trace explains the problem to you immediately.
I am no Python expert, but won't you loose the piece of information that it was actually ValueError, that caused program crash? You will see only SettingsLoadDataException in a trace without any real reason of it, right?
Also, if you do not rethrow exceptions, you should catch only those, you know how to deal with. It is always better to have your program crash, than to leave it in an unexpected state.

How to prevent "too broad exception" in this case?

I have a list of functions that may fail and, if one fails, I don't want the script to stop, but to continue with next function.
I am executing it with something like this :
list_of_functions = [f_a, f_b, f_c]
for current_function in list_of_functions:
try:
current_function()
except Exception:
print(traceback.format_exc())
It's working fine, but it is not PEP8 compliant:
When catching exceptions, mention specific exceptions whenever
possible instead of using a bare except: clause.
For example, use:
try:
import platform_specific_module
except ImportError:
platform_specific_module = None
A bare except: clause will catch SystemExit and KeyboardInterrupt
exceptions, making it harder to interrupt a program with Control-C,
and can disguise other problems. If you want to catch all exceptions
that signal program errors, use except Exception: (bare except is
equivalent to except BaseException: ).
A good rule of thumb is to limit use of bare 'except' clauses to two
cases:
If the exception handler will be printing out or logging the traceback; at least the user will be aware that an error has occurred.
If the code needs to do some cleanup work, but then lets the exception propagate upwards with raise . try...finally can be a better
way to handle this case.
How can I do this the good way?

The PEP8 guide you quote suggests that it is okay to use a bare exception in your case provided you are logging the errors. I would think that you should cover as many exceptions as you can/know how to deal with and then log the rest and pass, e.g.
import logging
list_of_functions = [f_a,f_b,f_c]
for current_function in list_of_functions:
try:
current_function()
except KnownException:
raise
except Exception as e:
logging.exception(e)

Use this to cheat PEP8:
try:
"""code"""
except (Exception,):
pass

I think in some rare cases catching general exception is just justified and there is a way to trick PEP8 inspection:
list_of_functions = [f_a,f_b,f_c]
for current_function in list_of_functions:
try:
current_function()
except (ValueError, Exception):
print(traceback.format_exc())
You can replace ValueError by any other. It works for me (at least in PyCharm).

You can just put a comment like except Exception as error: # pylint: disable=broad-except that's worked for me actually. I hope it could be work for you.

From issue PY-9715 on yourtrack.jetbrains.com:
From pep-0348:
BaseException
The superclass that all exceptions must inherit from. It's name was
chosen to reflect that it is at the base of the exception hierarchy
while being an exception itself. "Raisable" was considered as a name,
it was passed on because its name did not properly reflect the fact
that it is an exception itself.
Direct inheritance of BaseException is not expected, and will be
discouraged for the general case. Most user-defined exceptions should
inherit from Exception instead. This allows catching Exception to
continue to work in the common case of catching all exceptions that
should be caught. Direct inheritance of BaseException should only be
done in cases where an entirely new category of exception is desired.
But, for cases where all exceptions should be caught blindly, except
BaseException will work.

You can avoid the error if you then re-raise the Exception. This way you are able to do damage control and not endanger loosing track of its occurance.

Do you perhaps mean that each function can raise different exceptions? When you name the exception type in the except clause it can be any name that refers to an exception, not just the class name.
eg.
def raise_value_error():
raise ValueError
def raise_type_error():
raise TypeError
def raise_index_error():
doesnt_exist
func_and_exceptions = [(raise_value_error, ValueError), (raise_type_error, TypeError),
(raise_index_error, IndexError)]
for function, possible_exception in func_and_exceptions:
try:
function()
except possible_exception as e:
print("caught", repr(e), "when calling", function.__name__)
prints:
caught ValueError() when calling raise_value_error
caught TypeError() when calling raise_type_error
Traceback (most recent call last):
File "run.py", line 14, in <module>
function()
File "run.py", line 8, in raise_index_error
doesnt_exist
NameError: name 'doesnt_exist' is not defined
Of course that leaves you with not knowing what to do when each exception occurs. But since you just want to ignore it and carry on then that's not a problem.

First, generate the pylintrc using the below command
pylint --generate-rcfile > .pylintrc
For reference:
https://learn.microsoft.com/en-us/visualstudio/python/linting-python-code?view=vs-2022
Search for disable (uncomment if needed) in the generate pylintrc file and add the below exception.
broad-except
Rerun the pylint command and see the magic