Develop Reference

Logging and exceptions in Python

I'm currently coding my first larger script which is a console based GUI where the user selects options by entering numbers to start several tasks:
I've recently implemented a lot of error handling to prevent the window from closing if something goes wrong in the background. I'm a little bit confused whether or not my approach is correct.
The basic structure of my code is as following:
There is a function read_excel() which loads some Excel files with pandas:
def read_excel(excel_path):
try:
df = pd.read_excel(excel_path, encoding="utf-8")
except FileNotFoundError:
raise FileNotFoundError('Unable to load assignment file, maybe choose custom')
return
# do stuff....
if not all(len(x) == len(signalnames) for x in [frequencies, cans]):
raise ValueError('Frequency and can number must be given for every signal in assignment file!')
else:
logging.info("Successfully loaded assignment file")
return signalnames, frequencies, group_names, cans, can_paths
This function is then used together with others in the function ft_14() which is called by the GUI:
def ft_14(files, draft assignement_path):
try:
signalnames, frequencies, group_names, cans, can_paths = read_excel(assignement_path)
except (ValueError, FileNotFoundError) as e:
logging.error(e)
return
# do stuff..
try:
wb.save(os.path.join(os.path.dirname(files[0]), "FT.14_results.xlsx"))
wb.close()
except Exception:
logging.error('Unable to save report excel')
So my attempt is to raise exceptions in the backend and then except them in functions which are called by the GUI and use logging to display them for the user. So my question is if this approach is the correct way to use exceptions and logging together or if there is a smarter way, because calling:
try:
# some function()
except Exection as e:
logging.error(e)
doesn't feel right to me.

The code you posted at the end, which makes you uncomfortable, is correct in that it does what it says it will do. If there is an exception, it logs the error.
The concern is that it also handles the exception. In many cases, you don't want to change anything else in the exception handling process -- you just want to log, and let the exception propagate normally.
https://docs.python.org/3/tutorial/errors.html#raising-exceptions
import logging
logger = logging.getLogger(__name__)
def fn(x):
try:
return x / 0
except Exception as e:
logger.error(str(e))
raise e
print ("let's do something risky")
try:
fn(20)
except Exception as e:
pass
print ("it has been done")
Notice how fn(x) can detect and handle exceptions, then re-raise them as if it had not done anything to the exception at all? You do that with "raise" and no arguments.
In your first code example, you catch an exception, and then raise a completely different exception, which happens to be of the same type:
except FileNotFoundError:
raise FileNotFoundError('Unable to load assignment file, maybe choose custom')
This can be fair game if you want to hide info, for example if the exception has internal details you do not want to expose to the outside. But you don't know what info you lost by replacing the exception entirely with a new instance. But none of those is "right" or "wrong," they're just choices you make with how to handle exceptions. (edit: it is good practice to handle them eventually or document what it can throw, but we're off topic)
None of that matters to the logger. You could have logged it right there and re-raised. You could log the original exception, then raise the sanitized version. You could handle it there, log it, and not raise (as the example you posted). The logger doesn't care what you do with the exception. If you want it logged, log it immediately.

Keeping track of exceptions and whether or not they have been handled

How would I keep track of exceptions so that I could find out whether
or not they have been handled or not?
Lets say I have some error handling code:
def errorWindow(self, error_):
self.filewin = Toplevel(self.master)
self.NothingLabel = Label(self.filewin,text=error_,justify=CENTER)
self.NothingLabel.pack()
self.nbutton = Button(self.filewin, text="OK", command=self.filewin.destroy)
self.nbutton.pack(pady=5)
# When it is called it looks something like this:
try:
# Function that is being checked for exceptions
except Exception as e:
error_ = 'Failure the following error was encountered: \n {}'.format(e)
self.errorWindow(error_)
What I want to do is use the function I have built to handle errors I expect will happen on the users end. I would like to have a function that also creates an error window for users when an error occurs that I do not expect. Ideally this should only happen if an exception has occurred. So is there a way to identify and track specific exceptions so that I may handle the unexpected errors in a different way than the ones that I do expect?

Canceling a celery task from inside itself?

I know I could return, but I'm wondering if there's something else, especially for helper methods where the task where return None would force the caller to add boilerplate checking at each invocation.
I found InvalidTaskError, but no real documentation - is this an internal thing? Is it appropriate to raise this?
I was looking for something like a self.abort() similar to the self.retry(), but didn't see anything.
Here's an example where I'd use it.
def helper(task, arg):
if unrecoverable_problems(arg):
# abort the task
raise InvalidTaskError()
#task(bind=True)
task_a(self, arg):
helper(task=self, arg=arg)
do_a(arg)
#task(bind=True)
task_b(self, arg):
helper(task=self, arg=arg)
do_b(arg)

After doing more digging, I found an example using Reject;
(copied from doc page)
The task may raise Reject to reject the task message using AMQPs basic_reject method. This will not have any effect unless Task.acks_late is enabled.
Rejecting a message has the same effect as acking it, but some brokers may implement additional functionality that can be used. For example RabbitMQ supports the concept of Dead Letter Exchanges where a queue can be configured to use a dead letter exchange that rejected messages are redelivered to.
Reject can also be used to requeue messages, but please be very careful when using this as it can easily result in an infinite message loop.
Example using reject when a task causes an out of memory condition:
import errno
from celery.exceptions import Reject
#app.task(bind=True, acks_late=True)
def render_scene(self, path):
file = get_file(path)
try:
renderer.render_scene(file)
# if the file is too big to fit in memory
# we reject it so that it's redelivered to the dead letter exchange
# and we can manually inspect the situation.
except MemoryError as exc:
raise Reject(exc, requeue=False)
except OSError as exc:
if exc.errno == errno.ENOMEM:
raise Reject(exc, requeue=False)
# For any other error we retry after 10 seconds.
except Exception as exc:
raise self.retry(exc, countdown=10)

try-except-raise clause, good behaviour?

I have noticed me writing try-except clauses like the following very much in the past. The main reason for this is to write less code.
class Synchronizer(object):
# ...
def _assert_dir(self, dirname, argname, argnum):
""" *Private*. Raises OSError if the passed string does not point
to an existing directory on the file-system. """
if not os.path.isdir(dirname):
message = 'passed `%s` argument (%d) does not point to a ' \
'directory on the file-system.'
raise OSError(message % (argname, argnum))
def synchronize(self, source_dir, dest_dir, database):
# Ensure the passed directories do exist.
try:
self._assert_dir(source_dir, 'source_dir', 2)
self._assert_dir(dest_dir, 'dest_dir', 3)
except OSError:
raise
# ...
I was doing it this way, because otherwise I would've needed to write
class Synchronizer(object):
# ...
def synchronize(self, source_dir, dest_dir, database):
# Ensure the passed directories do exist.
if not os.path.isdir(source_dir):
message = 'passed `source_dir` argument (2) does not point to a ' \
'directory on the file-system.'
raise OSError(message)
if not os.path.isdir(dest_dir):
message = 'passed `dest_dir` argument (3) does not point to a ' \
'directory on the file-system.'
raise OSError(message)
# ...
I actually like the idea of writing methods doing check-and-raise operations, but I see one big disadvantage: Readability. Especially for editors that do code-folding, the try statement is not very much telling the reader what happens inside of it, while if not os.path.isdir(source_dir) is quite a good hint.
IMHO the try-except clause is required because it would confuse the catcher of the exception (reader of the traceback) where the exception comes from.
What do you think about this design? Is it awful, great or confusing to you? Or do you have any ideas on how to improve the situation?

There are two questions that I ask myself before using try for handling exceptional conditions and if the answer is YES to both, only then I will try to handle the exception.
Q1. Is this truly an exception scenario? I do not want to execute try blocks if the condition occurs 90% of the time. It is better to use if - else in such a case.
Q2. Can I recover from the error? It makes little sense to handle the exception if I cannot recover from it. It's better to propagate it to a higher level which happens automatically without me having to write extra code.
The code posted by you does not do anything to recover if the directory does not exist and it does not appear that you can do much about it. Why not let the error propagate to a higher level? Why do you even need a try block there?

This depends upon your requirement..
If you want to catch some exception, and continue with the code in your method, then you should use the 2nd scenario. Have yout try-except block inside your method.
def function():
try:
raise IOError
except IOError e:
// Handle
//continue with reset of the function
print "This will get printed"
function()
But if you want to handle all the exception at one place, with specific action for specific type, or you just want to halt your function, if one exception is raised, you can better handle them outside your function: -
def function():
raise IOError
// subsequent code Will not execute
print "This will not get printed"
try:
function()
except IOError e:
// Handle IOError
except EOFError e1:
// Handle EOF Error
By using the 2nd way, you are actually increasing the chance of some of your codes not getting executed. In general, your try-except block should be small. They should be separated for handling exception at different points and not all the exceptions should be handled at one place.
As far as I'm concerned, I generally like to minimize my try-except block as much as possible. That way I know where exactly my exception was raised.

How should I use try...except while defining a function?

I find I've been confused by the problem that when I needn't to use try..except.For last few days it was used in almost every function I defined which I think maybe a bad practice.For example:
class mongodb(object):
def getRecords(self,tname,conditions=''):
try:
col = eval("self.db.%s" %tname)
recs = col.find(condition)
return recs
except Exception,e:
#here make some error log with e.message
What I thought is ,exceptions may be raised everywhere and I have to use try to get them.
And my question is,is it a good practice to use it everywhere when defining functions?If not are there any principles for it?Help would be appreciated!
Regards

That may not be the best thing to do. Whole point of exceptions is that you can catch them on very different level than it's raised. It's best to handle them in the place where you have enough information to make something useful with them (that is very application and context dependent).
For example code below can throw IOError("[Errno 2] No such file or directory"):
def read_data(filename):
return open(filename).read()
In that function you don't have enough information to do something with it, but in place where you actually using this function, in case of such exception, you may decide to try different filename or display error to the user, or something else:
try:
data = read_data('data-file.txt')
except IOError:
data = read_data('another-data-file.txt')
# or
show_error_message("Data file was not found.")
# or something else

This (catching all possible exceptions very broadly) is indeed considered bad practice. You'll mask the real reason for the exception.
Catch only 'explicitely named' types of exceptions (which you expect to happen and you can/will handle gracefully). Let the rest (unexpected ones) bubble as they should.
You can log these (uncaught) exceptions (globally) by overriding sys.excepthook:
import sys
import traceback
# ...
def my_uncaught_exception_hook(exc_type, exc_value, exc_traceback):
msg_exc = "".join( \
traceback.format_exception(exc_type, exc_value, exc_traceback) )
# ... log here...
sys.excepthook = my_uncaught_exception_hook # our uncaught exception hook

You must find a balance between several goals:
An application should recover from as many errors as possible by itself.
An application should report all unrecoverable errors with enough detail to fix the cause of the problem.
Errors can happen everywhere but you don't want to pollute your code with all the error handling code.
Applications shouldn't crash
To solve #3, you can use an exception hook. All unhandled exceptions will cause the current transaction to abort. Catch them at the highest level, roll back the transaction (so the database doesn't become inconsistent) and either throw them again or swallow them (so the app doesn't crash). You should use decorators for this. This solves #4 and #1.
The solution for #2 is experience. You will learn with time what information you need to solve problems. The hard part is to still have the information when an error happens. One solution is to add debug logging calls in the low level methods.
Another solution is a dictionary per thread in which you can store some bits and which you dump when an error happens.

another option is to wrap a large section of code in a try: except: (for instance in a web application, one specific GUI page) and then use sys.exc_info() to print out the error and also the stack where it occurred
import sys
import traceback
try:
#some buggy code
x = ??
except:
print sys.exc_info()[0] #prints the exception class
print sys.exc_info()[1] #prints the error message
print repr(traceback.format_tb(sys.exc_info()[2])) #prints the stack