I've got (multiple) functions which I need to call a large number of times with essentially random arguments, and I need to create a log of what is returned each time, and with what arguments. Usually the function returns something or raises an error, in which case I can handle it fine.
However, I've now found some arguments that cause the function to cause a core dump, which also kills my script. What I would prefer is to raise an exception, which could then get handled by my existing exception handling code. Then it would get recorded as normal, and continue testing other inputs. Is there a way to do this?
Related
Imagine a Python code like this
results = long_task(some_arguments)
# this task takes long time
unpolished_task(results)
# this unpolished task can raise unpredicted Exceptions
What is the proper way to debug this, without waiting that long time to check how poorly writen the unpolished_task is? I know I could write some tests and save the results object to run unpolished_task later, but is there any way to modify an object after the faulty line is reached and the Exception thrown?
Or, as soon as I see this do I have to accept the execution stops?
https://i.stack.imgur.com/B2rAX.jpg
I created a big program that does a lot of different stuff. In this program, I added some error management but I would like to add management for critical errors which should start the critical_error_function().
So basically, I've used :
try :
//some fabulous code
except :
critical error(error_type)
But I am here to ask if a better way to do this...
In Python exceptions are the intended way of error handling. Assuming you wrap your whole program in one try-except block, a better way would be to
only try-except-wrap the lines that can generate exceptions instead of your complete program
catch them with a specific exception such as ValueError or even your own custom exception instead of the blank except statement
handle them appropriately. Handling could mean skipping this value, logging the error or calling your critical_error_function.
Is this idiomatic/pythonic to do like this or is there a better way? I want all the errors to get in log for in case I don't have access to the console output. Also I want to abort this code path in case the problem arises.
try:
with open(self._file_path, "wb") as out_f:
out_f.write(...)
...
except OSError as e:
log("Saving %s failed: %s" % (self._file_path, str(e)))
raise
EDIT: this question is about handling exceptions in a correct place/with correct idiom. It is not about logging class.
A proven, working scheme is to have a generic except clause at the top level of your application code to make sure any unhandled error will be logged (and re-raised fo course) - and it also gives you an opportunity to try and do some cleanup before crashing)
Once you have this, adding specific "log and re-reraise" exception handlers in your code makes sense if and when you want to capture more contextual informations in your log message, as in your snippet example. This means the exception might end up logged twice but this is hardly and issue .
If you really want to be pythonic (or if you value your error logs), use the stdlib's logging module and it's logger.exception() method that will automagically add the full traceback to the log.
Some (other) benefits of the logging module are the ability to decouple the logging configuration (which should be handled by the app itself, and can be quite fine-grained) from the logging calls (which most often happen at library code level), the compatibility with well-written libs (which already use logging so you just have to configure your loggers to get infos from 3rd-part libs - and this can really save your ass), and the ability to use different logging mechanisms (to stderr, to file, to syslog, via email alerts, whatever, and you're not restricted to a single handler) according to the log source and severity and the deployment environment.
Update:
What would you say about re-raising the same exception (as in example) or re-raising custom exception (MyLibException) instead of original one?
This is a common pattern indeed, but beware of overdoing it - you only want to do this for exceptions that are actually expected and where you really know the cause. Some exception classes can have different causes - cf OSError, 'IOErrorandRuntimeError- so never assume anything about what really caused the exception, either check it with a decently robust condition (for example the.errnofield forIOError`) or let the exception propagate. I once wasted a couple hours trying to understand why some lib complained about a malformed input file when the real reason was a permission issue (which I found out tracing the library code...).
Another possible issue with this pattern is that (in Python2 at least) you will loose the original exception and traceback, so better to log them appropriately before raising your own exception. IIRC Python3 has some mechanism to handle this situation in a cleaner way that let you preserve some of the original exception infos.
I'm creating a library that caches function return values to pkl files. However, sometimes when I terminate the program while writing to the pkl files, I wind up with corrupt pkl files (not always). I'm setting up the library to deal with these corrupt files (that lead mostly to an EOFError, but may also lead to an IOError). However, I need to create files that I know are corrupt to test this, and the method of terminating the program is not consistent. Is there some other way to write to a pkl file and be guaranteed an EOFError or IOError when I subsequently read from it?
Short answer: You don't need them.
Long answer: There's a better way to handle this, take a look below.
Ok, let's start by understanding each of these exception separately:
The EOFError happens whenever the parser reaches the end of file
without a complete representation of an object and, therefore, is unable to rebuild
the object.
An IOError represents a reading error, the file could be deleted or have it's permissions revoked during the process.
Now, let's develop a strategy for testing it.
One common idiom is to encapsulate the offending method, pickle.Pickler for example, with a method that may randomly throw these exceptions. Here is an example:
import pickle
from random import random
def chaos_pickle(obj, file, io_error_chance=0, eof_error_chance=0):
if random < io_error_chance:
raise IOError("Chaotic IOError")
if random < eof_error_chance:
raise EOFError("Chaotic EOFError")
return pickle.Pickler(obj, file)
Using this instead of the traditional pickle.Pickler ensures that your code randomly throws both of the exceptions (notice that there's a caveat, though, if you set io_error_chance to 1, it will never raise a EOFError.
This trick is quite useful when used along the mock library (unittest.mock) to create faulty objects for testing purposes.
Enjoy!
Take a bunch of your old, corrupted pickles and use those. If you don't have any, take a bunch of working pickles, truncate them quasi-randomly, and see which ones give errors when you try to load them. Alternatively, if the "corrupt" files don't need to even resemble valid pickles, you could just unpickle random crap you wouldn't expect to work. For example, mash the keyboard and try to unpickle the result.
Note that the docs say
The pickle module is not intended to be secure against erroneous or
maliciously constructed data. Never unpickle data received from an
untrusted or unauthenticated source.
What's the point of using raise if it exits the program?
Wouldn't it be just as effective to allow the crash to happen?
If I leave out the try-except block, the function crashes when I divide by zero and displays the reason. Or is there some other use that I don't know about?
def div(x,y):
try:
return(x/y)
except ZeroDivisionError as problem:
raise (problem)
I your case effect would be the same. But you may want to perform some additional logic in case of error (cleanup etc.) and perhaps raise a different (perhaps custom) error instead of original system low-level one, like with a message "Incorrect data, please check your input". And this can be done catching the error and raising a different one.
There is no point (in this case) in using raise. Normally, you'd have some code in there to do "something else" - that could include outputting some more debug information, writing some log data out, retrying the operation with a different set of parameters, etc. etc. etc.
I'm not sure there's much value in your case, where when an exception occurs it just re-raises it - it seems like someone (perhaps) intended to write some sort of handling code there, but just never got around to it.
Some great examples of the use cases for exception handling are in the Python Exception Handling Wiki --> http://wiki.python.org/moin/HandlingExceptions
The reason to re-raise an exception is to allow whatever code is calling you the opportunity to handle it after you have done something to handle it yourself. For example, you have closed a file that you were using (because cleanliness is a virtue) but your code cannot continue.
If you are not going to do anything to handle the exception, then no, there is no reason to write an exception handler for it!
The correct way to re-raise an exception is to simply use raise without any arguments. This way, whoever catches the exception (or the user of the script, if nobody catches it) gets a correct stack trace that tells where the exception was originally raised.