Let's say I want to be able to log to file every time any exception is raised, anywhere in my program. I don't want to modify any existing code.
Of course, this could be generalized to being able to insert a hook every time an exception is raised.
Would the following code be considered safe for doing such a thing?
class MyException(Exception):
def my_hook(self):
print('---> my_hook() was called');
def __init__(self, *args, **kwargs):
global BackupException;
self.my_hook();
return BackupException.__init__(self, *args, **kwargs);
def main():
global BackupException;
global Exception;
BackupException = Exception;
Exception = MyException;
raise Exception('Contrived Exception');
if __name__ == '__main__':
main();
If you want to log uncaught exceptions, just use sys.excepthook.
I'm not sure I see the value of logging all raised exceptions, since lots of libraries will raise/catch exceptions internally for things you probably won't care about.
Your code as far as I can tell would not work.
__init__ has to return None and you are trying to return an instance of backup exception. In general if you would like to change what instance is returned when instantiating a class you should override __new__.
Unfortunately you can't change any of the attributes on the Exception class. If that was an option you could have changed Exception.__new__ and placed your hook there.
the "global Exception" trick will only work for code in the current module. Exception is a builtin and if you really want to change it globally you need to import __builtin__; __builtin__.Exception = MyException
Even if you changed __builtin__.Exception it will only affect future uses of Exception, subclasses that have already been defined will use the original Exception class and will be unaffected by your changes. You could loop over Exception.__subclasses__ and change the __bases__ for each one of them to insert your Exception subclass there.
There are subclasses of Exception that are also built-in types that you also cannot modify, although I'm not sure you would want to hook any of them (think StopIterration).
I think that the only decent way to do what you want is to patch the Python sources.
This code will not affect any exception classes that were created before the start of main, and most of the exceptions that happen will be of such kinds (KeyError, AttributeError, and so forth). And you can't really affect those "built-in exceptions" in the most important sense -- if anywhere in your code is e.g. a 1/0, the real ZeroDivisionError will be raised (by Python's own internals), not whatever else you may have bound to that exceptions' name.
So, I don't think your code can do what you want (despite all the semicolons, it's still supposed to be Python, right?) -- it could be done by patching the C sources for the Python runtime, essentially (e.g. by providing a hook potentially caught on any exception even if it's later caught) -- such a hook currently does not exist because the use cases for it would be pretty rare (for example, a StopIteration is always raised at the normal end of every for loop -- and caught, too; why on Earth would one want to trace that, and the many other routine uses of caught exceptions in the Python internals and standard library?!).
Download pypy and instrument it.
Related
Lets assume there is a method, and I want this method to be called only at certain conditions of a class variable, and otherwise I want to throw an exception:
def foo(self):
if not self.somecondition:
raise ????
else:
#do the thing it supposed to do
Now I am looking for a correct exception for that. I went through them all at python built-in Exceptions docs and found nothing suitable. The classic valueerror is not quite suitable because there is not necessarily a problem with the values but rather with the method being called in this
circumstances.
What exception should I use then? Do I need to create my own in this case or does python has a built in answer?
According to a given protocol (which I cannot change, only implement), some function initialize_foo() is supposed to be called only once:
def initialize_foo():
"""
...
Note:
You must call this function exactly once.
"""
I would like to recognize a protocol abuse where it is called twice, and raise an exception:
_foo_initialized = False
def initialize_foo():
"""
...
Note:
You must call this function exactly once.
"""
if _foo_initialized:
raise <what>?
...
_foo_initialized = True
The problem is what class's object to raise. Looking at the standard exceptions, I can't find anything to subclass except Exception, which seems too general.
What is the general practice in this case?
I'd use RuntimeError.
It is often used for that sort of stuff, even in the standard library. You can find an example very similar to your use case in the warnings module:
if self._entered:
raise RuntimeError("Cannot enter %r twice" % self)
Another example is in threading:
if self._started.is_set():
raise RuntimeError("threads can only be started once")
You can also consider raising an ad-hoc exception (possibly a subclass of RuntimeError) if that error is supposed to be caught and if you feel that RuntimeError may be ambiguous.
I would recommend you to subclass a warning, instead of having an exception, since I have a feeling that a lot of times you'd rather continue running after this happens.
Say I have this code:
def wait_for_x(timeout_at=None):
while condition_that_could_raise_exceptions
if timeout_at is not None and time.time() > timeout_at:
raise SOMEEXCEPTIONHERE
do_some_stuff()
try:
foo()
wait_for_x(timeout_at=time.time() + 10)
bar()
except SOMEEXCEPTIONHERE:
# report timeout, move on to something else
How do I pick an exception type SOMEEXCEPTIONHERE for the function? Is it reasonable to create a unique exception type for that function, so that there's no danger of condition_that_could_raise_exceptions raising the same exception type?
wait_for_x.Timeout = type('Timeout', (Exception,), {})
If distinguishing exceptions from wait_for_x from those from condition_that_could_raise_exceptions is important enough, then sure, define a new exception type. After all, the type is the main way of distinguishing different kinds of exceptions, and parsing the message tends to get messy pretty quickly.
Yes, you should certainly define your own exception class whenever none of the built-in exception types are appropriate. In some cases (say, if you're building some kind of HTML munger on top of LXML or BeautifulSoup) it might also be appropriate to use an exception from some other module.
Python Standard Library defines a lot of its own custom exceptions. It seems good practice to do that as well for personal modules or functions.
In trying to eliminate potential race condition in a python module I wrote to monitor some specialized workflows, I learned about python's "easier to ask forgiveness than permission" (EAFP) coding style, and I'm now raising lots of custom exceptions with try/except blocks where I used to use if/thens.
I'm new to python and This EAFP style makes sense logically and seems make my code more robust, but something about this feels way overboard. Is is bad practice to define one or more exceptions per method?
These custom exceptions tend to be useful only to a single method and, while it feels like a functionally correct solution, it seems like a lot of code to maintain.
Here a sample method for example:
class UploadTimeoutFileMissing(Exception):
def __init__(self, value):
self.parameter = value
def __str__(self):
return repr(self.parameter)
class UploadTimeoutTooSlow(Exception):
def __init__(self, value):
self.parameter = value
def __str__(self):
return repr(self.parameter)
def check_upload(file, timeout_seconds, max_age_seconds, min_age_seconds):
timeout = time.time() + timeout_seconds
## Check until file found or timeout
while (time.time() < timeout):
time.sleep(5)
try:
filetime = os.path.getmtime(file)
filesize = os.path.getsize(file)
except OSError:
print "File not found %s" % file
continue
fileage = time.time() - filetime
## Make sure file isn't pre-existing
if fileage > max_age_seconds:
print "File too old %s" % file
continue
## Make sure file isn't still uploading
elif fileage <= min_age_seconds:
print "File too new %s" % file
continue
return(filetime, filesize)
## Timeout
try:
filetime
filesize
raise UploadTimeoutTooSlow("File still uploading")
except NameError:
raise UploadTimeoutFileMissing("File not sent")
define one or more exceptions per method
If you mean that the exception is actually defined per method as in "within the method body", then yes. That is bad practice. This is true also if you define two exceptions that would relate to the same error but you create two because two different methods raise them.
If you ask whether it is bad practice to raise more than one exception per method, then no, that is good practice. And if the errors are not of the same category, it's perfectly ok to define several exceptions per module.
In general, for larger modules you will define more than one exception. If you would work on some arithmetic library and you would define a ZeroDivisionError and an OverflowError (if they weren't already defined in python, because you can of course re-use those) that would be perfectly fine.
Is is bad practice to define one or more exceptions per method?
Yes.
One per module is more typical. It depends, of course, on the detailed semantics. The question boils down to this: "What will you really try to catch?"
If you're never going to use except ThisVeryDetailedException: in your code, then your very detailed exception isn't very helpful.
If you can do this: except Error as e: if e.some_special_case for the very few times it matters, then you can easily simplify to one exception per module and handle your special cases as attributes of the exception rather than different types of exceptions.
The common suggestions (one per module, named Error) means that your code will often look like this.
try:
something
except some_module.Error as e:
carry on
This gives you a nice naming convention: module.Error. This covers numerous sins.
On an unrelated note, if you think you've got "potential race condition" you should probably redesign things correctly or stop trying to use threads or switch to multiprocessing. If you use multiprocessing, you'll find that it's very easy to avoid race conditions.
I'm going to weigh in on this because custom exceptions are dear to my heart. I'll explain my circumstances and the reader can weigh them against their own.
I'm the pipeline architect for a visual effects company - most of what I do involves developing what I call the "Facility API" - it's a system of a great many modules which handle everything from locating things on the filesystem, managing module/tool/project configuration, to handling datatypes from various CG applications to enable collaboration.
I go to great lengths to try to ensure that Python's built-in exceptions never bubble up. Since our developers will be relying on an ecosystem of existing modules to build their own tools on top of, having the API let a generic IOError escape is counterproductive - especially since the calling routine might not even be aware that it's reading the filesystem (abstraction is a beautiful thing). If the underlying module is unable to express something meaningful about that error, more work needs to be done.
My approach to solving this is to create a facility exception class from which all other facility exceptions are derived. There are subclasses of that for specific types of task or specific host applications - which allows me to customize error handling (for instance, exceptions raised in Maya will launch a UI to aid in troubleshooting since the usual exception would be raised in an inconspicuous console and would often be missed).
All sorts of reporting is built into the facility exception class - exceptions don't appear to a user without also being reported internally. For a range of exceptions, I get an IM any time one is raised. Others simply report quietly into a database that I can query for recent (daily or weekly) reports. Each links to EXTENSIVE data captured from the user session - typically including a screenshot, stack trace, system configuration, and a whole lot more. This means I can effectively troubleshoot problems before they're reported - and have more information at my fingertips than most users are likely able to provide.
Very fine gradations in purpose are discouraged - the exceptions accept passed values (sometimes even a dictionary instead of a string, if we want to provide plenty of data for troubleshooting) to provide with their formatted output.
So no - I don't think defining an exception or two per module is unreasonable - but they need to be meaningful and add something to the project. If you're just wrapping an IOError to raise MyIOError("I got an IO error!"), then you may want to rethink that.
I don't think it's necessary to have an extremely specific exception for every possible scenario. A single UploadTimeoutError would probably be fine, and you can just customize the exception string - that's what the strings are for, after all. Note how python doesn't have a separate exception for every possible type of syntax error, just a general SyntaxError.
Also - is it actually necessary to define the __init__ and __str__ methods for each of your custom exceptions? As far as I can tell, if you're not implementing any unusual behavior, you don't need to add any code:
>>> class MyException(Exception): pass
...
>>> raise MyException("oops!")
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
MyException: oops!
>>> str(MyException("oops!"))
'oops!'
I have a class (see this previous question if you are interested) which tracks, amongst other things, errors. The class is called in a variety of situations, one of them is during an exception. Even though my class calls sys.exc_clear() as part of the regular course of events, the next time the class is called (even when there is no error, such as when I am just throwing some statistical information in one of the class functions) the sys.exc_info() tuple is still full of the original non-None objects.
I can call sys.exc_clear() and sys.exc_info() is just a bunch of Nones while the thread is executing in that class, but as soon as execution returns to the main program, this ceases to be valid. My reading of the documentation suggests that this is because the execution stack has returned to another frame. This seems to be a situation tangentially mentioned in this question previously.
So, my only option appears to be tacking sys.exc_clear() after each except in my main program. I have tried it in a few places and it works. I can do this, but it seems tedious and ugly. Is another way?
ADDITION:
Imagine the main program as
import tracking
def Important_Function():
try:
something that fails
except:
myTrack.track(level='warning', technical='Failure in Important_Function' ...)
return
def Other_Function():
myTrack.track(level='info', technical='Total=0' ...)
return
myTrack = tracking.Tracking()
myTrack.track(level='debug', parties=['operator'], technical='Started the program.')
Important_Function()
Other_Function()
Then the Tracking code as:
import sys
import inspect
import traceback
... lots of initialization stuff
def track(self, level='info', technical=None, parties=None, clear=True ...):
# What are our errors?
errors = {}
errortype, errorvalue, errortraceback = sys.exc_info()
errortype, errorvalue = sys.exc_info()[:2]
errors['type'] = None
errors['class'] = errortype
errors['value'] = errorvalue
errors['arguments'] = None
errors['traceback'] = None
try:
errors['type'] = str(errortype.__name__)
try:
errors['arguments'] = str(errorvalue.__dict__['args'])
except KeyError:
pass
errors['traceback'] = traceback.format_tb(errortraceback, maxTBlevel)
except:
pass
if clear == True:
sys.exc_clear()
No multi-threading that I'm aware of. If I print sys.exc_info() right after calling sys.exc_clear(), everything has been cleared. But once I return from the track function and then re-enter it, even without errors, sys.exc_info() is back with a tuple full of the previous, old errors.
Please note that last exception information is a per-thread construct. An excerpt from sys.exc_info:
This function returns a tuple of three values that give information about the exception that is currently being handled. The information returned is specific both to the current thread and to the current stack frame.
So, running sys.exc_clear in a thread does not affect other threads.
UPDATE:
Quoting from the documentation:
Warning
Assigning the traceback return value to a local variable in a function that is handling an exception will cause a circular reference. This will prevent anything referenced by a local variable in the same function or by the traceback from being garbage collected. Since most functions don’t need access to the traceback, the best solution is to use something like exctype, value = sys.exc_info()[:2] to extract only the exception type and value. If you do need the traceback, make sure to delete it after use (best done with a try ... finally statement) or to call exc_info() in a function that does not itself handle an exception.
You do assign the traceback to a local variable, and that is why I commented your question with the suggestion to remove the “offending” line.
I believe the error is caused by a misuse of sys.exc_clear() and exceptions. The actual problem is that you're calling it after handling another exception. This is the exception that actually gets cleared, not the one in the you recorded.
A solution to your problem would be to create a different method for tracking exceptions and call it in the except clause and call it only in the except clause -- that way the exception will always be the right one.
Problems I can see in the code above:
You're calling exc.clear_exc() after another exception has been handled, clearing it not the one you want.
You want to call exc.clear_exc() to clear someone else's exceptions - this is wrong and it can break the program (e.g. if a bare raise is called after the call to a fixed version track, it will fail). Whoever is handling the exception likes to use those values, clearing them like this can do no good.
You're expecting that if there was no error, sys.exc_info() will not be set as long as you clear it each time. That is not true -- there might be data for a previous exception there in a completely unrelated call to track. Don't rely on that.
All of those are fixed by using separate methods and never using sys.clear_exc().
Oh, another thing, if those except: clauses without an exceptions are not just examples, it's advisable to handle only exceptions that you know about, and not all of them like that.