The script I am writing should exit back to the shell prompt with a helpful message if the data to be processed is not exactly right. The user should fix the problems flagged until the script is happy and no longer exits with error messages. I am developing the script with TTD, so I write a pytest test before I write the function.
The most heavily up-voted answer here suggests that scripts be edited by calling sys.exit or raising SystemExit.
The function:
def istext(file_to_test):
try:
open(file_to_test).read(512)
except UnicodeDecodeError:
sys.exit('File {} must be encoded in UTF-8 (Unicode); try converting.'.format(file_to_test))
passes this test (where _non-text.png is a PNG file, i.e., not encoded in UTF-8):
def test_istext():
with pytest.raises(SystemExit):
istext('_non-text.png')
However, the script continues to run, and statements placed after the try/except block execute.
I want the script to completely exit every time so that the user can debug the data until it is correct, and the script will do what it is supposed to do (which is to process a directory full of UTF-8 text files, not PNG, JPG, PPTX... files).
Also tried:
The following also passes the test above by raising an exception that is a sub-class of SystemExit, but it also does not exit the script:
def istext(file_to_test):
class NotUTF8Error(SystemExit): pass
try:
open(file_to_test).read(512)
except UnicodeDecodeError:
raise NotUTF8Error('File {} must be UTF-8.'.format(file_to_test))
You can use raise Exception from exception syntax:
class MyException(SystemExit):
pass
def istext(file_to_test):
try:
open(file_to_test).read(512)
except UnicodeDecodeError as exception:
raise MyException(f'File {file_to_test} must be encoded in UTF-8 (Unicode); try converting.') \
from exception
I this case you doesn't change original error message and add your own message.
The try...except block is for catching an error and handling it internally. What you want to do is to re-raise the error.
def istext(file_to_test):
try:
open(file_to_test).read(512)
except UnicodeDecodeError:
print(('File {} must be encoded in UTF-8 (Unicode); try converting.'.format(file_to_test)))
raise
This will print your message, then automatically re-raise the error you've caught.
Instead of just re-raising the old error, you might want to change the error type as well. For this case, you specify raise further, e.g.:
raise NameError('I'm the shown error message')
You problem is not how to exit a program (sys.exit() works fine). You problem is that your test scenario is not raising a UnicodeDecodeError.
Here's a simplified version of your example. It works as expected:
import pytest
import sys
def foo(n):
try:
1/n
except ZeroDivisionError as e:
sys.exit('blah')
def test_foo():
# Assertion passes.
with pytest.raises(SystemExit):
foo(0)
# Assertion fails: "DID NOT RAISE <type 'exceptions.SystemExit'>"
with pytest.raises(SystemExit):
foo(9)
Add some diagnostic printing to your code to learn more. For example:
def istext(file_to_test):
try:
content = open(file_to_test).read(512)
# If you see this, no error occurred. Maybe your source
# file needs different content to trigger UnicodeDecodeError.
print('CONTENT len()', len(content))
except UnicodeDecodeError:
sys.exit('blah')
except Exception as e:
# Maybe some other type of error should also be handled?
...
In the end, what worked is similar to what #ADR proposed, with one difference: I was not able to get the formatted string syntax shown above to work correctly (f'File {file_to_test} must...'), nor could I find documentation of the f prefix for strings.
My slightly less elegant solution, then, for the (renamed) function:
def is_utf8(file):
class NotUTF8Error(SystemExit): pass
try:
open(file).read(512)
except UnicodeDecodeError as e:
raise NotUTF8Error('File {} not UTF-8: convert or delete, then retry.'.format(file)) from e
passes the pytest:
def test_is_utf81():
with pytest.raises(SystemExit):
is_utf8('/Users/tbaker/github/tombaker/mklists/mklists/_non-text.png')
Related
Is it possible to tell if there was an exception once you're in the finally clause? Something like:
try:
funky code
finally:
if ???:
print('the funky code raised')
I'm looking to make something like this more DRY:
try:
funky code
except HandleThis:
# handle it
raised = True
except DontHandleThis:
raised = True
raise
else:
raised = False
finally:
logger.info('funky code raised %s', raised)
I don't like that it requires to catch an exception, which you don't intend to handle, just to set a flag.
Since some comments are asking for less "M" in the MCVE, here is some more background on the use-case. The actual problem is about escalation of logging levels.
The funky code is third party and can't be changed.
The failure exception and stack trace does not contain any useful diagnostic information, so using logger.exception in an except block is not helpful here.
If the funky code raised then some information which I need to see has already been logged, at level DEBUG. We do not and can not handle the error, but want to escalate the DEBUG logging because the information needed is in there.
The funky code does not raise, most of the time. I don't want to escalate logging levels for the general case, because it is too verbose.
Hence, the code runs under a log capture context (which sets up custom handlers to intercept log records) and some debug info gets re-logged retrospectively:
try:
with LogCapture() as log:
funky_code() # <-- third party badness
finally:
# log events are buffered in memory. if there was an exception,
# emit everything that was captured at a WARNING level
for record in log.captured:
if <there was an exception>:
log_fn = mylogger.warning
else:
log_fn = getattr(mylogger, record.levelname.lower())
log_fn(record.msg, record.args)
Using a contextmanager
You could use a custom contextmanager, for example:
class DidWeRaise:
__slots__ = ('exception_happened', ) # instances will take less memory
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
# If no exception happened the `exc_type` is None
self.exception_happened = exc_type is not None
And then use that inside the try:
try:
with DidWeRaise() as error_state:
# funky code
finally:
if error_state.exception_happened:
print('the funky code raised')
It's still an additional variable but it's probably a lot easier to reuse if you want to use it in multiple places. And you don't need to toggle it yourself.
Using a variable
In case you don't want the contextmanager I would reverse the logic of the trigger and toggle it only in case no exception has happened. That way you don't need an except case for exceptions that you don't want to handle. The most appropriate place would be the else clause that is entered in case the try didn't threw an exception:
exception_happened = True
try:
# funky code
except HandleThis:
# handle this kind of exception
else:
exception_happened = False
finally:
if exception_happened:
print('the funky code raised')
And as already pointed out instead of having a "toggle" variable you could replace it (in this case) with the desired logging function:
mylog = mylogger.WARNING
try:
with LogCapture() as log:
funky_code()
except HandleThis:
# handle this kind of exception
else:
# In case absolutely no exception was thrown in the try we can log on debug level
mylog = mylogger.DEBUG
finally:
for record in log.captured:
mylog(record.msg, record.args)
Of course it would also work if you put it at the end of your try (as other answers here suggested) but I prefer the else clause because it has more meaning ("that code is meant to be executed only if there was no exception in the try block") and may be easier to maintain in the long run. Although it's still more to maintain than the context manager because the variable is set and toggled in different places.
Using sys.exc_info (works only for unhandled exceptions)
The last approach I want to mention is probably not useful for you but maybe useful for future readers who only want to know if there's an unhandled exception (an exception that was not caught in any except block or has been raised inside an except block). In that case you can use sys.exc_info:
import sys
try:
# funky code
except HandleThis:
pass
finally:
if sys.exc_info()[0] is not None:
# only entered if there's an *unhandled* exception, e.g. NOT a HandleThis exception
print('funky code raised')
raised = True
try:
funky code
raised = False
except HandleThis:
# handle it
finally:
logger.info('funky code raised %s', raised)
Given the additional background information added to the question about selecting a log level, this seems very easily adapted to the intended use-case:
mylog = WARNING
try:
funky code
mylog = DEBUG
except HandleThis:
# handle it
finally:
mylog(...)
You can easily assign your caught exception to a variable and use it in the finally block, eg:
>>> x = 1
>>> error = None
>>> try:
... x.foo()
... except Exception as e:
... error = e
... finally:
... if error is not None:
... print(error)
...
'int' object has no attribute 'foo'
Okay, so what it sounds like you actually just want to either modify your existing context manager, or use a similar approach: logbook actually has something called a FingersCrossedHandler that would do exactly what you want. But you could do it yourself, like:
#contextmanager
def LogCapture():
# your existing buffer code here
level = logging.WARN
try:
yield
except UselessException:
level = logging.DEBUG
raise # Or don't, if you just want it to go away
finally:
# emit logs here
Original Response
You're thinking about this a bit sideways.
You do intend to handle the exception - you're handling it by setting a flag. Maybe you don't care about anything else (which seems like a bad idea), but if you care about doing something when an exception is raised, then you want to be explicit about it.
The fact that you're setting a variable, but you want the exception to continue on means that what you really want is to raise your own specific exception, from the exception that was raised:
class MyPkgException(Exception): pass
class MyError(PyPkgException): pass # If there's another exception type, you can also inherit from that
def do_the_badness():
try:
raise FileNotFoundError('Or some other code that raises an error')
except FileNotFoundError as e:
raise MyError('File was not found, doh!') from e
finally:
do_some_cleanup()
try:
do_the_badness()
except MyError as e:
print('The error? Yeah, it happened')
This solves:
Explicitly handling the exception(s) that you're looking to handle
Making the stack traces and original exceptions available
Allowing your code that's going to handle the original exception somewhere else to handle your exception that's thrown
Allowing some top-level exception handling code to just catch MyPkgException to catch all of your exceptions so it can log something and exit with a nice status instead of an ugly stack trace
If it was me, I'd do a little re-ordering of your code.
raised = False
try:
# funky code
except HandleThis:
# handle it
raised = True
except Exception as ex:
# Don't Handle This
raise ex
finally:
if raised:
logger.info('funky code was raised')
I've placed the raised boolean assignment outside of the try statement to ensure scope and made the final except statement a general exception handler for exceptions that you don't want to handle.
This style determines if your code failed. Another approach might me to determine when your code succeeds.
success = False
try:
# funky code
success = True
except HandleThis:
# handle it
pass
except Exception as ex:
# Don't Handle This
raise ex
finally:
if success:
logger.info('funky code was successful')
else:
logger.info('funky code was raised')
If exception happened --> Put this logic in the exception block(s).
If exception did not happen --> Put this logic in the try block after the point in code where the exception can occur.
Finally blocks should be reserved for "cleanup actions," according to the Python language reference. When finally is specified the interpreter proceeds in the except case as follows: Exception is saved, then the finally block is executed first, then lastly the Exception is raised.
I've written a piece of code that uses a config file (in JSON-format)
def test_read_config_file(self):
self.assertRaises(ValueError, self.read_config_file('no_json.txt')
The original function looks like this:
def read_config_file(file_name)
config_data = None
try:
with open(file_name, 'r') as infile:
config_data = json.load(infile)
except ValueError as err:
LOGGER.error(str(err))
return config_data
When i run my testcase i get this:
2016-07-27 12:41:09,616 ERROR read_config_file(158) No JSON object could be decoded
2016-07-27 12:41:09,616 ERROR read_config_file(158) No JSON object could be decoded
2016-07-27 12:41:09,616 ERROR read_config_file(158) No JSON object could be decoded
2016-07-27 12:41:09,616 ERROR read_config_file(158) No JSON object could be decoded
no_json.txt just contains "Hi". Why am i getting 4 error here?
Thanks,
You are not using the unittest library correctly. When you write this:
def test_read_config_file(self):
self.assertRaises(ValueError, self.read_config_file('no_json.txt'))
# Btw. there was a missing closing `)`
the self.read_config_file() method is executed before the self.assertRaises. If it fails the self.assertRaises will never be called. Instead the exception bubbles up until something else catches it.
You want the self.assertRaises method to execute the self.read_config_file method. Because then and only then it can catch a potential ValueError. To do this you have two options:
Pass the method to test and the arguments separately:
self.assertRaises(ValueError, self.read_config_file, "no_json.txt")
Like this self.assertRaises will call the function you passed into it with the arguments specified. Then the exception occurs inside self.assertRaises where it can be caught and let the test succeed.
The second option is to use a context manager:
def test_read_config_file(self):
with self.assertRaises(ValueError):
self.read_config_file("no_json.txt")
Like this the exception will happen inside the with statement. In the cleanup step of the context manager the presence of such an exception can then again let the test succeed.
EDIT:
From your edit i can see that you already handle the ValueError in your self.read_config_file method. So any self.assertRaises approach will fail anyway. Either let the self.read_config_file raise the error or change your test.
The problem is that your function catches the ValueError exception that json.load raises, performs the logging, then simply goes on to return the value of config_data. Your test asserts that the function should raise an exception, but there is no code to ensure that the function does that.
The easiest way to fix this would be to modify the code by adding a raise statement to ensure that the ValueError is re-raised to be trapped by the assertRaises call:
def read_config_file(file_name)
config_data = None
try:
with open(file_name, 'r') as infile:
config_data = json.load(infile)
except ValueError as err:
LOGGER.error(str(err))
raise
return config_data
Let's say I have a program that runs continuously, waiting for order from a program with standard input. The method that keeps waiting for order is called "run" using while.
As you see, when run() gets certain order, they pass the order to certain function.
When I run the program, every time I give a command that can cause an error (say: Index error), it breaks and shut down (obviously)
I decided to try to catch the error with try/except
def a(order):
try:
<some algorithm>
return something
except Exception, error:
stderr.write(error)
stderr.flush()
def b(order):
try:
<some algorithm>
return something
except Exception, error:
stderr.write(error)
stderr.flush()
def run(order)
while stdin.notclosed:
try:
read stdin
if stdin==specific order :
x=a(stdin order)
else:
x=b(stdin order)
except Exception,error:
stderr.write(error)
stderr.flush()
run()
However, it seems the program that gives the order can't read the error. From my analyst, it seems the program that gives order only start reading stderr after the program that reads the order ends. However, due to try/catch, the program never ends. Is there anyway that to catch the error, write it, then end it. (The error can came from any function)
PS: Let's assume you can't modify or read the program that gives order. (This is competition, the reason I said this, is since that when I access the stderr, it's empty.)
Not sure if this does what you need, but you could re-raise the exception being handled by adding an emptyraisestatement at the end of theexceptblock as shown below. This will either cause the exception to be handled by the next higher-uptry/exceptblock, if there is one, or terminate the program if there isn't.
Example:
def a(order):
try:
<some algorithm>
return something
except Exception, error:
stderr.write(error)
stderr.flush()
raise # re-raise exception
def a(order):
try:
<some algorithm>
return something
except Exception, error:
import traceback
trace = traceback.format_exc()
return trace
def b(order):
try:
<some algorithm>
return something
except Exception, error:
import traceback
trace = traceback.format_exc()
return trace
def run(order)
while stdin.notclosed:
try:
read stdin
if stdin==specific order :
x=a(stdin order)
else:
x=b(stdin order)
#check if x == trace then sys.exit()
except Exception,error:
stderr.write(error)
stderr.flush()
run()
I'm using Fabric to automate, including the task of creating a directory. Here is my fabfile.py:
#!/usr/bin/env python
from fabric.api import *
def init():
try:
local('mkdir ./www')
except ##what exception?##:
#print exception name to put in above
Run fab fabfile.py and f I already have ./www created an error is raised, but I don't know what kind, so I don't know how to handle the error yet. Fabric only prints out the following:
mkdir: cannot create directory ‘./www’: File exists
Fatal error: local() encountered an error (return code 1) while executing 'mkdir ./www'
Aborting.
What I want to do is be able to find out the error type so that I can except my errors properly without blanket statements. It would be really helpful if an answer does not just tell me how to handle a mkdir exception, but print (or otherwise find the name to) any exception I may run into down the line (mkdir is just an example).
Thank you!
The issue is that fabric uses subprocess for doing these sorts of things. If you look at the source code for local you can see it doesn't actually raise an exception. It calls suprocess.Popen and uses communicate() to read stdout and stderr. If there is a non-zero return code then it returns a call to either warn or abort. The default is abort. So, to do what you want, try this:
def init():
with settings(warn_only=True):
local('mkdir ./www')
If you look at the source for abort, it looks like this:
10 def abort(msg):
21 from fabric.state import output
22 if output.aborts:
23 sys.stderr.write("\nFatal error: %s\n" % str(msg))
24 sys.stderr.write("\nAborting.\n")
25 sys.exit(1)
So, the exception would be a SystemExit exception. While you could catch this, the proper way to do it is outlined above using settings.
It is nothing to handle with exception, it is from the fabric api
try to set the entire script's warn_only setting to be true with
env.warn_only = True
Normally, when you get an uncaught exception, Python will print the exception type along with the error message:
>>> raise IOError("Error message.")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: Error message.
If that's not happening, you're probably not getting an exception at all.
If you really want to catch an arbitrary exception and print it, you want to catch Exception or BaseException. BaseException will include even things like KeyboardInterrupt, though, so be careful with that.
def init():
try:
local('mkdir ./www')
except BaseException as e:
print "local() threw a", type(e).__name__
raise # Reraise the exception
In general:
try:
some_code()
except Exception, e:
print 'Hit An Exception', e
raise
Will tell you what the exception was but if you are not planning on actually handling some of the exceptions then simply getting rid of the try: except: lines will have exactly the same effect.
Also if you run your code under a debugger then you can look at the exception(s) that you hit in more detail.
def init():
try:
local('mkdir ./www')
except Exception as e:
print e.__class__.__name__
That's all there is to it!
edit: Just re-read your question and realized that my code would only print "Fatal" in your case. It looks like fabric is throwing an error and returning their own error code so you would have to look at the documentation. I don't have any experience with fabric so I'd suggest to look here if you haven't already. Sorry if this isn't helpful!
try:
content = my_function()
except:
exit('Could not complete request.')
I want to modify the above code to check the value of content to see if it contains string. I thought of using if 'stuff' in content: or regular expressions, but I don't know how to fit it into the try; so that if the match is False, it raises the exception. Of course, I could always just add an if after that code, but is there a way to squeeze it in there?
Pseudocode:
try:
content = my_function()
if 'stuff' in content == False:
# cause the exception to be raised
except:
exit('Could not complete request.')
To raise an exception, you need to use the raise keyword. I suggest you read some more about exceptions in the manual. Assuming my_function() sometimes throws IndexError, use:
try:
content = my_function()
if 'stuff' not in content:
raise ValueError('stuff is not in content')
except (ValueError, IndexError):
exit('Could not complete request.')
Also, you should never use just except as it will catch more than you intend. It will, for example, catch MemoryError, KeyboardInterrupt and SystemExit. It will make your program harder to kill (Ctrl+C won't do what it's supposed to), error prone on low-memory conditions, and sys.exit() won't work as intended.
UPDATE: You should also not catch just Exception but a more specific type of exception. SyntaxError also inherits from Exception. That means that any syntax errors you have in your files will be caught and not reported properly.
try:
content = my_function()
if 'stuff' not in content:
raise ValueError('stuff not in content')
content2 = my_function2()
if 'stuff2' not in content2:
raise ValueError('stuff2 not in content2')
except ValueError, e:
exit(str(e))
If your code can have several possible exceptions, you can define each with a specific value. Catching it and exiting will then use this error value.
A nicer way to do this would be just to assert that the key is there:
assert 'stuff' in content, 'Stuff not in content'
If the assertion is not true, an AssertionError will be raised with the given message.
You can raise an exception with raise if that's what you're asking:
if 'stuff' not in content:
raise ValueError("stuff isn't there")
Note that you need to decide what kind of exception to raise. Here I raised ValueError. Likewise, you shouldn't use a bare except, but should use except ValueError or the like, to catch only the type of error you want to handle. In fact, in this case that's especially important. You presumably want to distinguish between a real error raised by my_function and the "stuff not in content" condition that you're testing.