I find I've been confused by the problem that when I needn't to use try..except.For last few days it was used in almost every function I defined which I think maybe a bad practice.For example:
class mongodb(object):
def getRecords(self,tname,conditions=''):
try:
col = eval("self.db.%s" %tname)
recs = col.find(condition)
return recs
except Exception,e:
#here make some error log with e.message
What I thought is ,exceptions may be raised everywhere and I have to use try to get them.
And my question is,is it a good practice to use it everywhere when defining functions?If not are there any principles for it?Help would be appreciated!
Regards
That may not be the best thing to do. Whole point of exceptions is that you can catch them on very different level than it's raised. It's best to handle them in the place where you have enough information to make something useful with them (that is very application and context dependent).
For example code below can throw IOError("[Errno 2] No such file or directory"):
def read_data(filename):
return open(filename).read()
In that function you don't have enough information to do something with it, but in place where you actually using this function, in case of such exception, you may decide to try different filename or display error to the user, or something else:
try:
data = read_data('data-file.txt')
except IOError:
data = read_data('another-data-file.txt')
# or
show_error_message("Data file was not found.")
# or something else
This (catching all possible exceptions very broadly) is indeed considered bad practice. You'll mask the real reason for the exception.
Catch only 'explicitely named' types of exceptions (which you expect to happen and you can/will handle gracefully). Let the rest (unexpected ones) bubble as they should.
You can log these (uncaught) exceptions (globally) by overriding sys.excepthook:
import sys
import traceback
# ...
def my_uncaught_exception_hook(exc_type, exc_value, exc_traceback):
msg_exc = "".join( \
traceback.format_exception(exc_type, exc_value, exc_traceback) )
# ... log here...
sys.excepthook = my_uncaught_exception_hook # our uncaught exception hook
You must find a balance between several goals:
An application should recover from as many errors as possible by itself.
An application should report all unrecoverable errors with enough detail to fix the cause of the problem.
Errors can happen everywhere but you don't want to pollute your code with all the error handling code.
Applications shouldn't crash
To solve #3, you can use an exception hook. All unhandled exceptions will cause the current transaction to abort. Catch them at the highest level, roll back the transaction (so the database doesn't become inconsistent) and either throw them again or swallow them (so the app doesn't crash). You should use decorators for this. This solves #4 and #1.
The solution for #2 is experience. You will learn with time what information you need to solve problems. The hard part is to still have the information when an error happens. One solution is to add debug logging calls in the low level methods.
Another solution is a dictionary per thread in which you can store some bits and which you dump when an error happens.
another option is to wrap a large section of code in a try: except: (for instance in a web application, one specific GUI page) and then use sys.exc_info() to print out the error and also the stack where it occurred
import sys
import traceback
try:
#some buggy code
x = ??
except:
print sys.exc_info()[0] #prints the exception class
print sys.exc_info()[1] #prints the error message
print repr(traceback.format_tb(sys.exc_info()[2])) #prints the stack
Related
For example i have a program with this structure:
Domain logic module -> Settings module -> Settings store backend
Next is a part of Settings module.
def load_from_json(self, json_str):
try:
self.load_from_dict(json.loads(json_str))
except ValueError as e:
raise SettingsLoadDataException('Error loading json')
Need I a custom exception SettingsLoadDataException here, or I could just skip catching json.loads errors?
def load_from_json(self, json_str):
self.load_from_dict(json.loads(json_str))
Update.
Also good variant is:
def load_from_json(self, json_str):
try:
self.load_from_dict(json.loads(json_str))
except ValueError as e:
raise ValueError('Error loading json')
That is a problem only you can answer. You could catch all exceptions, or you could let the program crash if it throws an exception you don't handle. If it is vital that the program doesn't crash, catch the exception. However, you should implement a recovery method then. If the Json doesn't load properly, can your program do anything useful without it ? If it can, I would catch the exception, otherwise you could just display an error and terminate.
You should work with exceptions in such a way, that seeing a stack trace explains the problem to you immediately.
I am no Python expert, but won't you loose the piece of information that it was actually ValueError, that caused program crash? You will see only SettingsLoadDataException in a trace without any real reason of it, right?
Also, if you do not rethrow exceptions, you should catch only those, you know how to deal with. It is always better to have your program crash, than to leave it in an unexpected state.
I'm trying to catch some exceptions thrown by the requests library, with the following try-except block:
try:
get = requests.get((requester.batchesUrl)+str(id)+'/', auth=requester.auth)
except (ConnectionRefusedError, ConnectionError, MaxRetryError) as e:
print("CAUGHT ECONNECTION ERROR")
raise type(e)(str(e) + "Additional Info: Method couldn't connect to website, check that your server is running"
).with_traceback(sys.exc_info()[2])
But instead of catching the exceptions and adding "Additional Info:" to the args, I get NameError: global name 'MaxRetryError' is not defined. Now I know MaxRetryError is an exception in urllib3.exceptions.MaxRetryError. Must I import these non-built-in exception in order to catch them? This seems verbose to me considering the number of possible exceptions one is likely to need to watch out for.
Running: Python 3.3, Windows 7.
You must always always always have a name in an accessible scope before you can refer to it without throwing an exception. If that means importing, then so be it.
If you want to handle these three exceptions differently from other exceptions, then yes, you will have to import them. (You already have at least one of them, since it's at the top level of requests, but the others you may not.)
But do you really need to do that? Trying to restrict yourself to the most detailed exception possible can be as bad as just handling everything. Sometimes both extremes are the right thing to do, but think about what you really want to happen.
If you get a RequestException, but it's not a ConnectionError, do you really want that to print a traceback and abort, or do you want to log that CAUGHT ECONNECTION ERROR and reraise it wrapped in your own type? For that matter, what about an OSError that isn't a RequestException?
As a side note, do you really get urllib3.exception.*Errors from requests? I've only seen them wrapped in a requests.exceptions.RequestException. (And from a quick look at the source to HTTPAdapter.send, there's a except MaxRetryError as e: raise ConnectionError(e, request=request), so I think they're supposed to be wrapped, and if you're seeing them unwrapped that may be a bug you need to file.)
As another side note, in Python 3.3, you don't have to hack with exception messages like that anymore; you can chain and wrap exceptions, add arguments, etc. See Exceptions in the docs for more details. But I think what you want here may be to define your own exception type, then do this:
try:
get = requests.get((requester.batchesUrl)+str(id)+'/', auth=requester.auth)
except OSError as e:
print("CAUGHT ECONNECTION ERROR")
raise EConnectionError("Method couldn't connect to website, check that your server is running")
That will put the original exception in the __context__ of your wrapper exception. If you want it in the __cause__ instead, then raise EConnectionError(…) from e. Either way, it'll show up if you format the traceback.
You must import, etc. in order to catch specific exceptions.
Of course you could simply catch everything.
But the real issue may be that you need to understand why you should catch exceptions.
There are really only few cases I can think of.
1) You catch specific exceptions so that you can respond to them effectively -- If you don't know how to respond to them (requiring an understanding of what they mean and how to respond) -- Since these are generally few in number, this is not a real problem in having to import a number of modules.
2) You catch exceptions so that you can continue processing the next transaction, etc. -- though you abandon the current operation, you may still be able to continue processing acceptablely if the transactions are independent.
3) You catch the exception so that you can log detail, etc. Often re-raising the transaction after you have logged it.
4) You have a top-level exception handler so that you can report the problem in a clean manner to the end-user and possibly do some thing like saving work-in-progress before it is lost.
ADDED
You don't just willy nilly catch exceptions so that you can ignore them unless you like pain.
There are many at times we write code inside try: and except: block in Django. But my confusion is what to write inside exception: block or how to know what is the exact error and raise it.
For ex: In this particular case I am invoking an utility function from my views.py, and I have written like this.
try:
res = process_my_data(request, name, email)
except:
import sys
print sys.exc_value
Inside the process_my_data() definition I am doing some kind of DB operations. If it fails, and comes to except block what should I write here. I am not sure what to write that is why written print sys.exc_value
I think the biggest question you have to ask yourself is "why am I writing this as a try except in the first place?" Do you know that what is inside the try is capable of throwing an exception? If the answer to that is yes, then surely you know what type of exception to catch. In this instance it seems like it depends on what backend DB library you are using. Look up the documentation for the one you're using. From there it's really up to you what you want to do within the except - how do you want your program to behave if an exception is thrown? At the absolute minimum I would catch a general exception and print the output to the console with something like this:
except Exception, e:
message = str(e)
print (message)
I have noticed me writing try-except clauses like the following very much in the past. The main reason for this is to write less code.
class Synchronizer(object):
# ...
def _assert_dir(self, dirname, argname, argnum):
""" *Private*. Raises OSError if the passed string does not point
to an existing directory on the file-system. """
if not os.path.isdir(dirname):
message = 'passed `%s` argument (%d) does not point to a ' \
'directory on the file-system.'
raise OSError(message % (argname, argnum))
def synchronize(self, source_dir, dest_dir, database):
# Ensure the passed directories do exist.
try:
self._assert_dir(source_dir, 'source_dir', 2)
self._assert_dir(dest_dir, 'dest_dir', 3)
except OSError:
raise
# ...
I was doing it this way, because otherwise I would've needed to write
class Synchronizer(object):
# ...
def synchronize(self, source_dir, dest_dir, database):
# Ensure the passed directories do exist.
if not os.path.isdir(source_dir):
message = 'passed `source_dir` argument (2) does not point to a ' \
'directory on the file-system.'
raise OSError(message)
if not os.path.isdir(dest_dir):
message = 'passed `dest_dir` argument (3) does not point to a ' \
'directory on the file-system.'
raise OSError(message)
# ...
I actually like the idea of writing methods doing check-and-raise operations, but I see one big disadvantage: Readability. Especially for editors that do code-folding, the try statement is not very much telling the reader what happens inside of it, while if not os.path.isdir(source_dir) is quite a good hint.
IMHO the try-except clause is required because it would confuse the catcher of the exception (reader of the traceback) where the exception comes from.
What do you think about this design? Is it awful, great or confusing to you? Or do you have any ideas on how to improve the situation?
There are two questions that I ask myself before using try for handling exceptional conditions and if the answer is YES to both, only then I will try to handle the exception.
Q1. Is this truly an exception scenario? I do not want to execute try blocks if the condition occurs 90% of the time. It is better to use if - else in such a case.
Q2. Can I recover from the error? It makes little sense to handle the exception if I cannot recover from it. It's better to propagate it to a higher level which happens automatically without me having to write extra code.
The code posted by you does not do anything to recover if the directory does not exist and it does not appear that you can do much about it. Why not let the error propagate to a higher level? Why do you even need a try block there?
This depends upon your requirement..
If you want to catch some exception, and continue with the code in your method, then you should use the 2nd scenario. Have yout try-except block inside your method.
def function():
try:
raise IOError
except IOError e:
// Handle
//continue with reset of the function
print "This will get printed"
function()
But if you want to handle all the exception at one place, with specific action for specific type, or you just want to halt your function, if one exception is raised, you can better handle them outside your function: -
def function():
raise IOError
// subsequent code Will not execute
print "This will not get printed"
try:
function()
except IOError e:
// Handle IOError
except EOFError e1:
// Handle EOF Error
By using the 2nd way, you are actually increasing the chance of some of your codes not getting executed. In general, your try-except block should be small. They should be separated for handling exception at different points and not all the exceptions should be handled at one place.
As far as I'm concerned, I generally like to minimize my try-except block as much as possible. That way I know where exactly my exception was raised.
I can't understand what sort of exceptions I should handle 'here and now', and what sort of exceptions I should re-raise or just don't handle here, and what to do with them later (on higher tier). For example: I wrote client/server application using python3 with ssl communication. Client is supposed to verify files on any differences on them, and if diff exists then it should send this 'updated' file to server.
class BasicConnection:
#blablabla
def sendMessage(self, sock, url, port, fileToSend, buffSize):
try:
sock.connect((url, port))
while True:
data = fileToSend.read(buffSize)
if not data: break
sock.send(data)
return True
except socket.timeout as toErr:
raise ConnectionError("TimeOutError trying to send File to remote socket: %s:%d"
% (url,port)) from toErr
except socket.error as sErr:
raise ConnectionError("Error trying to send File to remote socket: %s:%d"
% (url,port)) from sErr
except ssl.SSLError as sslErr:
raise ConnectionError("SSLError trying to send File to remote socket: %s:%d"
% (url,port)) from sslErr
finally:
sock.close()
Is it right way to use exceptions in python? The problem is: what if file.read() throws IOError? Should I handle it here, or just do nothing and catch it later? And many other possible exceptions?
Client use this class (BasicConnection) to send updated files to server:
class PClient():
def __init__(self, DATA):
'''DATA = { 'sendTo' : {'host':'','port':''},
'use_ssl' : {'use_ssl':'', 'fileKey':'', 'fileCert':'', 'fileCaCert':''},
'dirToCheck' : '',
'localStorage': '',
'timeToCheck' : '',
'buffSize' : '',
'logFile' : ''} '''
self._DATA = DATA
self._running = False
self.configureLogging()
def configureLogging(self):
#blablabla
def isRun(self):
return self._running
def initPClient(self):
try:
#blablabla
return True
except ConnectionError as conErr:
self._mainLogger.exception(conErr)
return False
except FileCheckingError as fcErr:
self._mainLogger.exception(fcErr)
return False
except IOError as ioErr:
self._mainLogger.exception(ioErr)
return False
except OSError as osErr:
self._mainLogger.exception(osErr)
return False
def startPClient(self):
try:
self._running = True
while self.isRun():
try :
self._mainLogger.debug("Checking differences")
diffFiles = FileChecker().checkDictionary(self._dict)
if len(diffFiles) != 0:
for fileName in diffFiles:
try:
self._mainLogger.info("Sending updated file: %s to remote socket: %s:%d"
% (fileName,self._DATA['sendTo']['host'],self._DATA['sendTo']['port']))
fileToSend = io.open(fileName, "rb")
result = False
result = BasicConnection().sendMessage(self._sock, self._DATA['sendTo']['host'],
self._DATA['sendTo']['port'], fileToSend, self._DATA['buffSize'])
if result:
self._mainLogger.info("Updated file: %s was successfully delivered to remote socket: %s:%d"
% (fileName,self._DATA['sendTo']['host'],self._DATA['sendTo']['port']))
except ConnectionError as conErr:
self._mainLogger.exception(conErr)
except IOError as ioErr:
self._mainLogger.exception(ioErr)
except OSError as osErr:
self._mainLogger.exception(osErr)
self._mainLogger.debug("Updating localStorage %s from %s " %(self._DATA['localStorage'], self._DATA['dirToCheck']))
FileChecker().updateLocalStorage(self._DATA['dirToCheck'],
self._DATA['localStorage'])
self._mainLogger.info("Directory %s were checked" %(self._DATA['dirToCheck']))
time.sleep(self._DATA['timeToCheck'])
except FileCheckingError as fcErr:
self._mainLogger.exception(fcErr)
except IOError as ioErr:
self._mainLogger.exception(ioErr)
except OSError as osErr:
self._mainLogger.exception(osErr)
except KeyboardInterrupt:
self._mainLogger.info("Shutting down...")
self.stopPClient()
except Exception as exc:
self._mainLogger.exception(exc)
self.stopPClient()
raise RuntimeError("Something goes wrong...") from exc
def stopPClient(self):
self._running = False
Is it correct? May be someone spend his own time and just help me to understand pythonic style of handling exceptions? I can't understand what to do with such exceptions as NameError, TypeError, KeyError, ValueError...and so on.......They could be thrown at any statement, at any time... and what to do with them, if I want to logged everything.
And what information should people usually log? If error occurs, what info about it I should log? All traceback, or just relevant message about it or something else?
I hope somebody helps me.
Thanks a lot.
In general, you should "catch" the exceptions that you expect to happen (because they may be caused by user error, or other environmental problems outside of your program's control), especially if you know what your code might be able to do about them. Just giving more details in an error report is a marginal issue, though some programs' specs may require doing that (e.g. a long-running server that's not supposed to crash due to such problems, but rather log a lot of state information, give the user a summary explanation, and just keep working for future queries).
NameError, TypeError, KeyError, ValueError, SyntaxError, AttributeError, and so on, can be thought of as due to errors in the program -- bugs, not problems outside of the programmer's control. If you're releasing a library or framework, so that your code is going to be called by other code outside of your control, then such bugs may quite likely be in that other code; you should normally let the exception propagate to help the other programmer debug their own bugs. If you're releasing an application, you own the bugs, and you must pick the strategy that helps you find them.
If your bugs show up while an end-user is running the program, you should log a lot of state information, and give the user a summary explanation and apologies (perhaps with a request to send you the log info, if you can't automate that -- or, at least, ask permission before you send anything from the user's machine to yours). You may be able to save some of the user's work so far, but often (in a program that's known to be buggy) that may not work anyway.
Most bugs should show up during your own testing of course; in that case, propagating the exception is useful as you can hook it up to a debugger and explore the bug's details.
Sometimes some exceptions like these show up just because "it's easier to ask forgiveness than permission" (EAFP) -- a perfectly acceptable programming technique in Python. In that case of course you should handle them at once. For example:
try:
return mylist[theindex]
except IndexError:
return None
here you might expect that theindex is generally a valid index into mylist, but occasionally outside of mylist's bounds -- and the latter case, by the semantics of the hypothetic app in which this snippet belongs, is not an error, just a little anomaly to be fixed by considering the list to be conceptually extended on both sides with infinite numbers of Nones. It's easier to just try/except than to properly check for positive and negative values of the index (and faster, if being out of bounds is a truly rare occurrence).
Similarly appropriate cases for KeyError and AttributeError happen less frequently, thanks to the getattr builtin and get method of dicts (which let you provide a default value), collections.defaultdict, etc; but lists have no direct equivalent of those, so the try/except is seen more frequently for IndexError.
Trying to catch syntax errors, type errors, value errors, name errors, etc, is a bit rarer and more controversial -- though it would surely be appropriate if the error was diagnosed in a "plug-in", third-party code outside your control which your framework/application is trying to load and execute dynamically (indeed that's the case where you're supplying a library or the like and need to coexist peacefully with code out of your control which might well be buggy). Type and value errors may sometimes occur within an EAFP pattern -- e.g. when you try to overload a function to accept either a string or a number and behave slightly differently in each case, catching such errors may be better than trying to check types -- but the very concept of functions thus overloaded is more often than not quite dubious.
Back to "user and environmental errors", users will inevitably make mistakes when they give you input, indicate a filename that's not actually around (or that you don't have permission to read, or to write if that's what you're supposed to be doing), and so on: all such errors should of course be caught and result in a clear explanation to the user about what's gone wrong, and another chance to get the input right. Networks sometime go down, databases or other external servers may not respond as expected, and so forth -- sometimes it's worth catching such problems and retrying (maybe after a little wait -- maybe with an indication to the user about what's wrong, e.g. they may have accidentally unplugged a cable and you want to give them a chance to fix things and tell you when to try again), sometimes (especially in unattended long-running programs) there's nothing much you can do except an ordered shutdown (and detailed logging of every possibly-relevant aspect of the environment).
So, in brief, the answer to your Q's title is, "it depends";-). I hope I have been of use in listing many of the situations and aspects on which it can depend, and recommending what's generally the most useful attitude to take towards such issues.
To start with, you don't need any _mainLogger.
If you want to catch any exceptions, maybe to log or send them by email or whatever, do that at the highest possible level -- certainly not inside this class.
Also, you definitely don't want to convert every Exception to a RuntimeError. Let it emerge. The stopClient() method has no purpose right now. When it has, we'll look at it..
You could basically wrap the ConnectionError, IOError and OSError together (like, re-raise as something else), but not much more than that...