Develop Reference

What is the difference between except and except BaseException

What is the difference between those two:
except:
# do something
and
except BaseException as be:
print(be)
I mean in the first case all possible exception are caught, but is this true for the second?
Also can the error message be printed using the first case?

The accepted answer is incorrect incomplete (at least for Python 3.6 and above).
By catching Exception you catch most errors - basically all the errors that any module you use might throw.
By catching BaseException, in addition to all the above exceptions, you also catch exceptions of the types SystemExit, KeyboardInterrupt, and GeneratorExit.
By catching KeyboardInterrupt, for example, you may stop your code from exiting after an initiated exit by the user (like pressing ^C in the console, or stopping launched application on some interpreters). This could be a wanted behavior (for example - to log an exit), but should be used with extreme care!
In the above example, by catching BaseException, you may cause your application to hang when you want it to exit.

Practically speaking, there is no difference between except: and except BaseException:, for any current Python release.
That's because you can't just raise any type of object as an exception. The raise statement explicitly disallows raising anything else:
[...] raise evaluates the first expression as the exception object. It must be either a subclass or an instance of BaseException.
Bold emphasis mine. This has not always been the case however, in older Python releases (2.4 and before) you could use strings as exceptions too.
The advantage then is that you get to have easy access to the caught exception. In order to be able to add as targetname, you must catch a specific class of exceptions, and only BaseException is going to do that.
You can still access the currently active exception by using sys.exc_info() though:
except:
be = sys.exc_info()[1]
Pick what you feel is more readable for your future self and for your colleagues.

Which exceptions i should catch and which should not in Python

For example i have a program with this structure:
Domain logic module -> Settings module -> Settings store backend
Next is a part of Settings module.
def load_from_json(self, json_str):
try:
self.load_from_dict(json.loads(json_str))
except ValueError as e:
raise SettingsLoadDataException('Error loading json')
Need I a custom exception SettingsLoadDataException here, or I could just skip catching json.loads errors?
def load_from_json(self, json_str):
self.load_from_dict(json.loads(json_str))
Update.
Also good variant is:
def load_from_json(self, json_str):
try:
self.load_from_dict(json.loads(json_str))
except ValueError as e:
raise ValueError('Error loading json')

That is a problem only you can answer. You could catch all exceptions, or you could let the program crash if it throws an exception you don't handle. If it is vital that the program doesn't crash, catch the exception. However, you should implement a recovery method then. If the Json doesn't load properly, can your program do anything useful without it ? If it can, I would catch the exception, otherwise you could just display an error and terminate.

You should work with exceptions in such a way, that seeing a stack trace explains the problem to you immediately.
I am no Python expert, but won't you loose the piece of information that it was actually ValueError, that caused program crash? You will see only SettingsLoadDataException in a trace without any real reason of it, right?
Also, if you do not rethrow exceptions, you should catch only those, you know how to deal with. It is always better to have your program crash, than to leave it in an unexpected state.

Python: Must non-built-in exceptions be imported in order to catch them?

I'm trying to catch some exceptions thrown by the requests library, with the following try-except block:
try:
get = requests.get((requester.batchesUrl)+str(id)+'/', auth=requester.auth)
except (ConnectionRefusedError, ConnectionError, MaxRetryError) as e:
print("CAUGHT ECONNECTION ERROR")
raise type(e)(str(e) + "Additional Info: Method couldn't connect to website, check that your server is running"
).with_traceback(sys.exc_info()[2])
But instead of catching the exceptions and adding "Additional Info:" to the args, I get NameError: global name 'MaxRetryError' is not defined. Now I know MaxRetryError is an exception in urllib3.exceptions.MaxRetryError. Must I import these non-built-in exception in order to catch them? This seems verbose to me considering the number of possible exceptions one is likely to need to watch out for.
Running: Python 3.3, Windows 7.

You must always always always have a name in an accessible scope before you can refer to it without throwing an exception. If that means importing, then so be it.

If you want to handle these three exceptions differently from other exceptions, then yes, you will have to import them. (You already have at least one of them, since it's at the top level of requests, but the others you may not.)
But do you really need to do that? Trying to restrict yourself to the most detailed exception possible can be as bad as just handling everything. Sometimes both extremes are the right thing to do, but think about what you really want to happen.
If you get a RequestException, but it's not a ConnectionError, do you really want that to print a traceback and abort, or do you want to log that CAUGHT ECONNECTION ERROR and reraise it wrapped in your own type? For that matter, what about an OSError that isn't a RequestException?
As a side note, do you really get urllib3.exception.*Errors from requests? I've only seen them wrapped in a requests.exceptions.RequestException. (And from a quick look at the source to HTTPAdapter.send, there's a except MaxRetryError as e: raise ConnectionError(e, request=request), so I think they're supposed to be wrapped, and if you're seeing them unwrapped that may be a bug you need to file.)
As another side note, in Python 3.3, you don't have to hack with exception messages like that anymore; you can chain and wrap exceptions, add arguments, etc. See Exceptions in the docs for more details. But I think what you want here may be to define your own exception type, then do this:
try:
get = requests.get((requester.batchesUrl)+str(id)+'/', auth=requester.auth)
except OSError as e:
print("CAUGHT ECONNECTION ERROR")
raise EConnectionError("Method couldn't connect to website, check that your server is running")
That will put the original exception in the __context__ of your wrapper exception. If you want it in the __cause__ instead, then raise EConnectionError(…) from e. Either way, it'll show up if you format the traceback.

You must import, etc. in order to catch specific exceptions.
Of course you could simply catch everything.
But the real issue may be that you need to understand why you should catch exceptions.
There are really only few cases I can think of.
1) You catch specific exceptions so that you can respond to them effectively -- If you don't know how to respond to them (requiring an understanding of what they mean and how to respond) -- Since these are generally few in number, this is not a real problem in having to import a number of modules.
2) You catch exceptions so that you can continue processing the next transaction, etc. -- though you abandon the current operation, you may still be able to continue processing acceptablely if the transactions are independent.
3) You catch the exception so that you can log detail, etc. Often re-raising the transaction after you have logged it.
4) You have a top-level exception handler so that you can report the problem in a clean manner to the end-user and possibly do some thing like saving work-in-progress before it is lost.
ADDED
You don't just willy nilly catch exceptions so that you can ignore them unless you like pain.

How to know and raise exact exception in Django views

There are many at times we write code inside try: and except: block in Django. But my confusion is what to write inside exception: block or how to know what is the exact error and raise it.
For ex: In this particular case I am invoking an utility function from my views.py, and I have written like this.
try:
res = process_my_data(request, name, email)
except:
import sys
print sys.exc_value
Inside the process_my_data() definition I am doing some kind of DB operations. If it fails, and comes to except block what should I write here. I am not sure what to write that is why written print sys.exc_value

I think the biggest question you have to ask yourself is "why am I writing this as a try except in the first place?" Do you know that what is inside the try is capable of throwing an exception? If the answer to that is yes, then surely you know what type of exception to catch. In this instance it seems like it depends on what backend DB library you are using. Look up the documentation for the one you're using. From there it's really up to you what you want to do within the except - how do you want your program to behave if an exception is thrown? At the absolute minimum I would catch a general exception and print the output to the console with something like this:
except Exception, e:
message = str(e)
print (message)

Using sys.exit or SystemExit; when to use which?

Some programmers use sys.exit, others use SystemExit.
What is the difference?
When do I need to use SystemExit or sys.exit inside a function?
Example:
ref = osgeo.ogr.Open(reference)
if ref is None:
raise SystemExit('Unable to open %s' % reference)
or:
ref = osgeo.ogr.Open(reference)
if ref is None:
print('Unable to open %s' % reference)
sys.exit(-1)

No practical difference, but there's another difference in your example code - print goes to standard out, but the exception text goes to standard error (which is probably what you want).

sys.exit(s) is just shorthand for raise SystemExit(s), as described in the former's docstring; try help(sys.exit). So, instead of either one of your example programs, you can do
sys.exit('Unable to open %s' % reference)

There are 3 exit functions, in addition to raising SystemExit.
The underlying one is os._exit, which requires 1 int argument, and exits immediately with no cleanup. It's unlikely you'll ever want to touch this one, but it is there.
sys.exit is defined in sysmodule.c and just runs PyErr_SetObject(PyExc_SystemExit, exit_code);, which is effectively the same as directly raising SystemExit. In fine detail, raising SystemExit is probably faster, since sys.exit requires an LOAD_ATTR and CALL_FUNCTION vs RAISE_VARARGS opcalls. Also, raise SystemExit produces slightly smaller bytecode (4bytes less), (1 byte extra if you use from sys import exit since sys.exit is expected to return None, so includes an extra POP_TOP).
The last exit function is defined in site.py, and aliased to exit or quit in the REPL. It's actually an instance of the Quitter class (so it can have a custom __repr__, so is probably the slowest running. Also, it closes sys.stdin prior to raising SystemExit, so it's recommended for use only in the REPL.
As for how SystemExit is handled, it eventually causes the VM to call os._exit, but before that, it does some cleanup. It also runs atexit._run_exitfuncs() which runs any callbacks registered via the atexit module. Calling os._exit directly bypasses the atexit step.

My personal preference is that at the very least SystemExit is raised (or even better - a more meaningful and well documented custom exception) and then caught as close to the "main" function as possible, which can then have a last chance to deem it a valid exit or not. Libraries/deeply embedded functions that have sys.exit is just plain nasty from a design point of view. (Generally, exiting should be "as high up" as possible)

According to documentation sys.exit(s) effectively does raise SystemExit(s), so it's pretty much the same thing.

While the difference has been answered by many answers, Cameron Simpson makes an interesting point in https://mail.python.org/pipermail/python-list/2016-April/857869.html:
TL;DR: It's better to just raise a "normal" exception, and use SystemExit or sys.exit only at the top levels of a script.
I m on python 2.7 and Linux , I have a simple code need suggestion if I
I could replace sys.exit(1) with raise SystemExit .
==Actual code==
def main():
try:
create_logdir()
create_dataset()
unittest.main()
except Exception as e:
logging.exception(e)
sys.exit(EXIT_STATUS_ERROR)
if __name__ == '__main__': main()
==Changed Code==
def main():
try:
create_logdir()
create_dataset()
unittest.main()
except Exception as e:
logging.exception(e)
raise SystemExit
if __name__ == '__main__':
main()
I am against both of these personally. My preferred pattern is like
this:
def main(argv):
try:
...
except Exception as e:
logging.exception(e)
return 1
if __name__ == '__main__':
sys.exit(main(sys.argv))
Notice that main() is back to being a normal function with normal
returns.
Also, most of us would avoid the "except Exception" and just let a top
level except bubble out: that way you get a stack backtrace for
debugging. I agree it prevents logging the exception and makes for
uglier console output, but I think it is a win. And if you do want
to log the exception there is always this:
try:
... except Exception as e:
logging.exception(e)
raise
to recite the exception into the log and still let it bubble out
normally.
The problem with the "except Exception" pattern is that it catches and
hides
every exception, not merely the narrow set of specific exceptions that you understand.
Finally, it is frowned upon to raise a bare Exception class. In
python 3 I believe it is actually forbidden, so it is nonportable
anyway. But even In Python to it is best to supply an Exception
instance, not the class:
raise SystemExit(1)
All the functions in try block have exception bubbled out using raise
Example for create_logdir() here is the function definition
def create_logdir():
try:
os.makedirs(LOG_DIR)
except OSError as e:
sys.stderr.write("Failed to create log directory...Exiting !!!")
raise
print "log file: " + corrupt_log
return True
def main():
try:
create_logdir()
except Exception as e:
logging.exception(e)
raise SystemExit
(a) In case if create_logdir() fails we will get the below error ,is
this fine or do I need to improve this code.
Failed to create log directory...Exiting !!!ERROR:root:[Errno 17] File
exists: '/var/log/dummy'
Traceback (most recent call last):
File "corrupt_test.py", line 245, in main
create_logdir()
File "corrupt_test.py", line 53, in create_logdir
os.makedirs(LOG_DIR)
File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
OSError: [Errno 17] File exists: '/var/log/dummy'
I prefer the bubble out approach, perhap with a log or warning
messages as you have done, eg:
logging.exception("create_logdir failed: makedirs(%r): %s" %
(LOG_DIR, e)) raise
(Also not that that log message records more context: context is very
useful when debugging problems.)
For very small scripts sys.stderr.write is ok, but in general any of
your functions that turned out to be generally useful might migrate
into a library in order to be reused; consider that stderr is not
always the place for messages; instead reading for the logging module
with error() or wanr() or exception() as appropriate. There is more
scope for configuring where the output goes that way without wiring
it into your inner functions.
Can I have just raise , instead of SystemExit or sys.exit(1) . This
looks wrong to me
def main():
try:
create_logdir()
except Exception as e
logging.exception(e)
raise
This is what I would do, myself.
Think: has the exception been "handled", meaning has the situation
been dealt with because it was expected? If not, let the exception
bubble out so that the user knows that something not understood by
the program has occurred.
Finally, it is generally bad to SystemExit or sys.exit() from inside
anything other than the outermost main() function. And I resist it
even there; the main function, if written well, may often be called
from somewhere else usefully, and that makes it effectively a library
function (it has been reused). Such a function should not
unilaterally abort the program. How rude! Instead, let the exception
bubble out: perhaps the caller of main() expects it and can handle
it. By aborting and not "raise"ing, you have deprived the caller of
the chance to do something appropriate, even though you yourself
(i.e. "main") do not know enough context to handle the exception.
So I am for "raise" myself. And then only because you want to log the
error. If you didn't want to log the exception you could avoid the
try/except entirely and have simpler code: let the caller worry
about unhandled exceptions!

SystemExit is an exception, which basically means that your progam had a behavior such that you want to stop it and raise an error. sys.exit is the function that you can call to exit from your program, possibily giving a return code to the system.
EDIT: they are indeed the same thing, so the only difference is in the logic behind in your program. An exception is some kind of "unwanted" behaviour, whether a call to a function is, from a programmer point of view, more of a "standard" action.