Develop Reference

Python, read log exception from logging.exception

I use the scrapy library for scraping and it handels all the logging for me. The log file is very big and I was hoping for another way than to read through the whole file. I'm only interested in the exception.
Is it posible to get the raw logging.exception and read from that without creating another log file?

Use the python logging facility which is what Scrapy uses. At the program entry point you need to customize this logging facility, it can be in the __init__.py file for example or anything else before you use any log related calls.
What you need to do is set different handlers for different levels (ERROR, WARNING, ...). Exception level do not exist, you might be mistaking with Error or Critical levels.
This is the "Logging How To", you should check it.
Your solution would look like :
import logging
#this is the logger Scrapy uses throughout the app.
logger = logging.getLogger()
my_exception_handler = logging.FileHandler()
my_exception_handler.setLevel(Logging.ERROR)
logger.addHandler(my_exception_handler)
Finally, if you are really talking about exception and not error or critical levels. You'll need to inherit from your Exception class or overwrite it so you can use the same logging approach I just explained.

Logging just arised exception

Is this idiomatic/pythonic to do like this or is there a better way? I want all the errors to get in log for in case I don't have access to the console output. Also I want to abort this code path in case the problem arises.
try:
with open(self._file_path, "wb") as out_f:
out_f.write(...)
...
except OSError as e:
log("Saving %s failed: %s" % (self._file_path, str(e)))
raise
EDIT: this question is about handling exceptions in a correct place/with correct idiom. It is not about logging class.

A proven, working scheme is to have a generic except clause at the top level of your application code to make sure any unhandled error will be logged (and re-raised fo course) - and it also gives you an opportunity to try and do some cleanup before crashing)
Once you have this, adding specific "log and re-reraise" exception handlers in your code makes sense if and when you want to capture more contextual informations in your log message, as in your snippet example. This means the exception might end up logged twice but this is hardly and issue .
If you really want to be pythonic (or if you value your error logs), use the stdlib's logging module and it's logger.exception() method that will automagically add the full traceback to the log.
Some (other) benefits of the logging module are the ability to decouple the logging configuration (which should be handled by the app itself, and can be quite fine-grained) from the logging calls (which most often happen at library code level), the compatibility with well-written libs (which already use logging so you just have to configure your loggers to get infos from 3rd-part libs - and this can really save your ass), and the ability to use different logging mechanisms (to stderr, to file, to syslog, via email alerts, whatever, and you're not restricted to a single handler) according to the log source and severity and the deployment environment.
Update:
What would you say about re-raising the same exception (as in example) or re-raising custom exception (MyLibException) instead of original one?
This is a common pattern indeed, but beware of overdoing it - you only want to do this for exceptions that are actually expected and where you really know the cause. Some exception classes can have different causes - cf OSError, 'IOErrorandRuntimeError- so never assume anything about what really caused the exception, either check it with a decently robust condition (for example the.errnofield forIOError`) or let the exception propagate. I once wasted a couple hours trying to understand why some lib complained about a malformed input file when the real reason was a permission issue (which I found out tracing the library code...).
Another possible issue with this pattern is that (in Python2 at least) you will loose the original exception and traceback, so better to log them appropriately before raising your own exception. IIRC Python3 has some mechanism to handle this situation in a cleaner way that let you preserve some of the original exception infos.

Python - Define custom unittest.TextTestRunner

In python unittest, can how do i define a custom unittest.TextTestRunner class? I need to use the Python logger module and ensure that all logs go to the log file created by the Python logger module. Especially when exceptions and assert errors are thrown up they need to go to this log file. I need to be able to user the logger.info() and logger.warning() functions and so on.
Can anyone give me some sample code or link to sample code or steps to do so.

How can I mirror Django logging to a second file temporarily?

I want to create task-specific log files that reflect what's happening during a particular operation, but I want these logging messages to also go to the primary Django log.
My current solution, which seems to work fine at first glance, is something like:
logger = getLogger("%s:%s" % (__name__, task_id))
handler = FileHandler(task_log_file)
logger.addHandler(handler)
# Work
logger.removeHandler(handler)
As I said, this works, but the main issue that occurs to me is that this logger isn't really temporary -- from what I've read of logging.Manager each logger will just hang around indefinitely until shutdown. In this case, when I'm done I know I won't use the logger again (okay, technically I might, but that will be rare), and assuming the system is stable this could be running through hundreds of thousands of tasks.
Is there a "right" way to do this?

You could have a way to mark-and-sweep your logging resource files, or just use a singleton pattern.

Is there any python exception notifier present?

What is my requirement ?
--> I need Exception notifier which will email to some specific configured user, about any sort of exception occurring in plain python app and web.py.
I want something similar to this http://matharvard.ca/posts/2011/jul/31/exception-notification-for-rails-3/
Is there anything same sort present ??
Please reply asap.
Thanks.

You can get what you want by:
Wrapping your code in try..except clause.
Using logging module to log the exceptions with a certain level of severity e.g ERROR.
Setting an SMTPHandler for exceptions of and above certain level.
This way is quite flexible. Your messages can be send to several places (like log files) and you can reconfigure your settings easily.

If you are not using any python heavy weight framework, try: https://github.com/fossilet/exception-notifier , it seems to be similar to the Rails' Exception notification, but quite simple.
If you are using django, seems you can use its built-in feature:https://docs.djangoproject.com/en/dev/howto/error-reporting/ ( and also see this: https://github.com/pinax/django-notification)
If using tornado, try this: https://github.com/kanevski/tornadotoad this is the most similar solution in python comparing to rails. )

You can overwrite the excepthook function from the sys module, and handle any uncought exceptions there.

I have the same requirement with you. I write a simple module to mail uncaught exceptions to the developers, as well as to record them to log files. It is used in our teams' cron scripts written in Python. Hope it can be useful for you too.