Develop Reference

python: Handling log output of module during program execution

I'm setting up a logger in my script like shown at the bottom. This works fine for my purposes and logs my __main__ log messages and those of any modules I use to stdout and a log file.
During program execution a module call that I'm using xarray.open_dataset(file, engine="cfgrib") raises an Error in some conditions and produces the following log output:
2023-02-18 10:02:06,731 cfgrib.dataset ERROR skipping variable: paramId==228029 shortName='i10fg'
Traceback (most recent call last):
...
How can I access this output during program execution?
The raised error in the cfgrib module is handled there gracefully and program execution can continue, but the logic of my program requires that I access the error message, in particular the part saying shortName='i10fg' in order to handle the error exhaustively.
Here is how my logger is set up:
def init_log():
"""initialize logging
returns logger using log settings from the config file (settings.toml)
"""
# all settings from a settings file with reasonable defaults
lg.basicConfig(
level=settings.logging.log_level,
format=settings.logging.format,
filemode=settings.logging.logfile_mode,
filename=settings.logging.filename,
)
mylogger = lg.getLogger(__name__)
stream = lg.StreamHandler()
mylogger.addHandler(stream)
clg.install(
level=settings.logging.log_level,
logger=mylogger,
fmt="%(asctime)s %(levelname)s:\t%(message)s",
)
return mylogger
# main
log = init_log()
log.info('...reading files...')
I went through the python logging documentation and cookbook. While this contains ample examples on how to modify logging for various purposes, I could not find an example for accessing and reacting to a log message during program execution.
The Exception in my logs look this:
2023-02-20 12:22:37,209 cfgrib.dataset ERROR skipping variable: paramId==228029 shortName='i10fg'
Traceback (most recent call last):
File "/home/foo/projects/windgrabber/.venv/lib/python3.10/site-packages/cfgrib/dataset.py", line 660, in build_dataset_components
dict_merge(variables, coord_vars)
File "/home/foo/projects/windgrabber/.venv/lib/python3.10/site-packages/cfgrib/dataset.py", line 591, in dict_merge
raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='time' value=Variable(dimensions=('time',), data=array([1640995200, 1640998800, 1641002400, ..., 1672520400, 1672524000,
1672527600])) new_value=Variable(dimensions=('time',), data=array([1640973600, 1641016800, 1641060000, 1641103200, 1641146400,
1641189600, 1641232800, 1641276000, 1641319200, 1641362400,
I cannot catch the Exception directly for some reason:
...
import sys
from cfgrib.dataset import DatasetBuildError
...
try:
df = xr.open_dataset(file, engine="cfgrib").to_dataframe()
# triggering error manually like with the two lines below works as expected
# raise Exception()
# raise DatasetBuildError()
except Exception as e:
print('got an Exception')
print(e)
print(e.args)
except BaseException as e:
print('got a BaseException')
print(e.args)
except DatasetBuildError as e:
print(e)
except:
print('got any and all exception')
type, value, traceback = sys.exc_info()
print(type)
print(value)
print(traceback)
Unless I uncomment the two lines where I raise the exception manually, the except clauses are never triggered, event though I can see the DatabaseBuildError in my logs.
Not sure if this has any bearing, but while I can see the Exception as quoted above in my file log, it is not printed to stdout.

Python: How to write error in the console in txt file?

I have a python script which every 10 minutes sends me an email with everything written in the console. I am running this with the crontab in my ubuntu 18.04 vps.
Sometimes it doesn't send the mail so I assume that when an error happens execution stops but how can I get the errors to be written in a txt file so I can analyze the error ?

Logging Module
To demonstrate the approach with the logging module, this would be the general approach
import logging
# Create a logging instance
logger = logging.getLogger('my_application')
logger.setLevel(logging.INFO) # you can set this to be DEBUG, INFO, ERROR
# Assign a file-handler to that instance
fh = logging.FileHandler("file_dir.txt")
fh.setLevel(logging.INFO) # again, you can set this differently
# Format your logs (optional)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter) # This will set the format to the file handler
# Add the handler to your logging instance
logger.addHandler(fh)
try:
raise ValueError("Some error occurred")
except ValueError as e:
logger.exception(e) # Will send the errors to the file
And if I cat file_dir.txt
2019-03-14 14:52:50,676 - my_application - ERROR - Some error occurred
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ValueError: Some error occurred
Print to File
As I pointed out in the comments, you could accomplish this with print as well (I'm not sure you will be applauded for it)
# Set your stdout pointer to a file handler
with open('my_errors.txt', 'a') as fh:
try:
raise ValueError("Some error occurred")
except ValueError as e:
print(e, file=fh)
cat my_errors.txt
Some error occurred
Note that logging.exception includes the traceback in this case, which is one of the many huge benefits of that module
Edit
In the interest of completeness, the traceback module leverages a similar approach as print, where you can supply a file handle:
import traceback
import sys
with open('error.txt', 'a') as fh:
try:
raise ValueError("Some error occurred")
except ValueError as e:
e_type, e_val, e_tb = sys.exc_info()
traceback.print_exception(e_type, e_val, e_tb, file=fh)
This will include all of the information you want from logging

You can use the logging module as suggested in the comments (possibly superior but outside the scope of my knowledge), or catch the errors with try and except like:
try:
pass
#run the code you currently have
except Exception as e: # catch ALLLLLL errors!!!
print(e) # or more likely you'd want something like "email_to_me(e)"
Although this is generally frowned upon to catch all exceptions, because then should your program fail for whatever reason it will get gobbled up in the except clause so a better approach is to figure out what specific error you are encountering like IndexError and then just catch this specific error like:
try:
pass
#run the code you currently have
except IndexError as e: # catch only indexing errors!!!
print(e) # or more likely you'd want something like "email_to_me(e)"

To be able to debug and not only know the kind of error that happened, you can also get the error stack using traceback module (usually in the starting package of modules):
import traceback
try:
my_function()
except Exception as e:
print(e)
traceback.print_exc()
And then run your code 'my_code.py' in console usig >>
python my_code.py >> my_prints.txt
All the prints of your code will then be written in this .txt file, including the printed error and its stack. This is very interesting in your case or while running code on a docker if you want to detach yourself from it with ctrl+p+q and still know what is printed.

Catching unpickleable exceptions and re-raising

This is a followup to my question Hang in Python script using SQLAlchemy and multiprocessing. As discussed in that question, pickling exceptions is problematic in Python. This is usually not a issue, but one case when it is, is when errors occur in the python multiprocessing module. Since multiprocessing moves objects around by pickling, if an error occurs inside a multiprocessing process, the entire process may hang, as demonstrated in that question.
One possible approach is to fix all the problematic exceptions, as discussed in that question. This is not easy, since one cannot easily know in advance which exceptions may be called. An alternative approach, which was suggested by lbolla in an answer to the question, is to catch the exception, construct an equivalent harmless exception, and then rethrow.
However, I'm not sure of exactly how to do this. Consider the following code.
class BadExc(Exception):
def __init__(self, message, a):
'''Non-optional param in the constructor.'''
Exception.__init__(self, message)
self.a = a
import sys
try:
try:
#print foo
raise BadExc("bad exception error message", "a")
except Exception, e:
raise Exception(e.__class__.__name__ + ": " +str(e)), None, sys.exc_info()[2]
except Exception, f:
pass
import cPickle
a = cPickle.dumps(f)
l = cPickle.loads(a)
print "raising error"
raise sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2]
This code pickles and unpickles the exception, and then throws it, giving the error
raising error
Traceback (most recent call last):
File "<stdin>", line 11, in <module>
Exception: BadExc: bad exception error message
Credits to Glenn Maynard's answer to "“Inner exception” (with traceback) in Python?".
This has the important stuff, namely the traceback, the error message, and the exception type, so this might be the best one can do. But ideally I'd like something that looks exactly like the original exception, namely
Traceback (most recent call last):
File "<stdin>", line 11, in <module>
__main__.BadExc: bad exception error message
or more generally, with the name of the exception in the front, rather than Exception. Is this possible?
Alternatively, instead of the BadExc class, one can use the print foo statement instead, which gives a NameError. However, this exception does not require special handling.

You can override sys.excepthook to achieve what you want. It at least works for this example, but it's pretty hacky so please test and no promises :-)
import sys
def excepthook_wrapper(type, value, traceback):
if len(value.args) == 2:
name, msg = value.args
value.args = (msg,)
sys.__excepthook__(name, value, traceback)
else:
sys.__excepthook__(type, value, traceback)
sys.excepthook = excepthook_wrapper
(Edit: I'm not really happy with this because now 'normal' Exceptions with two arguments will get handled differently too. Possible solution, 'tag' your special Exceptions by passing "PICKLED" as a first argument and then check for that, instead of checking for the length of the args.)
And then create the Exception with two arguments, the name (__module__.__class__) and the Exception message (str(e)):
try:
try:
#print foo
raise BadExc("bad exception error message", "a")
except Exception, e:
cls = e.__class__
if hasattr(cls, '__module__'):
name = '{0}.{1}'.format(cls.__module__, cls.__name__)
else:
name = cls.__name__
raise Exception(name, str(e)), None, sys.exc_info()[2]
except Exception, f:
pass
Then this:
import cPickle
a = cPickle.dumps(f)
l = cPickle.loads(a)
print "raising error"
raise sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2]
Prints:
raising error
Traceback (most recent call last):
File "test.py", line 18, in <module>
raise BadExc("bad exception error message", "a")
__main__.BadExc: bad exception error message

"Inner exception" (with traceback) in Python?

My background is in C# and I've just recently started programming in Python. When an exception is thrown I typically want to wrap it in another exception that adds more information, while still showing the full stack trace. It's quite easy in C#, but how do I do it in Python?
Eg. in C# I would do something like this:
try
{
ProcessFile(filePath);
}
catch (Exception ex)
{
throw new ApplicationException("Failed to process file " + filePath, ex);
}
In Python I can do something similar:
try:
ProcessFile(filePath)
except Exception as e:
raise Exception('Failed to process file ' + filePath, e)
...but this loses the traceback of the inner exception!
Edit: I'd like to see both exception messages and both stack traces and correlate the two. That is, I want to see in the output that exception X occurred here and then exception Y there - same as I would in C#. Is this possible in Python 2.6? Looks like the best I can do so far (based on Glenn Maynard's answer) is:
try:
ProcessFile(filePath)
except Exception as e:
raise Exception('Failed to process file' + filePath, e), None, sys.exc_info()[2]
This includes both the messages and both the tracebacks, but it doesn't show which exception occurred where in the traceback.

Python 3
In python 3 you can do the following:
try:
raise MyExceptionToBeWrapped("I have twisted my ankle")
except MyExceptionToBeWrapped as e:
raise MyWrapperException("I'm not in a good shape") from e
This will produce something like this:
Traceback (most recent call last):
...
MyExceptionToBeWrapped: ("I have twisted my ankle")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
...
MyWrapperException: ("I'm not in a good shape")

Python 2
It's simple; pass the traceback as the third argument to raise.
import sys
class MyException(Exception): pass
try:
raise TypeError("test")
except TypeError, e:
raise MyException(), None, sys.exc_info()[2]
Always do this when catching one exception and re-raising another.

Python 3 has the raise ... from clause to chain exceptions. Glenn's answer is great for Python 2.7, but it only uses the original exception's traceback and throws away the error message and other details. Here are some examples in Python 2.7 that add context information from the current scope into the original exception's error message, but keep other details intact.
Known Exception Type
try:
sock_common = xmlrpclib.ServerProxy(rpc_url+'/common')
self.user_id = sock_common.login(self.dbname, username, self.pwd)
except IOError:
_, ex, traceback = sys.exc_info()
message = "Connecting to '%s': %s." % (config['connection'],
ex.strerror)
raise IOError, (ex.errno, message), traceback
That flavour of raise statement takes the exception type as the first expression, the exception class constructor arguments in a tuple as the second expression, and the traceback as the third expression. If you're running earlier than Python 2.2, see the warnings on sys.exc_info().
Any Exception Type
Here's another example that's more general purpose if you don't know what kind of exceptions your code might have to catch. The downside is that it loses the exception type and just raises a RuntimeError. You have to import the traceback module.
except Exception:
extype, ex, tb = sys.exc_info()
formatted = traceback.format_exception_only(extype, ex)[-1]
message = "Importing row %d, %s" % (rownum, formatted)
raise RuntimeError, message, tb
Modify the Message
Here's another option if the exception type will let you add context to it. You can modify the exception's message and then reraise it.
import subprocess
try:
final_args = ['lsx', '/home']
s = subprocess.check_output(final_args)
except OSError as ex:
ex.strerror += ' for command {}'.format(final_args)
raise
That generates the following stack trace:
Traceback (most recent call last):
File "/mnt/data/don/workspace/scratch/scratch.py", line 5, in <module>
s = subprocess.check_output(final_args)
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory for command ['lsx', '/home']
You can see that it shows the line where check_output() was called, but the exception message now includes the command line.

In Python 3.x:
raise Exception('Failed to process file ' + filePath).with_traceback(e.__traceback__)
or simply
except Exception:
raise MyException()
which will propagate MyException but print both exceptions if it will not be handled.
In Python 2.x:
raise Exception, 'Failed to process file ' + filePath, e
You can prevent printing both exceptions by killing the __context__ attribute. Here I write a context manager using that to catch and change your exception on the fly:
(see http://docs.python.org/3.1/library/stdtypes.html for expanation of how they work)
try: # Wrap the whole program into the block that will kill __context__.
class Catcher(Exception):
'''This context manager reraises an exception under a different name.'''
def __init__(self, name):
super().__init__('Failed to process code in {!r}'.format(name))
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
if exc_type is not None:
self.__traceback__ = exc_tb
raise self
...
with Catcher('class definition'):
class a:
def spam(self):
# not really pass, but you get the idea
pass
lut = [1,
3,
17,
[12,34],
5,
_spam]
assert a().lut[-1] == a.spam
...
except Catcher as e:
e.__context__ = None
raise

I don't think you can do this in Python 2.x, but something similar to this functionality is part of Python 3. From PEP 3134:
In today's Python implementation, exceptions are composed of three
parts: the type, the value, and the traceback. The 'sys' module,
exposes the current exception in three parallel variables, exc_type,
exc_value, and exc_traceback, the sys.exc_info() function returns a
tuple of these three parts, and the 'raise' statement has a
three-argument form accepting these three parts. Manipulating
exceptions often requires passing these three things in parallel,
which can be tedious and error-prone. Additionally, the 'except'
statement can only provide access to the value, not the traceback.
Adding the 'traceback' attribute to exception values makes all
the exception information accessible from a single place.
Comparison to C#:
Exceptions in C# contain a read-only 'InnerException' property that
may point to another exception. Its documentation [10] says that
"When an exception X is thrown as a direct result of a previous
exception Y, the InnerException property of X should contain a
reference to Y." This property is not set by the VM automatically;
rather, all exception constructors take an optional 'innerException'
argument to set it explicitly. The 'cause' attribute fulfills
the same purpose as InnerException, but this PEP proposes a new form
of 'raise' rather than extending the constructors of all exceptions.
C# also provides a GetBaseException method that jumps directly to
the end of the InnerException chain; this PEP proposes no analog.
Note also that Java, Ruby and Perl 5 don't support this type of thing either. Quoting again:
As for other languages, Java and Ruby both discard the original
exception when another exception occurs in a 'catch'/'rescue' or
'finally'/'ensure' clause. Perl 5 lacks built-in structured
exception handling. For Perl 6, RFC number 88 [9] proposes an exception
mechanism that implicitly retains chained exceptions in an array
named ##.

For maximum compatibility between Python 2 and 3, you can use raise_from in the six library. https://six.readthedocs.io/#six.raise_from . Here is your example (slightly modified for clarity):
import six
try:
ProcessFile(filePath)
except Exception as e:
six.raise_from(IOError('Failed to process file ' + repr(filePath)), e)

You could use my CausedException class to chain exceptions in Python 2.x (and even in Python 3 it can be useful in case you want to give more than one caught exception as cause to a newly raised exception). Maybe it can help you.

Maybe you could grab the relevant information and pass it up? I'm thinking something like:
import traceback
import sys
import StringIO
class ApplicationError:
def __init__(self, value, e):
s = StringIO.StringIO()
traceback.print_exc(file=s)
self.value = (value, s.getvalue())
def __str__(self):
return repr(self.value)
try:
try:
a = 1/0
except Exception, e:
raise ApplicationError("Failed to process file", e)
except Exception, e:
print e

Assuming:
you need a solution, which works for Python 2 (for pure Python 3 see raise ... from solution)
just want to enrich the error message, e.g. providing some additional context
need the full stack trace
you can use a simple solution from the docs https://docs.python.org/3/tutorial/errors.html#raising-exceptions:
try:
raise NameError('HiThere')
except NameError:
print 'An exception flew by!' # print or log, provide details about context
raise # reraise the original exception, keeping full stack trace
The output:
An exception flew by!
Traceback (most recent call last):
File "<stdin>", line 2, in ?
NameError: HiThere
It looks like the key piece is the simplified 'raise' keyword that stands alone. That will re-raise the Exception in the except block.

Find module name of the originating exception in Python

Example:
>>> try:
... myapp.foo.doSomething()
... except Exception, e:
... print 'Thrown from:', modname(e)
Thrown from: myapp.util.url
In the above example, the exception was actually thrown at myapp/util/url.py module. Is there a way to get the __name__ of that module?
My intention is to use this in logging.getLogger function.

This should work:
import inspect
try:
some_bad_code()
except Exception, e:
frm = inspect.trace()[-1]
mod = inspect.getmodule(frm[0])
print 'Thrown from', mod.__name__
EDIT: Stephan202 mentions a corner case. In this case, I think we could default to the file name.
import inspect
try:
import bad_module
except Exception, e:
frm = inspect.trace()[-1]
mod = inspect.getmodule(frm[0])
modname = mod.__name__ if mod else frm[1]
print 'Thrown from', modname
The problem is that if the module doesn't get loaded (because an exception was thrown while reading the code in that file), then the inspect.getmodule call returns None. So, we just use the name of the file referenced by the offending frame. (Thanks for pointing this out, Stephan202!)

You can use the traceback module, along with sys.exc_info(), to get the traceback programmatically:
try:
myapp.foo.doSomething()
except Exception, e:
exc_type, exc_value, exc_tb = sys.exc_info()
filename, line_num, func_name, text = traceback.extract_tb(exc_tb)[-1]
print 'Thrown from: %s' % filename

This should do the trick:
import inspect
def modname():
t=inspect.trace()
if t:
return t[-1][1]

Python's logging package already supports this - check the documentation. You just have to specify %(module)s in the format string. However, this gives you the module where the exception was caught - not necessarily the same as the one where it was raised. The traceback, of course, gives you the precise location where the exception was raised.

I have a story about how CrashKit computes class names and package names from Python stack traces on the company blog: “Python stack trace saga”. Working code included.