The docs don't seem to be very clear on how to address the following ...
def test():
"""
>>> import doctest
>>> doctest.ELLIPSIS_MARKER = '<ignore>'
>>> import pandas as pd
>>> raise pd.errors.InvalidIndexError # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
Traceback (most recent call last):
<ignore>
<ignore>InvalidIndexError
"""
import doctest
doctest.run_docstring_examples(test, globals())
This will work fine but is not addressing the wildcard at the front of <ignore>InvalidIndexError
def test():
"""
>>> import doctest
>>> doctest.ELLIPSIS_MARKER = '<ignore>'
>>> import pandas as pd
>>> raise pd.errors.InvalidIndexError # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
Traceback (most recent call last):
<ignore>
pandas.errors.InvalidIndexError
"""
import doctest
doctest.run_docstring_examples(test, globals())
Note pandas version is 1.1.3
References
https://docs.python.org/3/library/doctest.html#option-flags
How enable ellipsis when calling Python doctest?
Can I have an ellipsis at the beginning of the line in a Python doctest?
doctest requires exceptions to look a certain way. From the docs:
Each line of the traceback stack (if present) must be indented further than the first line of the example, or start with a non-alphanumeric character. The first line following the traceback header indented the same and starting with an alphanumeric is taken to be the start of the exception detail.
(added bold)
This means if you make the ELLIPSIS_MARKER start with an alphanumeric, it'll work properly. Here's an example using re.error:
def test():
"""
>>> import doctest
>>> doctest.ELLIPSIS_MARKER = 'MODULE.'
>>> import re
>>> raise re.error(None) # doctest: +ELLIPSIS
Traceback (most recent call last):
...
MODULE.error: None
"""
By the way:
Note that tracebacks are treated very specially. In particular, in the rewritten example, the use of ... is independent of doctest's ELLIPSIS option. The ellipsis in that example could be left out, or could just as well be three (or three hundred) commas or digits, or an indented transcript of a Monty Python skit.
For context, here's an example with no exception that uses two ellipses:
def test():
r"""
>>> print('foo\nbar\nbaz') # doctest: +ELLIPSIS
foo
...
...
"""
That said, IGNORE_EXCEPTION_DETAIL may be a better solution. (I just learned about it myself.)
When specified, an example that expects an exception passes if an exception of the expected type is raised, even if the exception detail does not match. For example, an example expecting ValueError: 42 will pass if the actual exception raised is ValueError: 3*14, but will fail, e.g., if TypeError is raised.
It will also ignore the module name used in Python 3 doctest reports.
(added bold)
For example:
def test():
"""
>>> import re
>>> raise re.error(None) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
error: foobar
"""
Note that both the exception module and exception details are ignored in this example. That's on purpose, to show a side-effect of this solution.
I'm porting some Python code from 2.7 to 3.x. The original code fails with this error:
class myTextLog(wx.PyLog):
AttributeError: 'module' object has no attribute 'PyLog'
When checking for the presence of wx.PyLog it is indeed not there:
>>> import wx
>>> wx.PyLog
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'PyLog'
>>> wx.version()
u'4.1.0 msw (phoenix) wxWidgets 3.1.4'
When I check for the presence of wx.PyLog in an older version of wx, it is there:
>>> import wx
>>> wx.PyLog
<class 'wx._misc.PyLog'>
>>> wx._misc
<module 'wx._misc' from '/usr/lib/python2.7/site-packages/wx-3.0-gtk2/wx/_misc.pyc'>
>>> wx.version()
'3.0.2.0 gtk2 (classic)'
I see it being used in this old sample code: http://www2.geog.ucl.ac.uk/~plewis/bpms/src/start/Main.py
#---------------------------------------------------------------------------
# Show how to derive a custom wxLog class
class MyLog(wx.PyLog):
def __init__(self, textCtrl, logTime=0):
wx.PyLog.__init__(self)
self.tc = textCtrl
self.logTime = logTime
def DoLogString(self, message, timeStamp):
#print message, timeStamp
#if self.logTime:
# message = time.strftime("%X", time.localtime(timeStamp)) + \
# ": " + message
if self.tc:
self.tc.AppendText(message + '\n')
Somewhere along the line it appears to have been removed. What would be a suitable replacement for the code above given wx.PyLog is no longer available?
In python 3.x you should use: logging
You will want to take a look at Logging HOWTO
import logging
logging.basicConfig(filename='example.log', encoding='utf-8', level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
logging.error('And non-ASCII stuff, too, like Øresund and Malmö')
Python 2.7
I have my own exception:
class NoSourceFileError(Exception):
def __init__(self, logger, massage):
Exception.__init__(self, massage)
logger.logger.info(massage)
And it's calls with:
...
else:
raise NoSourceFileError('ERROR: can not find file %s for %s' % (add_file, add_name))
Problem here as you see - that I pass two variables (add_file, add_name) - but __init__ can accept only one var (message).
How can I pass both of them?
I tried play with *args - but can't make it work.
Logger - my additional class for logging.
The issue is not the string, which is fine. It is that you are passing logger into the exception.
Remove the logger parameter (and usage) and it will work:
>>> class NoSourceFileError(Exception):
... def __init__(self, message):
... Exception.__init__(self, message)
...
>>> raise NoSourceFileError('ERROR: can not find file %s for %s' % ('x', 'y'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
__main__.NoSourceFileError: ERROR: can not find file x for y
>>>
I'm not sure what logger is meant to be. But if you want to use it, you need to pass it as an argument to the exception when raising it.
On string formatting, it on it's own will always count as a single argument assuming it is done right. So can't count for more than 1.
As #werkritter has said, if you do want to use logger without having to pass it in as a parameter - define it globally. I will assume that it is meant to log errors, in which case it would make sense to have it defined globally anyway.
I am printing Python exception messages to a log file with logging.error:
import logging
try:
1/0
except ZeroDivisionError as e:
logging.error(e) # ERROR:root:division by zero
Is it possible to print more detailed information about the exception and the code that generated it than just the exception string? Things like line numbers or stack traces would be great.
logger.exception will output a stack trace alongside the error message.
For example:
import logging
try:
1/0
except ZeroDivisionError:
logging.exception("message")
Output:
ERROR:root:message
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ZeroDivisionError: integer division or modulo by zero
#Paulo Cheque notes, "be aware that in Python 3 you must call the logging.exception method just inside the except part. If you call this method in an arbitrary place you may get a bizarre exception. The docs alert about that."
Using exc_info options may be better, to allow you to choose the error level (if you use exception, it will always be at the error level):
try:
# do something here
except Exception as e:
logging.critical(e, exc_info=True) # log exception info at CRITICAL log level
One nice thing about logging.exception that SiggyF's answer doesn't show is that you can pass in an arbitrary message, and logging will still show the full traceback with all the exception details:
import logging
try:
1/0
except ZeroDivisionError:
logging.exception("Deliberate divide by zero traceback")
With the default (in recent versions) logging behaviour of just printing errors to sys.stderr, it looks like this:
>>> import logging
>>> try:
... 1/0
... except ZeroDivisionError:
... logging.exception("Deliberate divide by zero traceback")
...
ERROR:root:Deliberate divide by zero traceback
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ZeroDivisionError: integer division or modulo by zero
Quoting
What if your application does logging some other way – not using the logging module?
Now, traceback could be used here.
import traceback
def log_traceback(ex, ex_traceback=None):
if ex_traceback is None:
ex_traceback = ex.__traceback__
tb_lines = [ line.rstrip('\n') for line in
traceback.format_exception(ex.__class__, ex, ex_traceback)]
exception_logger.log(tb_lines)
Use it in Python 2:
try:
# your function call is here
except Exception as ex:
_, _, ex_traceback = sys.exc_info()
log_traceback(ex, ex_traceback)
Use it in Python 3:
try:
x = get_number()
except Exception as ex:
log_traceback(ex)
You can log the stack trace without an exception.
https://docs.python.org/3/library/logging.html#logging.Logger.debug
The second optional keyword argument is stack_info, which defaults to False. If true, stack information is added to the logging message, including the actual logging call. Note that this is not the same stack information as that displayed through specifying exc_info: The former is stack frames from the bottom of the stack up to the logging call in the current thread, whereas the latter is information about stack frames which have been unwound, following an exception, while searching for exception handlers.
Example:
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> logging.getLogger().info('This prints the stack', stack_info=True)
INFO:root:This prints the stack
Stack (most recent call last):
File "<stdin>", line 1, in <module>
>>>
If you use plain logs - all your log records should correspond this rule: one record = one line. Following this rule you can use grep and other tools to process your log files.
But traceback information is multi-line. So my answer is an extended version of solution proposed by zangw above in this thread. The problem is that traceback lines could have \n inside, so we need to do an extra work to get rid of this line endings:
import logging
logger = logging.getLogger('your_logger_here')
def log_app_error(e: BaseException, level=logging.ERROR) -> None:
e_traceback = traceback.format_exception(e.__class__, e, e.__traceback__)
traceback_lines = []
for line in [line.rstrip('\n') for line in e_traceback]:
traceback_lines.extend(line.splitlines())
logger.log(level, traceback_lines.__str__())
After that (when you'll be analyzing your logs) you could copy / paste required traceback lines from your log file and do this:
ex_traceback = ['line 1', 'line 2', ...]
for line in ex_traceback:
print(line)
Profit!
This answer builds up from the above excellent ones.
In most applications, you won't be calling logging.exception(e) directly. Most likely you have defined a custom logger specific for your application or module like this:
# Set the name of the app or module
my_logger = logging.getLogger('NEM Sequencer')
# Set the log level
my_logger.setLevel(logging.INFO)
# Let's say we want to be fancy and log to a graylog2 log server
graylog_handler = graypy.GELFHandler('some_server_ip', 12201)
graylog_handler.setLevel(logging.INFO)
my_logger.addHandler(graylog_handler)
In this case, just use the logger to call the exception(e) like this:
try:
1/0
except ZeroDivisionError, e:
my_logger.exception(e)
If "debugging information" means the values present when exception was raised, then logging.exception(...) won't help. So you'll need a tool that logs all variable values along with the traceback lines automatically.
Out of the box you'll get log like
2020-03-30 18:24:31 main ERROR File "./temp.py", line 13, in get_ratio
2020-03-30 18:24:31 main ERROR return height / width
2020-03-30 18:24:31 main ERROR height = 300
2020-03-30 18:24:31 main ERROR width = 0
2020-03-30 18:24:31 main ERROR builtins.ZeroDivisionError: division by zero
Have a look at some pypi tools, I'd name:
tbvaccine
traceback-with-variables
better-exceptions
Some of them give you pretty crash messages:
But you might find some more on pypi
A little bit of decorator treatment (very loosely inspired by the Maybe monad and lifting). You can safely remove Python 3.6 type annotations and use an older message formatting style.
fallible.py
from functools import wraps
from typing import Callable, TypeVar, Optional
import logging
A = TypeVar('A')
def fallible(*exceptions, logger=None) \
-> Callable[[Callable[..., A]], Callable[..., Optional[A]]]:
"""
:param exceptions: a list of exceptions to catch
:param logger: pass a custom logger; None means the default logger,
False disables logging altogether.
"""
def fwrap(f: Callable[..., A]) -> Callable[..., Optional[A]]:
#wraps(f)
def wrapped(*args, **kwargs):
try:
return f(*args, **kwargs)
except exceptions:
message = f'called {f} with *args={args} and **kwargs={kwargs}'
if logger:
logger.exception(message)
if logger is None:
logging.exception(message)
return None
return wrapped
return fwrap
Demo:
In [1] from fallible import fallible
In [2]: #fallible(ArithmeticError)
...: def div(a, b):
...: return a / b
...:
...:
In [3]: div(1, 2)
Out[3]: 0.5
In [4]: res = div(1, 0)
ERROR:root:called <function div at 0x10d3c6ae8> with *args=(1, 0) and **kwargs={}
Traceback (most recent call last):
File "/Users/user/fallible.py", line 17, in wrapped
return f(*args, **kwargs)
File "<ipython-input-17-e056bd886b5c>", line 3, in div
return a / b
In [5]: repr(res)
'None'
You can also modify this solution to return something a bit more meaningful than None from the except part (or even make the solution generic, by specifying this return value in fallible's arguments).
In your logging module(if custom module) just enable stack_info.
api_logger.exceptionLog("*Input your Custom error message*",stack_info=True)
If you look at the this code example (which works for Python 2 and 3) you'll see the function definition below which can extract
method
line number
code context
file path
for an entire stack trace, whether or not there has been an exception:
def sentry_friendly_trace(get_last_exception=True):
try:
current_call = list(map(frame_trans, traceback.extract_stack()))
alert_frame = current_call[-4]
before_call = current_call[:-4]
err_type, err, tb = sys.exc_info() if get_last_exception else (None, None, None)
after_call = [alert_frame] if err_type is None else extract_all_sentry_frames_from_exception(tb)
return before_call + after_call, err, alert_frame
except:
return None, None, None
Of course, this function depends on the entire gist linked above, and in particular extract_all_sentry_frames_from_exception() and frame_trans() but the exception info extraction totals less than around 60 lines.
Hope that helps!
I wrap all functions around my custom designed logger:
import json
import timeit
import traceback
import sys
import unidecode
def main_writer(f,argument):
try:
f.write(str(argument))
except UnicodeEncodeError:
f.write(unidecode.unidecode(argument))
def logger(*argv,logfile="log.txt",singleLine = False):
"""
Writes Logs to LogFile
"""
with open(logfile, 'a+') as f:
for arg in argv:
if arg == "{}":
continue
if type(arg) == dict and len(arg)!=0:
json_object = json.dumps(arg, indent=4, default=str)
f.write(str(json_object))
f.flush()
"""
for key,val in arg.items():
f.write(str(key) + " : "+ str(val))
f.flush()
"""
elif type(arg) == list and len(arg)!=0:
for each in arg:
main_writer(f,each)
f.write("\n")
f.flush()
else:
main_writer(f,arg)
f.flush()
if singleLine==False:
f.write("\n")
if singleLine==True:
f.write("\n")
def tryFunc(func, func_name=None, *args, **kwargs):
"""
Time for Successfull Runs
Exception Traceback for Unsuccessful Runs
"""
stack = traceback.extract_stack()
filename, codeline, funcName, text = stack[-2]
func_name = func.__name__ if func_name is None else func_name # sys._getframe().f_code.co_name # func.__name__
start = timeit.default_timer()
x = None
try:
x = func(*args, **kwargs)
stop = timeit.default_timer()
# logger("Time to Run {} : {}".format(func_name, stop - start))
except Exception as e:
logger("Exception Occurred for {} :".format(func_name))
logger("Basic Error Info :",e)
logger("Full Error TraceBack :")
# logger(e.message, e.args)
logger(traceback.format_exc())
return x
def bad_func():
return 'a'+ 7
if __name__ == '__main__':
logger(234)
logger([1,2,3])
logger(['a','b','c'])
logger({'a':7,'b':8,'c':9})
tryFunc(bad_func)
My approach was to create a context manager, to log and raise Exceptions:
import logging
from contextlib import AbstractContextManager
class LogError(AbstractContextManager):
def __init__(self, logger=None):
self.logger = logger.name if isinstance(logger, logging.Logger) else logger
def __exit__(self, exc_type, exc_value, traceback):
if exc_value is not None:
logging.getLogger(self.logger).exception(exc_value)
with LogError():
1/0
You can either pass a logger name or a logger instance to LogError(). By default it will use the base logger (by passing None to logging.getLogger).
One could also simply add a switch for raising the error or just logging it.
If you can cope with the extra dependency then use twisted.log, you don't have to explicitly log errors and also it returns the entire traceback and time to the file or stream.
A clean way to do it is using format_exc() and then parse the output to get the relevant part:
from traceback import format_exc
try:
1/0
except Exception:
print 'the relevant part is: '+format_exc().split('\n')[-2]
Regards
This is more of a curiosity question than anything else. I'm new with Python and playing around with it. I've just looked at the base64 module. What if instead of doing:
import base64
string = 'Foo Bar'
encoded = base664.b64encode
I wanted to do something like:
>>> class b64string():
>>> <something>
>>>
>>> string = b64string('Foo Bar')
>>> string
'Foo Bar'
>>> string.encode64()
'Rm9vIEJhcg=='
>>> string
'Rm9vIEJhcg=='
>>> string.assign('QmFyIEZvbw==')
>>> string
'QmFyIEZvbw=='
>>> string.b64decode()
'Bar Foo'
>>> string
'Bar Foo'
Is there a simple, pythonic way to create that class?
I've begun with this:
>>> class b64string(base64):
... def __init__(self, v):
... self.value=v
And already I get:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
module.__init__() takes at most 2 arguments (3 given)
And don't get me started on (just to see what would happen):
>>> class b64string(str, base64): pass
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
I know how to do it manually by listing all of the attributes of base64 in a new class and calling them with the stored value as argument. But is there a neat, pythonic way to do this? Is it a bad idea to do it? The idea would be, if needed, to do it with many such modules and have "super strings" that would have as modules all the things I would need to do with them. Is that bad? Is it un-pythonic? If it is pythonic, how is it done?
I don't think creating such complex string-like classes is a good idea, but if you really want to, here's a simple snippet that runs your examples.
First, we define a class that's a generic string-wrapper. Its core is a __getattr__ function that forwards every method call to a given self.module, adding self.string as the first parameter and remembering the result on self.string.
import base64
class ModuledString(object):
def __init__(self, string):
self.string = string
def __getattr__(self, attrname):
def func(*args, **kwargs):
result = getattr(self.module, attrname)(self.string, *args, **kwargs)
self.string = result
return result
return func
def __str__(self):
return str(self.string)
Creating a string-wrapper with base64 capabilities is then easy:
class B64String(ModuledString):
module = base64
if __name__ == '__main__':
string = B64String('Foo Bar')
print string
# 'Foo Bar'
print string.b64encode()
# 'Rm9vIEJhcg=='
print string
# 'Rm9vIEJhcg=='
string.string = 'QmFyIEZvbw=='
print string
# 'QmFyIEZvbw=='
print string.b64decode()
# 'Bar Foo'
Note that the above examples work only because b64encode and b64decode take a string as the first argument and return a string as the result (there is no validation in my __getattr__ function). A random function from some random module would probably raise some kind of exception. So, after all, it would be better to restrict the usage to a predefined set of functions from a given module, but it should be easy now.
I repeat, I don't recommend using such code in any serious project, only for fun.