The docs don't seem to be very clear on how to address the following ...
def test():
"""
>>> import doctest
>>> doctest.ELLIPSIS_MARKER = '<ignore>'
>>> import pandas as pd
>>> raise pd.errors.InvalidIndexError # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
Traceback (most recent call last):
<ignore>
<ignore>InvalidIndexError
"""
import doctest
doctest.run_docstring_examples(test, globals())
This will work fine but is not addressing the wildcard at the front of <ignore>InvalidIndexError
def test():
"""
>>> import doctest
>>> doctest.ELLIPSIS_MARKER = '<ignore>'
>>> import pandas as pd
>>> raise pd.errors.InvalidIndexError # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
Traceback (most recent call last):
<ignore>
pandas.errors.InvalidIndexError
"""
import doctest
doctest.run_docstring_examples(test, globals())
Note pandas version is 1.1.3
References
https://docs.python.org/3/library/doctest.html#option-flags
How enable ellipsis when calling Python doctest?
Can I have an ellipsis at the beginning of the line in a Python doctest?
doctest requires exceptions to look a certain way. From the docs:
Each line of the traceback stack (if present) must be indented further than the first line of the example, or start with a non-alphanumeric character. The first line following the traceback header indented the same and starting with an alphanumeric is taken to be the start of the exception detail.
(added bold)
This means if you make the ELLIPSIS_MARKER start with an alphanumeric, it'll work properly. Here's an example using re.error:
def test():
"""
>>> import doctest
>>> doctest.ELLIPSIS_MARKER = 'MODULE.'
>>> import re
>>> raise re.error(None) # doctest: +ELLIPSIS
Traceback (most recent call last):
...
MODULE.error: None
"""
By the way:
Note that tracebacks are treated very specially. In particular, in the rewritten example, the use of ... is independent of doctest's ELLIPSIS option. The ellipsis in that example could be left out, or could just as well be three (or three hundred) commas or digits, or an indented transcript of a Monty Python skit.
For context, here's an example with no exception that uses two ellipses:
def test():
r"""
>>> print('foo\nbar\nbaz') # doctest: +ELLIPSIS
foo
...
...
"""
That said, IGNORE_EXCEPTION_DETAIL may be a better solution. (I just learned about it myself.)
When specified, an example that expects an exception passes if an exception of the expected type is raised, even if the exception detail does not match. For example, an example expecting ValueError: 42 will pass if the actual exception raised is ValueError: 3*14, but will fail, e.g., if TypeError is raised.
It will also ignore the module name used in Python 3 doctest reports.
(added bold)
For example:
def test():
"""
>>> import re
>>> raise re.error(None) # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
error: foobar
"""
Note that both the exception module and exception details are ignored in this example. That's on purpose, to show a side-effect of this solution.
Related
Is is possible to have optional keys in string formats using '%' operator?
I’m using the logging API with Python 2.7, so I can't use Advanced String Formatting.
My problem is as follow:
>>> import logging
>>> FORMAT = '%(asctime)-15s %(message)s %(user)s'
>>> logging.basicConfig(format=FORMAT)
>>> logging.warning("It works for:", extra={'user': 'me'})
2016-08-29 11:24:31,262 It works for: me
>>> logging.warning("It does't work!")
Traceback (most recent call last):
...
KeyError: 'user'
Logged from file <input>, line 1
I want to have an empty string for user if missing. How can I do that?
I tried with a defaultdict, but it fails:
>>> import collections
>>> extra = collections.defaultdict(unicode)
>>> logging.warning("It does't work!", extra=extra)
Traceback (most recent call last):
...
KeyError: 'user'
Logged from file <input>, line 1
By contrast, with Jinja2, we can do:
>>> import jinja2
>>> jinja2.Template('name: {{ name }}, email: {{ email }}').render(name="me")
u'name: me, email: '
=> no exception here, just an empty string (for "email").
A) The defaultdict approach works fine, but only if used directly.
>>> import collections
>>> dd=collections.defaultdict(str)
>>> dd['k'] = 22
>>> '%(k)s %(nn)s' % dd
'22 '
B) The extra argument to a log function is used as described in the docs, i.e. not directly as shown above. That's why using a defaultdict instead of a regular dict does not make a difference.
The third keyword argument is extra which can be used to pass a
dictionary which is used to populate the dict of the LogRecord
created for the logging event with user-defined attributes.
C) You can use a logging filter to take care of the missing extra data:
import logging
class UserFilter:
def filter(self, record):
try:
record.user
except AttributeError:
record.user = '<N/A>'
return True
FORMAT = '%(asctime)-15s %(message)s %(user)s'
logging.basicConfig(format=FORMAT)
logging.getLogger().addFilter(UserFilter())
logging.warning("It works for:", extra={'user': 'me'})
logging.warning("It doesn't work!")
# DATE TIME It doesn't work! <N/A>
Any class with a filter method is fine. It can modify the record in-place and it must return True for accepting the record or False for filtering it out.
I need to determine MIME-types from files without suffix in python3 and I thought of python-magic as a fitting solution therefor.
Unfortunately it does not work as described here:
https://github.com/ahupp/python-magic/blob/master/README.md
What happens is this:
>>> import magic
>>> magic.from_file("testdata/test.pdf")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'from_file'
So I had a look at the object, which provides me with the class Magic for which I found documentation here:
http://filemagic.readthedocs.org/en/latest/guide.html
I was surprised, that this did not work either:
>>> with magic.Magic() as m:
... pass
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __init__() missing 1 required positional argument: 'ms'
>>> m = magic.Magic()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __init__() missing 1 required positional argument: 'ms'
>>>
I could not find any information about how to use the class Magic anywhere, so I went on doing trial and error, until I figured out, that it accepts instances of LP_magic_set only for ms.
Some of them are returned by the module's methods
magic.magic_set() and magic_t().
So I tried to instanciate Magic with either of them.
When I then call the file() method from the instance, it will always return an empty result and the errlvl() method tells me error no. 22.
So how do I use magic anyway?
I think that you are confusing different implementations of "python-magic"
You appear to have installed python-magic-5.19.1, however, you reference firstly the documentation for python-magic-0.4.6, and secondly filemagic-1.6. I think that you are better off using python-magic-0.4.6 as it is readily available at PYPI and easily installed via pip into virtualenv environments.
Documentation for python-magic-5.19.1 is hard to come by, but I managed to get it to work like this:
>>> import magic
>>> m=magic.open(magic.MAGIC_NONE)
>>> m.load()
0
>>> m.file('/etc/passwd')
'ASCII text'
>>> m.file('/usr/share/cups/data/default.pdf')
'PDF document, version 1.5'
You can also get different magic descriptions, e.g. MIME type:
>>> m=magic.open(magic.MAGIC_MIME)
>>> m.load()
0
>>> m.file('/etc/passwd')
'text/plain; charset=us-ascii'
>>> m.file('/usr/share/cups/data/default.pdf')
'application/pdf; charset=binary'
or for more recent versions of python-magic-5.30
>>> import magic
>>> magic.detect_from_filename('/etc/passwd')
FileMagic(mime_type='text/plain', encoding='us-ascii', name='ASCII text')
>>> magic.detect_from_filename('/etc/passwd').mime_type
'text/plain'
I've seen similar questions to this one but none of them really address the trackback.
If I have a class like so
class Stop_if_no_then():
def __init__(self, value one, operator, value_two, then, line_or_label, line_number):
self._firstvalue = value_one
self._secondvalue = value_two
self._operator = operator
self._gohere = line_or_label
self._then = then
self._line_number = line_number
def execute(self, OtherClass):
"code comparing the first two values and making changes etc"
What I want my execute method to be able to do is if self._then is not equal to the string "THEN" (in allcaps) then I want it to raise a custom error message and terminate the whole program while also not showing a traceback.
If the error is encountered the only thing that should print out would look something like (I'm using 3 as an example, formatting is not a problem) this.
`Syntax Error (Line 3): No -THEN- present in the statement.`
I'm not very picky about it actually being an exception class object, so there's no issue in that aspect. Since I will be using this in a while loop, simple if, elif just repeats the message over and over (because obviously I am not closing the loop). I have seen sys.exit() but that also prints out a giant block of red text, unless I am not using it correctly. I don't want to catch the exception in my loop because there are other classes in the same module in which I need to implement something like this.
You can turn off the traceback by limiting its depth.
Python 2.x
import sys
sys.tracebacklimit = 0
Python 3.x
In Python 3.5.2 and 3.6.1, setting tracebacklimit to 0 does not seem to have the intended effect. This is a known bug. Note that -1 doesn't work either. Setting it to None does however seem to work, at least for now.
In Python 3.6.2 and above you should set tracebacklimit to 0 or -1, as setting it to None does not disable the traceback output.
Python 3.6.1 and bellow results:
>>> import sys
>>> sys.tracebacklimit = 0
>>> raise Exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception
>>> sys.tracebacklimit = -1
>>> raise Exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception
>>> sys.tracebacklimit = None
>>> raise Exception
Exception
Python 3.6.2 and above results:
>>> import sys
>>> sys.tracebacklimit = 0
>>> raise Exception
Exception
>>> sys.tracebacklimit = -1
>>> raise Exception
Exception
>>> sys.tracebacklimit = None
>>> raise Exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception
Nevertheless, for better or worse, if multiple exceptions are raised, they can all still be printed. For example:
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
You can use SystemExit exception:
except Exception as err:
raise SystemExit(err)
https://docs.python.org/3/library/exceptions.html
You can use a try: and then except Exception as inst:
What that will do is give you your error message in a variable named inst and you can print out the arguments on the error with inst.args. Try printing it out and seeing what happens, and is any item in inst.args is the one you are looking for.
EDIT Here is an example I tried with pythons IDLE:
>>> try:
open("epik.sjj")
except Exception as inst:
d = inst
>>> d
FileNotFoundError(2, 'No such file or directory')
>>> d.args
(2, 'No such file or directory')
>>> d.args[1]
'No such file or directory'
>>>
EDIT 2: as for closing the program you can always raise and error or you can use sys.exit()
The cleanest way that I know is to use sys.excepthook.
You implement a three argument function that accepts type, value, and traceback and does whatever you like (say, only prints the value) and assign that function to sys.excepthook.
Here is an example:
import sys
def excepthook(type, value, traceback):
print(value)
sys.excepthook = excepthook
raise ValueError('hello')
This is available in both python 2 and python 3.
If you want to get rid of any traceback for customs exceptions and have line number,
you can do this trick
Python 3
import sys
import inspect
class NoTraceBackWithLineNumber(Exception):
def __init__(self, msg):
try:
ln = sys.exc_info()[-1].tb_lineno
except AttributeError:
ln = inspect.currentframe().f_back.f_lineno
self.args = "{0.__name__} (line {1}): {2}".format(type(self), ln, msg),
sys.exit(self)
class MyNewError(NoTraceBackWithLineNumber):
pass
raise MyNewError("Now TraceBack Is Gone")
Will give this output, and make the raise keyword useless
MyNewError (line 16): Now TraceBack Is Gone
"Exception chaining can be disabled by using from None " - Python docs
>>> try:
... open('database.sqlite')
... except IOError:
... raise RuntimeError from None
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
In general, if you want to catch any exception except SystemExit, and exit with the exception's message without the traceback, define your main function as below:
>>> import sys
>>> def main():
... try:
... # Run your program from here.
... raise RandomException # For testing
... except (Exception, KeyboardInterrupt) as exc:
... sys.exit(exc)
...
>>> main()
name 'RandomException' is not defined
$ echo $?
1
Note that in the case of multiple exceptions being raised, only one message is printed.
This answer is meant to improve upon the one by The-IT.
Is there an easy way to get the message of the exception to be colored on the command line? For example
def g(): f()
def f(): 1/0
g()
Gives the error
Traceback (most recent call last):
File "test.py", line 3, in <module>
g()
File "test.py", line 1, in g
def g(): f()
File "test.py", line 2, in f
def f(): 1/0
ZeroDivisionError: integer division or modulo by zero
I would like "integer division or modulo by zero" to be colored or highlighted on the terminal so that I can quickly pick it out of a long traceback (Linux only). Ideally, I wouldn't want to write a custom class for each Exception, but somehow catch and format all kinds.
EDIT: The question linked in the comments gives examples on how to solve the problem with external software, but I'm interested in an internal Python solution.
You can assign a custom function to the sys.excepthook handler. The function is called whenever there is a unhandled exception (so one that exits the interpreter).
def set_highlighted_excepthook():
import sys, traceback
from pygments import highlight
from pygments.lexers import get_lexer_by_name
from pygments.formatters import TerminalFormatter
lexer = get_lexer_by_name("pytb" if sys.version_info.major < 3 else "py3tb")
formatter = TerminalFormatter()
def myexcepthook(type, value, tb):
tbtext = ''.join(traceback.format_exception(type, value, tb))
sys.stderr.write(highlight(tbtext, lexer, formatter))
sys.excepthook = myexcepthook
set_highlighted_excepthook()
This version uses the pygments library to convert the traceback text into one formatted with ANSI coloring, before writing it to stderr.
Someone turned this into a project that detects terminal support and lets you set the pygments style, see colored-traceback.py.
Found another way to do this using the IPython module which is likely a dependency that everyone already has installed:
from IPython.core.ultratb import ColorTB
c = ColorTB()
exc = sys.exc_info()
print(''.join(c.structured_traceback(*exc)))
This takes the solution #freakish shared and makes the colorization part of the exception instead of requiring the user to add color to each exception message. Obviously, it only works for custom exceptions, so it may not be exactly what OP was looking for.
from colorama import Fore, init
init()
class Error (Exception):
def __init__ (self, message):
super().__init__(Fore.RED + message)
class BadConfigFile (Error):
pass
raise BadConfigFile("some error message")
This will print the traceback with "some error message" in red. Having 'Error' as a base class means you can create other exceptions that will all inherit the colorization of the message.
Have a look at colorama ( or any other coloring ) module. Then you can wrap you're entire app with:
import traceback
from colorama import Fore, init
init( )
try:
// your app
except Exception:
print Fore.RED + traceback.format_exc( ) + Fore.RESET
// possibly raise again or log to db
I'm writing some unit tests for a Python library and would like certain warnings to be raised as exceptions, which I can easily do with the simplefilter function. However, for one test I'd like to disable the warning, run the test, then re-enable the warning.
I'm using Python 2.6, so I'm supposed to be able to do that with the catch_warnings context manager, but it doesn't seem to work for me. Even failing that, I should also be able to call resetwarnings and then re-set my filter.
Here's a simple example which illustrates the problem:
>>> import warnings
>>> warnings.simplefilter("error", UserWarning)
>>>
>>> def f():
... warnings.warn("Boo!", UserWarning)
...
>>>
>>> f() # raises UserWarning as an exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in f
UserWarning: Boo!
>>>
>>> f() # still raises the exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in f
UserWarning: Boo!
>>>
>>> with warnings.catch_warnings():
... warnings.simplefilter("ignore")
... f() # no warning is raised or printed
...
>>>
>>> f() # this should raise the warning as an exception, but doesn't
>>>
>>> warnings.resetwarnings()
>>> warnings.simplefilter("error", UserWarning)
>>>
>>> f() # even after resetting, I'm still getting nothing
>>>
Can someone explain how I can accomplish this?
EDIT: Apparently this is a known bug: http://bugs.python.org/issue4180
Reading through the docs and few times and poking around the source and shell I think I've figured it out. The docs could probably improve to make clearer what the behavior is.
The warnings module keeps a registry at __warningsregistry__ to keep track of which warnings have been shown. If a warning (message) is not listed in the registry before the 'error' filter is set, any calls to warn() will not result in the message being added to the registry. Also, the warning registry does not appear to be created until the first call to warn:
>>> import warnings
>>> __warningregistry__
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
NameError: name '__warningregistry__' is not defined
>>> warnings.simplefilter('error')
>>> __warningregistry__
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
NameError: name '__warningregistry__' is not defined
>>> warnings.warn('asdf')
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
UserWarning: asdf
>>> __warningregistry__
{}
Now if we ignore warnings, they will get added to the warnings registry:
>>> warnings.simplefilter("ignore")
>>> warnings.warn('asdf')
>>> __warningregistry__
{('asdf', <type 'exceptions.UserWarning'>, 1): True}
>>> warnings.simplefilter("error")
>>> warnings.warn('asdf')
>>> warnings.warn('qwerty')
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
UserWarning: qwerty
So the error filter will only apply to warnings that aren't already in the warnings registry. To make your code work you'll need to clear the appropriate entries out of the warnings registry when you're done with the context manager (or in general any time after you've used the ignore filter and want a prev. used message to be picked up the error filter). Seems a bit unintuitive...
Brian Luft is correct about __warningregistry__ being the cause of the problem. But I wanted to clarify one thing: the way the warnings module appears to work is that it sets module.__warningregistry__ for each module where warn() is called. Complicating things even more, the stacklevel option to warnings causes the attribute to be set for the module the warning was issued "in the name of", not necessarily the one where warn() was called... and that's dependent on the call stack at the time the warning was issued.
This means you may have a lot of different modules where the __warningregistry__ attribute is present, and depending on your application, they may all need clearing before you'll see the warnings again. I've been relying on the following snippet of code to accomplish this... it clears the warnings registry for all modules whose name matches the regexp (which defaults to everything):
def reset_warning_registry(pattern=".*"):
"clear warning registry for all match modules"
import re
import sys
key = "__warningregistry__"
for mod in sys.modules.values():
if hasattr(mod, key) and re.match(pattern, mod.__name__):
getattr(mod, key).clear()
Update: CPython issue 21724 addresses issue that resetwarnings() doesn't clear warning state. I attached an expanded "context manager" version to this issue, it can be downloaded from reset_warning_registry.py.
Brian is spot on about the __warningregistry__. So you need to extend catch_warnings to save/restore the global __warningregistry__ too
Something like this may work
class catch_warnings_plus(warnings.catch_warnings):
def __enter__(self):
super(catch_warnings_plus,self).__enter__()
self._warningregistry=dict(globals.get('__warningregistry__',{}))
def __exit__(self, *exc_info):
super(catch_warnings_plus,self).__exit__(*exc_info)
__warningregistry__.clear()
__warningregistry__.update(self._warningregistry)
Following on from Eli Collins' helpful clarification, here is a modified version of the catch_warnings context manager that clears the warnings registry in a given sequence of modules when entering the context manager, and restores the registry on exit:
from warnings import catch_warnings
class catch_warn_reset(catch_warnings):
""" Version of ``catch_warnings`` class that resets warning registry
"""
def __init__(self, *args, **kwargs):
self.modules = kwargs.pop('modules', [])
self._warnreg_copies = {}
super(catch_warn_reset, self).__init__(*args, **kwargs)
def __enter__(self):
for mod in self.modules:
if hasattr(mod, '__warningregistry__'):
mod_reg = mod.__warningregistry__
self._warnreg_copies[mod] = mod_reg.copy()
mod_reg.clear()
return super(catch_warn_reset, self).__enter__()
def __exit__(self, *exc_info):
super(catch_warn_reset, self).__exit__(*exc_info)
for mod in self.modules:
if hasattr(mod, '__warningregistry__'):
mod.__warningregistry__.clear()
if mod in self._warnreg_copies:
mod.__warningregistry__.update(self._warnreg_copies[mod])
Use with something like:
import my_module_raising_warnings
with catch_warn_reset(modules=[my_module_raising_warnings]):
# Whatever you'd normally do inside ``catch_warnings``
I've run into the same issues, and while all of the other answers are valid I choose a different route. I don't want to test the warnings module, nor know about it's inner workings. So I just mocked it instead:
import warnings
import unittest
from unittest.mock import patch
from unittest.mock import call
class WarningTest(unittest.TestCase):
#patch('warnings.warn')
def test_warnings(self, fake_warn):
warn_once()
warn_twice()
fake_warn.assert_has_calls(
[call("You've been warned."),
call("This is your second warning.")])
def warn_once():
warnings.warn("You've been warned.")
def warn_twice():
warnings.warn("This is your second warning.")
if __name__ == '__main__':
__main__=unittest.main()
This code is Python 3, for 2.6 you need the use an external mocking library as unittest.mock was only added in 2.7.