Consider the following example that uses __subclasscheck__ for a custom exception type:
class MyMeta(type):
def __subclasscheck__(self, subclass):
print(f'__subclasscheck__({self!r}, {subclass!r})')
class MyError(Exception, metaclass=MyMeta):
pass
Now when raising an exception of this type, the __subclasscheck__ method gets invoked; i.e. raise MyError() results in:
__subclasscheck__(<class '__main__.MyError'>, <class '__main__.MyError'>)
Traceback (most recent call last):
File "test.py", line 8, in <module>
raise MyError()
__main__.MyError
Here the first line of the output shows that __subclasscheck__ got invoked to check whether MyError is a subclass of itself, i.e. issubclass(MyError, MyError). I'd like to understand why that's necessary and how it's useful in general.
I'm using CPython 3.8.1 to reproduce this behavior. I also tried PyPy3 (3.6.9) and here __subclasscheck__ is not invoked.
I guess this is a CPython implementation detail. As stated in documentation to PyErr_NormalizeException:
Under certain circumstances, the values returned by PyErr_Fetch()
below can be “unnormalized”, meaning that *exc is a class object but
*val is not an instance of the same class.
So sometime during the processing of the raised error, CPython will normalize the exception, because otherwise it cannot assume that the value of the error is of the right type.
In your case it happens as follows:
Eventually while processing the exception, PyErr_Print is called, where it calls _PyErr_NormalizeException.
_PyErr_NormaliizeException calls PyObject_IsSubclass.
PyObject_IsSubclass uses __subclasscheck__ if it is provided.
I cannot say what those "certain circumstances" for "*exc is a class object but *val is not an instance of the same class" are (maybe needed for backward compatibility - I don't know).
My first assumption was, that it happens, when CPython ensures (i.e. here), that the exception is derived from BaseException.
The following code
class OldStyle():
pass
raise OldStyle
would raise OldStyle for Python2, but TypeError: exceptions must be old-style classes or derived from BaseException, not type for
class NewStyle(object):
pass
raise NewStyle
or TypeError: exceptions must derive from BaseException in Python3 because in Python3 all classes are "new style".
However, for this check not PyObject_IsSubclass but PyType_FastSubclass is used:
#define PyExceptionClass_Check(x) \
(PyType_Check((x)) && \
PyType_FastSubclass((PyTypeObject*)(x), Py_TPFLAGS_BASE_EXC_SUBCLASS))
i.e. only the tpflags are looked at.
Related
When messing with Python dataclasses, I ran into this odd error that's pretty easy to reproduce.
from __future__ import annotations
import dataclasses as dc
import typing
#dc.dataclass
class Test:
foo: dc.InitVar[int]
print(typing.get_type_hints(Test))
Running this gets you the following:
Traceback (most recent call last):
File "test.py", line 11, in <module>
print(typing.get_type_hints(Test))
File "C:\Program Files\Python310\lib\typing.py", line 1804, in get_type_hints
value = _eval_type(value, base_globals, base_locals)
File "C:\Program Files\Python310\lib\typing.py", line 324, in _eval_type
return t._evaluate(globalns, localns, recursive_guard)
File "C:\Program Files\Python310\lib\typing.py", line 687, in _evaluate
type_ =_type_check(
File "C:\Program Files\Python310\lib\typing.py", line 173, in _type_check
raise TypeError(f"{msg} Got {arg!r:.100}.")
TypeError: Forward references must evaluate to types. Got dataclasses.InitVar[int].
Without from __future__ import annotations, it seems to work fine; but in the actual code I'm making use of that import in a couple different type hints. Is there no way to make it so that the annotations import doesn't break this?
So I was actually able to replicate this exact same behavior in my Python 3.10 environment, and frankly was sort of surprised that I was able to do so. The issue, at least from the surface, seems to be with InitVar and with how typing.get_type_hints resolves such non-generic types.
Anyways, before we get too deep into the weeds, it's worth clarifying a bit about how the from __future__ import annotations works. You can read more about it in the PEP that introduces it into the wild, but essentially the story "in a nutshell" is that the __future__ import converts all annotations in the module where it is used into forward-declared annotations, i.e. ones that are wrapped in a single quotes ' to render all type annotations as string values.
So then with all type annotations converted to strings, what typing.get_type_hints actually does is to resolve those ForwardRef types -- which is essentially the typing library's way of identifying annotations that are wrapped in strings -- using a class or module's globals namespace, along with an optional locals namespace if provided.
Here's a simple example to basically bring home all that was discussed above. All I'm doing here, is instead of using from __future__ import annotations at the top of the module, I'm manually going in and forward declaring all annotations by wrapping them in strings. It's worth noting that this is essentially the same as how it appears in the question above.
import typing
from dataclasses import dataclass, InitVar
#dataclass
class Test:
foo: 'InitVar[int]'
print(typing.get_type_hints(Test))
If curious, you can also try with a __future__ import and without forward declaring the annotations manually, and then inspect the Test.__annotations__ object to confirm that the end result is the same as how I've defined it above.
In either case, we run into the same error below, also as noted in the OP above:
Traceback (most recent call last):
print(typing.get_type_hints(Test))
File "C:\Users\USER\.pyenv\pyenv-win\versions\3.10.0\lib\typing.py", line 1804, in get_type_hints
value = _eval_type(value, base_globals, base_locals)
File "C:\Users\USER\.pyenv\pyenv-win\versions\3.10.0\lib\typing.py", line 324, in _eval_type
return t._evaluate(globalns, localns, recursive_guard)
File "C:\Users\USER\.pyenv\pyenv-win\versions\3.10.0\lib\typing.py", line 687, in _evaluate
type_ =_type_check(
File "C:\Users\USER\.pyenv\pyenv-win\versions\3.10.0\lib\typing.py", line 173, in _type_check
raise TypeError(f"{msg} Got {arg!r:.100}.")
TypeError: Forward references must evaluate to types. Got dataclasses.InitVar[int].
Let's note the stack trace as it's certainly to useful to know where things went wrong. However, we'll likely want to explore exactly why the dataclasses.InitVar usage resulted in this strange and unusual error in the first place, which is actually what we'll look at to start with.
So what's up with dataclasses.InitVar?
The TL;DR here is there's a problem with subscripted dataclasses.InitVar usage specifically. Anyway, let's look at only the relevant parts of how InitVar is defined in Python 3.10:
class InitVar:
def __init__(self, type):
self.type = type
def __class_getitem__(cls, type):
return InitVar(type)
Note that the __class_getitem__ is the method that is called when we subscript the class in an annotation, for example like InitVar[str]. This calls InitVar.__class_getitem__(str) which returns InitVar(str).
So the actual problem here is, the subscripted InitVar[int] usage returns an InitVar object, rather than the underlying type, which is the InitVar class itself.
So typing.get_type_hints is causing an error here because it sees an InitVar instance in the resolved type annotation, rather than the InitVar class itself, which is a valid type as it's a Python class essentially.
Hmm... but what seems to be the most straightforward way to resolve this?
The (Patchwork) Road to a Solution
If you check out the source code of typing.get_type_hints at least in Python 3.10, you'll notice that it's converting all string annotations to ForwardRef objects explictly, and then calling ForwardRef._evaluate on each one:
for name, value in ann.items():
...
if isinstance(value, str):
value = ForwardRef(value, is_argument=False)
>> value = _eval_type(value, base_globals, base_locals)
What the ForwardRef._evaluate method does is eval the contained reference using the class or module globals, and then internally call typing._type_check to check the reference contained in the ForwardRef object. This does a couple things like validating that the reference is of a Generic type from the typing module, which definitely aren't of interest here, since InitVar is explicitly defined is a non-generic type, at least in 3.10.
The relevant bits of typing._type_check are shown below:
if isinstance(arg, _SpecialForm) or arg in (Generic, Protocol):
raise TypeError(f"Plain {arg} is not valid as type argument")
if isinstance(arg, (type, TypeVar, ForwardRef, types.UnionType, ParamSpec)):
return arg
if not callable(arg):
>> raise TypeError(f"{msg} Got {arg!r:.100}.")
It's the last line shown above, raise TypeError(...) which seems to return the error message that we're running into. If you check the last condition that the _type_check function checks, you can kind of guess how we can implement the simplest possible workaround in our case:
if not callable(arg):
If we glance a little briefly into the documentation for the callable builtin, we get our first concrete hint of a possible solution we can use:
def callable(i_e_, some_kind_of_function): # real signature unknown; restored from __doc__
"""
Return whether the object is callable (i.e., some kind of function).
Note that classes are callable, as are instances of classes with a
__call__() method.
"""
So, simply put, all we need to do is define a __call__ method under the dataclasses.InitVar class. This can be a stub method, essentially a no-op, but at a minimum the class must define this method so that it can be considered a callable, and thus the typing module can accept it as a valid reference type in a ForwardRef object.
Finally, here's the same example as in the OP, but slightly modified to add a new line which patches dataclasses.InitVar to add the necessary method, as a stub:
from __future__ import annotations
import typing
from dataclasses import dataclass, InitVar
#dataclass
class Test:
foo: InitVar[int]
# can also be defined as:
# setattr(InitVar, '__call__', lambda *args: None)
InitVar.__call__ = lambda *args: None
print(typing.get_type_hints(Test))
The example now seems to work as expected, without any errors raised by the typing.get_type_hints method, when forward declaring any subscripted InitVar annotations.
I'm a minor contributor to a package where people are meant to do this (Foo.Bar.Bar is a class):
>>> from Foo.Bar import Bar
>>> s = Bar('a')
Sometimes people do this by mistake (Foo.Bar is a module):
>>> from Foo import Bar
>>> s = Bar('a')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'module' object is not callable
This might seems simple, but users still fail to debug it, I would like to make it easier. I can't change the names of Foo or Bar but I would like to add a more informative traceback like:
TypeError("'module' object is not callable, perhaps you meant to call 'Bar.Bar()'")
I read the Callable modules Q&A, and I know that I can't add a __call__ method to a module (and I don't want to wrap the whole module in a class just for this). Anyway, I don't want the module to be callable, I just want a custom traceback. Is there a clean solution for Python 3.x and 2.7+?
Add this to top of Bar.py: (Based on this question)
import sys
this_module = sys.modules[__name__]
class MyModule(sys.modules[__name__].__class__):
def __call__(self, *a, **k): # module callable
raise TypeError("'module' object is not callable, perhaps you meant to call 'Bar.Bar()'")
def __getattribute__(self, name):
return this_module.__getattribute__(name)
sys.modules[__name__] = MyModule(__name__)
# the rest of file
class Bar:
pass
Note: Tested with python3.6 & python2.7.
What you want is to change the error message when is is displayed to the user. One way to do that is to define your own excepthook.
Your own function could:
search the calling frame in the traceback object (which contains informations about the TypeError exception and the function which does that),
search the Bar object in the local variables,
alter the error message if the object is a module instead of a class or function.
In Foo.__init__.py you can install a your excepthook
import inspect
import sys
def _install_foo_excepthook():
_sys_excepthook = sys.excepthook
def _foo_excepthook(exc_type, exc_value, exc_traceback):
if exc_type is TypeError:
# -- find the last frame (source of the exception)
tb_frame = exc_traceback
while tb_frame.tb_next is not None:
tb_frame = tb_frame.tb_next
# -- search 'Bar' in the local variable
f_locals = tb_frame.tb_frame.f_locals
if 'Bar' in f_locals:
obj = f_locals['Bar']
if inspect.ismodule(obj):
# -- change the error message
exc_value.args = ("'module' object is not callable, perhaps you meant to call 'Foo.Bar.Bar()'",)
_sys_excepthook(exc_type, exc_value, exc_traceback)
sys.excepthook = _foo_excepthook
_install_foo_excepthook()
Of course, you need to enforce this algorithm…
With the following demo:
# coding: utf-8
from Foo import Bar
s = Bar('a')
You get:
Traceback (most recent call last):
File "/path/to/demo_bad.py", line 5, in <module>
s = Bar('a')
TypeError: 'module' object is not callable, perhaps you meant to call 'Foo.Bar.Bar()'
There are a lot of ways you could get a different error message, but they all have weird caveats and side effects.
Replacing the module's __class__ with a types.ModuleType subclass is probably the cleanest option, but it only works on Python 3.5+.
Besides the 3.5+ limitation, the primary weird side effects I've thought of for this option are that the module will be reported callable by the callable function, and that reloading the module will replace its class again unless you're careful to avoid such double-replacement.
Replacing the module object with a different object works on pre-3.5 Python versions, but it's very tricky to get completely right.
Submodules, reloading, global variables, any module functionality besides the custom error message... all of those are likely to break if you miss some subtle aspect of the implementation. Also, the module will be reported callable by callable, just like with the __class__ replacement.
Trying to modify the exception message after the exception is raised, for example in sys.excepthook, is possible, but there isn't a good way to tell that any particular TypeError came from trying to call your module as a function.
Probably the best you could do would be to check for a TypeError with a 'module' object is not callable message in a namespace where it looks plausible that your module would have been called - for example, if the Bar name is bound to the Foo.Bar module in either the frame's locals or globals - but that's still going to have plenty of false negatives and false positives. Also, sys.excepthook replacement isn't compatible with IPython, and whatever mechanism you use would probably conflict with something.
Right now, the problems you have are easy to understand and easy to explain. The problems you would have with any attempt to change the error message are likely to be much harder to understand and harder to explain. It's probably not a worthwhile tradeoff.
I'm looking at this minimal valid(?) program:
import sys
def f():
try:
raise Exception()
except Exception:
raise Exception(), None, sys.exc_info()[2]
f()
This program executes and behaves as expected , preserving the stack trace of the inner exception, as documented by help("raise"). However, when I run pylint on it, this is what I get:
$ pylint program.py
************* Module tmp
E: 7, 0: invalid syntax (<string>, line 7) (syntax-error)
The syntax-error disappears when I remove the second and third expressions to raise.
Is this a bug in pylint, or am I overlooking something?
Your pylint binary testing for Python 3 syntax, your code is valid for Python 2 only. Pylint tests code following the syntax of the Python binary you installed it with (it uses Python's own parser).
In Python 3, you'd use:
raise Exception().with_traceback(sys.exc_info()[2])
See the raise statement documentation for Python 3.
While your syntax may be correct for Python 2, you are technically using raise wrong. When passing in 3 elements, the first must be a class, not an instance. The second is an instance of that class, the third the traceback:
raise Exception, Exception(), sys.exc_info()[2]
or you can pass in None for an empty argument list passed to the first (the class) to create an instance:
raise Exception, None, sys.exc_info()[2]
Your code still happens to work, but only because Python isn't being too strict and takes that first argument as the instance when it is not a class.
If you want to test Python 2 code with pylint, install a copy into your Python 2 binary, and run that version. See Specify which python version pylint should evaluate for
I'm using a Class provided by a client (I have no access to the object code), and I'm trying to check if a object has a attribute. The attribute itself is write only, so the hasattr fails:
>>> driver.console.con.input = 'm'
>>> hasattr(driver.console.con, 'input')
False
>>> simics> #driver.console.con.input
Traceback (most recent call last):
File "<string>", line 1, in <module>
Attribute: Failed converting 'input' attribute in object
'driver.console.con' to Python: input attribute in driver.console.con
object: not readable.
Is there a different way to check if an attribute exists?
You appear to have some kind of native code proxy that bridges Python to an extension, and it is rather breaking normal Python conventions
There are two possibilities:
The driver.console.con object has a namespace that implements attributes as descriptors, and the input descriptor only has a __set__ method (and possibly a __delete__ method). In that case, look for the descriptor:
if 'input' in vars(type(driver.console.con)):
# there is an `input` name in the namespace
attr = vars(type(driver.console.con))['input']
if hasattr(attr, '__set__'):
# can be set
...
Here the vars() function retrieves the namespace for the class used for driver.console.con.
The proxy uses __getattr__ (or even __getattribute__) and __setattr__ hooks to handle arbitrary attributes. You are out of luck here, you can't detect what attributes either method will support outside of hasattr() and trying to set the attribute directly. Use try...except guarding:
try:
driver.console.con.input = 'something'
except Attribute: # exactly what exception object does this throw?
# can't be set, not a writable attribute
pass
You may have to use a debugger or print() statement to figure out exactly what exception is being thrown (use a try...except Exception as ex: block to capture all exceptions then inspect ex); in the traceback in your question the exception message at the end looks decidedly non-standard. That project really should raise an AttributeError at that point.
Given the rather custom exception being thrown, my money is on option 2 (but option 1 is still a possibility if the __get__ method on the descriptor throws the exception).
I'm trying to run this test: self.assertRaises(AttributeError, branch[0].childrennodes), and branch[0] does not have an attribute childrennodes, so it should be throwing an AttributeError, which the assertRaises should catch, but when I run the test, the test fails because it is throwing an AttributeError.
Traceback (most recent call last):
File "/home/tttt/../tttt/tests.py", line 504, in test_get_categories_branch
self.assertRaises(AttributeError, branch[0].children_nodes)
AttributeError: 'Category' object has no attribute 'children_nodes'
Any ideas?
When the test is running, before calling self.assertRaises, Python needs to find the value of all the method's arguments. In doing so, it evaluates branch[0].children_nodes, which raises an AttributeError. Since we haven't invoked assertRaises yet, this exception is not caught, causing the test to fail.
The solution is to wrap branch[0].children_nodes in a function or a lambda:
self.assertRaises(AttributeError, lambda: branch[0].children_nodes)
assertRaises can also be used as a context manager (Since Python 2.7, or in PyPI package 'unittest2'):
with self.assertRaises(AttributeError):
branch[0].children_nodes
# etc
This is nice because it can be used on arbitrary blocks of code in the middle of a test, rather than having to create a new function just to define the block of code to which it applies.
It can give you access to the raised exception for further processing, if needed:
with self.assertRaises(AttributeError) as cm:
branch[0].children_nodes
self.assertEquals(cm.exception.special_attribute, 123)
I think its because assert raises only accepts a callable. It evalutes to see if the callable raises an exception, not if the statement itself does.
self.assertRaises(AttributeError, getattr, branch[0], "childrennodes")
should work.
EDIT:
As THC4k correctly says it gathers the statements at collection time and will error then, not at testing time.
Also this is a reason why I like nose, it has a decorator (raises) that is useful and clearer for these kind of tests.
#raises(AttributeError)
def test_1(self)
branch[0].childrennodes
pytest also has a similar context manager:
from pytest import raises
def test_raising():
with raises(AttributeError):
branch[0].childrennodes