Develop Reference

Altering traceback of a non-callable module

I'm a minor contributor to a package where people are meant to do this (Foo.Bar.Bar is a class):
>>> from Foo.Bar import Bar
>>> s = Bar('a')
Sometimes people do this by mistake (Foo.Bar is a module):
>>> from Foo import Bar
>>> s = Bar('a')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'module' object is not callable
This might seems simple, but users still fail to debug it, I would like to make it easier. I can't change the names of Foo or Bar but I would like to add a more informative traceback like:
TypeError("'module' object is not callable, perhaps you meant to call 'Bar.Bar()'")
I read the Callable modules Q&A, and I know that I can't add a __call__ method to a module (and I don't want to wrap the whole module in a class just for this). Anyway, I don't want the module to be callable, I just want a custom traceback. Is there a clean solution for Python 3.x and 2.7+?

Add this to top of Bar.py: (Based on this question)
import sys
this_module = sys.modules[__name__]
class MyModule(sys.modules[__name__].__class__):
def __call__(self, *a, **k): # module callable
raise TypeError("'module' object is not callable, perhaps you meant to call 'Bar.Bar()'")
def __getattribute__(self, name):
return this_module.__getattribute__(name)
sys.modules[__name__] = MyModule(__name__)
# the rest of file
class Bar:
pass
Note: Tested with python3.6 & python2.7.

What you want is to change the error message when is is displayed to the user. One way to do that is to define your own excepthook.
Your own function could:
search the calling frame in the traceback object (which contains informations about the TypeError exception and the function which does that),
search the Bar object in the local variables,
alter the error message if the object is a module instead of a class or function.
In Foo.__init__.py you can install a your excepthook
import inspect
import sys
def _install_foo_excepthook():
_sys_excepthook = sys.excepthook
def _foo_excepthook(exc_type, exc_value, exc_traceback):
if exc_type is TypeError:
# -- find the last frame (source of the exception)
tb_frame = exc_traceback
while tb_frame.tb_next is not None:
tb_frame = tb_frame.tb_next
# -- search 'Bar' in the local variable
f_locals = tb_frame.tb_frame.f_locals
if 'Bar' in f_locals:
obj = f_locals['Bar']
if inspect.ismodule(obj):
# -- change the error message
exc_value.args = ("'module' object is not callable, perhaps you meant to call 'Foo.Bar.Bar()'",)
_sys_excepthook(exc_type, exc_value, exc_traceback)
sys.excepthook = _foo_excepthook
_install_foo_excepthook()
Of course, you need to enforce this algorithm…
With the following demo:
# coding: utf-8
from Foo import Bar
s = Bar('a')
You get:
Traceback (most recent call last):
File "/path/to/demo_bad.py", line 5, in <module>
s = Bar('a')
TypeError: 'module' object is not callable, perhaps you meant to call 'Foo.Bar.Bar()'

There are a lot of ways you could get a different error message, but they all have weird caveats and side effects.
Replacing the module's __class__ with a types.ModuleType subclass is probably the cleanest option, but it only works on Python 3.5+.
Besides the 3.5+ limitation, the primary weird side effects I've thought of for this option are that the module will be reported callable by the callable function, and that reloading the module will replace its class again unless you're careful to avoid such double-replacement.
Replacing the module object with a different object works on pre-3.5 Python versions, but it's very tricky to get completely right.
Submodules, reloading, global variables, any module functionality besides the custom error message... all of those are likely to break if you miss some subtle aspect of the implementation. Also, the module will be reported callable by callable, just like with the __class__ replacement.
Trying to modify the exception message after the exception is raised, for example in sys.excepthook, is possible, but there isn't a good way to tell that any particular TypeError came from trying to call your module as a function.
Probably the best you could do would be to check for a TypeError with a 'module' object is not callable message in a namespace where it looks plausible that your module would have been called - for example, if the Bar name is bound to the Foo.Bar module in either the frame's locals or globals - but that's still going to have plenty of false negatives and false positives. Also, sys.excepthook replacement isn't compatible with IPython, and whatever mechanism you use would probably conflict with something.
Right now, the problems you have are easy to understand and easy to explain. The problems you would have with any attempt to change the error message are likely to be much harder to understand and harder to explain. It's probably not a worthwhile tradeoff.

Add __getitem__ to module [duplicate]

This question already has an answer here:
Python: subscript a module
(1 answer)
Closed 7 years ago.
So it's quite a simple question. how do I add __getitem__ to a Python module. I mostly just want it as an ease of use, however it's confusing why it wouldn't let me 'just set it'. Below is a simple example of __getitem__ semi-working, however I wish for the other['test'] to work.
Here's the full output:
hello
hello
Traceback (most recent call last):
File "main.py", line 4, in <module>
print other['test']
TypeError: 'module' object has no attribute '__getitem__'
main.py
import other
print other.get('test')
print other.__getitem__('test')
print other['test']
other.py
test = 'hello'
def __getitem__(name):
return globals()[name]
get = __getitem__
I've tried to set __getitem__ using globals() aswell, globals()['__getitem__'] = __getitem__. It didn't work. And I tried to set it in main.py. So I'm confused as to why it's so adamant in not allowing me to use other['test'].
If it's impossible, then a short reason would be good.

Special methods are looked up on the type, not on an instance. Python looks for type(other).__getitem__() and that isn't available. You'd have to add the __getitem__ method to the module type; you can't in Python.
You'd have to replace the whole module instance in sys.modules with an instance of your own class to achieve what you want:
class MyModule(object):
def __init__(self, namespace):
self.__dict__.update(namespace)
def __getitem__(name):
return self.__dict__[name]
import other
import sys
sys.modules[other.__name__] = MyModule(other.__dict__)

This limitation doesn't just apply for modules, it applies for anything such that the type is not object or some subclass of object, or something with a metaclass that never bottoms out with object in the mro.
For example, you can also see this happening with type type:
In [32]: class Foo(type):
....: pass
....:
In [33]: type(Foo)
Out[33]: type
In [34]: Foo.__getitem__ = lambda x, y: x.__dict__.get(y)
In [35]: Foo.foo = "hello"
In [36]: Foo['foo']
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-38-e354ca231ddc> in <module>()
----> 1 Foo['foo']
TypeError: 'type' object has no attribute '__getitem__'
In [37]: Foo.__dict__.get('foo')
Out[37]: 'hello'
The reason is that at the C-API level, both module and type are particular instances of PyTypeObject which don't implement the required protocol for inducing the same search mechanism that the PyTypeObject implementation of object and friends does implement.
To change this aspect of the language itself, rather than hacking a replacement of sys.modules, you would need to change the C source definitions for PyModule_Type and PyType_Type such that there were C functions created for __getitem__ and added to the appropriate location in the C-API big PyTypeObject struct-o-magic-functions (a lot of which is expanded by the macro PyObject_HEAD) instead of 0 (which is the sentinel for does not exist), and recompile Python itself with these modified implementations of module and type.

Python's threading.local() isn't working for me

This is pretty simple:
import threading as t
t.local().x = 1
print t.local().x
When I run it, I get this:
Traceback (most recent call last):
File "C:\Documents and Settings\e272nk\Desktop\tst.py", line 3, in <module>
print t.local().x
AttributeError: 'thread._local' object has no attribute 'x'
The attribute assignment seems to be working ok, why can't I retrieve the value?
This is Python v2.7.5 on Windows XP.

You're creating an instance of the class local, setting an attribute on it, then discarding it. Then in the next line, you are creating another instance of local and trying to print a certain attribute from it. Since you didn't set the attribute on that instance, it doesn't have it, and you get an error.
As shown in the documentation, you should be creating an instance of local and keeping it around:
import threading as t
locs = t.local()
locs.x = 1
print locs.x

How to know if object gets deleted in Python

I have an object in the heap and a reference to it. There are certain circumstances in which the object gets deleted but the reference that points to its location doesn't know that. How can I check if there is real data in the heap?
For example:
from PySide import *
a = QProgressBar()
b = QProgressBar()
self.setIndexWidget(index,a)
self.setIndexWidget(index,b)
Then the a object gets deleted but print(a) returns a valid address. However if you try a.value() - runtime error occurs (C++ object already deleted).
a is None returns False.

For the PySide objects you'll need the shiboken module to perform object queries. For Pyside2, you'll need shiboken2.
import shiboken # shiboken2
print shiboken.isValid(a)

use sip module, read more about sip here
import sip
a = QProgressBar()
sip.isdeleted(a)
False
sip.delete(a)
a
<PyQt4.QtCore.QObject object at 0x017CCA98>
sip.isdeleted(a)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: underlying C/C++ object has been deleted

It is explicitly mentioned in the documentation when an object takes the responsibility for the deletion of another object. In your example, you can see this in the Qt doc :
If index widget A is replaced with index widget B, index widget A will be deleted.

How to generate a module object from a code object in Python

Given that I have the code object for a module, how do I get the corresponding module object?
It looks like moduleNames = {}; exec code in moduleNames does something very close to what I want. It returns the globals declared in the module into a dictionary. But if I want the actual module object, how do I get it?
EDIT:
It looks like you can roll your own module object. The module type isn't conveniently documented, but you can do something like this:
import sys
module = sys.__class__
del sys
foo = module('foo', 'Doc string')
foo.__file__ = 'foo.pyc'
exec code in foo.__dict__

As a comment already indicates, in today's Python the preferred way to instantiate types that don't have built-in names is to call the type obtained via the types module from the standard library:
>>> import types
>>> m = types.ModuleType('m', 'The m module')
note that this does not automatically insert the new module in sys.modules:
>>> import sys
>>> sys.modules['m']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'm'
That's a task you must perform by hand:
>>> sys.modules['m'] = m
>>> sys.modules['m']
<module 'm' (built-in)>
This can be important, since a module's code object normally executes after the module's added to sys.modules -- for example, it's perfectly correct for such code to refer to sys.modules[__name__], and that would fail (KeyError) if you forgot this step. After this step, and setting m.__file__ as you already have in your edit,
>>> code = compile("a=23", "m.py", "exec")
>>> exec code in m.__dict__
>>> m.a
23
(or the Python 3 equivalent where exec is a function, if Python 3 is what you're using, of course;-) is correct (of course, you'll normally have obtained the code object by subtler means than compiling a string, but that's not material to your question;-).
In older versions of Python you would have used the new module instead of the types module to make a new module object at the start, but new is deprecated since Python 2.6 and removed in Python 3.