I am trying to debug a multi-threaded program that uses a third-party package.
At some point, one of the attributes of an object (that is not created directly by me) is changed and I can't figure out what changed it. I could not find anything in my code that changes it.
Since this is a third-party package, I prefer not to change its code directly, but rather patch it from the outside as necessary.
My plan was to somehow tap into or wrap the code that sets the attribute and set a breakpoint or print the stack trace from there.
I tried monkey-patching the __setattr__ method of the instance, but it was not triggered.
I also tried to patch the class itself:
def patch_class(target):
def method(self, name, value):
print(name, value)
print("called from", target)
setattr(self, name, value) # break or print trace here
target.__setattr__ = types.MethodType(method, target)
patch_class(WebSocket)
but then all of the attributes are set on the class itself, as the method is bound to it.
Wrapping the class with a proxy does not really help either, since I am not instantiating it myself, but rather get the instance at some point after its creation.
If it matters, the said class is ws4py's WebSocket that is created by another third-party package, but I consider this an exercise in general debugging techniques.
Is there a more "pythonic" way of tapping into the mutation of an existing instance? (hack-ish ways will be appreciated as well)
I ended up creating a __setattr__ for the class.
def setter_fun(self, name, value):
print('setting', name, value)
self.__dict__[name] = value
if name is 'problematic_prop' and value is 'problematicValue':
traceback.print_stack()
# and set the class setter magic method
instance.__class__.__setattr__ = setter_fun
It is also possible to use setattr instead of using the __dict__ magic property:
setattr(self, name, value)
Now, when something sets the instance's problematic_prop to problematicValue, the stack trace will be printed:
>>> class A(object):
def __init__(self):
self.foo = 1
def set_problematic(self):
self.problematic_prop = 'problematicValue'
>>> a = A()
>>> a.__class__.__setattr__ = setter_fun
>>> a.foo = 2
setting foo 2
>>> print(a.foo)
2
>>> a.set_problematic()
setting problematic_prop problematicValue
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<input>", line 6, in set_problematic
File "<input>", line 5, in setter_fun
NameError: name 'traceback' is not defined
My failed attempts included either trying to attach the __setattr__ to the instance instead of the class, or trying to attach a bound method:
class MyClass(object):
def setter_fun(self, name, value):
print('setting', name, value)
self.__dict__[name] = value
if name is 'problematic_prop' and value is 'problematicValue':
traceback.print_stack()
def set_my_function(self):
# won't work, the function is bound to the current instance (self)
some.instace.__class__.__setattr__ = self.setter_fun
Related
The below Python fails for some reason.
class NetVend:
def blankCallback(data):
pass
def sendCommand(command, callback=NetVend.blankCallback):
return NetVend.sendSignedCommand(command, NetVend.signCommand(command), callback)
def sendSignedCommand(command, signature, callback):
pass
I get the following error:
Traceback (most recent call last):
File "module.py", line 1, in <module>
class NetVend:
File "module.py", line 5, in NetVend
def sendCommand(command, callback=NetVend.blankCallback):
NameError: name 'NetVend' is not defined
You cannot refer to a class name while still defining it.
The class body is executed as a local namespace; you can refer to functions and attributes as local names instead.
Moreover, default values to function keyword parameters are bound at definition time, not when the method is called. Use None as a sentinel instead.
Instead of:
def sendCommand(command, callback=NetVend.blankCallback):
return NetVend.sendSignedCommand(command, NetVend.signCommand(command), callback)
use:
def sendCommand(command, callback=None):
if callback is None:
callback = NetVend.blankCallback
return NetVend.sendSignedCommand(command, NetVend.signCommand(command), callback)
You probably wanted to use the class as a factory for instances instead of as a namespace for what are essentially functions. Even if you only used one instance (a singleton) there are benefits in actually creating an instance first.
Well, I wouldn't say the first, but the second option is certainly true :-)
The trouble is that the default argument is evaluated at compile time, but at that point NetVend does not exist in that scope, because (obviously) the class itself has not yet been fully evaluated.
The way round it is to set the default to None, and check within the method:
def sendCommand(command, callback=None):
if callback is None:
callback=NetVend.blankCallback
I am using a 3rd party Python library (wxPython), which has a buggy class in one of its modules.
The problematic code section looks like this:
def OnText(self, event):
value = self.GetValue()
if value != self.__oldvalue:
pass # Here some more code follows ...
self.__oldvalue = value
The problem is the if statement, because at the first call to this method self.__oldvalue has not been initialized yet. So for a workaround until this bug has been fixed by the library devs I thought I could fix this with a little workaround. I simply wanted to derive a child class from that faulty class and initialize self.__oldvalue in this constructor:
class MyIntCtrl(wx.lib.intctrl.IntCtrl):
def __init__(self, *args, **kw):
self.__oldvalue = None
super().__init__(*args, **kw)
However, now when I use this new class MyIntCtrl instead of the original IntCtrl class, I do get exactly the same error as before:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/wx/lib/intctrl.py", line 509, in OnText
if value != self.__oldvalue:
AttributeError: 'MyIntCtrl' object has no attribute '_IntCtrl__oldvalue'
Now I am wondering: What am I doing wrong, how else can I fix this issue in a child class?
Any member of class which starts with __ (double underscore) is private, you can use single underscore _ or not use underscores in naming for access them in derived classes.
class Parent:
def __init__(self):
self.__private_field = "private field"
self._protected_field = "protected field"
self.public_field = "public field"
class Child(Parent):
def __init__(self):
pass
def do(self):
print(self.__private_field) # It will throw exception
print(self._protected_field) # It will not throw exception
print(self.public_field) # It will not throw exception
Or you can bypass private/protected members by calling them like:
print(_Parent__private_field)
I'm trying to use super in a subclass which is wrapped in another class using a class decorator:
def class_decorator(cls):
class WrapperClass(object):
def make_instance(self):
return cls()
return WrapperClass
class MyClass(object):
def say(self, x):
print(x)
#class_decorator
class MySubclass(MyClass):
def say(self, x):
super(MySubclass, self).say(x.upper())
However, the call to super fails:
>>> MySubclass().make_instance().say('hello')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in say
TypeError: super(type, obj): obj must be an instance or subtype of type
The problem is that, when say is called, MySubclass doesn't refer to the original class anymore, but to the return value of the decorator.
One possible solution would be to store the value of MySubclass before decorating it:
class MySubclass(MyClass):
def say(self, x):
super(_MySubclass, self).say(x.upper())
_MySubclass = MySubclass
MySubclass = class_decorator(MySubclass)
This works, but isn't intuitive and would need to be repeated for each decorated subclass. I'm looking for a way that doesn't need additional boilerplate for each decorated subclass -- adding more code in one place (say, the decorator) would be OK.
Update: In Python 3 this isn't a problem, since you can use __class__ (or the super variant without arguments), so the following works:
#class_decorator
class MySubclass(MyClass):
def say(self, x):
super().say(x.upper())
Unfortunately, I'm stuck with Python 2.7 for this project.
The problem is that your decorator returns a different class than python (or anyone who uses your code) expects. super not working is just one of the many unfortunate consequences:
>>> isinstance(MySubclass().make_instance(), MySubclass)
False
>>> issubclass(MySubclass, MyClass)
False
>>> pickle.dumps(MySubclass().make_instance())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
_pickle.PicklingError: Can't pickle <class '__main__.MySubclass'>: it's not the same object as __main__.MySubclass
This is why a class decorator should modify the class instead of returning a different one. The correct implementation would look like this:
def class_decorator(wrapped_cls):
#classmethod
def make_instance(cls):
return cls()
wrapped_cls.make_instance = make_instance
return wrapped_cls
Now super and everything else will work as expected:
>>> MySubclass().make_instance().say('hello')
HELLO
The problem occurs because at the time when MySubclass.say() is called, the global symbol MySubclass no longer refers to what's defined in your code as 'class MySubclass'. It is an instance of WrapperClass, which isn't in any way related to MySubclass.
If you are using Python3, you can get around this by NOT passing any arguments to 'super', like this:
super().say(x.upper())
I don't really know why you use the specific construct that you have, but it does look strange that a sub-class of MyClass that defines 'say()' - and has itself a 'say()' method in the source code would have to end up as something that does not have that method - which is the case in your code.
Note you could change the class WrapperClass line to make it read
class WrapperClass(cls):
this will make your wrapper a sub-class of the one you just decorated. This doesn't help with your super(SubClass, self) call - you still need to remove the args (which is OK only on Python3), but at least an instance created as x=MySubclass() would have a 'say' method, as one would expect at first glance.
EDIT: I've come up with a way around this, but it really looks odd and has the disadvantage of making the 'wrapped' class know that it is being wrapped (and it becomes reliant on that, making it unusable if you remove the decorator):
def class_decorator(cls):
class WrapperClass(object):
def make_instance(self):
i = cls()
i._wrapped = cls
return i
return WrapperClass
class MyClass(object):
def say(self, x):
print(x)
#class_decorator
class MySubclass(MyClass):
def say(self, x):
super(self._wrapped, self).say(x.upper())
# make_instance returns inst of the original class, non-decorated i = MySubclass().make_instance() i.say('hello')
In essence, _wrapped saves a class reference as it was at declaration time, consistent with using the regular super(this_class_name, self) builtin call.
Suppose you want to call a method foo on object bar, but somehow while typing the method invocation you intuitively treated foo as a property and you typed bar.foo instead of bar.foo() (with parenthesis). Now, both are syntactically correct, so no error is raised, but semantically very different. It happened to me several times already (my experience in Ruby makes it even worse) and caused me dearly in terms of long and confusing debugging sessions.
Is there a way to make Python interpreter print a warning in such cases - whenever you access an attribute which is callable, but you haven't actually called it?
For the record - I thought about overriding __getattribute__ but it's messy and ultimately won't achieve the goal since function invocation via () happens after __getattribute__ has returned.
This can't be done in all cases because sometimes you don't want to call the method, e.g. you might want to store it as a callable to be used later, like callback = object.method.
But you can use static analysis tools such as pylint or PyCharm (my recommendation) that warn you if you write a statement that looks pointless, e.g. object.method without any assignment.
Furthermore if you write x = obj.get_x but meant get_x(), then later when you try to use x a static analysis tool may be able to warn you (if you're lucky) that x is a method but an instance of X is expected.
It was quite challenging, but I think i get it done! My code isn't very complicated, but you need to be well aware of metaclasses.
Metaclass and wrapper (WarnIfNotCalled.py):
class Notifier:
def __init__(self, name, obj, callback):
self.callback = callback
self.name = name
self.obj = obj
self.called = False
def __call__(self, *args, **kwargs):
self.callback(self.obj, *args, **kwargs)
self.called = True
def __del__(self):
if not self.called:
print("Warning! {} function hasn't been called!".format(self.name))
class WarnIfNotCalled(type):
def __new__(cls, name, bases, dct):
dct_func = {}
for name, val in dct.copy().items():
if name.startswith('__') or not callable(val):
continue
else:
dct_func[name] = val
del dct[name]
def getattr(self, name):
if name in dct_func:
return Notifier(name, self, dct_func[name])
dct['__getattr__'] = getattr
return super(WarnIfNotCalled, cls).__new__(cls, name, bases, dct)
It's very easy to use - just specify a metaclass
from WarnIfNotCalled import WarnIfNotCalled
class A(metaclass = WarnIfNotCalled):
def foo(self):
print("foo has been called")
def bar(self, x):
print("bar has been called and x =", x)
If you didn't forget to call these functions, everything works as usual
a = A()
a.foo()
a.bar(5)
Output:
foo has been called
bar has been called and x = 5
But if you DID forget:
a = A()
a.foo
a.bar
You see the following
Warning! foo function hasn't been called!
Warning! bar function hasn't been called!
Happy debugging!
For one of the project I am currently working I was thinking of creating a class that could not be instantiate by a client and only be supplied an instance of through a particular interface i.e. the client would not be able create further instance out of it by some hackery such as:
>>> try:
... raise WindowsError
... except:
... foo = sys.exc_info()
...
>>> foo
(<type 'exceptions.WindowsError'>, WindowsError(), <traceback object at 0x0000000005503A48>)
>>> type(foo[2])()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: cannot create 'traceback' instances
once he has one.
I was successfully able to create a class that couldn't be instantiated. i.e.
>>> class Foo():
... def __init__(self):
... raise TypeError("cannot create 'Foo' instances")
...
>>> bar = Foo()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __init__
TypeError: cannot create 'Foo' instances
>>> bar
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'bar' is not defined
But how could I use this every same definition to create an instance of the class?
Of course I could do something like this:
>>> class Foo():
... def __init__(self, instantiate = False):
... if not instantiate:
... raise TypeError("cannot create 'Foo' instances")
but I don't find it elegant enough nor does it completely prevent the client from further instantiating it. And no I aint going down the road of building a C++ module for it.
Any suggestions on how to achieve such a thing? import abc?
A brief rational to answer Martijn's question and for completeness:
Actual you could consider the instance of the particular, and related, classes, in question, as nodes in a tree and that both the parent and the children to remain connected, dependent on and cognizant of each other and have a single unique root throughout any instance of python(insured by the use package). Any state changes in a particular node would cause others to update themselves and the database to which they are connect, accordingly. Apart from that I was being curious to know how such a thing could be put in place (the traceback class was teasing me).
What you're doing is a bad idea, you shouldn't do it.
I'm sure there's an other, better solution.
If you do decide to go with your way anyways (you shouldn't), here's how you can create an object without using __init__():
Objects in python are created with the __new__() method. The method __init__() only edits the object which was created by __new__(). For example, __init__() usually initializes some attributes for the object.
When declaring something like x = Foo() what happens is this:
x = object.__new__(Foo) gets called first and creates the object.
Foo.__init__(x) gets called second, it simply initializes some attributes etc. to the already existing object.
This means that you are not required to call Foo() (and as a result, call __init__() too). Instead, you can just call __new__() directly:
class Foo(object):
def __init__(self):
raise TypeError("Cannot create 'Foo' instances.")
>>> x = object.__new__(Foo)
>>> x
<__main__.Foo object at 0x02B074F0>
Our x is now an instance of Foo, without any attributes that is, and it can use any methods defined in Foo class.
If you want, you can create your own replacement function of __init__ for initializing attributes:
def init_foo(foo, name):
foo.name = name
>>> init_foo(x, "Mike")
>>> x.name
'Mike'
This could of course be Foo's instance method too:
class Foo(object):
def __init__(self):
raise TypeError("Cannot create 'Foo' instances.")
def init(self, name):
self.name = name
>>> x = object.__new__(Foo)
>>> x.init("Mike")
>>> x.name
'Mike'
Going even step further, you can even use a classmethod for creating your object with only one call:
class Foo(object):
def __init__(self):
raise TypeError("Cannot create 'Foo' instances.")
#classmethod
def new(cls, name):
obj = object.__new__(cls)
obj.name = name
return obj
>>> x = Foo.new("Mike")
>>> x.name
'Mike'