Develop Reference

Pickling dynamically created types

I've been trying to get some dynamically created types (i.e. ones created by calling 3-arg type()) to pickle and unpickle nicely. I've been using this module switching trick to hide the details from users of the module and give clean semantics.
I've learned several things already:
The type must be findable with getattr on the module itself
The type must be consistent with what getattr finds, that is to say if we call pickle.dumps(o) then it must be true that type(o) == getattr(module, 'name of type')
Where I'm stuck though is that there still seems to be something odd going on - it seems to be calling __getstate__ on something unexpected.
Here's the simplest setup I've got that reproduces the issue, testing with Python 3.5, but I'd like to target back to 3.3 if possible:
# module.py
import sys
import functools
def dump(self):
return b'Some data' # Dummy for testing
def undump(self, data):
print('Undump: %r' % data) # Do nothing for testing
# Cheaty demo way to make this consistent
#functools.lru_cache(maxsize=None)
def make_type(name):
return type(name, (), {
'__getstate__': dump,
'__setstate__': undump,
})
class Magic(object):
def __init__(self, path):
self.path = path
def __getattr__(self, name):
print('Getting thing: %s (from: %s)' % (name, self.path))
# for simple testing all calls to make_type must end in last x.y.z.last
if name != 'last':
if self.path:
return Magic(self.path + '.' + name)
else:
return Magic(name)
return make_type(self.path + '.' + name)
# Make the switch
sys.modules[__name__] = Magic('')
And then a quick way to exercise that:
import module
import pickle
f=module.foo.bar.woof.last()
print(f.__getstate__()) # See, *this* works
print('Pickle starts here')
print(pickle.dumps(f))
Which then gives:
Getting thing: foo (from: )
Getting thing: bar (from: foo)
Getting thing: woof (from: foo.bar)
Getting thing: last (from: foo.bar.woof)
b'Some data'
Pickle starts here
Getting thing: __spec__ (from: )
Getting thing: _initializing (from: __spec__)
Getting thing: foo (from: )
Getting thing: bar (from: foo)
Getting thing: woof (from: foo.bar)
Getting thing: last (from: foo.bar.woof)
Getting thing: __getstate__ (from: foo.bar.woof)
Traceback (most recent call last):
File "test.py", line 7, in <module>
print(pickle.dumps(f))
TypeError: 'Magic' object is not callable
I wasn't expecting to see anything looking up __getstate__ on module.foo.bar.woof, but even if we force that lookup to fail by adding:
if name == '__getstate__': raise AttributeError()
into our __getattr__ it still fails with:
Traceback (most recent call last):
File "test.py", line 7, in <module>
print(pickle.dumps(f))
_pickle.PicklingError: Can't pickle <class 'module.Magic'>: it's not the same object as module.Magic
What gives? Am I missing something with __spec__? The docs for __spec__ pretty much just stress setting it appropriately, but don't seem to actually explain much.
More importantly the bigger question is how am I supposed to go about making types I programatically generated via a pseudo module's __getattr__ implementation pickle properly?
(And obviously once I've managed to get pickle.dumps to produce something I expect pickle.loads to call undump with the same thing)

To pickle f, pickle needs to pickle f's class, module.foo.bar.woof.last.
The docs don't claim support for pickling arbitrary classes. They claim the following:
The following types can be pickled:
...
classes that are defined at the top level of a module
module.foo.bar.woof.last isn't defined at the top level of a module, even a pretend module like module. In this not-officially-supported case, the pickle logic ends up trying to pickle module.foo.bar.woof, either here:
elif parent is not module:
self.save_reduce(getattr, (parent, lastname))
or here
else if (parent != module) {
PickleState *st = _Pickle_GetGlobalState();
PyObject *reduce_value = Py_BuildValue("(O(OO))",
st->getattr, parent, lastname);
status = save_reduce(self, reduce_value, NULL);
module.foo.bar.woof can't be pickled for multiple reasons. It returns a non-callable Magic instance for all unsupported method lookups, like __getstate__, which is where your first error comes from. The module-switching thing prevents finding the Magic class to pickle it, which is where your second error comes from. There are probably more incompatibilities.

As it seems, and is already proven that making the class callable is just a drifting out another wrong direction, thankfully to this hack, I could find a getaround to make the class reiterable by its TYPE. following the context of the error <class 'module.Magic'>: it's not the same object as module.Magic the pickler doesn't iterate through the same call that renders a different type from the other one, this is a major common problem with pickling self instanciating classes, for this instance, an object by its class, there for the solution is patching the class with its type #mock.patch('module.Magic', type(module.Magic)) this is a short answer for a something.
Main.py
import module
import pickle
import mock
f=module1.foo.bar.woof.last
print(f().__getstate__()) # See, *this* works
print('Pickle starts here')
#mock.patch('module1.Magic', type(module1.Magic))
def pickleit():
return pickle.dumps(f())
print(pickleit())
Magic class
class Magic(object):
def __init__(self, value):
self.path = value
__class__: lambda x:x
def __getstate__(self):
print ("Shoot me! i'm at " + self.path )
return dump(self)
def __setstate__(self,value):
print ('something will never occur')
return undump(self,value)
def __spec__(self):
print ("Wrong side of the planet ")
def _initializing(self):
print ("Even farther lost ")
def __getattr__(self, name):
print('Getting thing: %s (from: %s)' % (name, self.path))
# for simple testing all calls to make_type must end in last x.y.z.last
if name != 'last':
if self.path:
return Magic(self.path + '.' + name)
else:
return Magic(name)
print('terminal stage' )
return make_type(self.path + '.' + name)
Even assuming this is not more of striking the ball by the edge of the bat, I could see the content dumped into my console.

How can I create an Exception in Python minus the last stack frame?

Not sure how possible this is, but here goes:
I'm trying to write an object with some slightly more subtle behavior - which may or may not be a good idea, I haven't determined that yet.
I have this method:
def __getattr__(self, attr):
try:
return self.props[attr].value
except KeyError:
pass #to hide the keyerror exception
msg = "'{}' object has no attribute '{}'"
raise AttributeError(msg.format(self.__dict__['type'], attr))
Now, when I create an instance of this like so:
t = Thing()
t.foo
I get a stacktrace containing my function:
Traceback (most recent call last):
File "attrfun.py", line 23, in <module>
t.foo
File "attrfun.py", line 15, in __getattr__
raise AttributeError(msg.format(self._type, attr))
AttributeError: 'Thing' object has no attribute 'foo'
I don't want that - I want the stack trace to read:
Traceback (most recent call last):
File "attrfun.py", line 23, in <module>
t.foo
AttributeError: 'Thing' object has no attribute 'foo'
Is this possible with a minimal amount of effort, or is there kind of a lot required? I found this answer which indicates that something looks to be possible, though perhaps involved. If there's an easier way, I'd love to hear it! Otherwise I'll just put that idea on the shelf for now.

You cannot tamper with traceback objects (and that's a good thing). You can only control how you process one that you've already got.
The only exceptions are: you can
substitute an exception with another or re-raise it with raise e (i.e make the traceback point to the re-raise statement's location)
raise an exception with an explicit traceback object
remove outer frame(s) from a traceback object by accessing its tb_next property (this reflects a traceback object's onion-like structure)
For your purpose, the way to go appears to be the 1st option: re-raise an exception from a handler one level above your function.
And, I'll say this again, this is harmful for yourself or whoever will be using your module as it deletes valuable diagnostic information. If you're dead set on making your module proprietary with whatever rationale, it's more productive for that goal to make it a C extension.

The traceback object is created during stack unwinding, not directly when you raise the exception, so you can not alter it right in your function. What you could do instead (though it's probably a bad idea) is to alter the top level exception hook so that it hides your function from the traceback.
Suppose you have this code:
class MagicGetattr:
def __getattr__(self, item):
raise AttributeError(f"{item} not found")
orig_excepthook = sys.excepthook
def excepthook(type, value, traceback):
iter_tb = traceback
while iter_tb.tb_next is not None:
if iter_tb.tb_next.tb_frame.f_code is MagicGetattr.__getattr__.__code__:
iter_tb.tb_next = None
break
iter_tb = iter_tb.tb_next
orig_excepthook(type, value, traceback)
sys.excepthook = excepthook
# The next line will raise an error
MagicGetattr().foobar
You will get the following output:
Traceback (most recent call last):
File "test.py", line 49, in <module>
MagicGetattr().foobar
AttributeError: foobar not found
Note that this ignores the __cause__ and __context__ members of the exception, which you would probably want to visit too if you were to implement this in real life.

You can get the current frame and any other level using the inspect module. For instance, here is what I use when I'd like to know where I'm in my code :
from inspect import currentframe
def get_c_frame(level = 0) :
"""
Return caller's frame
"""
return currentframe(level)
...
def locate_error(level = 0) :
"""
Return a string containing the filename, function name and line
number where this function was called.
Output is : ('file name' - 'function name' - 'line number')
"""
fi = get_c_frame(level = level + 2)
return '({} - {} - {})'.format(__file__,
fi.f_code,
fi.f_lineno)

Add extra information to all python exceptions

After another terrible bug hunt, I am wondering the following:
Is it possible to add some extra information to all exceptions, for example the name of an object.
This would increase the readability of the errors a lot and make looking for bugs (or input errors) a lot quicker. This is especially the case if one has many objects which are from the same class and therefore share much code, but have different attributes. In this case it can be very useful if the error message also states the name of the object in the error.
A simplfied example: I am trying to simulate different types of facilities, a pig farm and a cow farm. These are the same class, but do have different attributes. In the simulation many facilities are made and if an exception is raised, it would be very helpful if the name of the object is added to the exception.
class facility():
def __init__(self, name):
self.name = name
self.animals = []
farms = []
farms.append(facility('cow_farm'))
farms.append(facility('pig_farm'))
print farms[0].stock
This would yield
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: facility instance has no attribute 'stock'
But I would like to add the name of the facility:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: facility instance has no attribute 'stock'
Name of object: cow_farm
I tried something like
def print_name(exception):
try:
print self.name
except AttributeError:
pass
raise exception
#print_name
Exception
But that doesn't work. Is it possible to do such a thing, or are there good reasons not to do this?

If you want to handle errors and add information, you could do it as follows:
farm = farms[0]
try:
print farm.stock
except AttributeError:
raise AttributeError("{} has no attribute 'stock'".format(farm.name))
However, it might be wiser to add the empty stock in __init__ to avoid this error.
You should never use a bare except, as it hides useful information from you (particularly when developing and debugging!) Generally, each try block should be as short as possible, preferably doing only one thing. If multiple errors could stem from a single try block, you can add multiple handlers:
try:
print farm.stock["hay"]
except AttributeError:
raise AttributeError("{} has no attribute 'stock'".format(farm.name))
except KeyError:
raise KeyError("{} has no 'hay' in 'stock'".format(farm.name))
(Although note that adding self.stock in __init__ and checking if "hay" in farm.stock: would save you from this error handling.)
If an error happens that you weren't expecting, it is generally best for that error to propagate up the call stack until it is explicitly handled or you get to see it. Otherwise, you are heading for this foolish anti-pattern:
def some_func(*args, **kwargs):
try:
# all of some_func's content goes here
except:
raise Exception("Something went wrong in some_func().")
Which is no use to you and extremely frustrating for anyone trying to use your code.
If you want to handle AttributeErrors like this at the class level, you could do:
class Facility(object):
def __init__(self, ...):
...
def __getattr__(self, key):
"""Called on attempt to access attribute instance.key."""
if key not in self.__dict__:
message = "{} instance '{}' has no attribute '{}'."
message = message.format(type(self).__name__,
self.name, key)
raise AttributeError(message)
else:
return self.__dict__[key]
Then you will get
>>> farm = Facility("pig farm")
>>> print farm.stock
...
"AttributeError: Facility instance 'pig farm' has no attribute 'stock'."
If you want to use this pattern with multiple classes, you can make a superclass:
class ProtectedAttrs(object):
def __init__(self, name):
self.name = name
def __getattr__(self, key):
...
class Facility(ProtectedAttrs):
def __init__(self, name):
super(Facility, self).__init__(name)
self.animals = []
This sort of thing will work for some types of error. However, I am not aware of any general way to handle all errors with references to the instance involved.

Most of exceptions contain message attribute, that give you additional information about error
In [163]: try:
.....: farm = []
.....: farm.stock
.....: except AttributeError as err:
.....: print err.message
.....:
'list' object has no attribute 'stock'
Exception traceback points you to a code line, so usually it is not hard to figure out the problem

Python pickle crash when trying to return default value in __getattr__

I have a dictionary like class that I use to store some values as attributes. I recently added some logic(__getattr__) to return None if an attribute doesn't exist. As soon as I did this pickle crashed, and I wanted some insight into why?
Test Code:
import cPickle
class DictionaryLike(object):
def __init__(self, **kwargs):
self.__dict__.update(kwargs)
def __iter__(self):
return iter(self.__dict__)
def __getitem__(self, key):
if(self.__dict__.has_key(key)):
return self.__dict__[key]
else:
return None
''' This is the culprit...'''
def __getattr__(self, key):
print 'Retreiving Value ' , key
return self.__getitem__(key)
class SomeClass(object):
def __init__(self, kwargs={}):
self.args = DictionaryLike(**kwargs)
someClass = SomeClass()
content = cPickle.dumps(someClass,-1)
print content
Result:
Retreiving Value __getnewargs__
Traceback (most recent call last):
File <<file>> line 29, in <module>
content = cPickle.dumps(someClass,-1)
TypeError: 'NoneType' object is not callable`
Did I do something stupid? I had read a post that deepcopy() might require that I throw an exception if a key doesn't exist? If this is the case is there any easy way to achieve what I want without throwing an exception?
End result is that if some calls
someClass.args.i_dont_exist
I want it to return None.

Implementing __getattr__ is a bit tricky, since it is called for every non-existing attribute. In your case, the pickle module tests your class for the __getnewargs__ special method and receives None, which is obviously not callable.
You might want to alter __getattr__ to call the base implementation for magic names:
def __getattr__(self, key):
if key.startswith('__') and key.endswith('__'):
return super(DictionaryLike, self).__getattr__(key)
return self.__getitem__(key)
I usually pass through all names starting with an underscore, so that I can sidestep the magic for internal symbols.

You need to raise an AttributeError when an attribute is not present in your class:
def __getattr__(self, key):
i = self.__getitem__(key)
if i == None:
raise AttributeError
return self.__getitem__(key)
I am going to assume that this behavior is required. From the python documentation for getattr, "Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name. This method should return the (computed) attribute value or raise an AttributeError exception."
There is no way to tell pickle etc that the attribute it's looking for is not found unless you raise the exception. For example, in your error message pickle is looking for a special callable method called __getnewargs__, pickle expects that if the AttributeError exception is not found the return value is callable.
I guess one potential work around you could perhaps try defining all of the special methods pickle is looking for as dummy methods?

Python: dynamically add attributes to new-style class/obj

Can I dynamically add attributes to instances of a new-style class (one that derives from object)?
Details:
I'm working with an instance of sqlite3.Connection. Simply extending the class isn't an option because I don't get the instance by calling a constructor; I get it by calling sqlite3.connect().
Building a wrapper doesn't save me much of the bulk for the code I'm writing.
Python 2.7.1
Edit
Right answers all. But I still am not reaching my goal; instances of sqlite3.Connection bar my attempts to set attributes in the following ways (as do instances of object itself). I always get an AttributeError:
> conn = sqlite3.connect([filepath])
> conn.a = 'foo'
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
conn.a = 'foo'
AttributeError: 'object' object has no attribute 'a'
> conn.__setattr__('a','foo')
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
conn.__setattr__('a','foo')
AttributeError: 'object' object has no attribute 'a'
Help?

Yes, unless the class is using __slots__ or preventing attribute writing by overriding __setattr__, or an internal Python class, or a Python class implemented natively (usually in C).
You can always try setting an attribute. Except for seriously weird __setattr__ implementations, assigning an attribute to an instance of a class of one of the types mentioned above should raise an AttributeError.
In these cases, you'll have to use a wrapper, like this:
class AttrWrapper(object):
def __init__(self, wrapped):
self._wrapped = wrapped
def __getattr__(self, n):
return getattr(self._wrapped, n)
conn = AttrWrapper(sqlite3.connect(filepath))

Simple experimentation:
In []: class Tst(object): pass
..:
In []: t= Tst()
In []: t.attr= 'is this valid?'
In []: t.attr
Out[]: 'is this valid?'
So, indeed it seems to be possible to do that.
Update:
But from the documentation: SQLite is a C library that ..., so it seems that you really need to wrap it.

conn.a = 'foo',
or any dynamic assignment is valid, if conn is
<type 'classobj'>.
Things like:
c=object()
c.e=1
will raise an Attribute error. On the otherhand: Python allows you to do fantastic Metaclass programming:
>>>from new import classobj
>>>Foo2 = classobj('Foo2',(Foo,),{'bar':lambda self:'bar'})
>>>Foo2().bar()
>>>'bar'
>>>Foo2().say_foo()
>>>foo