Develop Reference

Pickling dynamically created types

I've been trying to get some dynamically created types (i.e. ones created by calling 3-arg type()) to pickle and unpickle nicely. I've been using this module switching trick to hide the details from users of the module and give clean semantics.
I've learned several things already:
The type must be findable with getattr on the module itself
The type must be consistent with what getattr finds, that is to say if we call pickle.dumps(o) then it must be true that type(o) == getattr(module, 'name of type')
Where I'm stuck though is that there still seems to be something odd going on - it seems to be calling __getstate__ on something unexpected.
Here's the simplest setup I've got that reproduces the issue, testing with Python 3.5, but I'd like to target back to 3.3 if possible:
# module.py
import sys
import functools
def dump(self):
return b'Some data' # Dummy for testing
def undump(self, data):
print('Undump: %r' % data) # Do nothing for testing
# Cheaty demo way to make this consistent
#functools.lru_cache(maxsize=None)
def make_type(name):
return type(name, (), {
'__getstate__': dump,
'__setstate__': undump,
})
class Magic(object):
def __init__(self, path):
self.path = path
def __getattr__(self, name):
print('Getting thing: %s (from: %s)' % (name, self.path))
# for simple testing all calls to make_type must end in last x.y.z.last
if name != 'last':
if self.path:
return Magic(self.path + '.' + name)
else:
return Magic(name)
return make_type(self.path + '.' + name)
# Make the switch
sys.modules[__name__] = Magic('')
And then a quick way to exercise that:
import module
import pickle
f=module.foo.bar.woof.last()
print(f.__getstate__()) # See, *this* works
print('Pickle starts here')
print(pickle.dumps(f))
Which then gives:
Getting thing: foo (from: )
Getting thing: bar (from: foo)
Getting thing: woof (from: foo.bar)
Getting thing: last (from: foo.bar.woof)
b'Some data'
Pickle starts here
Getting thing: __spec__ (from: )
Getting thing: _initializing (from: __spec__)
Getting thing: foo (from: )
Getting thing: bar (from: foo)
Getting thing: woof (from: foo.bar)
Getting thing: last (from: foo.bar.woof)
Getting thing: __getstate__ (from: foo.bar.woof)
Traceback (most recent call last):
File "test.py", line 7, in <module>
print(pickle.dumps(f))
TypeError: 'Magic' object is not callable
I wasn't expecting to see anything looking up __getstate__ on module.foo.bar.woof, but even if we force that lookup to fail by adding:
if name == '__getstate__': raise AttributeError()
into our __getattr__ it still fails with:
Traceback (most recent call last):
File "test.py", line 7, in <module>
print(pickle.dumps(f))
_pickle.PicklingError: Can't pickle <class 'module.Magic'>: it's not the same object as module.Magic
What gives? Am I missing something with __spec__? The docs for __spec__ pretty much just stress setting it appropriately, but don't seem to actually explain much.
More importantly the bigger question is how am I supposed to go about making types I programatically generated via a pseudo module's __getattr__ implementation pickle properly?
(And obviously once I've managed to get pickle.dumps to produce something I expect pickle.loads to call undump with the same thing)

To pickle f, pickle needs to pickle f's class, module.foo.bar.woof.last.
The docs don't claim support for pickling arbitrary classes. They claim the following:
The following types can be pickled:
...
classes that are defined at the top level of a module
module.foo.bar.woof.last isn't defined at the top level of a module, even a pretend module like module. In this not-officially-supported case, the pickle logic ends up trying to pickle module.foo.bar.woof, either here:
elif parent is not module:
self.save_reduce(getattr, (parent, lastname))
or here
else if (parent != module) {
PickleState *st = _Pickle_GetGlobalState();
PyObject *reduce_value = Py_BuildValue("(O(OO))",
st->getattr, parent, lastname);
status = save_reduce(self, reduce_value, NULL);
module.foo.bar.woof can't be pickled for multiple reasons. It returns a non-callable Magic instance for all unsupported method lookups, like __getstate__, which is where your first error comes from. The module-switching thing prevents finding the Magic class to pickle it, which is where your second error comes from. There are probably more incompatibilities.

As it seems, and is already proven that making the class callable is just a drifting out another wrong direction, thankfully to this hack, I could find a getaround to make the class reiterable by its TYPE. following the context of the error <class 'module.Magic'>: it's not the same object as module.Magic the pickler doesn't iterate through the same call that renders a different type from the other one, this is a major common problem with pickling self instanciating classes, for this instance, an object by its class, there for the solution is patching the class with its type #mock.patch('module.Magic', type(module.Magic)) this is a short answer for a something.
Main.py
import module
import pickle
import mock
f=module1.foo.bar.woof.last
print(f().__getstate__()) # See, *this* works
print('Pickle starts here')
#mock.patch('module1.Magic', type(module1.Magic))
def pickleit():
return pickle.dumps(f())
print(pickleit())
Magic class
class Magic(object):
def __init__(self, value):
self.path = value
__class__: lambda x:x
def __getstate__(self):
print ("Shoot me! i'm at " + self.path )
return dump(self)
def __setstate__(self,value):
print ('something will never occur')
return undump(self,value)
def __spec__(self):
print ("Wrong side of the planet ")
def _initializing(self):
print ("Even farther lost ")
def __getattr__(self, name):
print('Getting thing: %s (from: %s)' % (name, self.path))
# for simple testing all calls to make_type must end in last x.y.z.last
if name != 'last':
if self.path:
return Magic(self.path + '.' + name)
else:
return Magic(name)
print('terminal stage' )
return make_type(self.path + '.' + name)
Even assuming this is not more of striking the ball by the edge of the bat, I could see the content dumped into my console.

Mocking an entire class

Long story short, I'm perfectly able to mock class method, when it's just that method that's replaced by mock object, but I'm unable to mock that method when I'm trying to replace the whole class by the mock object
The #mock.patch.object successfully mocks the scan method but #mock.patch fails to do so. I've followed the example at
https://docs.python.org/3/library/unittest.mock.html#unittest.mock.patch
but apparently I'm doing something wrong.
I'm mocking the lexicon module in the same namespace in both cases (it's imported by import lexicon in the sentence_parser) but the mock_lexicon is lexicon.lexicon check fails
#!python
import sys;
sys.path.append('D:\python\lexicon');
import lexicon;
import sentence_parser;
import unittest2 as unittest;
import mock;
class ParserTestCases(unittest.TestCase) :
def setUp(self) :
self.Parser = sentence_parser.Parser();
#mock.patch('lexicon.lexicon')
def test_categorizedWordsAreAssigned_v1(self, mock_lexicon) :
print "mock is lexicon:";
print mock_lexicon is lexicon.lexicon + "\n";
instance = mock_lexicon.return_value;
instance.scan.return_value = "anything";
self.Parser.categorize_words_in_sentence("sentence");
instance.scan.assert_called_once_with("sentence");
#mock.patch.object(lexicon.lexicon, 'scan')
def test_categorizedWordsAreAssigned_v2(self, mock_scan) :
mock_scan.return_value = "anything";
self.Parser.categorize_words_in_sentence("sentence");
mock_scan.assert_called_once_with("sentence");
if (__name__ == '__main__') :
unittest.main()
Output :
mock is lexicon:
False
======================================================================
FAIL: test_categorizedWordsAreAssigned_v1 (__main__.ParserTestCases)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\python\get_img\getImage_env\lib\site-packages\mock\mock.py", line 1305, in patched
return func(*args, **keywargs)
File "./test_sentence_parser.py", line 26, in test_categorizedWordsAreAssigned_v1
instance.scan.assert_called_once_with("sentence");
File "D:\python\get_img\getImage_env\lib\site-packages\mock\mock.py", line 947, in assert_called_once_with
raise AssertionError(msg)
AssertionError: Expected 'scan' to be called once. Called 0 times.
----------------------------------------------------------------------
Ran 2 tests in 0.009s
FAILED (failures=1)
EDIT :
To clarify, the Parser is defined as follows
#!python
import sys;
sys.path.append('D:\python\lexicon');
import lexicon;
class Parser(object) :
my_lexicon = lexicon.lexicon()
def __init__(self) :
self.categorized_words = ['test'];
def categorize_words_in_sentence(self, sentence) :
self.categorized_words = self.my_lexicon.scan(sentence);
if (__name__ == '__main__') :
instance = Parser();
instance.categorize_words_in_sentence("bear");
print instance.categorized_words;

What is real relevant here is how categorize_words_in_sentence Parser's method use lexicon. But first of all we should remove the noise:
print mock_lexicon is lexicon.lexicon + "\n"
Is what can lead us to the wrong direction: try to replace it by
self.assertIs(mock_lexicon, lexicon.lexicon)
and you will understand that you are printing False because mock_lexicon is not lexicon.lexicon + "\n" but just lexicon.lexicon.
Now I cannot tell you why the first test doesn't work because the answer is in categorize_words_in_sentence method or more probably in sentence_parser module where I can guess you can have something like
from lexicon import lexicon
In both case take a look to Where to Patch documentation that can enlighten you on what can be the cause and what you really need to patch in your case.
The second version works just because you are patching the object and not the reference (that should be different).
Finally the more concise and general version can be:
#mock.patch('lexicon.lexicon.scan', return_value="anything")
def test_categorizedWordsAreAssigned_v3(self, mock_scan) :
self.Parser.categorize_words_in_sentence("sentence")
mock_scan.assert_called_once_with("sentence")
One more thing: remove unittest2 at least you're not using python 2.4 and you are interested on backported unittest features.
[EDIT]
Now I can stop to guess and point to you why the first version doesn't work and will never work:
class Parser(object) :
my_lexicon = lexicon.lexicon()
Parser.my_lexicon attribute is evaluated at the load time. That means when you import sentence_parser a lexicon is created and the reference associated to Parser.my_lexicon. When you patch lexicon.lexicon you leave this reference untouched and your parser object still use the original reference created when is imported.
What you can do is to patch the reference in Parser class by
#patch("sentence_parser.Parser.my_lexicon")
You can use create_autospect if you want give to your mock the same lexicon's signature.
#patch("sentence_parser.Parser.my_lexicon", create_autospec("lexicon.lexicon", instance=True))

__str__ method in python

I wonder why the python magic method (str) always looking for the return statement rather a print method ?
class test:
def __init__(self):
print("constructor called")
def __call__(self):
print("callable")
def __str__(self):
return "string method"
obj=test() ## print constructor called
obj() ### print callable
print(obj) ## print string method
my question is why i can't use something like this inside the str method
def __str__(self):
print("string method")

This is more to enable the conversion of an object into a str - your users don't necessary want all that stuff be printed into the terminal whenever they want to do something like
text = str(obj_instance)
They want text to contain the result, not printed out onto the terminal.
Doing it your way, the code would effectively be this
text = print(obj_instance)
Which is kind of nonsensical because the result of print isn't typically useful and text won't contain the stream of text that was passed into str type.
As you already commented (but since deleted), not providing the correct type for the return value will cause an exception to be raised, for example:
>>> class C(object):
... def __str__(self):
... return None
...
>>> str(C())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __str__ returned non-string (type NoneType)
>>>

Because __str__() is used when you print the object, so the user is already calling print which needs the String that represent the Object - as a variable to pass back to the user's print
In the example you provided above, if __str__ would print you would get:
print(obj)
translated into:
print(print("string method"))

What is the best way to pass variables to my unittest instances?

I've got test classes that inherit from unittest.TestCase. I load the classes multiple times like so:
tests = [TestClass1, TestClass2]
for test in tests:
for var in array:
# somehow indicate that this test should have the value of 'var'
suite.addTest(unittest.defaultTestLoader.loadTestsFromTestCase(test))
Thing is, I want to pass the value of 'var' to each test, but I cannot use class variables because they are shared between every instance of the class, and I don't have access to the code that actually does the instantiation of the objects. What is the best way of accomplishing this?

I think that even if you don't have access to the class that implement the test cases, you can subclass them and overload the setUp method.

I think you're going about this the wrong way. Rather than doing what you are trying there why dont you just do, say you have in class:
from my_tests.variables import my_array
class TestClass1(unittest.TestCase):
def setUp():
....initializations...
def tearDown():
....clean up after...
def my_test_that_should_use_value_from_array(self):
for value in my_array:
test_stuff(value)

UPDATE:
Since you need to:
feed some variable value to MyTestCase
run MyTestCase using this value
change value
If MyTestCase still running - use updated value.
Consider this:
keep values map in the file (.csv/.txt/.xml/etc.)
read values map from file in the setUp()
find value for your MyTestCase from values map using TestCase.id() method (as shown in the example below).
use it in the test cases.
unittest has handy id() method, which returns test case name in filename.testclassname.methodname format.
So you can use it like this:
import unittest
my_variables_map = {
'test_01': 'foo',
'test_02': 'bar',
}
class MyTest(unittest.TestCase):
def setUp(self):
test_method_name = self.id() # filename.testclassname.methodname
test_method_name = test_method_name.split('.')[-1] # method name
self.variable_value = my_variables_map.get(test_method_name)
self.error_message = 'No values found for "%s" method.' % test_method_name
def test_01(self):
self.assertTrue(self.variable_value is not None, self.error_message)
def test_02(self):
self.assertTrue(self.variable_value is not None, self.error_message)
def test_03(self):
self.assertTrue(self.variable_value is not None, self.error_message)
if __name__ == '__main__':
unittest.main()
This gives you:
$ python /tmp/ut.py
..F
======================================================================
FAIL: test_03 (__main__.MyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/ut.py", line 25, in test_03
self.assertTrue(self.variable_value is not None, self.error_message)
AssertionError: No values found for "test_03" method.
----------------------------------------------------------------------
Ran 3 tests in 0.000s
FAILED (failures=1)
$

I found the Data-Driven Tests (DDT - not the pesticide) package helpful for this.
http://ddt.readthedocs.org/en/latest/example.html

How to add abilities to strings in Python

This is more of a curiosity question than anything else. I'm new with Python and playing around with it. I've just looked at the base64 module. What if instead of doing:
import base64
string = 'Foo Bar'
encoded = base664.b64encode
I wanted to do something like:
>>> class b64string():
>>> <something>
>>>
>>> string = b64string('Foo Bar')
>>> string
'Foo Bar'
>>> string.encode64()
'Rm9vIEJhcg=='
>>> string
'Rm9vIEJhcg=='
>>> string.assign('QmFyIEZvbw==')
>>> string
'QmFyIEZvbw=='
>>> string.b64decode()
'Bar Foo'
>>> string
'Bar Foo'
Is there a simple, pythonic way to create that class?
I've begun with this:
>>> class b64string(base64):
... def __init__(self, v):
... self.value=v
And already I get:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
module.__init__() takes at most 2 arguments (3 given)
And don't get me started on (just to see what would happen):
>>> class b64string(str, base64): pass
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Error when calling the metaclass bases
metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
I know how to do it manually by listing all of the attributes of base64 in a new class and calling them with the stored value as argument. But is there a neat, pythonic way to do this? Is it a bad idea to do it? The idea would be, if needed, to do it with many such modules and have "super strings" that would have as modules all the things I would need to do with them. Is that bad? Is it un-pythonic? If it is pythonic, how is it done?

I don't think creating such complex string-like classes is a good idea, but if you really want to, here's a simple snippet that runs your examples.
First, we define a class that's a generic string-wrapper. Its core is a __getattr__ function that forwards every method call to a given self.module, adding self.string as the first parameter and remembering the result on self.string.
import base64
class ModuledString(object):
def __init__(self, string):
self.string = string
def __getattr__(self, attrname):
def func(*args, **kwargs):
result = getattr(self.module, attrname)(self.string, *args, **kwargs)
self.string = result
return result
return func
def __str__(self):
return str(self.string)
Creating a string-wrapper with base64 capabilities is then easy:
class B64String(ModuledString):
module = base64
if __name__ == '__main__':
string = B64String('Foo Bar')
print string
# 'Foo Bar'
print string.b64encode()
# 'Rm9vIEJhcg=='
print string
# 'Rm9vIEJhcg=='
string.string = 'QmFyIEZvbw=='
print string
# 'QmFyIEZvbw=='
print string.b64decode()
# 'Bar Foo'
Note that the above examples work only because b64encode and b64decode take a string as the first argument and return a string as the result (there is no validation in my __getattr__ function). A random function from some random module would probably raise some kind of exception. So, after all, it would be better to restrict the usage to a predefined set of functions from a given module, but it should be easy now.
I repeat, I don't recommend using such code in any serious project, only for fun.