Can anyone tell me why my code and function serialization handlers are not working below? The copyreg module is fairly unfamiliar to me, and it is not clear if the code below is written properly.
>>> import pickle, copyreg, types, marshal
>>> def average(*args):
return sum(args) / len(args)
>>> average_dump = pickle.dumps(average)
>>> del average
>>> average = pickle.loads(average_dump)
Traceback (most recent call last):
File "<pyshell#31>", line 1, in <module>
average = pickle.loads(average_dump)
AttributeError: 'module' object has no attribute 'average'
>>> copyreg.pickle(types.CodeType,
lambda code: (marshal.loads, (marshal.dumps(code),)),
marshal.loads)
>>> up = lambda co, ns, de, cl: types.FunctionType(co, globals(), na, de, cl)
>>> copyreg.pickle(types.FunctionType,
lambda function: (up, (function.__code__,
function.__name__,
function.__defaults__,
function.__closure__)),
up)
>>> def average(*args):
return sum(args) / len(args)
>>> average_dump
b'\x80\x03c__main__\naverage\nq\x00.'
>>> pickle.dumps(average)
b'\x80\x03c__main__\naverage\nq\x00.'
>>> del average; average = pickle.loads(average_dump)
Traceback (most recent call last):
File "<pyshell#39>", line 1, in <module>
del average; average = pickle.loads(average_dump)
AttributeError: 'module' object has no attribute 'average'
My expectation is that if the registered functions were working properly, then both code and function objects would be serialized. If that worked as expected, unpickling functions would also be possible.
Edit: subclassing Pickler as suggested in this answer. Does not seem to help either. The function from the example is still being serialized by name instead of the handlers from the copyreg module.
>>> import pickle, copyreg, types, marshal
>>> copyreg.pickle(types.CodeType,
lambda code: (marshal.loads, (marshal.dumps(code),)),
marshal.loads)
>>> up = lambda co, ns, de, cl: types.FunctionType(co, globals(), na, de, cl)
>>> copyreg.pickle(types.FunctionType,
lambda function: (up, (function.__code__,
function.__name__,
function.__defaults__,
function.__closure__)),
up)
>>> class MyPickler(pickle.Pickler):
def __init__(self, *args):
super().__init__(*args)
self.dispatch_table = copyreg.dispatch_table
>>> def average(*args):
return sum(args) / len(args)
>>> x = io.BytesIO(); y = MyPickler(x)
>>> y.dump(average)
>>> x.getvalue()
b'\x80\x03c__main__\naverage\nq\x00.'
If you want to serialize functions, please run the following command:
pip install dill
Once done, you can import dill and use it in place of the pickle module.
If you are running Python 3 and want easy access to pip, put a batch file in your Windows directory named pip.bat and put the following line in it (assumes you have a proxy that interferes with SSL):
py -3 -m pip %* --trusted-host pypi.org --trusted-host files.pythonhosted.org
Function's can't be pickled by-value:
Note that functions (built-in and user-defined) are pickled by “fully qualified” name reference, not by value. This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.
(http://docs.python.org/2/library/pickle.html#what-can-be-pickled-and-unpickled)
Related
I am attempting to do the following:
def test_fn(self):
cool_dict = {}
blah = Mock(spec=DictField, wraps=cool_dict)
blah['key'] = 'val'
print(cool_dict))
return False
Basically, I want to ensure that anything which happens to blah is allowed for a DictField, but I want anything that happens to blah to actually happen to cool_dict, so I can see assert that it has a certain state.
How can I do this? The above code fails:
FAILED (errors=1)
Error
Traceback (most recent call last):
File "C:\Program Files\Python 3.5\Lib\unittest\case.py", line 59, in testPartExecutor
yield
File "C:\Program Files\Python 3.5\Lib\unittest\case.py", line 605, in run
testMethod()
File "C:\Users\danie01.AD\PycharmProjects\component\component\Default\Functional\pam_team_management\test\test_team_create.py", line 23, in test_populate_info_page
blah['key'] = 'val'
TypeError: 'Mock' object does not support item assignment
I also tried it with a MagicMock:
def test_populate_info_page(self):
cool_dict = {}
blah = MagicMock(spec=DictField, wraps=cool_dict)
blah['key'] = 'val'
print(cool_dict)
return False
Which didn't fail, but cool_dict was still {}
In Python, you can also make use of magic methods which are the built-in methods on classes. For dict objects, you would want to override the __getitem__() and __setitem__() magic methods.
As an example, when you make the statement blah['key'] = 'var', it's actually calling blah.__setitem__('key', 'var'). So you'll want to mock those two magic methods, and then check on the status of those mocked methods.
Here's an example of how I might try what you're talking about:
>>> cool_dict = {'key': 'val'}
>>> blah = Mock()
>>> blah.__getitem__ = Mock()
>>> blah.__getitem__.side_effect = cool_dict.__getitem__
>>> blah['key']
'val'
>>> blah.__getitem__.assert_called() # No assertion raised, which means that it was called
>>>
So in this example, the __getitem__() method of the 'blah' object is the thing you're going to be using a Mock() to mock, and then you create a side effect: it triggers the same __getitem__() function on the cool_dict. After it's been called, you can inspect that Mock afterward to see whether it was called and what it was called with. See Mocking Magic Methods for more context.
The package joblib has a function delayed which captures the arguments passed to the function. It can be used for example like this:
from joblib import delayed
def f(n):
return n**2
delayed(f)(2)
# (<function f at 0x7f939eb3fe60>, (2,), {})
h = delayed(f)
h(2)
# (<function f at 0x7f939eb3fe60>, (2,), {})
The latter usage makes me think this could be used like any other decorator. This is supported by its documentation:
joblib.delayed(function, check_pickle=True)
Decorator used to capture the arguments of a function.
However, this fails:
from joblib import delayed
#delayed
def g(n):
return n**2
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# File "/usr/lib/python2.7/dist-packages/joblib/parallel.py", line 158, in delayed
# pickle.dumps(function)
# File "/usr/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
# raise TypeError, "can't pickle %s objects" % base.__name__
#TypeError: can't pickle function objects
I tried this with Python 2.7.12
It looks like it's a namespace issue, which causes joblib.delayed to attempt to pickle the output function (instead of the original function).
I haven't looked into the joblib code, but the thing you proposed would be the way to do it:
import joblib
def _func(*args, **kwargs):
'your code here'
func = joblib.delayed(_func)
It's not ideal, but it works.
Can I dynamically add attributes to instances of a new-style class (one that derives from object)?
Details:
I'm working with an instance of sqlite3.Connection. Simply extending the class isn't an option because I don't get the instance by calling a constructor; I get it by calling sqlite3.connect().
Building a wrapper doesn't save me much of the bulk for the code I'm writing.
Python 2.7.1
Edit
Right answers all. But I still am not reaching my goal; instances of sqlite3.Connection bar my attempts to set attributes in the following ways (as do instances of object itself). I always get an AttributeError:
> conn = sqlite3.connect([filepath])
> conn.a = 'foo'
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
conn.a = 'foo'
AttributeError: 'object' object has no attribute 'a'
> conn.__setattr__('a','foo')
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
conn.__setattr__('a','foo')
AttributeError: 'object' object has no attribute 'a'
Help?
Yes, unless the class is using __slots__ or preventing attribute writing by overriding __setattr__, or an internal Python class, or a Python class implemented natively (usually in C).
You can always try setting an attribute. Except for seriously weird __setattr__ implementations, assigning an attribute to an instance of a class of one of the types mentioned above should raise an AttributeError.
In these cases, you'll have to use a wrapper, like this:
class AttrWrapper(object):
def __init__(self, wrapped):
self._wrapped = wrapped
def __getattr__(self, n):
return getattr(self._wrapped, n)
conn = AttrWrapper(sqlite3.connect(filepath))
Simple experimentation:
In []: class Tst(object): pass
..:
In []: t= Tst()
In []: t.attr= 'is this valid?'
In []: t.attr
Out[]: 'is this valid?'
So, indeed it seems to be possible to do that.
Update:
But from the documentation: SQLite is a C library that ..., so it seems that you really need to wrap it.
conn.a = 'foo',
or any dynamic assignment is valid, if conn is
<type 'classobj'>.
Things like:
c=object()
c.e=1
will raise an Attribute error. On the otherhand: Python allows you to do fantastic Metaclass programming:
>>>from new import classobj
>>>Foo2 = classobj('Foo2',(Foo,),{'bar':lambda self:'bar'})
>>>Foo2().bar()
>>>'bar'
>>>Foo2().say_foo()
>>>foo
I'm writing some unit tests for a Python library and would like certain warnings to be raised as exceptions, which I can easily do with the simplefilter function. However, for one test I'd like to disable the warning, run the test, then re-enable the warning.
I'm using Python 2.6, so I'm supposed to be able to do that with the catch_warnings context manager, but it doesn't seem to work for me. Even failing that, I should also be able to call resetwarnings and then re-set my filter.
Here's a simple example which illustrates the problem:
>>> import warnings
>>> warnings.simplefilter("error", UserWarning)
>>>
>>> def f():
... warnings.warn("Boo!", UserWarning)
...
>>>
>>> f() # raises UserWarning as an exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in f
UserWarning: Boo!
>>>
>>> f() # still raises the exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in f
UserWarning: Boo!
>>>
>>> with warnings.catch_warnings():
... warnings.simplefilter("ignore")
... f() # no warning is raised or printed
...
>>>
>>> f() # this should raise the warning as an exception, but doesn't
>>>
>>> warnings.resetwarnings()
>>> warnings.simplefilter("error", UserWarning)
>>>
>>> f() # even after resetting, I'm still getting nothing
>>>
Can someone explain how I can accomplish this?
EDIT: Apparently this is a known bug: http://bugs.python.org/issue4180
Reading through the docs and few times and poking around the source and shell I think I've figured it out. The docs could probably improve to make clearer what the behavior is.
The warnings module keeps a registry at __warningsregistry__ to keep track of which warnings have been shown. If a warning (message) is not listed in the registry before the 'error' filter is set, any calls to warn() will not result in the message being added to the registry. Also, the warning registry does not appear to be created until the first call to warn:
>>> import warnings
>>> __warningregistry__
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
NameError: name '__warningregistry__' is not defined
>>> warnings.simplefilter('error')
>>> __warningregistry__
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
NameError: name '__warningregistry__' is not defined
>>> warnings.warn('asdf')
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
UserWarning: asdf
>>> __warningregistry__
{}
Now if we ignore warnings, they will get added to the warnings registry:
>>> warnings.simplefilter("ignore")
>>> warnings.warn('asdf')
>>> __warningregistry__
{('asdf', <type 'exceptions.UserWarning'>, 1): True}
>>> warnings.simplefilter("error")
>>> warnings.warn('asdf')
>>> warnings.warn('qwerty')
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
UserWarning: qwerty
So the error filter will only apply to warnings that aren't already in the warnings registry. To make your code work you'll need to clear the appropriate entries out of the warnings registry when you're done with the context manager (or in general any time after you've used the ignore filter and want a prev. used message to be picked up the error filter). Seems a bit unintuitive...
Brian Luft is correct about __warningregistry__ being the cause of the problem. But I wanted to clarify one thing: the way the warnings module appears to work is that it sets module.__warningregistry__ for each module where warn() is called. Complicating things even more, the stacklevel option to warnings causes the attribute to be set for the module the warning was issued "in the name of", not necessarily the one where warn() was called... and that's dependent on the call stack at the time the warning was issued.
This means you may have a lot of different modules where the __warningregistry__ attribute is present, and depending on your application, they may all need clearing before you'll see the warnings again. I've been relying on the following snippet of code to accomplish this... it clears the warnings registry for all modules whose name matches the regexp (which defaults to everything):
def reset_warning_registry(pattern=".*"):
"clear warning registry for all match modules"
import re
import sys
key = "__warningregistry__"
for mod in sys.modules.values():
if hasattr(mod, key) and re.match(pattern, mod.__name__):
getattr(mod, key).clear()
Update: CPython issue 21724 addresses issue that resetwarnings() doesn't clear warning state. I attached an expanded "context manager" version to this issue, it can be downloaded from reset_warning_registry.py.
Brian is spot on about the __warningregistry__. So you need to extend catch_warnings to save/restore the global __warningregistry__ too
Something like this may work
class catch_warnings_plus(warnings.catch_warnings):
def __enter__(self):
super(catch_warnings_plus,self).__enter__()
self._warningregistry=dict(globals.get('__warningregistry__',{}))
def __exit__(self, *exc_info):
super(catch_warnings_plus,self).__exit__(*exc_info)
__warningregistry__.clear()
__warningregistry__.update(self._warningregistry)
Following on from Eli Collins' helpful clarification, here is a modified version of the catch_warnings context manager that clears the warnings registry in a given sequence of modules when entering the context manager, and restores the registry on exit:
from warnings import catch_warnings
class catch_warn_reset(catch_warnings):
""" Version of ``catch_warnings`` class that resets warning registry
"""
def __init__(self, *args, **kwargs):
self.modules = kwargs.pop('modules', [])
self._warnreg_copies = {}
super(catch_warn_reset, self).__init__(*args, **kwargs)
def __enter__(self):
for mod in self.modules:
if hasattr(mod, '__warningregistry__'):
mod_reg = mod.__warningregistry__
self._warnreg_copies[mod] = mod_reg.copy()
mod_reg.clear()
return super(catch_warn_reset, self).__enter__()
def __exit__(self, *exc_info):
super(catch_warn_reset, self).__exit__(*exc_info)
for mod in self.modules:
if hasattr(mod, '__warningregistry__'):
mod.__warningregistry__.clear()
if mod in self._warnreg_copies:
mod.__warningregistry__.update(self._warnreg_copies[mod])
Use with something like:
import my_module_raising_warnings
with catch_warn_reset(modules=[my_module_raising_warnings]):
# Whatever you'd normally do inside ``catch_warnings``
I've run into the same issues, and while all of the other answers are valid I choose a different route. I don't want to test the warnings module, nor know about it's inner workings. So I just mocked it instead:
import warnings
import unittest
from unittest.mock import patch
from unittest.mock import call
class WarningTest(unittest.TestCase):
#patch('warnings.warn')
def test_warnings(self, fake_warn):
warn_once()
warn_twice()
fake_warn.assert_has_calls(
[call("You've been warned."),
call("This is your second warning.")])
def warn_once():
warnings.warn("You've been warned.")
def warn_twice():
warnings.warn("This is your second warning.")
if __name__ == '__main__':
__main__=unittest.main()
This code is Python 3, for 2.6 you need the use an external mocking library as unittest.mock was only added in 2.7.
Does Python have extension methods like C#? Is it possible to call a method like:
MyRandomMethod()
on existing types like int?
myInt.MyRandomMethod()
You can add whatever methods you like on class objects defined in Python code (AKA monkey patching):
>>> class A(object):
>>> pass
>>> def stuff(self):
>>> print self
>>> A.test = stuff
>>> A().test()
This does not work on builtin types, because their __dict__ is not writable (it's a dictproxy).
So no, there is no "real" extension method mechanism in Python.
It can be done with Forbidden Fruit (https://pypi.python.org/pypi/forbiddenfruit)
Install forbiddenfruit:
pip install forbiddenfruit
Then you can extend built-in types:
>>> from forbiddenfruit import curse
>>> def percent(self, delta):
... return self * (1 + delta / 100)
>>> curse(float, 'percent', percent)
>>> 1.0.percent(5)
1.05
Forbidden Fruit is fundamentally dependent on the C API, it works only on cpython implementations and won’t work on other python implementations, such as Jython, pypy, etc.
not sure if that what you're asking but you can extend existing types and then call whatever you like on the new thing:
class int(int):
def random_method(self):
return 4 # guaranteed to be random
v = int(5) # you'll have to instantiate all you variables like this
v.random_method()
class int(int):
def xkcd(self):
import antigravity
print(42)
>>>v.xkcd()
Traceback (most recent call last):
File "<pyshell#81>", line 1, in <module>
v.xkcd()
AttributeError: 'int' object has no attribute 'xkcd'
c = int(1)
>>> c.random_method()
4
>>> c.xkcd()
42
hope that clarifies your question
The following context manager adds the method like Forbidden Fruit would without the limitations of it. Besides that it has the additional benefit of removing the extension method afterwards:
class extension_method:
def __init__(self, obj, method):
method_name = method.__name__
setattr(obj, method_name, method)
self.obj = obj
self.method_name = method_name
def __enter__(self):
return self.obj
def __exit__(self, type, value, traceback):
# remove this if you want to keep the extension method after context exit
delattr(self.obj, self.method_name)
Usage is as follows:
class C:
pass
def get_class_name(self):
return self.__class__.__name__
with extension_method(C, get_class_name):
assert hasattr(C, 'get_class_name') # the method is added to C
c = C()
print(c.get_class_name()) # prints 'C'
assert not hasattr(C, 'get_class_name') # the method is gone from C
I've had great luck with the method described here:
http://mail.python.org/pipermail/python-dev/2008-January/076194.html
I have no idea if it works on builtins though.
Another option is to override the meta-class. This allows you to, among other things, specify functions that should exist in all classes.
This article starts to discuss it:
http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html