For the class I'm writing I want to implement methods to dump the data to a shelve file and also a classmethod that would create a new instance based on the values of the instance loaded from file. So for example if I have a class like that:
import shelve
class Foo:
def __init__(self, a):
self.a = a
def save(self, fname):
with shelve.open(fname, flag='c') as shlv:
shlv['data'] = self
At the moment I'm using an external function to load the shelve file and return a saved instance:
def load_data(filename) -> Foo:
with shelve.open(filename) as shlv:
return shlv['data']
And this works fine.
I was just trying to think of how I can implement a class method that would load the data and create a new instance.
I'm pretty sure that what I'm thinking is an ugly solution but I'm still interested out of pure curiosity.
My original idea was to do something like:
class Foo:
def __init__(self, a):
self.a = a
def save(self, fname):
with shelve.open(fname, flag='c') as shlv:
shlv['data'] = self
#classmethod
def from_saved_data(cls, data):
cls.__dict__ = data.__dict__
return cls
This doesn't work because I was trying to do dumb things to the metaclass and got this error:
AttributeError: attribute '__dict__' of 'type' objects is not writable
What are the other ways I can do this?
Related
I have a class that used to have a field data, but later the class was changed and now data is a property.
I would like to be able to unpickle instances that were pickled before the change, to preserve backward compatibility. A minimal example for illustration (in Python 2, though it should be the same in Python 3):
import pickle
class X(object):
def __init__(self):
self.data = 100
pickle.dump(X(), open("x-file",'w'))
# Redefine the class
class X(object):
def __init__(self):
self._data = 101
#property
def data(self):
return self._data
y = pickle.load(open("x-file")) # cannot access the original data through y
print(y.data)
What I would like is to define a function load that unpickles the object, detects it is old style (e.g. by seeing that it doesn't have the _data field), and return a new style instance with its data instead. However, since the field data now is a property, the old data field was overwritten by the class definition.
Is there any easy way I can access the old data (i.e. other than parsing the pickle file myself, for example)?
EDIT
After Peter Wood's answer, I got to this solution:
import pickle
class X(object):
def __init__(self):
self.data = 100
pickle.dump(X(), open("x-file",'w'))
# Redefine the class
class X(object):
def __init__(self):
self._data = 101
#property
def data(self):
return self._data
def __setstate__(self, state):
if not '_data' in state:
self._data = state['data']
del state['data']
self.__dict__.update(state)
y = pickle.load(open("x-file")) # cannot access the original data through y
print(y.data)
The documentation says in What can be pickled and unpickled?:
If you plan to have long-lived objects that will see many versions of
a class, it may be worthwhile to put a version number in the objects
so that suitable conversions can be made by the class’s __setstate__()
method.
This is one of four "magic" methods you can define on your classes to facilitate serialisation. See Pickling Class Instances.
I have a class MyClass with a complex __init__ function.
This class had a method my_method(self) which I would like to test.
my_method only needs attribute my_attribute from the class instance.
Is there a way I can mock class instances without calling __init__ and by setting the attributes of each class instance instead?
What I have:
# my_class.py
from utils import do_something
class MyClass(object):
def __init__(self, *args, **kwargs):
# complicated function which I would like to bypass when initiating a mocked instance class
pass
def my_method(self):
return do_something(self.my_attribute)
What I tried
#mock.patch("my_class.MyClass")
def test_my_method(class_mock, attribute):
instance = class_mock.return_value
instance.my_attribute = attribute
example_instance = my_class.MyClass()
out_my_method = example_instance.my_method()
# then perform some assertions on `out_my_method`
however this still makes usage of __init__ which I hope we can by-pass or mock.
As I mentioned in the comments, one way to test a single method without having to create an instance is:
MyClass.my_method(any_object_with_my_attribute)
The problem with this, as with both options in quamrana's answer, is that we have now expanded the scope of any future change just because of the tests. If a change to my_method requires access to an additional attribute, we now have to change both the implementation and something else (the SuperClass, the MockMyClass, or in this case any_object_with_my_attribute_and_another_one).
Let's have a more concrete example:
import json
class MyClass:
def __init__(self, filename):
with open(filename) as f:
data = json.load(f)
self.foo = data.foo
self.bar = data.bar
self.baz = data.baz
def my_method(self):
return self.foo ** 2
Here any test that requires an instance of MyClass. is painful because of the file access in __init__. A more testable implementation would split apart the detail of how the data is accessed and the initialisation of a valid instance:
class MyClass:
def __init__(self, foo, bar, baz):
self.foo = foo
self.bar = bar
self.baz = baz
def my_method(self):
return self.foo ** 2
#classmethod
def from_json(cls, filename):
with open(filename) as f:
data = json.load(f)
return cls(data.foo, data.bar, data.baz)
You have to refactor MyClass("path/to/file") to MyClass.from_json("path/to/file"), but wherever you already have the data (e.g. in your tests) you can use e.g. MyClass(1, 2, 3) to create the instance without requiring a file (you only need to consider the file in the tests of from_json itself). This makes it clearer what the instance actually needs, and allows the introduction of other ways to construct an instance without changing the interface.
There are at least two options I can see:
Extract a super class:
class SuperClass:
def __init__(self, attribute):
self.my_attribute = attribute
def my_method(self):
return do_something(self.my_attribute)
class MyClass(SuperClass):
def __init__(self, *args, **kwargs):
super().__init__(attribute) # I don't know where attribute comes from
# complicated function which I would like to bypass when initiating a mocked instance class
Your tests can instantiate SuperClass and call my_method().
Inherit from MyClass as is and make your own simple __init__():
class MockMyClass(MyClass):
def __init__(self, attribute):
self.my_attribute = attribute
Now your test code can instantiate MockMyClass with the required attribute and call my_method()
For instance, you could write the test as follows
def test_my_method(attribute):
class MockMyClass(MyClass):
def __init__(self, attribute):
self.my_attribute = attribute
out_my_method = MockMyClass(attribute).my_method()
# perform assertions on out_my_method
I have a Python project where most business logic is in class methods. Now I would like to reuse some of the class methods in an independent project.
Is it possible to write a class method that 'exports' other class methods to a new Python file to create a script with a bunch of exported functions?
class MyObject:
def __init__(self, value):
self.value = value
def method1(self):
# the method I want to use in another project
def method2(self):
...
def method3(self):
...
def export_method(self, target_file):
# export the code of method1 to a new python file
When I run export_method('myfile.py') I would like to create a Python file that contains method1 as a function:
def method1():
...
Note: I understand that the software should be restructured and method1 should be in another module where it can be imported from other projects. I'm just curious if there is a simple way to access the code of a Python program from the code itself.
Use inspect:
Either directly:
import inspect
lines = inspect.getsource(MyObject.method1)
with open(target_file, 'w') as file:
file.write(lines)
Or if you prefer to get it as a class method and print all methods in the class:
import inspect
class MyObject:
def __init__(self, value):
self.value = value
def method1(self):
pass
def method2(self):
pass
def method3(self):
pass
#classmethod
def export_method(cls, target_file):
# export the code of method1 to a new python file
methods = inspect.getmembers(cls, predicate=inspect.ismethod)
with open(target_file, 'w') as f:
for method in methods:
lines = inspect.getsource(method[1])
f.write(lines)
Because of the #classmethod decorator the following is allowed:
MyObject.export_method('code.txt')
I have a python pickle file and when I try to load it, I would like it to load one specific object before another (because I have some dependencies...).
Is that possible?
Here is the scenario:
class A:
def __init__(self):
self.known_names = ["Dan", "David"]
def __getattr__(self, name):
if name not in self.known_names:
raise UnknownName
else:
return self[name]
class B:
def __init__(self):
self.a_instance = A()
def __setattr__(self, name, value):
if self.a_instance.attr == "something":
do_something...
def __setstate__(self):
self.foo = "blah"
The problem occurs when loading the pickle file. class B instance is loaded beofe class A. In that scenario, class B's __setstate__ method tries to set self.foo. This results __settattr__ method call which checks self.a_instance's attr attribute. However, class A was not unpickled yet so self.known_names does not exist. So calling the __getattr__ of A's results in an infinite recursion (since known_names does not exist, it calls __getattr__ on it as well).
This is not a dependency problem per se since normally __init__ will not be called when unpickling an object.
As described in the Python pickle docs and this ticket your problem is most likely that A cannot be safely unpickled due to the use of __getattr__.
Implementing __getstate__ and __setstate__ in A might be sufficient:
def __getstate__(self):
"""Extract state to pickle."""
return self.__dict__
def __setstate__(self, d):
"""Restore from pickled state."""
self.__dict__.update(d)
I've been trying to pickle an object which contains references to static class methods.
Pickle fails (for example on module.MyClass.foo) stating it cannot be pickled, as module.foo does not exist.
I have come up with the following solution, using a wrapper object to locate the function upon invocation, saving the container class and function name:
class PicklableStaticMethod(object):
"""Picklable version of a static method.
Typical usage:
class MyClass:
#staticmethod
def doit():
print "done"
# This cannot be pickled:
non_picklable = MyClass.doit
# This can be pickled:
picklable = PicklableStaticMethod(MyClass.doit, MyClass)
"""
def __init__(self, func, parent_class):
self.func_name = func.func_name
self.parent_class = parent_class
def __call__(self, *args, **kwargs):
func = getattr(self.parent_class, self.func_name)
return func(*args, **kwargs)
I am wondering though, is there a better - more standard way - to pickle such an object?
I do not want to make changes to the global pickle process (using copy_reg for example), but the following pattern would be great:
class MyClass(object):
#picklable_staticmethod
def foo():
print "done."
My attempts at this were unsuccessful, specifically because I could not extract the owner class from the foo function. I was even willing to settle for explicit specification (such as #picklable_staticmethod(MyClass)) but I don't know of any way to refer to the MyClass class right where it's being defined.
Any ideas would be great!
Yonatan
This seems to work.
class PickleableStaticMethod(object):
def __init__(self, fn, cls=None):
self.cls = cls
self.fn = fn
def __call__(self, *args, **kwargs):
return self.fn(*args, **kwargs)
def __get__(self, obj, cls):
return PickleableStaticMethod(self.fn, cls)
def __getstate__(self):
return (self.cls, self.fn.__name__)
def __setstate__(self, state):
self.cls, name = state
self.fn = getattr(self.cls, name).fn
The trick is to snag the class when the static method is gotten from it.
Alternatives: You could use metaclassing to give all your static methods a .__parentclass__ attribute. Then you could subclass Pickler and give each subclass instance its own .dispatch table which you can then modify without affecting the global dispatch table (Pickler.dispatch). Pickling, unpickling, and calling the method might then be a little faster.
EDIT: modified after Jason comment.
I think python is correct in not letting pickling a staticmethod object - as it is impossible to pickle instance or class methods! Such an object would make very little sense outside of its context:
Check this: Descriptor Tutorial
import pickle
def dosomething(a, b):
print a, b
class MyClass(object):
dosomething = staticmethod(dosomething)
o = MyClass()
pickled = pickle.dumps(dosomething)
This works, and that's what should be done - define a function, pickle it, and use such function as a staticmethod in a certain class.
If you've got an use case for your need, please write it down and I'll be glad to discuss it.