Try as I might, I an unable to pickle kivy widgets using either pickle or dill. I've already viewed the similar question here, but the answer given does not seem to work. I've attempted every protocol for both dill and pickle. Attempting this:
from kivy.uix.widget import Widget
import pickle
widget=Widget()
pickle.dump(widget,open('example.pkl','wb'),protocol=4)
Gives an error like so:
Traceback (most recent call last):
File "nonsense.py", line 9, in <module>
pickle.dump(widget,open('test.pkl','wb'),protocol=4)
_pickle.PicklingError: Can't pickle <class 'weakref'>: attribute lookup weakref on builtins failed
Different protocols give different errors. I've attempted to use dill as well (since it can pickle a greater number of types) and I get a different set of errors, depending on the protocol used.
For protocol 0:
TypeError: __init__() takes exactly 0 positional arguments (1 given)
for protocol 4:
_pickle.PicklingError: args[0] from __newobj_ex__ args has the wrong class
with variations on this theme for the protocols between the two.
Are widgets unable to be pickled in any way? Failing that, is it possible to save a widget to file in some other manner? It would be exceptionally difficult to save the information I need without being able to save any instance of the widget class as well.
Related
I have an object in the heap and a reference to it. There are certain circumstances in which the object gets deleted but the reference that points to its location doesn't know that. How can I check if there is real data in the heap?
For example:
from PySide import *
a = QProgressBar()
b = QProgressBar()
self.setIndexWidget(index,a)
self.setIndexWidget(index,b)
Then the a object gets deleted but print(a) returns a valid address. However if you try a.value() - runtime error occurs (C++ object already deleted).
a is None returns False.
For the PySide objects you'll need the shiboken module to perform object queries. For Pyside2, you'll need shiboken2.
import shiboken # shiboken2
print shiboken.isValid(a)
use sip module, read more about sip here
import sip
a = QProgressBar()
sip.isdeleted(a)
False
sip.delete(a)
a
<PyQt4.QtCore.QObject object at 0x017CCA98>
sip.isdeleted(a)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: underlying C/C++ object has been deleted
It is explicitly mentioned in the documentation when an object takes the responsibility for the deletion of another object. In your example, you can see this in the Qt doc :
If index widget A is replaced with index widget B, index widget A will be deleted.
Python 3.7.1, filters 1.3.2
>>> import filters
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "E:\anaconda3\lib\site-packages\filters\__init__.py", line 27, in <modu
from filters.extensions import FilterExtensionRegistry
File "E:\anaconda3\lib\site-packages\filters\extensions.py", line 11, in <mo
from class_registry import EntryPointClassRegistry
File "E:\anaconda3\lib\site-packages\class_registry\__init__.py", line 5, in
from .registry import *
File "E:\anaconda3\lib\site-packages\class_registry\registry.py", line 33, i
class BaseRegistry(with_metaclass(ABCMeta, Mapping)):
File "E:\anaconda3\lib\site-packages\six.py", line 827, in __new__
return meta(name, bases, d)
File "E:\anaconda3\lib\abc.py", line 126, in __new__
cls = super().__new__(mcls, name, bases, namespace, **kwargs)
TypeError: type() doesn't support MRO entry resolution; use types.new_class()
Looks like it is related to this and got fixed. However, I have the latest Python 3.7 and filters package. Any ideas?
Maintainer of filters and class-registry here. I apologise that it took me so long to find this!
The issue is caused by a couple of lines in the class-registry package:
class BaseRegistry(with_metaclass(ABCMeta, Mapping)):
...
class MutableRegistry(with_metaclass(ABCMeta, BaseRegistry)):
...
The error occurs because with_metaclass() creates a dynamic type internally, which conflicts when using it with a generic type like Mapping and MutableMapping.
This issue was discussed on https://bugs.python.org/issue33188 and it appears that the outcome was, "works as intended":
This is not a bug but an explicit design decision. Generic classes are static typing concept and therefore are not supposed to work freely with dynamic class creation. During discussion of PEP 560 it was decided that there should be at least one way to dynamically create generic classes, types.new_class was chosen for this, see https://www.python.org/dev/peps/pep-0560/#dynamic-class-creation-and-types-resolve-bases
Also the exception message is quite clear about this. Unfortunately, PEPs 560 and 557 were discussed in parallel so not every possible interactions where thought out. But is it critical for dataclasses to call type? I believe there should be no other differences with types.new_class. I would say the latter is even better than type because it correctly treats __prepare__ on the metaclass IIRC. So I would propose to switch from type() to types.new_class() for dynamic creation of dataclasses.
There are two possible ways to resolve this issue:
Use add_metaclass() instead of with_metaclass().
Drop support for Python 2 and replace with_metaclass() with Python-3-style base/metaclass declarations.
Both solutions are represented in pull requests submitted by stj.
I no longer work at EFL Global [now LenddoEFL], so I don't have direct access to that repo any more. It may be some time before a new version is released; in the meantime, I have forked the project and released a new version that explicitly supports Python 3.7 (and drops support for Python 2):
https://pypi.org/project/phx-filters/
I'm a minor contributor to a package where people are meant to do this (Foo.Bar.Bar is a class):
>>> from Foo.Bar import Bar
>>> s = Bar('a')
Sometimes people do this by mistake (Foo.Bar is a module):
>>> from Foo import Bar
>>> s = Bar('a')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'module' object is not callable
This might seems simple, but users still fail to debug it, I would like to make it easier. I can't change the names of Foo or Bar but I would like to add a more informative traceback like:
TypeError("'module' object is not callable, perhaps you meant to call 'Bar.Bar()'")
I read the Callable modules Q&A, and I know that I can't add a __call__ method to a module (and I don't want to wrap the whole module in a class just for this). Anyway, I don't want the module to be callable, I just want a custom traceback. Is there a clean solution for Python 3.x and 2.7+?
Add this to top of Bar.py: (Based on this question)
import sys
this_module = sys.modules[__name__]
class MyModule(sys.modules[__name__].__class__):
def __call__(self, *a, **k): # module callable
raise TypeError("'module' object is not callable, perhaps you meant to call 'Bar.Bar()'")
def __getattribute__(self, name):
return this_module.__getattribute__(name)
sys.modules[__name__] = MyModule(__name__)
# the rest of file
class Bar:
pass
Note: Tested with python3.6 & python2.7.
What you want is to change the error message when is is displayed to the user. One way to do that is to define your own excepthook.
Your own function could:
search the calling frame in the traceback object (which contains informations about the TypeError exception and the function which does that),
search the Bar object in the local variables,
alter the error message if the object is a module instead of a class or function.
In Foo.__init__.py you can install a your excepthook
import inspect
import sys
def _install_foo_excepthook():
_sys_excepthook = sys.excepthook
def _foo_excepthook(exc_type, exc_value, exc_traceback):
if exc_type is TypeError:
# -- find the last frame (source of the exception)
tb_frame = exc_traceback
while tb_frame.tb_next is not None:
tb_frame = tb_frame.tb_next
# -- search 'Bar' in the local variable
f_locals = tb_frame.tb_frame.f_locals
if 'Bar' in f_locals:
obj = f_locals['Bar']
if inspect.ismodule(obj):
# -- change the error message
exc_value.args = ("'module' object is not callable, perhaps you meant to call 'Foo.Bar.Bar()'",)
_sys_excepthook(exc_type, exc_value, exc_traceback)
sys.excepthook = _foo_excepthook
_install_foo_excepthook()
Of course, you need to enforce this algorithm…
With the following demo:
# coding: utf-8
from Foo import Bar
s = Bar('a')
You get:
Traceback (most recent call last):
File "/path/to/demo_bad.py", line 5, in <module>
s = Bar('a')
TypeError: 'module' object is not callable, perhaps you meant to call 'Foo.Bar.Bar()'
There are a lot of ways you could get a different error message, but they all have weird caveats and side effects.
Replacing the module's __class__ with a types.ModuleType subclass is probably the cleanest option, but it only works on Python 3.5+.
Besides the 3.5+ limitation, the primary weird side effects I've thought of for this option are that the module will be reported callable by the callable function, and that reloading the module will replace its class again unless you're careful to avoid such double-replacement.
Replacing the module object with a different object works on pre-3.5 Python versions, but it's very tricky to get completely right.
Submodules, reloading, global variables, any module functionality besides the custom error message... all of those are likely to break if you miss some subtle aspect of the implementation. Also, the module will be reported callable by callable, just like with the __class__ replacement.
Trying to modify the exception message after the exception is raised, for example in sys.excepthook, is possible, but there isn't a good way to tell that any particular TypeError came from trying to call your module as a function.
Probably the best you could do would be to check for a TypeError with a 'module' object is not callable message in a namespace where it looks plausible that your module would have been called - for example, if the Bar name is bound to the Foo.Bar module in either the frame's locals or globals - but that's still going to have plenty of false negatives and false positives. Also, sys.excepthook replacement isn't compatible with IPython, and whatever mechanism you use would probably conflict with something.
Right now, the problems you have are easy to understand and easy to explain. The problems you would have with any attempt to change the error message are likely to be much harder to understand and harder to explain. It's probably not a worthwhile tradeoff.
Following the suggestion here, my package (or the directory containing my modules) is located at C:/Python34/Lib/site-packages. The directory contains an __init__.py and sys.path contains a path to the directory as shown.
Still I am getting the following error:
Traceback (most recent call last):
File "C:/Python34/Lib/site-packages/toolkit/window.py", line 6, in <module>
from catalogmaker import Catalog
File "C:\Python34\Lib\site-packages\toolkit\catalogmaker.py", line 1, in <module>
from patronmaker import Patron
File "C:\Python34\Lib\site-packages\toolkit\patronmaker.py", line 4, in <module>
class Patron:
File "C:\Python34\Lib\site-packages\toolkit\patronmaker.py", line 11, in Patron
patrons = pickle.load(f)
ImportError: No module named 'Patron'
I have a class in patronmaker.py named 'Patron' but no module named Patron so I am not sure what the last statement in the error message means. I very much appreciate your thoughts on what I am missing.
Python Version 3.4.1 on a Windows 32 bits machine.
You are saving all patron instances (i.e. self) to the Patron class attribute Patron.patrons. Then you are trying to pickle a class attribute from within the class. This can choke pickle, however I believe dill should be able to handle it. Is it really necessary to save all the class instances to a list in Patrons? It's a bit of an odd thing to do…
pickle serializes classes by reference, and doesn't play well with __main__ for many objects. In dill, you don't have to serialize classes by reference, and it can handle issues with __main__, much better. Get dill here: https://github.com/uqfoundation
Edit:
I tried your code (with one minor change) and it worked.
dude#hilbert>$ python patronmaker.py
Then start python…
>>> import dill
>>> f = open('patrons.pkl', 'rb')
>>> p = dill.load(f)
>>> p
[Julius Caeser, Kunte Kinta, Norton Henrich, Mother Teresa]
The only change I made was to uncomment the lines at the end of patronmaker.py so that it saved some patrons…. and I also replaced import pickle with import dill as pickle everywhere.
So, even by downloading and running your code, I can't produce an error with dill. I'm using the latest dill from github.
Additional Edit:
Your traceback above is from an ImportError. Did you install your module? If you didn't use setup.py to install it, or if you don't have your module on your PYTHONPATH, then you won't find your module regardless of how you are serializing things.
Even more edits:
Looking at your code, you should be using the singleton pattern for patrons… it should not be inside the class Patron. The block of code at the class level to load the patrons into Patron.patrons is sure to cause problems… and probably bound to be the source of some form of errors. I also see that you are pickling the attribute Patrons.patrons (not even the class itself) from inside the Patrons class -- this is madness -- don't do it. Also notice that when you are trying to obtain the patrons, you use Patron.patrons… this is calling the class object and not an instance. Move patrons outside of the class, and use the singleton directly as a list of patrons. Also you should typically be using the patrons instance, so if you wanted to have each patron know who all the other patrons are, p = Patron('Joe', 'Blow'), then p.patrons to get all patrons… but you'd need to write a Patrons.load method that reads the singleton list of patrons… you could also use a property to make the load give you something that looks like an attribute.
If you build a singleton of patrons (as a list)… or a "registry" of patrons (as a dict) if you like, then just check if a patrons pickle file exists… to load to the registry… and don't do it from inside the Patrons class… things should go much better. Your code currently is trying to load a class instance on a class definition while it builds that class object. That's bad...
Also, don't expect people to go downloading your code and debugging it for you, when you don't present a minimal test case or sufficient info for how the traceback was created.
You may have hit on a valid pickling error in dill for some dark corner case, but I can't tell b/c I can't reproduce your error. However, I can tell that you need some refactoring.
And just to be explicit:
Move your patrons initializing mess from Patrons into a new file patrons.py
import os
import dill as pickle
#Initialize patrons with saved pickle data
if os.path.isfile('patrons.pkl'):
with open("patrons.pkl", 'rb') as f:
patrons = pickle.load(f)
else: patrons = []
Then in patronmaker.py, and everywhere else you need the singleton…
import dill as pickle
import os.path
import patrons as the
class Patron:
def __init__(self, lname, fname):
self.lname = lname.title()
self.fname = fname.title()
self.terrCheckedOutHistory = {}
#Add any created Patron to patrons list
the.patrons.append(self)
#Preserve this person via pickle
with open('patrons.pkl', 'wb') as f:
pickle.dump(the.patrons, f)
And you should be fine unless your code is hitting one of the cases that attributes on modules can't be serialized because they were added dynamically (see https://github.com/uqfoundation/dill/pull/47), which should definitely make pickle fail, and in some cases dill too… probably with an AtrributeError on the module. I just can't reproduce this… and I'm done.
I have an object in the heap and a reference to it. There are certain circumstances in which the object gets deleted but the reference that points to its location doesn't know that. How can I check if there is real data in the heap?
For example:
from PySide import *
a = QProgressBar()
b = QProgressBar()
self.setIndexWidget(index,a)
self.setIndexWidget(index,b)
Then the a object gets deleted but print(a) returns a valid address. However if you try a.value() - runtime error occurs (C++ object already deleted).
a is None returns False.
For the PySide objects you'll need the shiboken module to perform object queries. For Pyside2, you'll need shiboken2.
import shiboken # shiboken2
print shiboken.isValid(a)
use sip module, read more about sip here
import sip
a = QProgressBar()
sip.isdeleted(a)
False
sip.delete(a)
a
<PyQt4.QtCore.QObject object at 0x017CCA98>
sip.isdeleted(a)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: underlying C/C++ object has been deleted
It is explicitly mentioned in the documentation when an object takes the responsibility for the deletion of another object. In your example, you can see this in the Qt doc :
If index widget A is replaced with index widget B, index widget A will be deleted.