I'm trying to put some sites i crawled into a shelve, but the shelve won't accept any Site-objects. It will accept lists, strings, tuples, what have you, but as soon as i put in a Site-object, it crashes when i try to get the contents of the shelve
So when i fill up my shelve like this:
def add_to_shelve(self, site):
db = shelve.open("database")
print site, site.url
for word in site.content:
db[word] = site.url #site.url is a string, word has to be one too
shelve.open("database")['whatever'] works perfectly.
But if I do this:
def add_to_shelve(self, site):
db = shelve.open("database")
print site, site.url
for word in site.content:
db[word] = site #site is now an object of Site
shelve.open("database")['whatever'] errors out with this error message:
AttributeError: 'module' object has no attribute 'Site'
I'm completely stumped, and the pythondocs, strangely, don't have much info either. All they say is that the key in a shelve has to be a string, but the value or data can be "an arbitrary object"
It looks like you refactored your code after saving objects in the shelve. When retrieving objects from the shelve, Python rebuilds the object, and it needs to find the original class that, presumably, you have moved. This problem is typical when working with pickle (as the shelve module does).
The solution, as pduel suggests, is to provide a backwards-compatibility reference to the class in the same location that it used to be, so that pickle can find it. If you re-save all the objects, thereby rebuilding the pickles, you can remove that backwards-comatibility referece.
It seems that Python is looking for a constructor for a 'Site' object, and not finding it. I have not used shelve, but I recall the rules for what can be pickled are byzantine, and suspect the shelve rules are similar.
Try adding the line:
Site = sitemodule.Site
(with the name of the module providing 'Site') before you try unshelving. This ensures that a Site class can be found.
Related
I am working with this object SharedMemoryDisplay in the above script and I want to return/retrieve self.camera_container, where self.camera_container = {camera_id: (camera_id, frame, frame_properties)}
I tried to create a method to return the this but it gives two errors
prop, camera_container = monitor_memory.get_frame()
TypeError: 'NoneType' object is not iterable
AttributeError: Can't pickle local object 'SharedMemoryDisplay.__init__.<locals>.<lambda>'
I am only able to able to get self.camera_container[key] if I just do which is fine but I want to get self.camera_container also.
return self.camera_container[key]
In the below script is where, I am using this object to display in a cv2 named window, my ultimate motive is to retrieve frames of all the cameras seperately what it currently does is joins all the camera frames and returns that via self.display_frame which is added to webdisplay_memory in the below script (for displaying in the html) that's why I created a method to retrieve dictionary camera_container.
webdisplay_memory.add_frame(0, self.display_frame, None)
rather than messing with this variable I was thinking of creating a method that returns self.camera_container than using this to get frames of each camera seperately.
how can I overcome this, kindly help if you have better and efficient solutions!
multiprocessing uses pickle under the hood. pickle can serialize only a certain set of objects. Specifically, it can not serialize defaultdict (a type for your camera_container). So, either use a normal dict and replace all the lookups to self.camera_container.get(key, None), or look into this question and try to use pathos.multiprocessing with dill. The latter approach is not tested by me, though.
I've tried to encode a set object and it failed to do it, claiming an AttributeError code.
Is there a way for it to work?
NOTE: I've been using the socket module.
albums = set()
for key, val in data.items():
albums.add(val['album'])
msg = albums.encode()
Are you trying to send some form of representation of albums via a socket? Then you need a binary representation of that object first. Using .encode() suggests that you want the string-representation of albumns, which you can get using msg = repr(albums).encode().
There are a few options here, but probably the easiest is to encode your dictionary using the Python pickle protocol.
To do that, use sending code like this:
import pickle
data_to_send = pickle.dumps(albums)
mysocket.send(data_to_send)
and receiving code like this:
import pickle
albums = pickle.loads(data_received_from_socket)
However, I want to warn you: using the socket module at all is a can of worms. It is low level and meant for experienced programmers. Compared to the question you've just asked, the other problems you'll have to deal with are probably a lot harder. For example, you have to think about how to delimit your messages before you figure out what to pass as the second parameter to socket.recv (the bufsize parameter).
I suggest you try something higher-level like Python's xmlrpc modules.
I have a queryset that I need to pickle lazily and I am having some serious troubles. cPickle.dumps(queryset.query) throws the following error:
Can't pickle <class 'myproject.myapp.models.myfile.QuerySet'>: it's not the same object as myproject.myapp.models.myfile.QuerySet
Strangely (or perhaps not so strangely), I only get that error when I call cPcikle from another method or a view, but not when I call it from the command line.
I made the method below after reading PicklingError: Can't pickle <class 'decimal.Decimal'>: it's not the same object as decimal.Decimal and Django mod_wsgi PicklingError while saving object:
def dump_queryset(queryset, model):
from segment.segmentengine.models.segment import QuerySet
memo = {}
new_queryset = deepcopy(queryset, memo)
memo = {}
new_query = deepcopy(new_queryset.query, memo)
queryset = QuerySet(model=model, query=new_query)
return cPickle.dumps(queryset.query)
As you can see, I am getting extremely desperate -- that method still yields the same error. Is there a known, non-hacky solution to this problem?
EDIT: Tried using --noreload running on the django development server, but to no avail.
EDIT2: I had a typo in the error I displayed above -- it was models.QuerySet, not models.mymodel.QuerySet that it was complaining about. There is another nuance here, which is that my models file is broken out into multiple modules, so the error is ACTUALLY:
Can't pickle <class 'myproject.myapp.models.myfile.QuerySet'>: it's not the same object as myproject.myapp.models.myfile.QuerySet
Where myfile is one of the modules under models. I have an __ini__.py in models with the following line:
from myfile import *
I wonder if this is contributing to my issue. Is there some way to change my init to protect myself against this? Are there any other tests to try?
EDIT3: Here is a little more background on my use case: I have a model called Context that I use to populate a UI element with instances of mymodel. The user can add/remove/manipulate the objects on the UI side, changing their context, and when they return, they can keep their changes, because the context serialized everything. A context has a generic foreign key to different types of filters/ways the user can manipulate the object, all of which must implement a few methods that the context uses to figure out what it should display. One such filter takes a queryset that can be passed in and displays all of the objects in that queryset. This provides a way to pass in arbitrary querysets that are produced elsewhere and have them displayed in the UI element. The model that uses the Context is hierarchical (using mptt for this), and the UI element makes a request to get children each time the user clicks around, we can then take the children and determine if they should be displayed based on whether or not they are included in the Context. Hope that helps!
EDIT4: I am able to dump an empty queryset, but as soon as I add anything of value, it fails.
EDIT4: I am on Django 1.2.3
This may not be the case for everyone, but I was using Ipython notebook and having a similar issue pickling my own class.
The problem turned out to be from a reload call
from dir.my_module import my_class
reload(dir.my_module)
Removing the reload call and then re-running the import and the cell where the instance of that object was created then allowed it to be pickled.
not so elegant but perhaps it works:
add the directory of the myfile -module to os.sys.path and use only import myfile in each module where you use myfile. (remove any from segment.segmentengine.models.segment import, anywhere in your project)
According to this doc, pickling a QuerySet should be not a problem. Thus, the problem should come from other place.
Since you mentined:
EDIT2: I had a typo in the error I displayed above -- it was models.QuerySet, not models.mymodel.QuerySet that it was complaining about. There is another nuance here, which is that my models file is broken out into multiple modules, so the error is ACTUALLY:
The second error message you provided look like the same as previous one, is that what you mean?
The error message you provided looks weird. Since you are pickling "queryset.query", the error should related to the django.db.models.sql.Query class instead of the QuerySet class.
Some modules or classes may have the same name. They will override each other then cause this kind of issue. To make thing easier, I will recommend you to use "import ooo.xxx" instead of "from ooo import *".
Your could also try
import ooo.xxx as othername
I have an object with an __init__ procedure that requires at least one parameter and
I want to store in the cache.
When trying to getting the object from the cache I get an error that the I didn't pass enough parameters to the ___init___ method.
Someone told me I need to pickle the object before sending it to the cache but all the examples I saw were using .dat files and on appengine you cannot use any file system.
You can use pickle without any filesystem, using pickle.loads / pickle.dumps. For example:
import pickle
obj = YourClass(yourparam=...)
data = pickle.dumps(obj)
# and now, store "data" into the cache
# later, get "data" from the cache
obj = pickle.loads(data)
# and tada, obj if the same as before :)
I think you are trying to use memcache in appengine. This blog will help you a lot
http://blog.notdot.net/2009/9/Efficient-model-memcaching
I'm currently learning about pickling in Python and I'm confused with an error I'm getting complaing that the unpickled variable I'm initalizing does not have a particular attribute I'm wanting to use.
I'm pickling a dictionary of the script's data and then trying to unpickle.load it back using the following code:
def loadData():
global script_data_filepath
with open(script_data_filepath) as script_data_file:
data_to_load = pickle.load(script_data_file)
for data_item in data_to_load.items():
print(data_item[0])
print(data_item[1])
The problem is that Python is saying that items() is not an attribute of data_to_load because data_to_load is of type 'str'. In the code I've given this is the first time that data_to_load is declared, I pressumed that it would dynamically take whatever type is assigned to it (which should be a dictionary as that's what I know will be loaded from this file).
That should work. The code you posted is fine. How are you dumping it? You're using pickle.dump, not pickle.dumps for a string? Your problem is almost certainly in the dumping code.
>>> import cPickle
>>> foo = {4:2}
>>> cPickle.dump(foo, open('foo.pickle', 'wb'))
>>> data_to_load = cPickle.load(open('foo.pickle'))
>>> data_to_load.items()
[(4, 2)]