Develop Reference

AttributeError: Can't pickle local object 'SharedMemoryDisplay.__init__.<locals>.<lambda>'

I am working with this object SharedMemoryDisplay in the above script and I want to return/retrieve self.camera_container, where self.camera_container = {camera_id: (camera_id, frame, frame_properties)}
I tried to create a method to return the this but it gives two errors
prop, camera_container = monitor_memory.get_frame()
TypeError: 'NoneType' object is not iterable
AttributeError: Can't pickle local object 'SharedMemoryDisplay.__init__.<locals>.<lambda>'
I am only able to able to get self.camera_container[key] if I just do which is fine but I want to get self.camera_container also.
return self.camera_container[key]
In the below script is where, I am using this object to display in a cv2 named window, my ultimate motive is to retrieve frames of all the cameras seperately what it currently does is joins all the camera frames and returns that via self.display_frame which is added to webdisplay_memory in the below script (for displaying in the html) that's why I created a method to retrieve dictionary camera_container.
webdisplay_memory.add_frame(0, self.display_frame, None)
rather than messing with this variable I was thinking of creating a method that returns self.camera_container than using this to get frames of each camera seperately.
how can I overcome this, kindly help if you have better and efficient solutions!

multiprocessing uses pickle under the hood. pickle can serialize only a certain set of objects. Specifically, it can not serialize defaultdict (a type for your camera_container). So, either use a normal dict and replace all the lookups to self.camera_container.get(key, None), or look into this question and try to use pathos.multiprocessing with dill. The latter approach is not tested by me, though.

deserialize RecognizedForm object from json (Azure Form Recognizer Python SDK)

I need to deserialize json-serialized Azure Form Recognizer results into python FormRecognizer objects (from azure-ai-formrecognizer==3.1.0b1 package), and I do not see any sort of api to perform this deserialization. Any help would be appreciated.

Depending on how you are parsing, if you just need to access RecognizedForm attributes and don't need a true RecognizedForm object (in other words, just the shape of a RecognizedForm object), this might work for you:
import json
from types import SimpleNamespace
recognized_form = json.loads(recognized_form_json, object_hook=lambda fields: SimpleNamespace(**fields))
Otherwise, we can manually deserialize it back to a RecognizedForm. See this gist for an example (not fully tested).

Save the optimization results from PyGMO by using pickle or dill

I generated a population using PyGMO. A population is a class which contains the individual results of the computation. I can iterate through the population and save the current function values and the parameter values. Unfortunately I cannot dump the whole class e.g. using pickle or dill. If I try:
with open('pop', 'wb') as f:
dill.dump(pop,f)
I do get:
RuntimeError: unregistered class - derived class not registered or exported
It would be great to serialize the whole object because I might be able to use it for a warm start.
Any Ideas ?

As a matter of fact, I had the same issue some days ago. Instead of saving the whole population, save the whole class (island or archipelago). I did use cPickle and pickle, and they both work fine.
The trick to declare the class of your problem before dumping or loading the object. You can see more here.
Regards!

dreaded "not the same object error" pickling a queryset.query object

I have a queryset that I need to pickle lazily and I am having some serious troubles. cPickle.dumps(queryset.query) throws the following error:
Can't pickle <class 'myproject.myapp.models.myfile.QuerySet'>: it's not the same object as myproject.myapp.models.myfile.QuerySet
Strangely (or perhaps not so strangely), I only get that error when I call cPcikle from another method or a view, but not when I call it from the command line.
I made the method below after reading PicklingError: Can't pickle <class 'decimal.Decimal'>: it's not the same object as decimal.Decimal and Django mod_wsgi PicklingError while saving object:
def dump_queryset(queryset, model):
from segment.segmentengine.models.segment import QuerySet
memo = {}
new_queryset = deepcopy(queryset, memo)
memo = {}
new_query = deepcopy(new_queryset.query, memo)
queryset = QuerySet(model=model, query=new_query)
return cPickle.dumps(queryset.query)
As you can see, I am getting extremely desperate -- that method still yields the same error. Is there a known, non-hacky solution to this problem?
EDIT: Tried using --noreload running on the django development server, but to no avail.
EDIT2: I had a typo in the error I displayed above -- it was models.QuerySet, not models.mymodel.QuerySet that it was complaining about. There is another nuance here, which is that my models file is broken out into multiple modules, so the error is ACTUALLY:
Can't pickle <class 'myproject.myapp.models.myfile.QuerySet'>: it's not the same object as myproject.myapp.models.myfile.QuerySet
Where myfile is one of the modules under models. I have an __ini__.py in models with the following line:
from myfile import *
I wonder if this is contributing to my issue. Is there some way to change my init to protect myself against this? Are there any other tests to try?
EDIT3: Here is a little more background on my use case: I have a model called Context that I use to populate a UI element with instances of mymodel. The user can add/remove/manipulate the objects on the UI side, changing their context, and when they return, they can keep their changes, because the context serialized everything. A context has a generic foreign key to different types of filters/ways the user can manipulate the object, all of which must implement a few methods that the context uses to figure out what it should display. One such filter takes a queryset that can be passed in and displays all of the objects in that queryset. This provides a way to pass in arbitrary querysets that are produced elsewhere and have them displayed in the UI element. The model that uses the Context is hierarchical (using mptt for this), and the UI element makes a request to get children each time the user clicks around, we can then take the children and determine if they should be displayed based on whether or not they are included in the Context. Hope that helps!
EDIT4: I am able to dump an empty queryset, but as soon as I add anything of value, it fails.
EDIT4: I am on Django 1.2.3

This may not be the case for everyone, but I was using Ipython notebook and having a similar issue pickling my own class.
The problem turned out to be from a reload call
from dir.my_module import my_class
reload(dir.my_module)
Removing the reload call and then re-running the import and the cell where the instance of that object was created then allowed it to be pickled.

not so elegant but perhaps it works:
add the directory of the myfile -module to os.sys.path and use only import myfile in each module where you use myfile. (remove any from segment.segmentengine.models.segment import, anywhere in your project)

According to this doc, pickling a QuerySet should be not a problem. Thus, the problem should come from other place.
Since you mentined:
EDIT2: I had a typo in the error I displayed above -- it was models.QuerySet, not models.mymodel.QuerySet that it was complaining about. There is another nuance here, which is that my models file is broken out into multiple modules, so the error is ACTUALLY:
The second error message you provided look like the same as previous one, is that what you mean?
The error message you provided looks weird. Since you are pickling "queryset.query", the error should related to the django.db.models.sql.Query class instead of the QuerySet class.
Some modules or classes may have the same name. They will override each other then cause this kind of issue. To make thing easier, I will recommend you to use "import ooo.xxx" instead of "from ooo import *".

Your could also try
import ooo.xxx as othername

shelve gives strange error

I'm trying to put some sites i crawled into a shelve, but the shelve won't accept any Site-objects. It will accept lists, strings, tuples, what have you, but as soon as i put in a Site-object, it crashes when i try to get the contents of the shelve
So when i fill up my shelve like this:
def add_to_shelve(self, site):
db = shelve.open("database")
print site, site.url
for word in site.content:
db[word] = site.url #site.url is a string, word has to be one too
shelve.open("database")['whatever'] works perfectly.
But if I do this:
def add_to_shelve(self, site):
db = shelve.open("database")
print site, site.url
for word in site.content:
db[word] = site #site is now an object of Site
shelve.open("database")['whatever'] errors out with this error message:
AttributeError: 'module' object has no attribute 'Site'
I'm completely stumped, and the pythondocs, strangely, don't have much info either. All they say is that the key in a shelve has to be a string, but the value or data can be "an arbitrary object"

It looks like you refactored your code after saving objects in the shelve. When retrieving objects from the shelve, Python rebuilds the object, and it needs to find the original class that, presumably, you have moved. This problem is typical when working with pickle (as the shelve module does).
The solution, as pduel suggests, is to provide a backwards-compatibility reference to the class in the same location that it used to be, so that pickle can find it. If you re-save all the objects, thereby rebuilding the pickles, you can remove that backwards-comatibility referece.

It seems that Python is looking for a constructor for a 'Site' object, and not finding it. I have not used shelve, but I recall the rules for what can be pickled are byzantine, and suspect the shelve rules are similar.
Try adding the line:
Site = sitemodule.Site
(with the name of the module providing 'Site') before you try unshelving. This ensures that a Site class can be found.