I'm working on a game in Pygame that includes a player class and an enemy class. Each class has multiple variables within it. I'm trying to figure out how I can save the data of these sprites by using Python's built-in pickle module. I thought of doing something similar to this:
data_file = open_file("save.dat","wb")
for i in enemyList:
pickle.dump(i.health)
pickle.dump(i.rect.x)
pickle.dump(i.rect.y)
pickle.dump(i.image)
and so on for each variable. How can I save the data and retrieve it in the same state it was in previously?
Answer
Since pickle is object serialization, you should just be able to dump your whole object. The b in wb is for binary. This is because you don't have to know how an object is represented in binary, you can just dump it like so:
data_file = open_file("save.dat","wb")
for i in enemyList:
pickle.dump(i, data_file)
Then when you load it back in you will have the whole object.
To open it:
with open('save.dat', 'rb') as fp:
i = pickle.load(fp)
I havn't used pickle before, but since it is all binary you should just be able to dump your enemyList if it an object:
data_file = open_file("save.dat","wb")
pickle.dump(enemyList, data_file)
with open('save.dat', 'rb') as fp:
enemyList = pickle.load(fp)
Excluding/Including Additional State
Pickle uses the __getstate__ and __setstate__ methods to alter state before reading and writing pickle serialized data. If you wish to omit un-serialization data you must override these methods. Here is the documentation to help you in doing so:
Pickle State
Consideration
Serialization (and therefor python pickle) is seen as an alternative to creating your own file format. Which often times, I find to be easier depending on the data types. If you are not in control of your object hierarchy, sometimes you don't want to create your own inherited object to try and gain control of all the data. Sometimes it is just easier to write your own file format.
Related
I generated a population using PyGMO. A population is a class which contains the individual results of the computation. I can iterate through the population and save the current function values and the parameter values. Unfortunately I cannot dump the whole class e.g. using pickle or dill. If I try:
with open('pop', 'wb') as f:
dill.dump(pop,f)
I do get:
RuntimeError: unregistered class - derived class not registered or exported
It would be great to serialize the whole object because I might be able to use it for a warm start.
Any Ideas ?
As a matter of fact, I had the same issue some days ago. Instead of saving the whole population, save the whole class (island or archipelago). I did use cPickle and pickle, and they both work fine.
The trick to declare the class of your problem before dumping or loading the object. You can see more here.
Regards!
So I just found out today that I can import variables from other Python files. For example, I can have a variable green = 1, and have a completely separate file use that variable. I've found that this is really helpful for functions.
So, here's my question. I'm sorry if the title didn't help very much, I wasn't entirely sure what this would be called.
I want my program to ask the player what his or her name is. Once the player has answered, I want that variable to be stored in a separate Python file, "player_variables.py", so every time I want to say the player's name, instead of having to go to from game import name, I can use from player_variables import name, just to make it easier.
I fully understand that this is a lazy man's question, but I'm just trying to learn as much as I could. I'm still very new, and I'm sorry if this question is ridiculous. :).
Thanks for the help, I appreciate it. (be nice to me!)
From your question, I think you're confusing some ideas about variables and the values of variables.
1) Simply creating a python file with variable names allows you to access the values of those variables defined therein. Example:
# myvariables.py
name = 'Steve'
# main.py
import myvariables
print myvariables.name
Since this requires that the variables themselves be defined as global, this solution is not ideal. See this link for more: https://docs.python.org/2/reference/simple_stmts.html#global
2) However, as your question states that you want your program to save the variable of some user-entered value in the python file, that is another thing entirely as that's metaprogramming in python - as your program is outputting Python code, in this case a file with the same name as your variables python file.
From the above, if this is the means by which you're accessing and updating variables values, it is a simpler idea to have a config file which can be loaded or changed at whim. Metaprogramming is only necessary when you need it.
I think you should use a data format for storing data, not a data manipulating programming language for this:
import json
import myconfig
def load_data():
with open(myconfig.config_loc, 'r') as json_data:
return json.loads(json_data.read())
def save_data(name, value):
existing_data = load_data()
existing_data[name] = value
with open(myconfig.config_loc, 'w') as w:
w.write(json.dumps(existing_data, indent=4, sort_keys=True))
You should only store the location of this json file in your myvariables.py file, but I'd call it something else (in this case, myconfig, or just config).
More on the json library can be found here.
I want to search surfs in all images in a given directory and save their keypoints and descriptors for future use. I decided to use pickle as shown below:
#!/usr/bin/env python
import os
import pickle
import cv2
class Frame:
def __init__(self, filename):
surf = cv2.SURF(500, 4, 2, True)
self.filename = filename
self.keypoints, self.descriptors = surf.detect(cv2.imread(filename, cv2.CV_LOAD_IMAGE_GRAYSCALE), None, False)
if __name__ == '__main__':
Fdb = open('db.dat', 'wb')
base_path = "img/"
frame_base = []
for filename in os.listdir(base_path):
frame_base.append(Frame(base_path+filename))
print filename
pickle.dump(frame_base,Fdb,-1)
Fdb.close()
When I try to execute, I get a following error:
File "src/pickle_test.py", line 23, in <module>
pickle.dump(frame_base,Fdb,-1)
...
pickle.PicklingError: Can't pickle <type 'cv2.KeyPoint'>: it's not the same object as cv2.KeyPoint
Does anybody know, what does it mean and how to fix it? I am using Python 2.6 and Opencv 2.3.1
Thank you a lot
The problem is that you cannot dump cv2.KeyPoint to a pickle file. I had the same issue, and managed to work around it by essentially serializing and deserializing the keypoints myself before dumping them with Pickle.
So represent every keypoint and its descriptor with a tuple:
temp = (point.pt, point.size, point.angle, point.response, point.octave,
point.class_id, desc)
Append all these points to some list that you then dump with Pickle.
Then when you want to retrieve the data again, load all the data with Pickle:
temp_feature = cv2.KeyPoint(x=point[0][0],y=point[0][1],_size=point[1], _angle=point[2],
_response=point[3], _octave=point[4], _class_id=point[5])
temp_descriptor = point[6]
Create a cv2.KeyPoint from this data using the above code, and you can then use these points to construct a list of features.
I suspect there is a neater way to do this, but the above works fine (and fast) for me. You might have to play around with your data format a bit, as my features are stored in format-specific lists. I tried to present the above using my idea at its generic base. I hope that this may help you.
Part of the issue is cv2.KeyPoint is a function in python that returns a cv2.KeyPoint object. Pickle is getting confused because, literally, "<type 'cv2.KeyPoint'> [is] not the same object as cv2.KeyPoint". That is, cv2.KeyPoint is a function object, while the type was cv2.KeyPoint. Why OpenCV is like that, I can only make guesses at unless I go digging. I have a feeling it has something to do with it being a wrapper around a C/C++ library.
Python does give you the ability to fix this yourself. I found the inspiration on this post about pickling methods of classes.
I actually use this clip of code, highly modified from the original in the post
import copyreg
import cv2
def _pickle_keypoints(point):
return cv2.KeyPoint, (*point.pt, point.size, point.angle,
point.response, point.octave, point.class_id)
copyreg.pickle(cv2.KeyPoint().__class__, _pickle_keypoints)
Key points of note:
In Python 2, you need to use copy_reg instead of copyreg and point.pt[0], point.pt[1] instead of *point.pt.
You can't directly access the cv2.KeyPoint class for some reason, so you make a temporary object and use that.
The copyreg patching will use the otherwise problematic cv2.KeyPoint function as I have specified in the output of _pickle_keypoints when unpickling, so we don't need to implement an unpickling routine.
And to be nauseatingly complete, cv2::KeyPoint::KeyPoint is an overloaded function in C++, but in Python, this isn't exactly a thing. Whereas in the C++, there's a function that takes the point for the first argument, in Python, it would try to interpret that as an int instead. The * unrolls the point into two arguments, x and y to match the only int argument constructor.
I had been using casper's excellent solution until I realized this was possible.
A similar solution to the one provided by Poik. Just call this once before pickling.
def patch_Keypoint_pickiling(self):
# Create the bundling between class and arguments to save for Keypoint class
# See : https://stackoverflow.com/questions/50337569/pickle-exception-for-cv2-boost-when-using-multiprocessing/50394788#50394788
def _pickle_keypoint(keypoint): # : cv2.KeyPoint
return cv2.KeyPoint, (
keypoint.pt[0],
keypoint.pt[1],
keypoint.size,
keypoint.angle,
keypoint.response,
keypoint.octave,
keypoint.class_id,
)
# C++ Constructor, notice order of arguments :
# KeyPoint (float x, float y, float _size, float _angle=-1, float _response=0, int _octave=0, int _class_id=-1)
# Apply the bundling to pickle
copyreg.pickle(cv2.KeyPoint().__class__, _pickle_keypoint)
More than for the code, this is for the incredibly clear explanation available there : https://stackoverflow.com/a/50394788/11094914
Please note that if you want to expand this idea to other "unpickable" class of openCV, you only need to build a similar function to "_pickle_keypoint". Be sure that you store attributes in the same order as the constructor. You can consider copying the C++ constructor, even in Python, as I did. Mostly C++ and Python constructors seems not to differ too much.
I has issue with the "pt" tuple. However, a C++ constructor exists for X and Y separated coordinates, and thus, allow this fix/workaround.
I have an object with an __init__ procedure that requires at least one parameter and
I want to store in the cache.
When trying to getting the object from the cache I get an error that the I didn't pass enough parameters to the ___init___ method.
Someone told me I need to pickle the object before sending it to the cache but all the examples I saw were using .dat files and on appengine you cannot use any file system.
You can use pickle without any filesystem, using pickle.loads / pickle.dumps. For example:
import pickle
obj = YourClass(yourparam=...)
data = pickle.dumps(obj)
# and now, store "data" into the cache
# later, get "data" from the cache
obj = pickle.loads(data)
# and tada, obj if the same as before :)
I think you are trying to use memcache in appengine. This blog will help you a lot
http://blog.notdot.net/2009/9/Efficient-model-memcaching
I'm trying to put some sites i crawled into a shelve, but the shelve won't accept any Site-objects. It will accept lists, strings, tuples, what have you, but as soon as i put in a Site-object, it crashes when i try to get the contents of the shelve
So when i fill up my shelve like this:
def add_to_shelve(self, site):
db = shelve.open("database")
print site, site.url
for word in site.content:
db[word] = site.url #site.url is a string, word has to be one too
shelve.open("database")['whatever'] works perfectly.
But if I do this:
def add_to_shelve(self, site):
db = shelve.open("database")
print site, site.url
for word in site.content:
db[word] = site #site is now an object of Site
shelve.open("database")['whatever'] errors out with this error message:
AttributeError: 'module' object has no attribute 'Site'
I'm completely stumped, and the pythondocs, strangely, don't have much info either. All they say is that the key in a shelve has to be a string, but the value or data can be "an arbitrary object"
It looks like you refactored your code after saving objects in the shelve. When retrieving objects from the shelve, Python rebuilds the object, and it needs to find the original class that, presumably, you have moved. This problem is typical when working with pickle (as the shelve module does).
The solution, as pduel suggests, is to provide a backwards-compatibility reference to the class in the same location that it used to be, so that pickle can find it. If you re-save all the objects, thereby rebuilding the pickles, you can remove that backwards-comatibility referece.
It seems that Python is looking for a constructor for a 'Site' object, and not finding it. I have not used shelve, but I recall the rules for what can be pickled are byzantine, and suspect the shelve rules are similar.
Try adding the line:
Site = sitemodule.Site
(with the name of the module providing 'Site') before you try unshelving. This ensures that a Site class can be found.