Develop Reference

Serializing a list of class instances in python

In python, I am trying to store a list to a file. I've tried pickle, json, etc, but none of them support classes being inside those lists. I can't sacrifice the lists or the classes, I must maintain both. How can I do it?
My current attempt:
try:
with open('file.json', 'r') as file:
allcards = json.load(file)
except:
allcards = []
def saveData(list):
with open('file.json', 'w') as file:
print(list)
json.dump(list, file, indent=2)
saveData is called elsewhere, and I've done all the testing I can and have determined the error comes from trying to save the list due to it's inclusion of classes. It throws me the error
Object of type Card is not JSON serializable
whenever I do the JSON method, and any other method doesn't even give errors but doesn't load the list when I reload the program.
Edit: As for the pickle method, here is what it looks like:
try:
with open('allcards.dat', 'rb') as file:
allcards = pickle.load(file)
print(allcards)
except:
allcards = []
class Card():
def __init__(self, owner, name, rarity, img, pack):
self.owner = str(owner)
self.name = str(name)
self.rarity = str(rarity)
self.img = img
self.pack = str(pack)
def saveData(list):
with open('allcards.dat', 'wb') as file:
pickle.dump(list, file)
When I do this, all that happens is the code runs as normal, but the list is not saved. And the print(allcards) does not trigger either which makes me believe it's somehow not detecting the file or causing some other error leading to it just going straight to the exception. Also, img is supposed to always a link, in case that changes anything.
I have no other way I believe I can help solve this issue, but I can post more code if need be.
Please help, and thanks in advance.

Python's built-in pickle module does not support serializing a python class, but there are libraries that extend the pickle module and provide this functionality. Drill and Cloudpickle both support serializing a python class and has the exact same interface as the pickle module.
Dill: https://github.com/uqfoundation/dill
Cloudpickle: https://github.com/cloudpipe/cloudpickle

//EDIT
The article linked below is good, but I've written a bad example.
This time I've created a new snippet from scratch -- sorry for making it earlier more complicated than it should.
import json
class Card(object):
#classmethod
def from_json(cls, data):
return cls(**data)
def __init__(self, figure, color):
self.figure = figure
self.color = color
def __repr__(self):
return f"<Card: [{self.figure} of {self.color}]>"
def save(cards):
with open('file.json', 'w') as f:
json.dump(cards, f, indent=4, default=lambda c: c.__dict__)
def load():
with open('file.json', 'r') as f:
obj_list = json.load(f)
return [Card.from_json(obj) for obj in obj_list]
cards = []
cards.append(Card("1", "clubs"))
cards.append(Card("K", "spades"))
save(cards)
cards_from_file = load()
print(cards_from_file)
Source

pickle is not working in a proper way

import nltk
import pickle
input_file=open('file.txt', 'r')
input_datafile=open('newskills1.txt', 'r')
string=input_file.read()
fp=(input_datafile.read().splitlines())
def extract_skills(string):
skills=pickle.load(fp)
skill_set=[]
for skill in skills:
skill= ''+skill+''
if skill.lower() in string:
skill_set.append(skill)
return skill_set
if __name__ == '__main__':
skills= extract_skills(string)
print(skills)
I want to print the skills from file but, here pickle is not working.
It shows the error:
_pickle.UnpicklingError: the STRING opcode argument must be quoted

The file containing the pickled data must be written and read as a binary file. See the documentation for examples.
Your extraction function should look like:
def extract_skills(path):
with open(path, 'rb') as inputFile:
skills = pickle.load(inputFile)
Of course, you will need to dump your data into a file open as binary as well:
def save_skills(path, skills):
with open(path, 'wb') as outputFile:
pickle.dump(outputFile, skills)
Additionally, the logic of your main seems a bit flawed.
While the code that follows if __name__ == '__main__' is only executed when the script is run as main module, the code that is not in the main should only be static, ie definitions.
Basically, your script should not do anything, unless run as main.
Here is a cleaner version.
import pickle
def extract_skills(path):
...
def save_skills(path, skills):
...
if __name__ == '__main__':
inputPath = "skills_input.pickle"
outputPath = "skills_output.pickle"
skills = extract_skills(inputPath)
# Modify skills
save_skills(outputPath, skills)

Python - error while pickling [duplicate]

I'm using python3.3 and I'm having a cryptic error when trying to pickle a simple dictionary.
Here is the code:
import os
import pickle
from pickle import *
os.chdir('c:/Python26/progfiles/')
def storvars(vdict):
f = open('varstor.txt','w')
pickle.dump(vdict,f,)
f.close()
return
mydict = {'name':'john','gender':'male','age':'45'}
storvars(mydict)
and I get:
Traceback (most recent call last):
File "C:/Python26/test18.py", line 31, in <module>
storvars(mydict)
File "C:/Python26/test18.py", line 14, in storvars
pickle.dump(vdict,f,)
TypeError: must be str, not bytes

The output file needs to be opened in binary mode:
f = open('varstor.txt','w')
needs to be:
f = open('varstor.txt','wb')

Just had same issue. In Python 3, Binary modes 'wb', 'rb' must be specified whereas in Python 2x, they are not needed. When you follow tutorials that are based on Python 2x, that's why you are here.
import pickle
class MyUser(object):
def __init__(self,name):
self.name = name
user = MyUser('Peter')
print("Before serialization: ")
print(user.name)
print("------------")
serialized = pickle.dumps(user)
filename = 'serialized.native'
with open(filename,'wb') as file_object:
file_object.write(serialized)
with open(filename,'rb') as file_object:
raw_data = file_object.read()
deserialized = pickle.loads(raw_data)
print("Loading from serialized file: ")
user2 = deserialized
print(user2.name)
print("------------")

pickle uses a binary protocol, hence only accepts binary files. As the document said in the first sentence, "The pickle module implements binary protocols for serializing and de-serializing".

how to use json on mac os

i have encountered a very strange issue: i use json.dump to write a file and then use json.load to read the file.
The same code can run succeed on windows 7 but it can not do on mac os x 10.7
Below is the code:
class Result:
def __init__(self,name,result):
self.name = name
self.result = result
def __repr__(self):
return 'Result name : %s , result : %s' % (self.name,self.result)
class MyEncoder(json.JSONEncoder):
def default(self,obj):
#convert object to a dict
d = {'CaseResult':{}}
d['CaseResult'][obj.name] = obj.result
return d
def save(name,result):
filename = 'basic.json'
obj = Result(name,result)
obj_json = MyEncoder().encode(obj)
with open(filename, mode='ab+') as fp:
json.dump(obj_json,fp)
s=json.load(fp)
save('aaa','bbb')
in mac os it give an error "ValueError:NO JSON object could be decoded"
who can tell me why this happen and how can i resolve it

This problem is unrelated to being run on a Mac; this code should never work:
with open(filename, mode='ab+') as fp:
json.dump(obj_json,fp)
s=json.load(fp)
This is because after json.dump, your file pointer is at the end of the file. You must call fp.seek to reset it to the initial position, like this:
import os
with open(filename, mode='rb+') as fp:
fp.seek(0, os.SEEK_END)
pos = fp.tell()
json.dump(obj_json,fp)
fp.seek(pos)
s=json.load(fp)

I'm not sure how this actually works on Windows, but you're missing a seek back to the beginning of the file before you read the object back. change your save/load to
with open(filename, mode='ab+') as fp:
json.dump(obj_json,fp)
fp.seek(0)
s=json.load(fp)
and it runs just fine on MacOS too. Note that you're appending to the file, so only the first run succeeds in loading the object back, the next one will find extra data after the end of the object.

Using Pickle object like an API call

I trained a NaiveBayes classifier to do elementary sentiment analysis. The model is 208MB . I want to load it only once and then use Gearman workers to keep calling the model to get the results. It takes rather long time to load it only once. How do i load the model only once and then keep calling it ?
Some code , hope this helps :
import nltk.data
c=nltk.data.load("/path/to/classifier.pickle")
This remains as the loader script.
Now i have a gearman worker script which should call this "c" object and then classify the text.
c.classify('features')
This is what i want to do .
Thanks.

If the question is how to use pickle, than that's the answer
import pickle
class Model(object):
#some crazy array of data
def getClass(sentiment)
#return class of sentiment
def loadModel(filename):
f = open(filename, 'rb')
res = pickle.load(f)
f.close()
return res
def saveModel(model, filename):
f = open(filename, 'wb')
pickle.dump(model, f)
f.close()
m = loadModel('bayesian.pickle')
if it's a problem to load large object in such a way, than I don't know whether pickle could help