I am trying to load a pickle file which i created using joblib.dump()
The dumping code looks like:
from sklearn.externals import joblib
with open('sample.pickle','wb') as f:
joblib.dump([x,y],f)
This works fine and my sample.pickle is saved successfully. But when i try to load this file:
with open('sample.pickle', 'rb') as f:
x, y = joblib.load(f)
I get the following error:
ValueError: EOF: reading array data, expected 1200 bytes got 0
The full error log looks like this:
Traceback (most recent call last):
File "model.py", line 16, in <module>
vec_x, vec_y = joblib.load(f)
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 568, in load
obj = _unpickle(fobj)
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 508, in _unpickle
obj = unpickler.load()
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\pickle.py", line 1050, in load
dispatch[key[0]](self)
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 341, in load_build
self.stack.append(array_wrapper.read(self))
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 184, in read
array = self.read_array(unpickler)
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 135, in read_array
read_size, "array data")
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle_utils.py", line 646, in _read_bytes
raise ValueError(msg % (error_template, size, len(data)))
ValueError: EOF: reading array data, expected 1200 bytes got 0
By the way, i am very new to dumping and pickling.
Any help will be appreciated. Thanks in advance.
Related
I am training Resnet50 on imagenet using the script provided from PyTorch (with a slight trivial tweak for my purpose). However, I am getting the following error after 14 epochs of training. I have allocated 4 gpus in the server I'm using to run this. Any pointers as to what this error is about would be appreciated. Thanks a lot!
Epoch: [14][5000/5005] Time 1.910 (2.018) Data 0.000 (0.191) Loss 2.6954 (2.7783) Total 2.6954 (2.7783) Reg 0.0000 Prec#1 42.969 (40.556) Prec#5 64.844 (65.368)
Test: [0/196] Time 86.722 (86.722) Loss 1.9551 (1.9551) Prec#1 51.562 (51.562) Prec#5 81.641 (81.641)
Traceback (most recent call last):
File "main_group.py", line 549, in <module>
File "main_group.py", line 256, in main
File "main_group.py", line 466, in validate
if args.gpu is not None:
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 801, in __next__
return self._process_data(data)
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torch/_utils.py", line 385, in reraise
raise self.exc_type(msg)
OSError: Caught OSError in DataLoader worker process 11.
Original Traceback (most recent call last):
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torchvision/datasets/folder.py", line 138, in __getitem__
sample = self.loader(path)
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torchvision/datasets/folder.py", line 174, in default_loader
return pil_loader(path)
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torchvision/datasets/folder.py", line 155, in pil_loader
with open(path, 'rb') as f:
OSError: [Errno 5] Input/output error: '/data/users2/oiler/github/imagenet-data/val/n02102973/ILSVRC2012_val_00009130.JPEG'
It is difficult to tell what the problem is just by looking at the error you have posted.
All we know is that there was an issue reading the file at '/data/users2/oiler/github/imagenet-data/val/n02102973/ILSVRC2012_val_00009130.JPEG'.
Try the following:
Confirm the file actually exists.
Confirm that it is infact a valid JPEG and not corrupted (by viewing it).
Confirm that you can open it with Python and also load it manually with PIL.
If none of that works, try deleting the file. Do you get the same error on another file in the folder?
My friend saved a decision tree model using joblib.dump(). But when I tried to predict using some data from the saved model, I got the following error. Can anyone advise as to why this is happening? My friend and I have the same versions of all required libraries
Traceback (most recent call last):
File "testing.py", line 12, in <module>
classifier = joblib.load('saved_model.pkl')
File "C:\Users\naiks\AppData\Local\Programs\Python\Python38-32\lib\site-packages\joblib\numpy_pickle.py", line 585, in load
obj = _unpickle(fobj, filename, mmap_mode)
File "C:\Users\naiks\AppData\Local\Programs\Python\Python38-32\lib\site-packages\joblib\numpy_pickle.py", line 504, in _unpickle
obj = unpickler.load()
File "C:\Users\naiks\AppData\Local\Programs\Python\Python38-32\lib\pickle.py", line 1210, in load
dispatch[key[0]](self)
File "C:\Users\naiks\AppData\Local\Programs\Python\Python38-32\lib\pickle.py", line 1587, in load_reduce
stack[-1] = func(*args)
File "sklearn\tree\_tree.pyx", line 607, in sklearn.tree._tree.Tree.__cinit__
ValueError: Buffer dtype mismatch, expected 'SIZE_t' but got 'long long'
I was using a 32 bit version of python, whereas the machine on which the model was trained was using 64 bit python. Hence the buffer size was different, resulting in the error
import tensorflow as tf
imagePath = '/home/sk/test_set'
image_list = tf.gfile.ListDirectory(imagePath)
for i in image_list:
image_data = tf.gfile.FastGFile(i, 'rb').read()
when excute this code error occured
Traceback (most recent call last):
File "/home/sk/tensorflow-classify/photos/as.py", line 18, in <module>
image_data = tf.gfile.FastGFile(i, 'rb').read()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py", line 115, in read
self._preread_check()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py", line 75, in _preread_check
compat.as_bytes(self.__name), 1024 * 512, status)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: 11.jpg
i think tf.gfile.FastGFile(i, 'rb').read() this is not properly working.
it works when file name is 1.jpg, but changing file name following error occurs
how can i fix this problem
It looks like the path imagePath = '/home/sk/test_set' is not correct or the method accepts relatives routes.
Check the example here
There are a FLAGS variable where defines all the routes, follow the leed based on your file structure, and where do you have the proyect
I write python layer for caffe that can read hdf5 with some manipulation needs for me. But I have some issue when open and read hdf5 file in python setup method. Here it is.
When I used tables with the
code in setup:
def setup(self, bottom, top):
h5file = tables.open_file("/home/titan/models/hdf5/train_small.h5", driver="H5FD_CORE")
I have this error when run net:
Traceback (most recent call last):
File "/home/titan/scripts/python_layers/pydata_hdf5.py", line 37, in setup
h5file = tables.open_file("/home/titan/models/hdf5/train_small.h5", driver="H5FD_CORE")
File "/home/titan/anaconda/lib/python2.7/site-packages/tables/file.py", line 318, in open_file
return File(filename, mode, title, root_uep, filters, **kwargs)
File "/home/titan/anaconda/lib/python2.7/site-packages/tables/file.py", line 784, in __init__
self._g_new(filename, mode, **params)
File "tables/hdf5extension.pyx", line 465, in tables.hdf5extension.File._g_new (tables/hdf5extension.c:4872)
tables.exceptions.HDF5ExtError: HDF5 error back trace
File "../../../src/H5FDcore.c", line 273, in H5Pset_fapl_core
not a file access property list
File "../../../src/H5Pint.c", line 3371, in H5P_object_verify
property list is not a member of the class
File "../../../src/H5Pint.c", line 3321, in H5P_isa_class
not a property list
End of HDF5 error back trace
When I used h5py
def setup(self, bottom, top):
self.data = h5py.File('/home/titan/models/hdf5_nose_mouth/train_small.h5', 'r')
I have the same error:
Traceback (most recent call last):
File "/home/titan/scripts/python_layers/pydata_hdf5.py", line 11, in <module>
import h5py
File "/home/titan/anaconda/lib/python2.7/site-packages/h5py/__init__.py", line 31, in <module>
from .highlevel import *
File "/home/titan/anaconda/lib/python2.7/site-packages/h5py/highlevel.py", line 13, in <module>
from ._hl.base import is_hdf5, HLObject
File "/home/titan/anaconda/lib/python2.7/site-packages/h5py/_hl/base.py", line 78, in <module>
dlapl = default_lapl()
File "/home/titan/anaconda/lib/python2.7/site-packages/h5py/_hl/base.py", line 65, in default_lapl
lapl = h5p.create(h5p.LINK_ACCESS)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2458)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2415)
File "h5py/h5p.pyx", line 130, in h5py.h5p.create (-------src-dir-------/h5py/h5p.c:2491)
ValueError: Not a property list class (Not a property list class)
When I used deepdish:
def setup(self, bottom, top):
self.data = dd.io.load('/home/titan/models/hdf5/train_smallest.h5')
I have error too:
Traceback (most recent call last):
File "/home/titan/scripts/python_layers/pydata_hdf5.py", line 36, in setup
self.data = dd.io.load('/home/titan/models/hdf5/train_smallest.h5')
File "/home/titan/anaconda/lib/python2.7/site-packages/deepdish/io/hdf5io.py", line 476, in load
with tables.open_file(path, mode='r') as h5file:
File "/home/titan/anaconda/lib/python2.7/site-packages/tables/file.py", line 318, in open_file
return File(filename, mode, title, root_uep, filters, **kwargs)
File "/home/titan/anaconda/lib/python2.7/site-packages/tables/file.py", line 784, in __init__
self._g_new(filename, mode, **params)
File "tables/hdf5extension.pyx", line 488, in tables.hdf5extension.File._g_new (tables/hdf5extension.c:5081)
tables.exceptions.HDF5ExtError: HDF5 error back trace
File "../../../src/H5F.c", line 1582, in H5Fopen
not file access property list
File "../../../src/H5Pint.c", line 3321, in H5P_isa_class
not a property list
End of HDF5 error back trace
But when I read simply *.txt file, all is ok. Also I can read this file from console and use it in hdf5 layer in caffe. Please help me, how I can read hdf5 file from python layer?
Try to install another version of h5py. I just solved it with:
pip install h5py==prev_version
I guess it has something to do with the linking but it would be interesting to know which is the exact cause for this problem.
I am new to python. I have a file data.pkl. What I would like to do is get the data from the file. I looked at http://docs.python.org/library/pickle.html, 11.1.7 example and tried exactly that.
My code looks like this:
import pprint, pickle
pkl_file = open('data.pkl', 'rb')
data1 = pickle.load(pkl_file)
pprint.pprint(data1)
pkl_file.close()
But it is giving me error:
Traceback (most recent call last):
File "/home/sadiksha/workspace/python/test.py", line 5, in <module>
data1 = pickle.load(pkl_file)
File "/usr/lib/python2.7/pickle.py", line 1378, in load
return Unpickler(file).load()
File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
File "/usr/lib/python2.7/pickle.py", line 966, in load_string
raise ValueError, "insecure string pickle"
Can anyone please tell me what am I doing wrong here?
It seems that your pickle file was either not written correctly (specifying 'wb') or the file was somehow corrupted. Try creating your own pickle file and reading that back in. That should do the trick.
As for the pickle file specified, it is definitely corrupted.