My friend saved a decision tree model using joblib.dump(). But when I tried to predict using some data from the saved model, I got the following error. Can anyone advise as to why this is happening? My friend and I have the same versions of all required libraries
Traceback (most recent call last):
File "testing.py", line 12, in <module>
classifier = joblib.load('saved_model.pkl')
File "C:\Users\naiks\AppData\Local\Programs\Python\Python38-32\lib\site-packages\joblib\numpy_pickle.py", line 585, in load
obj = _unpickle(fobj, filename, mmap_mode)
File "C:\Users\naiks\AppData\Local\Programs\Python\Python38-32\lib\site-packages\joblib\numpy_pickle.py", line 504, in _unpickle
obj = unpickler.load()
File "C:\Users\naiks\AppData\Local\Programs\Python\Python38-32\lib\pickle.py", line 1210, in load
dispatch[key[0]](self)
File "C:\Users\naiks\AppData\Local\Programs\Python\Python38-32\lib\pickle.py", line 1587, in load_reduce
stack[-1] = func(*args)
File "sklearn\tree\_tree.pyx", line 607, in sklearn.tree._tree.Tree.__cinit__
ValueError: Buffer dtype mismatch, expected 'SIZE_t' but got 'long long'
I was using a 32 bit version of python, whereas the machine on which the model was trained was using 64 bit python. Hence the buffer size was different, resulting in the error
Related
I've been doing the pytorch tutorial (https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html) and have been getting this error that I don't know how to fix. The full error is below:
Traceback (most recent call last):
File "main.py", line 146, in <module>
main()
File "main.py", line 138, in main
train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
File "/engine.py", line 26, in train_one_epoch
for images, targets in metric_logger.log_every(data_loader, print_freq, header):
File "/utils.py", line 180, in log_every
for obj in iterable:
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 521, in __next__
data = self._next_data()
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 425, in reraise
raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py", line 311, in __getitem__
return self.dataset[self.indices[idx]]
File "main.py", line 64, in __getitem__
img, target = self.transforms(img, target)
File "/transforms.py", line 26, in __call__
image, target = t(image, target)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/transforms.py", line 50, in forward
image = F.to_tensor(image)
File "/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py", line 129, in to_tensor
np.array(pic, mode_to_nptype.get(pic.mode, np.uint8), copy=True)
TypeError: __array__() takes 1 positional argument but 2 were given
I believe it means somewhere I'm using an array with 2 arguments which isn't allowed, but I don't really know where abouts that is happening - perhaps in one of their pre written libraries?
I can share the code in full if desired, but thought its a bit unwieldy. Does anyone know what might be causing this error?
PyTorch has already considered this issue. It does not seem to be a PyTorch problem.
As xwang233 mentioned in the issue, we can fix it by downgrading pillow:
pip install pillow==8.2.0
This issue could be fixed as well by upgrading Pillow from version 8.3.0 to 8.3.1. I had the same issue with
torch==1.9.0+cu111
torchvision==0.10.0+cu111
Pillow==8.3.0
After Pillow was upgraded to 8.3.1 (with no change to torch and torchvision) as below, the issue is gone:
pip install --upgrade pillow
Thanks to DRTorresRuiz for providing the clue about Pillow.
I had the same error when using:
torch==1.9.0
torchvision==0.10.0
In my requirements.txt file I downgraded the torch library, which forced me to downgrade torchvision, and that fixed the error for me. The library versions I ended up using that did not raise the error were:
torch==1.8.1
torchvision==0.9.1
change your code:
np.array(pic ,np.float32)
to:
np.array(pic).astype('float32')
I am training Resnet50 on imagenet using the script provided from PyTorch (with a slight trivial tweak for my purpose). However, I am getting the following error after 14 epochs of training. I have allocated 4 gpus in the server I'm using to run this. Any pointers as to what this error is about would be appreciated. Thanks a lot!
Epoch: [14][5000/5005] Time 1.910 (2.018) Data 0.000 (0.191) Loss 2.6954 (2.7783) Total 2.6954 (2.7783) Reg 0.0000 Prec#1 42.969 (40.556) Prec#5 64.844 (65.368)
Test: [0/196] Time 86.722 (86.722) Loss 1.9551 (1.9551) Prec#1 51.562 (51.562) Prec#5 81.641 (81.641)
Traceback (most recent call last):
File "main_group.py", line 549, in <module>
File "main_group.py", line 256, in main
File "main_group.py", line 466, in validate
if args.gpu is not None:
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 801, in __next__
return self._process_data(data)
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
data.reraise()
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torch/_utils.py", line 385, in reraise
raise self.exc_type(msg)
OSError: Caught OSError in DataLoader worker process 11.
Original Traceback (most recent call last):
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torchvision/datasets/folder.py", line 138, in __getitem__
sample = self.loader(path)
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torchvision/datasets/folder.py", line 174, in default_loader
return pil_loader(path)
File "/home/users/oiler/anaconda3/envs/ml/lib/python3.7/site-packages/torchvision/datasets/folder.py", line 155, in pil_loader
with open(path, 'rb') as f:
OSError: [Errno 5] Input/output error: '/data/users2/oiler/github/imagenet-data/val/n02102973/ILSVRC2012_val_00009130.JPEG'
It is difficult to tell what the problem is just by looking at the error you have posted.
All we know is that there was an issue reading the file at '/data/users2/oiler/github/imagenet-data/val/n02102973/ILSVRC2012_val_00009130.JPEG'.
Try the following:
Confirm the file actually exists.
Confirm that it is infact a valid JPEG and not corrupted (by viewing it).
Confirm that you can open it with Python and also load it manually with PIL.
If none of that works, try deleting the file. Do you get the same error on another file in the folder?
i try to use validate_on_lfw.py to evaluate LFW with model '20170512-110547'
i use anaconda and tensorflow(i am a beginner of cv ,so i am not sure if the problem description i gave is detailed)
tensorflow/core/graph/graph_constructor.cc:1244] Importing a graph
with a lower producer version 21 into an existing graph with producer
version 26. Shape inference will have run different parts of the graph
with different producer versions.
Traceback (most recent call last):
File "/home/wind/anaconda3/envs/tensorflow-cpu/lib/python3.6/site-
packages/tensorflow/python/framework/importer.py", line 489, in
import_graph_def graph._c_graph, serialized, options) # pylint:
disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input 0
of node cross_entropy_per_example/Shape was passed int32 from
batch_join:1 incompatible with expected int64.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "src/validate_on_lfw.py", line 164, in <module>
main(parse_arguments(sys.argv[1:]))
File "src/validate_on_lfw.py", line 73, in main
facenet.load_model(args.model, input_map=input_map)
File "/home/wind/facenet/src/facenet.py", line 381, in load_model
saver = tf.train.import_meta_graph(os.path.join(model_exp, meta_file), input_map=input_map)
File "/home/wind/anaconda3/envs/tensorflow-cpu/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1955, in import_meta_graph
**kwargs)
File "/home/wind/anaconda3/envs/tensorflow-cpu/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 743, in import_scoped_meta_graph
producer_op_list=producer_op_list)
File "/home/wind/anaconda3/envs/tensorflow-cpu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
return func(*args, **kwargs)
File "/home/wind/anaconda3/envs/tensorflow-cpu/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 493, in import_graph_def
raise ValueError(str(e))
ValueError: Input 0 of node cross_entropy_per_example/Shape was passed
int32 from batch_join:1 incompatible with expected int64.
Change the dtypes for data_flow_ops.FIFOQueue to tf.int64 from tf.int32. And also change the dtypes for labels_placeholder and control_placeholder to tf.int64 from tf.int32. That did it for me.
I am trying to load a pickle file which i created using joblib.dump()
The dumping code looks like:
from sklearn.externals import joblib
with open('sample.pickle','wb') as f:
joblib.dump([x,y],f)
This works fine and my sample.pickle is saved successfully. But when i try to load this file:
with open('sample.pickle', 'rb') as f:
x, y = joblib.load(f)
I get the following error:
ValueError: EOF: reading array data, expected 1200 bytes got 0
The full error log looks like this:
Traceback (most recent call last):
File "model.py", line 16, in <module>
vec_x, vec_y = joblib.load(f)
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 568, in load
obj = _unpickle(fobj)
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 508, in _unpickle
obj = unpickler.load()
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\pickle.py", line 1050, in load
dispatch[key[0]](self)
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 341, in load_build
self.stack.append(array_wrapper.read(self))
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 184, in read
array = self.read_array(unpickler)
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle.py", line 135, in read_array
read_size, "array data")
File "C:\Users\acer_pc\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\externals\joblib\numpy_pickle_utils.py", line 646, in _read_bytes
raise ValueError(msg % (error_template, size, len(data)))
ValueError: EOF: reading array data, expected 1200 bytes got 0
By the way, i am very new to dumping and pickling.
Any help will be appreciated. Thanks in advance.
I am trying to convert the pre-trained caffe model provided here into tensorflow this converter, which seems to be the most popular option out there. Nevertheless, when I run their script
$ python3 convert.py ../colorful-colorization/models/colorization_deploy_v2.prototxt --caffemodel ../colorful-colorization/models/colorization_release_v2.caffemodel --code-output-path ./out-code/mynet.py
I get the following error
Traceback (most recent call last):
File "convert.py", line 60, in <module>
main()
File "convert.py", line 56, in main
args.phase)
File "convert.py", line 27, in convert
transformer = TensorFlowTransformer(def_path, caffemodel_path, phase=phase)
File "/home/ebalda/Documents/Python/playground/caffe2tensorflow/kaffe/tensorflow/transformer.py", line 221, in __init__
self.load(def_path, data_path, phase)
File "/home/ebalda/Documents/Python/playground/caffe2tensorflow/kaffe/tensorflow/transformer.py", line 227, in load
graph = GraphBuilder(def_path, phase).build()
File "/home/ebalda/Documents/Python/playground/caffe2tensorflow/kaffe/graph.py", line 140, in __init__
self.load()
File "/home/ebalda/Documents/Python/playground/caffe2tensorflow/kaffe/graph.py", line 146, in load
text_format.Merge(def_file.read(), self.params)
File "/usr/local/lib/python3.6/dist-packages/google/protobuf/text_format.py", line 521, in Merge
text.split('\n'),
TypeError: a bytes-like object is required, not 'str'
After browsing around in github and stackoverflow, I could not find other persons having a similar issue. Does anyone know what could be the problem?