I have tried to run this segmentation model using spyder.
When I run data.py, I obtain this message
File "C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master/data.py", line 19, in create_train_data
imgs = np.ndarray((total, image_rows, image_cols), dtype=np.uint8)
TypeError: 'float' object cannot be interpreted as an integer
The code is:
**14** def create_train_data():
**15** train_data_path = os.path.join(data_path, 'train')
**16** images = os.listdir(train_data_path)
**17** total = len(images) / 2
**18** imgs = np.ndarray((total, image_rows, image_cols), dtype=np.uint8)
**19** imgs_mask = np.ndarray((total, image_rows, image_cols), dtype=np.uint8)
I have replaced line 17 by total = int(len(images) / 2) and np.uint8 by float in lines 18 and 19. The problem is solved.
When I run train.py I have this message
runfile('C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master/train.py', wdir='C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master')
------------------------------
Loading and preprocessing train data...
------------------------------
------------------------------
Creating and compiling model...
------------------------------
C:\Users\achaire\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py:3118: RuntimeWarning: Mean of empty slice.
out=out, **kwargs)
C:\Users\hamdi\Anaconda3\lib\site-packages\numpy\core\_methods.py:140: RuntimeWarning: Degrees of freedom <= 0 for slice
keepdims=keepdims)
------------------------------
Fitting model...
------------------------------
C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master/train.py:119: UserWarning: The `nb_epoch` argument in `fit` has been renamed `epochs`.
callbacks=[model_checkpoint])
Train on 0 samples, validate on 0 samples
Epoch 1/20
Traceback (most recent call last):
File "<ipython-input-15-f713d62eb4dc>", line 1, in <module>
runfile('C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master/train.py', wdir='C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master')
File "C:\Users\achaire\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)
File "C:\Users\achaire\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master/train.py", line 153, in <module>
train_and_predict()
File "C:/Users/achaire/Downloads/Compressed/ultrasound-nerve-segmentation-master/ultrasound-nerve-segmentation-master/train.py", line 119, in train_and_predict
callbacks=[model_checkpoint])
File "C:\Users\achaire\Anaconda3\lib\site-packages\keras\engine\training.py", line 1039, in fit
validation_steps=validation_steps)
File "C:\Users\achaire\Anaconda3\lib\site-packages\keras\engine\training_arrays.py", line 217, in fit_loop
callbacks.on_epoch_end(epoch, epoch_logs)
File "C:\Users\achaire\Anaconda3\lib\site-packages\keras\callbacks.py", line 79, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "C:\Users\achaire\Anaconda3\lib\site-packages\keras\callbacks.py", line 338, in on_epoch_end
self.progbar.update(self.seen, self.log_values)
AttributeError: 'ProgbarLogger' object has no attribute 'log_values'
I have the last version of anaconda, keras and python ...
Answer 1:
If the error only occurs when you use smaller datasets, you’re very likely using datasets small enough to not have a single sample in the validation set.
Thus it cannot calculate a validation loss.
Answer 2:
I up-voted the previous answer as it gave me the insight to verify the data and inputs to the fit_generator function and find out what the root cause of the issue actually was. In summary, in cases where my dataset was small, I calculated validation_steps and steps_per_epoch which turned out to be zero (0) which caused the error.
I suppose the better longer-term answer, perhaps for the Keras team, is to cause an error/exception in fit_generator when these values are zero, which would probably lead to a better understanding about how to address this issue.
Answer 3:
The error occurs to us because we forgot to set validation_data in fit() method, while used 'callbacks': [keras.callbacks.EarlyStopping(monitor='val_loss', patience=1)],
Code causing error is:
self.model.fit(
x=x_train,
y=y_train,
callbacks=[keras.callbacks.EarlyStopping(monitor='val_loss', patience=1)],
verbose=True)
Adding validation_data=(self.x_validate, self.y_validate), in fit() fixed:
self.model.fit(
x=x_train,
y=y_train,
callbacks=[keras.callbacks.EarlyStopping(monitor='val_loss', patience=1)],
validation_data=(x_validate, y_validate),
verbose=True)
Answer 4:
This error occurs due to the smaller dataset, to resolve this, increase the train times and split the train set in 80:20.
Reference: https://inneka.com/ml/kr/keras-early-stopping-callback-error-val_loss-metric-not-available/
I got the same error and I fixed it by setting verbose = 0 in while training.
I had this same problem. If the problem is during your fit_generator, then there is a high chance that you have some tiny bug in your data_generator code. Debug your fit_generator.
Related
This is WongKinYiu / PyTorch_YOLOv4 problem. I found many solutions of this problem in YOLOv7, and no solution in YOLOv4. Since, the community here is more active. Thus, I try to ask here also.
This is the link to WongKinYiu loss.py
Traceback (most recent call last):
File "/content/PyTorch_YOLOv4/train.py", line 537, in
train(hyp, opt, device, tb_writer, wandb)
File "/content/PyTorch_YOLOv4/train.py", line 288, in train
loss, loss_items = compute_loss(pred, targets.to(device), model) # loss scaled by batch_size
File "/content/PyTorch_YOLOv4/utils/loss.py", line 69, in compute_loss
tcls, tbox, indices, anchors = build_targets(p, targets, model) # targets
File "/content/PyTorch_YOLOv4/utils/loss.py", line 151, in build_targets
a, t = at[j], t.repeat(na, 1, 1)[j] # filter
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)
I am training a CNN model on the COCO dataset, and I am getting this error after a few iterations. The error is not consistent because I got this error in 1100 iterations, once in 4500 iterations and one time in 8900 iterations (all of them in 1 epoch).
I thought that this error might be a bug in the new version of imageio, so I changed the version to 2.3.0 but still, after 8900 iterations in 1 epoch, I am getting this error.
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-46-4b33bec4a89e> in <module>()
52
53 # train for one epoch
---> 54 train_loss = train(train_loader, model, [criterion1, criterion2], optimizer)
55 print('train_loss: ',train_loss)
56
4 frames
/usr/local/lib/python3.7/dist-packages/torch/_utils.py in reraise(self)
432 # instantiate since we don't know how to
433 raise RuntimeError(msg) from None
--> 434 raise exception
435
436
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
data = [self.dataset[idx] for idx in possibly_batched_index]
File "<ipython-input-34-4c8722b5b16b>", line 143, in __getitem__
image = imageio.imread(img_path, pilmode='RGB')
File "/usr/local/lib/python3.7/dist-packages/imageio/core/functions.py", line 206, in imread
reader = read(uri, format, 'i', **kwargs)
File "/usr/local/lib/python3.7/dist-packages/imageio/core/functions.py", line 129, in get_reader
return format.get_reader(request)
File "/usr/local/lib/python3.7/dist-packages/imageio/core/format.py", line 168, in get_reader
return self.Reader(self, request)
File "/usr/local/lib/python3.7/dist-packages/imageio/core/format.py", line 217, in __init__
self._open(**self.request.kwargs.copy())
TypeError: _open() got an unexpected keyword argument 'pilmode'
I've had this error before. The TLDR is that you can't assume all of your data is clean and able to be parsed. You aren't loading the data in order as far as I can tell either. You may even have data shuffling enabled. With all of that in mind you should not expect it to fail determinisitically at iteration 100 or 102 or anything.
The issue comes down to one (or more) of the files in COCO dataset is either corrupted or is of a different format. You can process the images in order with a batchsize of 1 and print out the file name to see which one it is.
To "fix" this issue you can do one of several things:
wrap the call to load the image in a try-except block and just skip it.
Convert the image yourself to another appropriate format.
Try a different way to load images in with pytorch.
See here as an example failure scenario when loading in images with imageio.
LSTM(
(embed): Embedding(139948, 12, padding_idx=0)
(lstm): LSTM(12, 12, num_layers=2, batch_first=True, bidirectional=True)
(lin): Linear(in_features=240, out_features=6, bias=True)
)
Train epoch : 1, loss : 771.319284286499, accuracy :0.590
=================================================================================================
Traceback (most recent call last):enter code here
File "C:/Users/Administrator/PycharmProjects/untitled/example.py", line 297, in <module>
scores = model(x_test, x_test_seq_length)
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "C:/Users/Administrator/PycharmProjects/untitled/example.py", line 141, in forward
x = self.embed(x) # sequence_length(max_len), batch_size, embed_size
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\sparse.py", line 117, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1506, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'
It works fine on the training set, but I keep getting that error in the test set. I've been thinking for 10 hours.
What is problem??
It seems that your program is expecting to run using the GPU but is instead being run on the CPU. Make sure you have your GPU settings for your program set correctly and the version of CUDA you're using is up-to-date.
You can find more information on this here (Assuming you're using tensorflow):
https://www.tensorflow.org/install/gpu
If you trained the model on your CPU, then the test data was loaded and converted to the CUDA datatype somehow. So, you can solve this issue by moving your input tensors to CPU device. And move the model as well (it won't hurt).
This can be done like so:
>>> import torch
>>> device = torch.device("cpu")
>>> # move the model
>>> model = model.to(device)
>>> # move any input tensors
>>> test_data = test_data.to(device)
I am new to CNN and machine learning in general and have been trying to follow Image Classification tutorial of TensorFlow.
Now, the Google Colab can be found here. I've followed the this official tutorial of TensorFlow. And I've changed it slightly so it saves the model as h5 instead of tf format so I can use the Keras' model.predict_classes.
Now, I have the model trained and the model reloaded from the saved model alright. But I'm repeatedly getting list index out of range error whenever I am trying to predict the image which I am doing so:
def predict():
image = tf.io.read_file('target.jpeg')
image = tf.image.decode_jpeg(image, channels=3)
image = tf.image.resize(image, [224, 224])
print(model.predict_classes(image)[0])
target.jpeg is one of the images I took from the flowers_photos dataset on which the model is trained.
The Traceback is:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in predict
File "/home/amitjoki/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/sequential.py", line 319, in predict_classes
proba = self.predict(x, batch_size=batch_size, verbose=verbose)
File "/home/amitjoki/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 821, in predict
use_multiprocessing=use_multiprocessing)
File "/home/amitjoki/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 712, in predict
callbacks=callbacks)
File "/home/amitjoki/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 187, in model_iteration
f = _make_execution_function(model, mode)
File "/home/amitjoki/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 555, in _make_execution_function
return model._make_execution_function(mode)
File "/home/amitjoki/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 2037, in _make_execution_function
self._make_predict_function()
File "/home/amitjoki/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 2027, in _make_predict_function
**kwargs)
File "/home/amitjoki/.local/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3544, in function
return EagerExecutionFunction(inputs, outputs, updates=updates, name=name)
File "/home/amitjoki/.local/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3468, in __init__
self.outputs[0] = array_ops.identity(self.outputs[0])
IndexError: list index out of range
I searched extensively but couldn't get any solution. It would be helpful if anyone can point me in the direction of getting this up and running.
All the predict functions in Keras expect batch of inputs. Therefore, since you are doing prediction on one single image, you need to add an axis at the beginning of image tensor to represent the batch axis:
image = tf.expand_dims(image, axis=0) # the shape would be (1, 224, 224, 3)
print(model.predict_classes(image)[0])
I have a trained model which I am loading using CNTK.load_model() function. I was looking at the MNIST Tutorial on the CNTK git repo as reference for model evaluation code. I have created a data reader (which is a MinibatchSource object) and trying to run model.eval(mb) where mb = minibatch_source.next_minibatch(...) (Similar to this answer)
But, I'm getting the following error message
Traceback (most recent call last):
File "LID_test.py", line 162, in <module>
test_and_evaluate()
File "LID_test.py", line 159, in test_and_evaluate
predictions = model.eval(mb)
File "/home/t-asbahe/anaconda3/envs/cntk-py35/lib/python3.5/site-packages/cntk/ops/functions.py", line 228, in eval
_, output_map = self.forward(arguments, self.outputs, device=device, as_numpy=as_numpy)
File "/home/t-asbahe/anaconda3/envs/cntk-py35/lib/python3.5/site-packages/cntk/utils/swig_helper.py", line 62, in wrapper
result = f(*args, **kwds)
File "/home/t-asbahe/anaconda3/envs/cntk-py35/lib/python3.5/site-packages/cntk/ops/functions.py", line 354, in forward
None, device)
File "/home/t-asbahe/anaconda3/envs/cntk-py35/lib/python3.5/site-packages/cntk/utils/__init__.py", line 393, in sanitize_var_map
if len(arguments) < len(op_arguments):
TypeError: object of type 'Variable' has no len()
I have no input_variable named 'Variable' in my model and I don't see any reason to get this error.
P.S.: My inputs are sparse inputs (one-hots)
You have a few options:
Pass a set of data as numpy array (instance in CNTK 202 tutorial) where onehot data is passed in as a numpy array.
pred = model.eval({model.arguments[0]:[onehot]})
Read the minibatch data and pass it to the eval function
eval_input_map = { input : reader_eval.streams.features }
eval_data = reader_eval.next_minibatch(eval_minibatch_size,
input_map = eval_input_map)
mydata = eval_data[input].value
predicted= model.eval(mydata)