Develop Reference

Tensorflow segmentation fault in dataset generation

I'm running batch-wise predictions on a pre-trained model. The error is raised during the dataset creation process, specifically in a tf.data.Dataset.map() function.
This is the offending function-
paths_dataset = tf.data.Dataset.from_tensor_slices(file_list)
# #tf.autograph.experimental.do_not_convert
def path_to_wav(path):
audio_bytes = tf.io.read_file(path)
wav, sr = tf.audio.decode_wav(audio_bytes, desired_channels=1, desired_samples=example_sample_length)
wav = tfio.audio.resample(wav, rate_in=tf.cast(sr, dtype=tf.int64), rate_out=resample_rate)
return tf.transpose(wav, perm=[1, 0])
wavs_dataset = paths_dataset.map(path_to_wav)
It has been tough to debug because the script works fine in a debugger and in a notebook. It only throws an error if I run the script via terminal or send the job to a cluster node-
WARNING:tensorflow:AutoGraph could not transform <function resample at 0x2b0dc6249160> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: std::bad_cast
To silence this warning, decorate the function with #tf.autograph.experimental.do_not_convert
srun: error: node416: task 0: Segmentation fault
Adding the #tf.autograph.experimental.do_not_convert decorator suppresses the warning but throws the following error-
File "/home/s/ss645/mlos/train/estimates_dataset.py", line 120, in simple_dataset
wavs_dataset = paths_dataset.map(path_to_wav)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2048, in map
return MapDataset(self, map_func, preserve_cardinality=True, name=name)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 5243, in __init__
self._map_func = structured_function.StructuredFunctionWrapper(
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/data/ops/structured_function.py", line 271, in __init__
self._function = fn_factory()
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2567, in get_concrete_function
graph_function = self._get_concrete_function_garbage_collected(
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2533, in _get_concrete_function_garbage_collected
graph_function, _ = self._maybe_define_function(args, kwargs)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2711, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 2627, in _create_graph_function
func_graph_module.func_graph_from_py_func(
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py", line 1141, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/data/ops/structured_function.py", line 248, in wrapped_fn
ret = wrapper_helper(*args)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/data/ops/structured_function.py", line 177, in wrapper_helper
ret = autograph.tf_convert(self._func, ag_ctx)(*nested_args)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 642, in wrapper
return func(*args, **kwargs)
File "/home/s/ss645/mlos/train/estimates_dataset.py", line 115, in path_to_wav
wav = tfio.audio.resample(wav, rate_in=sr, rate_out=resample_rate)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow_io/python/ops/audio_ops.py", line 462, in resample
value = tf.vectorized_map(f, input)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 549, in vectorized_map
return pfor(loop_fn, batch_size,
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 206, in pfor
outputs = f()
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 187, in f
return _pfor_impl(loop_fn,
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 286, in _pfor_impl
loop_fn_outputs = f(loop_var)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 642, in wrapper
return func(*args, **kwargs)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 530, in loop_fn
return fn(gathered_elems)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow_io/python/ops/audio_ops.py", line 458, in f
return core_ops.io_audio_resample(
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow_io/python/ops/__init__.py", line 88, in __getattr__
return getattr(self._load(), attrb)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow_io/python/ops/__init__.py", line 84, in _load
self._mod = _load_library(self._library)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow_io/python/ops/__init__.py", line 64, in _load_library
l = load_fn(f)
File "/home/s/ss645/miniconda3/envs/fsdm/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
RuntimeError: std::bad_cast
srun: error: node650: task 0: Exited with exit code 1
Any help would be much appreciated! I'm currently unable to file a Tensorflow issue because the error is not reproducible in a notebook.

Can't load the model from checkpoint

Using object detection API and mask-rcnn from the model zoo, when I start training with the default command:
python model_main_tf2.py --pipeline_config_path=pipeline.config --model_dir=model/v1/
I get this error
Two checkpoint references resolved to different objects (<tensorflow.python.keras.layers.convolutional.Conv2D object at 0x7f5e94075c10> and <tensorflow.python.keras.layers.convolutional.Conv2D object at 0x7f5e9403b710>).
Traceback (most recent call last):
File "model_main_tf2.py", line 115, in <module>
tf.compat.v1.app.run()
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/opt/anaconda/envs/tensorflow2.3/lib/python3.7/site-packages/absl/app.py", line 300, in run
_run_main(main, args)
File "/opt/anaconda/envs/tensorflow2.3/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "/home/TF/workspace/Antenna/model_main_tf2.py", line 112, in main
record_summaries=FLAGS.record_summaries)
File "/home/TF/models/research/object_detection/model_lib_v2.py", line 603, in train_loop
train_input, unpad_groundtruth_tensors)
File "/home/TF/models/research/object_detection/model_lib_v2.py", line 401, in load_fine_tune_checkpoint
checkpoint_path).expect_partial().assert_existing_objects_matched()
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/util.py", line 1721, in restore
status = self._saver.restore(save_path=save_path)
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/util.py", line 1320, in restore
checkpoint=checkpoint, proto_id=0).restore(self._graph_view.root)
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/base.py", line 209, in restore
restore_ops = trackable._restore_from_checkpoint_position(self) # pylint: disable=protected-access
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/base.py", line 914, in _restore_from_checkpoint_position
tensor_saveables, python_saveables))
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/training/tracking/util.py", line 297, in restore_saveables
validated_saveables).restore(self.save_path_tensor, self.options)
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/training/saving/functional_saver.py", line 340, in restore
restore_ops = restore_fn()
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/training/saving/functional_saver.py", line 316, in restore_fn
restore_ops.update(saver.restore(file_prefix, options))
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/training/saving/functional_saver.py", line 111, in restore
restored_tensors, restored_shapes=None)
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/distribute/values.py", line 890, in restore
for v in self._mirrored_variable.values))
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/distribute/values.py", line 890, in <genexpr>
for v in self._mirrored_variable.values))
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/distribute/values_util.py", line 195, in assign_on_device
return variable.assign(tensor)
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 858, in assign
self._shape.assert_is_compatible_with(value_tensor.shape)
File "/home/.local/lib/python3.7/site-packages/tensorflow/python/framework/tensor_shape.py", line 1134, in assert_is_compatible_with
raise ValueError("Shapes %s and %s are incompatible" % (self, other))
ValueError: Shapes (1, 1, 1088, 256) and (1, 1, 1024, 512) are incompatible
I use the default config from tf-api models/research/object-detection/configs
I have already tried to fix it by using config-file from the model zoo or fine_tune_checkpoint_type: "detection"

The input shape in the pipeline.config file doesn't match the one from the model checkpoint

ValueError: A `Concatenate` layer should be called on a list of at least 2 inputs

I'm trying to use a sigmoid to join the output of two models with different embedding matrix. but I keep getting the error at the concatenate line. I have tried other suggestions from similar questions but it keeps giving the same error. I feel I'm missing something but I can't find it. please help explain. Thanks
############################ MODEL 1 ######################################
input_tensor=Input(shape=(35,))
input_layer= Embedding(vocab_size, 300, input_length=35, weights=[embedding_matrix],trainable=True)(input_tensor)
conv_blocks = []
filter_sizes = (2,3,4)
for fx in filter_sizes:
conv_layer= Conv1D(100, kernel_size=fx, activation='relu', data_format='channels_first')(input_layer) #filters=100, kernel_size=3
maxpool_layer = MaxPooling1D(pool_size=4)(conv_layer)
flat_layer= Flatten()(maxpool_layer)
conv_blocks.append(flat_layer)
conc_layer=concatenate(conv_blocks, axis=1)
graph = Model(inputs=input_tensor, outputs=conc_layer)
model = Sequential()
model.add(graph)
model.add(Dropout(0.2))
############################ MODEL 2 ######################################
input_tensor_1=Input(shape=(35,))
input_layer_1= Embedding(vocab_size, 300, input_length=35, weights=[embedding_matrix_1],trainable=True)(input_tensor_1)
conv_blocks_1 = []
filter_sizes_1 = (2,3,4)
for fx in filter_sizes_1:
conv_layer_1= Conv1D(100, kernel_size=fx, activation='relu', data_format='channels_first')(input_layer_1) #filters=100, kernel_size=3
maxpool_layer_1 = MaxPooling1D(pool_size=4)(conv_layer_1)
flat_layer_1= Flatten()(maxpool_layer_1)
conv_blocks_1.append(flat_layer_1)
conc_layer_1=concatenate(conv_blocks_1, axis=1)
graph_1 = Model(inputs=input_tensor_1, outputs=conc_layer_1)
model_1 = Sequential()
model_1.add(graph_1)
model_1.add(Dropout(0.2))
fused = concatenate([graph, graph_1], axis=-1)
prediction = Dense(3, activation='sigmoid')(fused)
model = Model(inputs=[input_tensor,input_tensor_1], outputs=[prediction])
model.compile(loss='sparse_categorical_crossentropy',optimizer='Adagrad', metrics=['accuracy'])
model.summary()
This is the error trace
Traceback (most recent call last):
File "DL_Ensemble.py", line 145, in <module>
fused = concatenate([graph, graph_1], axis= 1 )
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/keras/layers/merge.py", line 705, in concatenate
return Concatenate(axis=axis, **kwargs)(inputs)
File "/usr/pkg/lib/python3.8/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 887, in __call__
self._maybe_build(inputs)
File "/usr/pkg/lib/python3.8/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2141, in _maybe_build
self.build(input_shapes)
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/keras/utils/tf_utils.py", line 306, in wrapper
output_shape = fn(instance, input_shape)
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/keras/layers/merge.py", line 378, in build
raise ValueError('A `Concatenate` layer should be called '
ValueError: A `Concatenate` layer should be called on a list of at least 2 inputs
UPDATE: I have reflected the answers given by #VivekMehta, however, I have this error.
File "DL_Ensemble.py", line 165, in <module>
model.fit([train_sequences,train_sequences], train_y, epochs=10,
verbose=False, batch_size=32, class_weight={0: 6.0, 1: 1.0, 2: 2.0})
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/keras/engine/training.py", line 709, in fit
return func.fit(
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/keras/engine/training_v2.py", line 313, in fit
training_result = run_one_epoch(
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/keras/engine/training_v2.py", line 123, in run_one_epoch
batch_outs = execution_function(iterator)
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/keras/engine/training_v2_utils.py",
line
86, in execution_function
distributed_function(input_fn))
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
result = self._call(*args, **kwds)
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/eager/def_function.py", line 520, in _call
return self._stateless_fn(*args, **kwds)
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/eager/function.py", line 1823, in __call__
return graph_function._filtered_call(args, kwargs) # pylint:
disable=protected-access
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/eager/function.py", line 1137, in _filtered_call
return self._call_flat(
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/eager/function.py", line 1223, in _call_flat
flat_outputs = forward_function.call(
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/eager/function.py", line 506, in call
outputs = execute.execute(
File "/usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError:
Conv2DCustomBackpropInputOp only supports NHWC.
[[node Conv2DBackpropInput (defined at /usr/pkg/lib/python3.8/site-
packages/tensorflow_core/python/framework/ops.py:1751) ]] [Op:__inference_distributed_function_2250]
Function call stack:
distributed_function
I also wanted to add that when the code is run on a GPU as opposed to a CPU, the error occurs on the same line as before but the message changes to :
File "DL_Ensemble.py", line 166, in <module>
model.fit([train_sequences,train_sequences], train_y, epochs=10, verbose=False, batch_size=32, class_weight={0: 6.0, 1: 1.0, 2: 2.0})
File "/home/kosimadukwe/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 880, in fit
validation_steps=validation_steps)
File "/home/kosimadukwe/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 329, in model_iteration
batch_outs = f(ins_batch)
File "/home/kosimadukwe/.local/lib/python3.7/site-packages/tensorflow/python/keras/backend.py", line 3073, in __call__
self._make_callable(feed_arrays, feed_symbols, symbol_vals, session)
File "/home/kosimadukwe/.local/lib/python3.7/site-packages/tensorflow/python/keras/backend.py", line 3019, in _make_callable
callable_fn = session._make_callable_from_options(callable_opts)
File "/home/kosimadukwe/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1471, in _make_callable_from_options
return BaseSession._Callable(self, callable_options)
File "/home/kosimadukwe/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1425, in __init__
session._session, options_ptr, status)
File "/home/kosimadukwe/.local/lib/python3.7/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC.
[[{{node training/Adagrad/gradients/conv1d_5/conv1d/Conv2D_grad/Conv2DBackpropInput}}]]
Exception ignored in: <function BaseSession._Callable.__del__ at 0x7fe4dd06a730>
Traceback (most recent call last):
File "/home/kosimadukwe/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1455, in __del__
self._session._session, self._handle, status)
File "/home/kosimadukwe/.local/lib/python3.7/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: No such callable handle: 94697914208640

So from you stack trace, code is throwing error at:
fused = concatenate([graph, graph_1], axis= 1 )
print(type(graph))
# output: <class 'tensorflow.python.keras.engine.training.Model'>
This error is coming because concatenate expects list of tensors to be concatenated. While you are passing graph and graph_1 which is not tensor but a Model instance.
So from your code I assume that you want to concatenate output of these two models. In that case you'll have to change above line to:
fused = concatenate([graph.outputs[0], graph_1.outputs[0]], axis=-1)
Here, graph.outputs gives list of outputs by given by Model. Since each model is giving us one output, we will take 0th index from each output.
Change this part and you'll get model summary as you are expecting.

Tensorflow Dataset map: input seems to be a placeholder which causes an error in tf.read_file

I'm trying to read in data using the Tensorflow Dataset API. I have loaded filenames and label filenames into arrays which I load into a dataset. I then try to map these filenames to the actual image files, but get an error that seems to state that the input to the mapping function recieves placeholders rather than actual tensors.
class DatasetReader:
def __init__(self, records_list, batch_size=1):
self.batch_size = batch_size
self.records = {}
self.records["image"] = tf.convert_to_tensor([record['image'] for record in records_list])
self.records["filename"] = tf.convert_to_tensor([record['filename'] for record in records_list])
self.records["annotation"] = tf.convert_to_tensor([record['annotation'] for record in records_list])
self.dataset = Dataset.from_tensor_slices(self.records)
self.dataset = self.dataset.map(self._input_parser)
self.dataset = self.dataset.batch(batch_size)
self.dataset = self.dataset.repeat()
def _input_parser(self, record):
filename = record['filename']
image_name = record['image']
annotation_file = record['annotation']
image = tf.image.decode_image(tf.read_file(filename))
annotation = tf.image.decode_image(tf.read_file(annotation_file))
return self._augment_image(image, annotation)
The error I'm getting is in the line image = tf.image.decode_image(tf.read_file(filename)). The stack trace is below.
File "FCN.py", line 269, in <module>
tf.app.run()
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "FCN.py", line 179, in main
train_records, valid_records, image_options_train, image_options_val, FLAGS.batch_size, FLAGS.batch_size)
File "/home/ubuntu/FCN.tensorflow/TFReader.py", line 89, in from_records
train_reader = DatasetReader(train_records, train_image_options, train_batch_size)
File "/home/ubuntu/FCN.tensorflow/TFReader.py", line 34, in __init__
self.dataset = self.dataset.map(self._input_parser)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py", line 964, in map
return MapDataset(self, map_func, num_threads, output_buffer_size)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py", line 1735, in __init__
self._map_func.add_to_graph(ops.get_default_graph())
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/function.py", line 449, in add_to_graph
self._create_definition_if_needed()
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/data/python/framework/function.py", line 168, in _create_definition_if_needed
outputs = self._func(*inputs)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py", line 1723, in tf_map_func
ret = map_func(nested_args)
File "/home/ubuntu/FCN.tensorflow/TFReader.py", line 42, in _input_parser
image = tf.image.decode_image(tf.read_file(filename))
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 223, in read_file
result = _op_def_lib.apply_op("ReadFile", filename=filename, name=name)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/data/python/framework/function.py", line 80, in create_op
data_types, **kwargs)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/function.py", line 665, in create_op
**kwargs)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2632, in create_op
set_shapes_for_outputs(ret)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1911, in set_shapes_for_outputs
shapes = shape_func(op)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1861, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.py", line 595, in call_cpp_shape_fn
require_shape_fn)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.py", line 659, in _call_cpp_shape_fn_impl
raise ValueError(err.message)
ValueError: Shape must be rank 0 but is rank 1 for 'ReadFile' (op: 'ReadFile') with input shapes: [?].

You cannot pass in a rank-1 tensor to tf.read_file. Here are some examples:
import tensorflow as tf
# Correct: input can be a string.
tf.image.decode_image(tf.read_file("filename"))
# Correct: input can be a rank-0 tensor.
tf.image.decode_image(tf.read_file(tf.convert_to_tensor("filename")))
# Wrong: input cannot be a list.
tf.image.decode_image(tf.read_file(["filename"]))
# Wrong: input cannot be a rank-1 tensor
tf.image.decode_image(tf.read_file(tf.convert_to_tensor(["filename"])))
In your code, it seems like self.records["filename"] is a rank-1 tensor; you might mistakenly passed it as a parameter to tf.read_file in _input_parser

Output files returned after training a alexnet model...?

Code is written in Python 3.5.X
Please try to make the answer simple for a 3rd year Computer Science Student
The output files from train_model.py seems to be a model.meta file but the test_model.py is asking for a .model file. The tutorial user has a .model file as well I can't seem to understand why i am getting a file with .model.meta
I am trying to play GTA San Andreas through Python or more precisely the car in GTA is driven by the model.
It takes screen frames as Input and recorded the key i Input during training. This training data is used to train the model.
Code for training the model
import numpy as np
from alexnet import alexnet
WIDTH = 80
HEIGHT = 60
LR = 1e-3
EPOCHS = 8
MODEL_NAME = 'pygta_sa-car-{}-{}-{}-epochs.model'.format(LR, 'alextnetv2', EPOCHS)
model = alexnet(WIDTH, HEIGHT, LR)
train_data = np.load('training_data_v2.npy')
train = train_data[:-500]
test = train_data[-500:]
X = np.array([i[0] for i in train]).reshape(-1,WIDTH,HEIGHT,1)
Y = [i[1] for i in train]
test_x = np.array([i[0] for i in test]).reshape(-1,WIDTH,HEIGHT,1)
test_y = [i[1] for i in test]
model.fit({'input': X}, {'targets': Y}, n_epoch=EPOCHS, validation_set=({'input': test_x}, {'targets': test_y}),
snapshot_step=500, show_metric=True, run_id=MODEL_NAME)
# tensorboard --logdir=foo:F:\play_gta_sa\log
model.save(MODEL_NAME)
training completes successfully and returns files
Files returned on the video of the Tutorial i am using to do this project
sent_dex files returned
Content of Checkpoint file
model_checkpoint_path: "F:\play_gta_sa\pygta_sa-car-0.001-alextnetv2-8-epochs.model"
all_model_checkpoint_paths: "F:\play_gta_sa\pygta_sa-car-0.001-alextnetv2-8-epochs.model"
Code for testing the model on the game
import numpy as np
import cv2
import time
from grabscreen import grab_screen
from getkeys import key_check
from directkeys import PressKey, ReleaseKey, W, A, S, D
from alexnet import alexnet
WIDTH = 80
HEIGHT = 60
LR = 1e-3
EPOCHS = 8
MODEL_NAME = 'pygta_sa-car-{}-{}-{}-epochs.model'.format(LR, 'alexnetv2',EPOCHS)
def straight():
PressKey(W)
ReleaseKey(A)
ReleaseKey(D)
def left():
PressKey(W)
PressKey(A)
ReleaseKey(D)
def right():
PressKey(W)
PressKey(D)
ReleaseKey(A)
model = alexnet(WIDTH, HEIGHT, LR)
model.load(MODEL_NAME)
def main():
for i in list(range(10))[::-1]:
print(i+1)
time.sleep(1)
last_time = time.time()
paused = False
while True:
if not paused:
screen = grab_screen(region=(0,40,800,640))
screen = cv2.cvtColor(screen,cv2.COLOR_BGR2GRAY)
screen = cv2.resize(screen,(80,60))
print('Frame took {} seconds'.format(time.time()-last_time))
last_time = time.time()
moves = list(np.around(model.predict([screen.reshape(80,60,1)])[0]))
print(moves, prediction)
if moves == [1,0,0]:
left()
elif moves == [0,1,0]:
straight()
elif moves == [0,0,1]:
right()
keys = key_check()
# p pauses game and can get annoying.
if 'T' in keys:
if paused:
paused = False
time.sleep(1)
else:
paused = True
ReleaseKey(A)
ReleaseKey(W)
ReleaseKey(D)
time.sleep(1)
main()
the error message on running test model
Traceback (most recent call last):
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1039, in _do_call
return fn(*args)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1021, in _run_fn
status, run_metadata)
File "C:\Program Files\Python35\lib\contextlib.py", line 66, in __exit__
next(self.gen)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for F:\play_gta_sa\pygta_sa-car-0.001-alexnetv2-8-epochs.model
[[Node: save_1/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/RestoreV2/tensor_names, save_1/RestoreV2/shape_and_slices)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "F:\play_gta_sa\test_model.py", line 33, in <module>
model.load(MODEL_NAME)
File "C:\Program Files\Python35\lib\site-packages\tflearn\models\dnn.py", line 282, in load
self.trainer.restore(model_file, weights_only, **optargs)
File "C:\Program Files\Python35\lib\site-packages\tflearn\helpers\trainer.py", line 452, in restore
self.restorer.restore(self.session, model_file)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 1457, in restore
{self.saver_def.filename_tensor_name: save_path})
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\client\session.py", line 778, in run
run_metadata_ptr)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\client\session.py", line 982, in _run
feed_dict_string, options, run_metadata)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1032, in _do_run
target_list, options, run_metadata)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1052, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for F:\play_gta_sa\pygta_sa-car-0.001-alexnetv2-8-epochs.model
[[Node: save_1/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/RestoreV2/tensor_names, save_1/RestoreV2/shape_and_slices)]]
Caused by op 'save_1/RestoreV2', defined at:
File "<string>", line 1, in <module>
File "C:\Program Files\Python35\lib\idlelib\run.py", line 124, in main
ret = method(*args, **kwargs)
File "C:\Program Files\Python35\lib\idlelib\run.py", line 351, in runcode
exec(code, self.locals)
File "F:\play_gta_sa\test_model.py", line 32, in <module>
model = alexnet(WIDTH, HEIGHT, LR)
File "F:\play_gta_sa\alexnet.py", line 40, in alexnet
max_checkpoints=1, tensorboard_verbose=0, tensorboard_dir='log')
File "C:\Program Files\Python35\lib\site-packages\tflearn\models\dnn.py", line 64, in __init__
best_val_accuracy=best_val_accuracy)
File "C:\Program Files\Python35\lib\site-packages\tflearn\helpers\trainer.py", line 147, in __init__
allow_empty=True)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 1056, in __init__
self.build()
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 1086, in build
restore_sequentially=self._restore_sequentially)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 691, in build
restore_sequentially, reshape)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 407, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 247, in restore_op
[spec.tensor.dtype])[0])
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 669, in restore_v2
dtypes=dtypes, name=name)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
op_def=op_def)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
self._traceback = _extract_stack()
NotFoundError (see above for traceback): Unsuccessful TensorSliceReader constructor: Failed to find any matching files for F:\play_gta_sa\pygta_sa-car-0.001-alexnetv2-8-epochs.model
[[Node: save_1/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save_1/Const_0, save_1/RestoreV2/tensor_names, save_1/RestoreV2/shape_and_slices)]]

Thanks for the update.
Edit:
Try add this line before tf.reset_default_graph() loading model. That is
import tensorflow as tf
tf.reset_default_graph()
model = alexnet(WIDTH, HEIGHT, LR)
model.load(MODEL_NAME)

The crux of this error is:
Unsuccessful TensorSliceReader constructor: Failed to find any matching files for F:\play_gta_sa\pygta_sa-car-0.001-alexnetv2-8-epochs.model
Okay, so a typical "file not found." Are we confident that we have this file? Maybe, but, if it was there, it would have been found. Our first guess should be we've typoed or otherwise made a mistake. Let's look at your model files:
For training the model, you have:
MODEL_NAME = 'pygta_sa-car-{}-{}-{}-epochs.model'.format(LR, 'alextnetv2', EPOCHS)
For testing the model, you have:
MODEL_NAME = 'pygta_sa-car-{}-{}-{}-epochs.model'.format(LR, 'alexnetv2',EPOCHS)
Do you see difference yet? There's a typo. alextnetv2 vs alexnetv2
Fix that, and the file will at least be found.