Develop Reference

Train machine learning model with JAX + ObJAX, met 'ValueError: Unable to cast Python instance to C++ type'

When I use JAX + ObJAX framework trained WRN model, there was an error: 'ValueError: Unable to cast Python instance to C++ type (compile in debug mode for details)', I don't know why...
Error information:
Traceback (most recent call last):
File "train.py", line 330, in <module>
app.run(main)
File "/home/shangjing/anaconda/yes/envs/python-tensorflow/lib/python3.6/site-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/home/shangjing/anaconda/yes/envs/python-tensorflow/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "train.py", line 306, in main
tm.train(FLAGS.epochs, len(xs), train, test, logdir, save_steps=FLAGS.save_steps, patience=FLAGS.patience)
File "train.py", line 91, in train
self.train_step(summary, next(train_iter), progress)
File "train.py", line 65, in train_step
kv = self.train_op(progress, np.array(data['image'].numpy()), np.array(data['label'].numpy()))
File "/home/shangjing/anaconda/yes/envs/python-tensorflow/lib/python3.6/site-packages/objax/module.py", line 258, in __call__
output, changes = self._call(self.vc.tensors(), kwargs, *args)
File "/home/shangjing/anaconda/yes/envs/python-tensorflow/lib/python3.6/site-packages/jax/api.py", line 416, in f_jitted
return cpp_jitted_f(context, *args, **kwargs)
ValueError: Unable to cast Python instance to C++ type (compile in debug mode for details)
The main code: https://github.com/tensorflow/privacy/tree/master/research/mi_lira_2021
I used python3.6.13+tensorflow2.4.0
I didn't change any of the code in train.py. And the command I used is: CUDA_VISIBLE_DEVICES='1' python3 -u train.py --dataset=cifar10 --epochs=100 --save_steps=20 --arch wrn28-2 --num_experiments 16 --expid 0
According to the Error Information, I tried to see line 65 in train.py: kv = self.train_op(progress, data['image'].numpy(), data['label'].numpy()) , and I thought maybe data['image'].numpy(), data['label'].numpy() have something wrong. But useless...

pyspark unable load pipelineModel

I met a problem, unable to load PipelineModel
I test my model in practice environment, but unable to apply this model and code on production environment
Traceback (most recent call last):
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 228, in <module>
main(xdr_input_file,model_file,xdr_output_file)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 215, in main
xdr_df_predict = xdr_predict(xdr_df,model_file)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 193, in xdr_predict
loadmodel = PipelineModel.load(model_input_path)
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 257, in load
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 197, in load
File "/usr/bch/1.5.0/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 79, in deco
pyspark.sql.utils.IllegalArgumentException: 'requirement failed: Error loading metadata: Expected class name org.apache.spark.ml.PipelineModel but found class name pyspark.ml.pipeline.PipelineModel'
21/12/01 12:01:06 INFO SparkContext: Invoking stop() from shutdown hook

Thanks all the help.I am a intern in bigdata industry.This is my first time to post in stackoverflow,i am sorry to post in unreguired method.
Finally,i sovled this problem to adjust my code from spark2.4 to spark2.2.
Here is details about this tracesback:
I test my code in test environment under the version of spark2.4 and python3.7; i meet error when i deploy it in product environment under the version of spark2.2 and python3.7.
I train model under product envirenment, this si Model generate error:
Traceback (most recent call last):
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/train/model_generate.py", line 331, in
main(xdr_file_path,jingfeng_file_path,save_model_path)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/train/model_generate.py", line 318, in main
tvs_piplineModel, gbdt_bestModel = generate_model(label_col, xdr_75109_String_title, union_df, save_model_path)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/train/model_generate.py", line 310, in generate_model
tvs_piplineModel.save(save_model_path)
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 217, in save
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 212, in write
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 100, in init
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 249, in _to_java
AttributeError: 'TrainValidationSplitModel' object has no attribute '_to_java'
when i skip model generate to model predict in model which i trained in test envirenment,this is Model predict error:
Traceback (most recent call last):
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 228, in
main(xdr_input_file,model_file,xdr_output_file)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 215, in main
xdr_df_predict = xdr_predict(xdr_df,model_file)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 193, in xdr_predict
loadmodel = PipelineModel.load(model_input_path)
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 257, in load
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 197, in load
File "/usr/bch/1.5.0/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in call
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 79, in deco
pyspark.sql.utils.IllegalArgumentException: 'requirement failed: Error loading metadata: Expected class name org.apache.spark.ml.PipelineModel but found class name pyspark.ml.pipeline.PipelineModel'
I check official document it explains ML persistence:version changes of ML persistence
So the error might casused by the vision of spark.
I ignore the "TrainValidationSplitModel" function which mention in first traceback,and it does work.My code run successfully.
Success run screenshot
Conclusion, my code aim to deploy a machine-learning classificaiton model in product environment. So i import pyspark.ml.gbdt to process dataframe. But i ignore the vision of test and product envirenoment.Thanks for all the help,this is the experience of an chinese intern.Forgive my pool expressive ability.

Running export_inference_graph.py throws value error

I am new to StackOverflow.I am trying to export a model by using export_inference_graph.py.
I trained my model locally using faster_rcnn_inception_v2.I am following this tutorial.
When in command prompt I type
python export_inference_graph.py --input_type image_tensor --pipeline_config_path CAPTCHA_training/faster_rcnn_inception_v2_coco.config --trained_checkpoint_prefix "CAPTCHA_training_dir/model.ckpt-51272" --output_directory CAPTCHA_inference_graph
All with correct paths I get following error.
File "export_inference_graph.py", line 206, in <module>
tf.app.run()
File "C:\Users\Jatin\anaconda3\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\Jatin\anaconda3\lib\site-packages\absl\app.py", line 303, in run
_run_main(main, args)
File "C:\Users\Jatin\anaconda3\lib\site-packages\absl\app.py", line 251, in _run_main
sys.exit(main(argv))
File "export_inference_graph.py", line 194, in main
exporter.export_inference_graph(
File "C:\Users\Jatin\anaconda3\lib\site-packages\object_detection\exporter.py", line 604, in export_inference_graph
detection_model = model_builder.build(pipeline_config.model,
File "C:\Users\Jatin\anaconda3\lib\site-packages\object_detection\builders\model_builder.py", line 1116, in build
return build_func(getattr(model_config, meta_architecture), is_training,
File "C:\Users\Jatin\anaconda3\lib\site-packages\object_detection\builders\model_builder.py", line 583, in _build_faster_rcnn_model
_check_feature_extractor_exists(frcnn_config.feature_extractor.type)
File "C:\Users\Jatin\anaconda3\lib\site-packages\object_detection\builders\model_builder.py", line 249, in _check_feature_extractor_exists
raise ValueError('{} is not supported. See `model_builder.py` for features '
ValueError: faster_rcnn_inception_v2 is not supported. See `model_builder.py` for features extractors compatible with different versions of Tensorflow
I am using Python 3.8.5 and tensorflow version 2.4.1
Thanks in advance

Looks like a Tensorflow version problem according to error. Looking into source faster_rcnn_inception_v2 exists under if tf_version.is_tf1():. Try using to TF 1.
https://github.com/tensorflow/models/blob/5a89897396aa8ecc7b3ef8919f987e96fc8d74db/research/object_detection/builders/model_builder.py#L70
https://github.com/tensorflow/models/blob/5a89897396aa8ecc7b3ef8919f987e96fc8d74db/research/object_detection/models/faster_rcnn_inception_resnet_v2_feature_extractor.py#L33
faster_rcnn_inception_v2_coco exists here in Tensorflow 1 Detection model zoo.
https://github.com/tensorflow/models/blob/5a89897396aa8ecc7b3ef8919f987e96fc8d74db/research/object_detection/g3doc/tf1_detection_zoo.md#coco-trained-models

Tensorflow- AttributeError: 'KeepAspectRatioResizer' object has no attribute 'per_channel_pad_value'

So, I trained an object detection model and now I want to export .ckpt files.
When I try to export the .ckpt files:
python export_inference_graph.py --input_type image_tensor --pipeline_config_path training/faster_rcnn_inception_v2_pets.config --trained_checkpoint_prefix training3/model.ckpt-47816 --output_directory inference_graph
I get this:
Traceback (most recent call last):
File "export_inference_graph.py", line 147, in <module>
tf.app.run()
File "/home/ubuntu/anaconda3/envs/tensorflow1/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "export_inference_graph.py", line 143, in main
FLAGS.output_directory, input_shape)
File "/home/ubuntu/tensorflow1/models/research/object_detection/exporter.py", line 454, in export_inference_graph
is_training=False)
File "/home/ubuntu/anaconda3/envs/tensorflow1/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/builders/model_builder.py", line 101, in build
add_summaries)
File "/home/ubuntu/anaconda3/envs/tensorflow1/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/builders/model_builder.py", line 274, in _build_faster_rcnn_model
image_resizer_fn = image_resizer_builder.build(frcnn_config.image_resizer)
File "/home/ubuntu/anaconda3/envs/tensorflow1/lib/python3.6/site-packages/object_detection-0.1-py3.6.egg/object_detection/builders/image_resizer_builder.py", line 83, in build
if keep_aspect_ratio_config.per_channel_pad_value:
AttributeError: 'KeepAspectRatioResizer' object has no attribute 'per_channel_pad_value'
It seems that everybody has this working fine and have no problems with this.
Could anyone please tell me what is going on here?

I know this is a few months later, but I just encountered this issue too!
It seems the image_resizer.proto is missing the per_channel_pad_value attribute.
Update the proto file to include the attribute, from here:
https://github.com/tensorflow/models/blob/master/research/object_detection/protos/image_resizer.proto
recompile it and then try again.
Should work this time.

Running export_inference_graph.py gives difference in types error

I am trying to export a model by executing export_inference_graph.py script.
I tried with my trained model.ckpt and official example files for ssd_mobilenet_v1_pets.
In cmd I type:
python export_inference_graph.py \ --input_type image_tensor \ --pipeline_config_path training/ssd_mobilenet_v1_pets.config \ --trained_checkpoint_prefix training/model.ckpt-2453 \ --output_directory heart_graph
I am using TensorFlow 1.4 and I always get the following error:
Traceback (most recent call last):
File "export_inference_graph.py", line 119, in <module>
tf.app.run()
File "C:\Users\<Name>\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "export_inference_graph.py", line 115, in main
FLAGS.output_directory, input_shape)
File "C:\Users\<Name>\AppData\Local\Programs\Python\Python35\Lib\site-packages\tensorflow\models\research\object_detection\exporter.py", line 427, in export_inference_graph
input_shape, optimize_graph, output_collection_name)
File "C:\Users\<Name>\AppData\Local\Programs\Python\Python35\Lib\site-packages\tensorflow\models\research\object_detection\exporter.py", line 353, in _export_inference_graph
postprocessed_tensors = detection_model.postprocess(output_tensors)
File "C:\Users\<Name>\AppData\Local\Programs\Python\Python35\Lib\site-packages\tensorflow\models\research\object_detection\meta_architectures\ssd_meta_arch.py", line 405, in postprocess
class_predictions_without_background)
File "C:\Users\<Name>\AppData\Local\Programs\Python\Python35\Lib\site-packages\tensorflow\models\research\object_detection\builders\post_processing_builder.py", line 94, in score_converter_fn
scaled_logits = tf.divide(logits, logit_scale, name='scale_logits')
File "C:\Users\<Name>\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\math_ops.py", line 309, in divide
return DivideDelegateWithName(x, name) / y
File "C:\Users\<Name>\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\math_ops.py", line 294, in __truediv__
return _truediv_python3(self.x, y, self.name)
File "C:\Users\<Name>\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\math_ops.py", line 981, in _truediv_python3
(x_dtype, y_dtype))
TypeError: x and y must have the same dtype, got tf.float32 != tf.int32
Where is the problem and how to solve this?

recently I used object detection to do some interesting things, and occur this error too, I look up the issues on the github and find the solutions.
In the https://github.com/tensorflow/models/issues/2774, there is a solution that changed the source code, and I try it. It WORKS!!
You can find the post_processing_builder.pyand change the function with
def _score_converter_fn_with_logit_scale(tf_score_converter_fn, logit_scale):
"""Create a function to scale logits then apply a Tensorflow function."""
def score_converter_fn(logits):
cr = logit_scale
cr = tf.constant([[cr]],tf.float32)
print(logit_scale)
print(logits)
scaled_logits = tf.divide(logits, cr, name='scale_logits') #change logit_scale
return tf_score_converter_fn(scaled_logits, name='convert_scores')
score_converter_fn.__name__ = '%s_with_logit_scale' % (
tf_score_converter_fn.__name__)
return score_converter_fn
Then go to the research folder, run
python setup.py install
Then it will be OK!
By the way, I don't know whether you should re-install the slim in the research folder, you'd better re-install it too.