The following line of code I am trying to run in PyCharm and I have tensorflow_hub installed and imported.
use = hub.load("https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3")
Any suggestions for the below error? As I need this for my project.
Traceback (most recent call last):
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3820, in _get_op_def
return self._op_def_cache[type]
KeyError: 'SentencepieceOp'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/Jon10/OneDrive/Documents/Computer Science/Dissertation/PythonPractice/TFTest/test.py", line 28, in <module>
use = hub.load("https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3")
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_hub\module_v2.py", line 102, in load
obj = tf_v1.saved_model.load_v2(module_path, tags=tags)
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\saved_model\load.py", line 517, in load
return load_internal(export_dir, tags)
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\saved_model\load.py", line 541, in load_internal
export_dir)
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\saved_model\load.py", line 114, in __init__
meta_graph.graph_def.library))
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\saved_model\function_deserialization.py", line 312, in load_function_def_library
copy, copy_functions=False)
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\framework\function_def_to_graph.py", line 61, in function_def_to_graph
fdef, input_shapes, copy_functions)
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\framework\function_def_to_graph.py", line 214, in function_def_to_graph_def
op_def = ops.get_default_graph()._get_op_def(node_def.op) # pylint: disable=protected-access
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3824, in _get_op_def
c_api.TF_GraphGetOpDef(self._c_graph, compat.as_bytes(type), buf)
tensorflow.python.framework.errors_impl.NotFoundError: Op type not registered 'SentencepieceOp' in binary running on DESKTOP-..... Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
You need to install tensorflow_text, and import it before using hub.load
Related
ERROR IS :
Traceback (most recent call last):
File "P:\Prashant\Python\Attendance\lib\site-packages\pyzbar\zbar_library.py", line 58, in load
dependencies, libzbar = load_objects(Path(''))
File "P:\Prashant\Python\Attendance\lib\site-packages\pyzbar\zbar_library.py", line 50, in load_objects
deps = [
File "P:\Prashant\Python\Attendance\lib\site-packages\pyzbar\zbar_library.py", line 51, in <listcomp>
cdll.LoadLibrary(str(directory.joinpath(dep)))
File "C:\Users\kalep\AppData\Local\Programs\Python\Python39\lib\ctypes\__init__.py", line 452, in LoadLibrary
return self._dlltype(name)
File "C:\Users\kalep\AppData\Local\Programs\Python\Python39\lib\ctypes\__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'libiconv.dll' (or one of its dependencies). Try using the full path with constructor syntax.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "P:\Prashant\CPP Project Planning\Rooturaj CPP PRoject\Attendance\attend.py", line 3, in <module>
from pyzbar .pyzbar import decode
File "P:\Prashant\Python\Attendance\lib\site-packages\pyzbar\pyzbar.py", line 7, in <module>
from .wrapper import (
File "P:\Prashant\Python\Attendance\lib\site-packages\pyzbar\wrapper.py", line 151, in <module>
zbar_version = zbar_function(
File "P:\Prashant\Python\Attendance\lib\site-packages\pyzbar\wrapper.py", line 148, in zbar_function
return prototype((fname, load_libzbar()))
File "P:\Prashant\Python\Attendance\lib\site-packages\pyzbar\wrapper.py", line 127, in load_libzbar
libzbar, dependencies = zbar_library.load()
File "P:\Prashant\Python\Attendance\lib\site-packages\pyzbar\zbar_library.py", line 60, in load
dependencies, libzbar = load_objects(Path(__file__).parent)
File "P:\Prashant\Python\Attendance\lib\site-packages\pyzbar\zbar_library.py", line 54, in load_objects
libzbar = cdll.LoadLibrary(str(directory.joinpath(fname)))
File "C:\Users\kalep\AppData\Local\Programs\Python\Python39\lib\ctypes\__init__.py", line 452, in LoadLibrary
return self._dlltype(name)
File "C:\Users\kalep\AppData\Local\Programs\Python\Python39\lib\ctypes\__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'P:\Prashant\Python\Attendance\lib\site-packages\pyzbar\libzbar-64.dll' (or one of its dependencies). Try using the full path with constructor syntax.
I am creating project on attendance system using QR Code scanning and I use Python to create it.
I used all needed libraries in the program
like, cv2, numpy, pybase64, pyzbar.
But I am getting this above Error in the Pyzbar.
I got solution for this issue. I was facing similar issue just installed Visual C++ 64 bit version on my computer and restarted. Choose between vcredist_x86.exe or vcredist_x64.exe based on your machine.
https://www.microsoft.com/en-us/download/confirmation.aspx?id=40784. Hope this solves your issue as well.
I met a problem, unable to load PipelineModel
I test my model in practice environment, but unable to apply this model and code on production environment
Traceback (most recent call last):
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 228, in <module>
main(xdr_input_file,model_file,xdr_output_file)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 215, in main
xdr_df_predict = xdr_predict(xdr_df,model_file)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 193, in xdr_predict
loadmodel = PipelineModel.load(model_input_path)
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 257, in load
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 197, in load
File "/usr/bch/1.5.0/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 79, in deco
pyspark.sql.utils.IllegalArgumentException: 'requirement failed: Error loading metadata: Expected class name org.apache.spark.ml.PipelineModel but found class name pyspark.ml.pipeline.PipelineModel'
21/12/01 12:01:06 INFO SparkContext: Invoking stop() from shutdown hook
Thanks all the help.I am a intern in bigdata industry.This is my first time to post in stackoverflow,i am sorry to post in unreguired method.
Finally,i sovled this problem to adjust my code from spark2.4 to spark2.2.
Here is details about this tracesback:
I test my code in test environment under the version of spark2.4 and python3.7; i meet error when i deploy it in product environment under the version of spark2.2 and python3.7.
I train model under product envirenment, this si Model generate error:
Traceback (most recent call last):
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/train/model_generate.py", line 331, in
main(xdr_file_path,jingfeng_file_path,save_model_path)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/train/model_generate.py", line 318, in main
tvs_piplineModel, gbdt_bestModel = generate_model(label_col, xdr_75109_String_title, union_df, save_model_path)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/train/model_generate.py", line 310, in generate_model
tvs_piplineModel.save(save_model_path)
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 217, in save
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 212, in write
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 100, in init
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 249, in _to_java
AttributeError: 'TrainValidationSplitModel' object has no attribute '_to_java'
when i skip model generate to model predict in model which i trained in test envirenment,this is Model predict error:
Traceback (most recent call last):
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 228, in
main(xdr_input_file,model_file,xdr_output_file)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 215, in main
xdr_df_predict = xdr_predict(xdr_df,model_file)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 193, in xdr_predict
loadmodel = PipelineModel.load(model_input_path)
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 257, in load
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 197, in load
File "/usr/bch/1.5.0/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in call
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 79, in deco
pyspark.sql.utils.IllegalArgumentException: 'requirement failed: Error loading metadata: Expected class name org.apache.spark.ml.PipelineModel but found class name pyspark.ml.pipeline.PipelineModel'
I check official document it explains ML persistence:version changes of ML persistence
So the error might casused by the vision of spark.
I ignore the "TrainValidationSplitModel" function which mention in first traceback,and it does work.My code run successfully.
Success run screenshot
Conclusion, my code aim to deploy a machine-learning classificaiton model in product environment. So i import pyspark.ml.gbdt to process dataframe. But i ignore the vision of test and product envirenoment.Thanks for all the help,this is the experience of an chinese intern.Forgive my pool expressive ability.
I want to load bookcorpus like this:
train_ds, test_ds = load_dataset('bookcorpus', split=['train', 'test']),
however, get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/marcelbraasch/.local/lib/python3.8/site-packages/datasets/load.py", line 1627, in load_dataset
builder_instance.download_and_prepare(
File "/home/marcelbraasch/.local/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
self._download_and_prepare(
File "/home/marcelbraasch/.local/lib/python3.8/site-packages/datasets/builder.py", line 709, in _download_and_prepare
verify_splits(self.info.splits, split_dict)
File "/home/marcelbraasch/.local/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 74, in verify_splits
raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=4853859824, num_examples=74004228, dataset_name='bookcorpus'), 'recorded': SplitInfo(name='train', num_bytes=2982081448, num_examples=45726619, dataset_name='bookcorpus')}]
I want to proceed to save this to disk as I don't want to download this every time I use it. What causes this error?
BookCorpus is no longer publicly available.
Here is a work around:
https://github.com/soskek/bookcorpus
I am trying to create connection to Hive hosted in HDInsight cluster through my python script and getting below error-
Traceback (most recent call last):
File "ClassLoader.java", line 357, in java.lang.ClassLoader.loadClass
File "Launcher.java", line 349, in sun.misc.Launcher$AppClassLoader.loadClass
File "ClassLoader.java", line 424, in java.lang.ClassLoader.loadClass
File "URLClassLoader.java", line 382, in java.net.URLClassLoader.findClass
java.lang.ClassNotFoundException: java.lang.ClassNotFoundException: org.apache.thrift.transport.TTransportException
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "org.jpype.JPypeContext.java", line 330, in org.jpype.JPypeContext.callMethod
File "Method.java", line 498, in java.lang.reflect.Method.invoke
File "DelegatingMethodAccessorImpl.java", line 43, in sun.reflect.DelegatingMethodAccessorImpl.invoke
File "NativeMethodAccessorImpl.java", line 62, in sun.reflect.NativeMethodAccessorImpl.invoke
File "NativeMethodAccessorImpl.java", line -2, in sun.reflect.NativeMethodAccessorImpl.invoke0
File "DriverManager.java", line 247, in java.sql.DriverManager.getConnection
File "DriverManager.java", line 664, in java.sql.DriverManager.getConnection
File "HiveDriver.java", line 105, in org.apache.hive.jdbc.HiveDriver.connect
Exception: Java Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "s.py", line 5, in <module>
"/root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar")
File "/usr/local/lib64/python3.6/site-packages/jaydebeapi/__init__.py", line 412, in connect
jconn = _jdbc_connect(jclassname, url, driver_args, jars, libs)
File "/usr/local/lib64/python3.6/site-packages/jaydebeapi/__init__.py", line 230, in _jdbc_connect_jpype
return jpype.java.sql.DriverManager.getConnection(url, *dargs)
java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: org/apache/thrift/transport/TTransportException
My Script is -
import jaydebeapi
conn = jaydebeapi.connect("org.apache.hive.jdbc.HiveDriver",
"jdbc:hive2://10.20.30.40:10001/default;transportMode=http;ssl=false;httpPath=/hive2",
["username", "password"],
"/root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar")
I have exported CLASSPATH wil all jar files.
The error is java.lang.ClassNotFoundException: java.lang.ClassNotFoundException which specifies that the execution is not able to find the jar /root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar.I believe it's only placed in a host from where you are executing the code. I would suggest placing the jar file in the same directory structure in all the nodes in the cluster and have a check on permissions so that the user executing the job has access to that path.
So I'm following sentdex's object detection tutorial and I have gotten to the step where you are supposed to export the inference graph. I'm using the "export_inference_graph.py" script from Tensorflow's object_detection folder.
The problem is that I'm getting this ValueError:
Traceback (most recent call last):
File "C:\Users\Zelcore-Dator\AppData\Local\Programs\Python\Python35\lib\site-packages\google\proto
buf\internal\python_message.py", line 545, in _GetFieldByName
return message_descriptor.fields_by_name[field_name]
KeyError: 'layout_optimizer'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "export_inference_graph.py", line 119, in <module>
tf.app.run()
File "C:\Users\Zelcore-Dator\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\p
ython\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "export_inference_graph.py", line 115, in main
FLAGS.output_directory, input_shape)
File "C:\Users\Zelcore-Dator\AppData\Local\Programs\Python\Python35\lib\site-packages\object_detec
tion-0.1-py3.5.egg\object_detection\exporter.py", line 427, in export_inference_graph
input_shape, optimize_graph, output_collection_name)
File "C:\Users\Zelcore-Dator\AppData\Local\Programs\Python\Python35\lib\site-packages\object_detec
tion-0.1-py3.5.egg\object_detection\exporter.py", line 391, in _export_inference_graph
initializer_nodes='')
File "C:\Users\Zelcore-Dator\AppData\Local\Programs\Python\Python35\lib\site-packages\object_detec
tion-0.1-py3.5.egg\object_detection\exporter.py", line 72, in freeze_graph_with_def_protos
layout_optimizer=rewriter_config_pb2.RewriterConfig.ON)
File "C:\Users\Zelcore-Dator\AppData\Local\Programs\Python\Python35\lib\site-packages\google\proto
buf\internal\python_message.py", line 484, in init
field = _GetFieldByName(message_descriptor, field_name)
File "C:\Users\Zelcore-Dator\AppData\Local\Programs\Python\Python35\lib\site-packages\google\proto
buf\internal\python_message.py", line 548, in _GetFieldByName
(message_descriptor.name, field_name))
ValueError: Protocol message RewriterConfig has no "layout_optimizer" field.
I'm guessing that it has something to do with protobuf, but I've reinstalled it several times already with no success.
All help appreciated
Happened to me too. Didn't happen few weeks ago.
Until the bug is fixed, you could use an earlier version that still works.
replace line 72 in 'object_detection/exporter.py':
layout_optimizer=rewriter_config_pb2.RewriterConfig.ON)
with the old and working line:
optimize_tensor_layout=True)
I used:
rewrite_options = rewriter_config_pb2.RewriterConfig(optimize_tensor_layout=True)
but kept running into the same issue UNTIL I went and reran
python setup.py install
from my "research" folder. Then I was able to get everything to work.
Remove optimize_tensor_layout=rewriter_config_pb2.RewriterConfig.ON
change the line 71 in exporter.py
rewrite_options = rewriter_config_pb2.RewriterConfig(optimize_tensor_layout=rewriter_config_pb2.RewriterConfig.ON)
to:
rewrite_options = rewriter_config_pb2.RewriterConfig()