I am using python and pymssql for SQL connection.
Basically i am fetching some data from 'table1' and doing some
Mathematical calculation and inserting into 'table2'
I have tested my python script with limited number of
records from 'table1' and it was working fine.
Then I have started my script with real time data,ie really huge
amount of data(122457 records)
Then I have faced given below issue
Traceback (most recent call last):
File "trakryt_main_mod.py", line 312, in <module>
create_visitwalkby_table()
File "trakryt_main_mod.py", line 126, in create_visitwalkby_table
scenarios.one_beacon(cursor,orgid)
File "C:\Nikhil\trak_reporting\python\scenarios.py", line 260, in one_beacon
close_visit()
File "C:\Nikhil\trak_reporting\python\scenarios.py", line 207, in close_visit
conn = pymssql.connect(server,user,password,db,port=port)
File "pymssql.pyx", line 645, in pymssql.connect (pymssql.c:10301)
File "pymssql.pyx", line 247, in pymssql.Connection.__init__ (pymssql.c:3771)
pymssql.OperationalError: Cannot start transaction: (20004, 'DB-Lib error messag
e 20004, severity 9:\nRead from the server failed\nNet-Lib error during Unknown
error (10054)\n')
Related
I met a problem, unable to load PipelineModel
I test my model in practice environment, but unable to apply this model and code on production environment
Traceback (most recent call last):
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 228, in <module>
main(xdr_input_file,model_file,xdr_output_file)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 215, in main
xdr_df_predict = xdr_predict(xdr_df,model_file)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 193, in xdr_predict
loadmodel = PipelineModel.load(model_input_path)
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 257, in load
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 197, in load
File "/usr/bch/1.5.0/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 79, in deco
pyspark.sql.utils.IllegalArgumentException: 'requirement failed: Error loading metadata: Expected class name org.apache.spark.ml.PipelineModel but found class name pyspark.ml.pipeline.PipelineModel'
21/12/01 12:01:06 INFO SparkContext: Invoking stop() from shutdown hook
Thanks all the help.I am a intern in bigdata industry.This is my first time to post in stackoverflow,i am sorry to post in unreguired method.
Finally,i sovled this problem to adjust my code from spark2.4 to spark2.2.
Here is details about this tracesback:
I test my code in test environment under the version of spark2.4 and python3.7; i meet error when i deploy it in product environment under the version of spark2.2 and python3.7.
I train model under product envirenment, this si Model generate error:
Traceback (most recent call last):
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/train/model_generate.py", line 331, in
main(xdr_file_path,jingfeng_file_path,save_model_path)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/train/model_generate.py", line 318, in main
tvs_piplineModel, gbdt_bestModel = generate_model(label_col, xdr_75109_String_title, union_df, save_model_path)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/train/model_generate.py", line 310, in generate_model
tvs_piplineModel.save(save_model_path)
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 217, in save
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 212, in write
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 100, in init
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 249, in _to_java
AttributeError: 'TrainValidationSplitModel' object has no attribute '_to_java'
when i skip model generate to model predict in model which i trained in test envirenment,this is Model predict error:
Traceback (most recent call last):
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 228, in
main(xdr_input_file,model_file,xdr_output_file)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 215, in main
xdr_df_predict = xdr_predict(xdr_df,model_file)
File "/home/fwfx_yaofei/telbd-yjy/src/ml/complain_user_it/predict/model_predict.py", line 193, in xdr_predict
loadmodel = PipelineModel.load(model_input_path)
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 257, in load
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/ml/util.py", line 197, in load
File "/usr/bch/1.5.0/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in call
File "/usr/bch/1.5.0/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 79, in deco
pyspark.sql.utils.IllegalArgumentException: 'requirement failed: Error loading metadata: Expected class name org.apache.spark.ml.PipelineModel but found class name pyspark.ml.pipeline.PipelineModel'
I check official document it explains ML persistence:version changes of ML persistence
So the error might casused by the vision of spark.
I ignore the "TrainValidationSplitModel" function which mention in first traceback,and it does work.My code run successfully.
Success run screenshot
Conclusion, my code aim to deploy a machine-learning classificaiton model in product environment. So i import pyspark.ml.gbdt to process dataframe. But i ignore the vision of test and product envirenoment.Thanks for all the help,this is the experience of an chinese intern.Forgive my pool expressive ability.
I am trying to create connection to Hive hosted in HDInsight cluster through my python script and getting below error-
Traceback (most recent call last):
File "ClassLoader.java", line 357, in java.lang.ClassLoader.loadClass
File "Launcher.java", line 349, in sun.misc.Launcher$AppClassLoader.loadClass
File "ClassLoader.java", line 424, in java.lang.ClassLoader.loadClass
File "URLClassLoader.java", line 382, in java.net.URLClassLoader.findClass
java.lang.ClassNotFoundException: java.lang.ClassNotFoundException: org.apache.thrift.transport.TTransportException
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "org.jpype.JPypeContext.java", line 330, in org.jpype.JPypeContext.callMethod
File "Method.java", line 498, in java.lang.reflect.Method.invoke
File "DelegatingMethodAccessorImpl.java", line 43, in sun.reflect.DelegatingMethodAccessorImpl.invoke
File "NativeMethodAccessorImpl.java", line 62, in sun.reflect.NativeMethodAccessorImpl.invoke
File "NativeMethodAccessorImpl.java", line -2, in sun.reflect.NativeMethodAccessorImpl.invoke0
File "DriverManager.java", line 247, in java.sql.DriverManager.getConnection
File "DriverManager.java", line 664, in java.sql.DriverManager.getConnection
File "HiveDriver.java", line 105, in org.apache.hive.jdbc.HiveDriver.connect
Exception: Java Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "s.py", line 5, in <module>
"/root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar")
File "/usr/local/lib64/python3.6/site-packages/jaydebeapi/__init__.py", line 412, in connect
jconn = _jdbc_connect(jclassname, url, driver_args, jars, libs)
File "/usr/local/lib64/python3.6/site-packages/jaydebeapi/__init__.py", line 230, in _jdbc_connect_jpype
return jpype.java.sql.DriverManager.getConnection(url, *dargs)
java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: org/apache/thrift/transport/TTransportException
My Script is -
import jaydebeapi
conn = jaydebeapi.connect("org.apache.hive.jdbc.HiveDriver",
"jdbc:hive2://10.20.30.40:10001/default;transportMode=http;ssl=false;httpPath=/hive2",
["username", "password"],
"/root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar")
I have exported CLASSPATH wil all jar files.
The error is java.lang.ClassNotFoundException: java.lang.ClassNotFoundException which specifies that the execution is not able to find the jar /root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar.I believe it's only placed in a host from where you are executing the code. I would suggest placing the jar file in the same directory structure in all the nodes in the cluster and have a check on permissions so that the user executing the job has access to that path.
I am trying to create a table using Happybase. To start I enter the following commands get Hbase and Thrift running:
start-hbase.sh
hbase thrift start &
Once this is running I open Python's command prompt and type the following:
import happybase as hb
connection = hb.Connection()
connection.open()
However when I try to create a table:
connection.create_table(
'mytable',
{'cf1': dict(max_versions=10),
'cf2': dict(max_versions=1, block_cache_enabled=False),
'cf3': dict(), # use defaults
}
I get the following error that I just don't understand.
Traceback (most recent call last):
File "<stdin>", line 5, in <module>
File "/usr/local/lib/python2.7/dist-packages/happybase/connection.py", line 309, in create_table
self.client.createTable(name, column_descriptors)
File "/usr/local/lib/python2.7/dist-packages/thriftpy/thrift.py", line 198, in _req
return self._recv(_api)
File "/usr/local/lib/python2.7/dist-packages/thriftpy/thrift.py", line 210, in _recv
fname, mtype, rseqid = self._iprot.read_message_begin()
File "thriftpy/protocol/cybin/cybin.pyx", line 429, in cybin.TCyBinaryProtocol.read_message_begin (thriftpy/protocol/cybin/cybin.c:6325)
File "thriftpy/protocol/cybin/cybin.pyx", line 60, in cybin.read_i32 (thriftpy/protocol/cybin/cybin.c:1546)
File "thriftpy/transport/buffered/cybuffered.pyx", line 65, in thriftpy.transport.buffered.cybuffered.TCyBufferedTransport.c_read (thriftpy/transport/buffered/cybuffered.c:1881)
File "thriftpy/transport/buffered/cybuffered.pyx", line 69, in thriftpy.transport.buffered.cybuffered.TCyBufferedTransport.read_trans (thriftpy/transport/buffered/cybuffered.c:1948)
File "thriftpy/transport/cybase.pyx", line 61, in thriftpy.transport.cybase.TCyBuffer.read_trans (thriftpy/transport/cybase.c:1472)
File "/usr/local/lib/python2.7/dist-packages/thriftpy/transport/socket.py", line 125, in read
message='TSocket read 0 bytes')
thriftpy.transport.TTransportException: TTransportException(message='TSocket read 0 bytes', type=4)
)
You need to specify the server address, and possibly port:
connection = hb.Connection(SERVER, PORT)
You can probably omit the port value, as most likely the default value will match, but just in case check on what port your thrift server is listening and specify that as a numeric value
This error just started to pop up in our pipelines.
I'm moving a dataframe that's about 1.5mil rows using the pandas.to_gbq method.
Any help would be greatly appreciated!
Code:
output.to_gbq('table_name',
'project-id',
chunksize=50000,
private_key='ga_auth.json',
if_exists='replace'
)
Error:
Traceback (most recent call last):
File ".\rfm_bigquery.py", line 175, in <module>
send_rfm_to_gbq()
File ".\rfm_bigquery.py", line 152, in send_rfm_to_gbq
if_exists='replace',
File "C:\Users\yyu\Desktop\env\rfm_bigquery\lib\site-packages\pandas\core\frame.py", line 1187, in to_gbq
table_schema=table_schema)
File "C:\Users\yyu\Desktop\env\rfm_bigquery\lib\site-packages\pandas\io\gbq.py", line 119, in to_gbq
table_schema=table_schema)
File "C:\Users\yyu\Desktop\env\rfm_bigquery\lib\site-packages\pandas_gbq\gbq.py", line 1036, in to_gbq
progress_bar=progress_bar,
File "C:\Users\yyu\Desktop\env\rfm_bigquery\lib\site-packages\pandas_gbq\gbq.py", line 513, in load_data
self.process_http_error(ex)
File "C:\Users\yyu\Desktop\env\rfm_bigquery\lib\site-packages\pandas_gbq\gbq.py", line 376, in process_http_error
raise GenericGBQException("Reason: {0}".format(ex))
pandas_gbq.gbq.GenericGBQException: Reason: 404 POST https://www.googleapis.com/upload/bigquery/v2/projects/hidden-moon-164616/jobs?uploadType=resumable: Not Found
I had the same problem with the latest version, I moved back to pandas==0.23.3 and pandas-gbq==0.5.0 and it is finally working ...
I am using Plone 4.3.3 for creating my Plone site but when I shut-down the server it shows the following error.
Traceback (most recent call last):
File "/Plone/zinstance/parts/instance/bin/interpreter", line 298, in <module>
exec(compile(__file__f.read(), __file__, "exec"))
File "/Plone/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/run.py", line 76, in <module>
run()
File "/Plone/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/run.py", line 26, in run
starter.run()
File "/Plone/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/__init__.py", line 108, in run
self.shutdown()
File "/Plone/buildout-cache/eggs/Zope2-2.13.22-py2.7.egg/Zope2/Startup/__init__.py", line 113, in shutdown
db.close()
File "/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-i686.egg/ZODB/DB.py", line 624, in close
user#user-Vostro-3300:~/Plone/zinstance$ #self._connectionMap
File "/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-i686.egg/ZODB/DB.py", line 506, in _connectionMap
self.pool.map(f)
File "/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-i686.egg/ZODB/DB.py", line 206, in map
self.all.map(f)
File "/Plone/buildout-cache/eggs/transaction-1.1.1-py2.7.egg/transaction/weakset.py", line 58, in map
f(elt)
File "/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-i686.egg/ZODB/DB.py", line 628, in _
c._release_resources()
File "/Plone/buildout-cache/eggs/ZODB3-3.10.5-py2.7-linux-i686.egg/ZODB/Connection.py", line 1075, in _release_resources
c._storage.release()
AttributeError: 'NoneType' object has no attribute 'release'
There is an issue with Zope2 shutdown that tries to close a database connection (and in turn, a storage). However, this late-running sequence has some cosmetic side-effect for users of RelStorage. This is annoying, but not fundamentally a problem that should cause any data integrity issues.
Users of FileStorage or ZEO should not see this.
References:
https://github.com/zopefoundation/Zope/commit/5032027470091957a6c0028da04c0fc0a1ed646b
https://mail.zope.org/pipermail/zodb-dev/2013-August/015119.html