Hive Server 2 error on python connect with hiveserver2 - python

I am using
Centos , Python2.7 , hive 2.1 ,Hadoop 2.7.2 ,pyHive
here is code
from pyhive import hive
from TCLIService.ttypes import TOperationState
cursor = hive.connect('localhost').cursor()
cursor.execute('SELECT * FROM my_awesome_data LIMIT 10', async=True)
#status = cursor.poll().operationState
#while status in (TOperationState.INITIALIZED_STATE, TOperationState.RUNNI$
# logs = cursor.fetch_logs()
# for message in logs:
# print message
# If needed, an asynchronous query can be cancelled at any time with:
# cursor.cancel()
# status = cursor.poll().operationState
#print cursor.fetchall()
when I run python /usr/local/py/test5.py in terminal its showing ....
Traceback (most recent call last):
File "/usr/local/py/test5.py", line 3, in <module>
cursor = hive.connect('localhost').cursor()
File "/usr/local/lib/python2.7/site-packages/pyhive/hive.py", line 63, in connect
return Connection(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pyhive/hive.py", line 104, in __init__
self._transport.open()
File "/usr/local/lib/python2.7/site-packages/thrift_sasl/__init__.py", line 80, in open
status, payload = self._recv_sasl_message()
File "/usr/local/lib/python2.7/site-packages/thrift_sasl/__init__.py", line 98, in _recv_sasl_message
header = read_all_compat(self._trans, 5)
File "/usr/local/lib/python2.7/site-packages/thrift_sasl/six.py", line 31, in <lambda>
read_all_compat = lambda trans, sz: trans.readAll(sz)
File "/usr/local/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 58, in readAll
chunk = self.read(sz - have)
File "/usr/local/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 120, in read
message='TSocket read 0 bytes')
thrift.transport.TTransport.TTransportException: TSocket read 0 bytes
Hive server error log showing after this...
ERROR [HiveServer2-Handler-Pool: Thread-41]: server.TThreadPoolServer (:()) - Thrift error occur$
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:228)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Also I had tried pyhs2 getting same error
what was going wrong ?
Thanks

I hava solved this error with these versions :
Centos 7 , Python2.7 , hive 2.1 ,Hadoop 2.7.3 and java java 1.7.0_91
using impyla
its working for me.

thrift.transport.TTransport.TTransportException: TSocket read 0 bytes
I had the same problem and solved it by
Start hiveserver2 (how to start hiveserver2)
Modify your port. The default port is 10000 or you can change it to follow your hive_site.xml

Related

Adafruit MAX 31856 communicating with raspberrypi

I have been trying to read a thermocouple temperature on my raspberrypi however cant get the raspi to communicate with the adafruit MAX 31856. I have downloaded necessary libraries and checked that all connections are secure and correct however keep running into this error when I try to execute this code.
Any help is greatly appreciated!!
CODE
import board
import digitalio
import adafruit_max31856
spi = board.SPI()
cs = digitalio.DigitalInOut(board.D5)
cs.direction = digitalio.Direction.OTUPUT
thermocouple = adafruit_max31856.MAX31856(spi,cs)
print(thermocouple.temperature)
ERROR:
Traceback (most recent call last):
File "/home/pi/test4.py", line 6, in <module>
spi = board.SPI()
File "/usr/local/lib/python3.7/dist-packages/board.py", line 299, in SPI
return busio.SPI(SCLK, MOSI, MISO)
File "/usr/local/lib/python3.7/dist-packages/busio.py", line 289, in __init__
self._spi = _SPI(portId)
File "/usr/local/lib/python3.7/dist-packages/adafruit_blinka/microcontroller/generic_linux/spi.py", line 25, in __init__
self._spi = spi.SPI(device=(portid, 0))
File "/usr/local/lib/python3.7/dist-packages/Adafruit_PureIO/spi.py", line 167, in __init__
raise IOError("{} does not exist".format(device))
OSError: /dev/spidev0.0 does not exist
Code
Errors
Most probably there is a device driver loaded for spi0-0 and that removes /dev/spidev0.0 device.
check your /boot/config.txt for overlays loading MAX 31856 driver module.
if you want to use adafruit_max31856 lib you shouldn't be loading this driver. hope that helps

Problem with hive thrift or with HiveMetastore by Apache Airflow

We have a problem when possible trying to connect with Hive Metastore by HiveMetastoreHook in Apache Airflow
thrift.transport.TTransport.TTransportException: b'Error in sasl_decode (-1) SASL(-1): generic failure: Unable to find a callback: 32775'
We googled this issue but still have not find any answer
But for temporary fix - We do REcreate our hive external table on which the issue happens and restart this task. Then after a few days this problem happens again and again and again. And we have no any idea where we need to fix.
NOTE:
This happens only on one big table ,
we have hive 3.1.0 and Airflow 1.10.5 ,
this issue reproduces from python3 cli by import airflow,
select this big table from hive is fine (data is fine too)
Full stack trace
[2021-06-30 18:05:41,143] {{base_hook.py:84}} INFO - Using connection to: id: our_hive_metastore_connection. Host: server_name, Port: someport, Schema: our_schema, Login: some_login, Password: None, extra: {'authMechanism': 'GSSAPI', 'kerberos_service_name': 'some_name'}
>>> hm.get_table(table_name='some_big_table', db='some_schema')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/site-packages/airflow/hooks/hive_hooks.py", line 608, in get_table
return client.get_table(dbname=db, tbl_name=table_name)
File "/usr/local/lib/python3.6/site-packages/hmsclient/genthrift/hive_metastore/ThriftHiveMetastore.py", line 2253, in get_table
return self.recv_get_table()
File "/usr/local/lib/python3.6/site-packages/hmsclient/genthrift/hive_metastore/ThriftHiveMetastore.py", line 2266, in recv_get_table
(fname, mtype, rseqid) = iprot.readMessageBegin()
File "/usr/local/lib64/python3.6/site-packages/thrift/protocol/TBinaryProtocol.py", line 134, in readMessageBegin
sz = self.readI32()
File "/usr/local/lib64/python3.6/site-packages/thrift/protocol/TBinaryProtocol.py", line 217, in readI32
buff = self.trans.readAll(4)
File "/usr/local/lib64/python3.6/site-packages/thrift/transport/TTransport.py", line 62, in readAll
chunk = self.read(sz - have)
File "/usr/local/lib/python3.6/site-packages/thrift_sasl/__init__.py", line 166, in read
self._read_frame()
File "/usr/local/lib/python3.6/site-packages/thrift_sasl/__init__.py", line 180, in _read_frame
message=self.sasl.getError())
thrift.transport.TTransport.TTransportException: b'Error in sasl_decode (-1) SASL(-1): generic failure: Unable to find a callback: 32775'```
Is anybody have some idea what we need to fix?
Appreciate for help!

threading error when calling pyodbc cursor

I am receiving "TypeError: bad argument type for built-in operation" error when calling cursor.columns from pyodbc, Traceback the issue to threading.py -> currentThread function...
I have installed my python environment on a new HP Z2 desktop PC running :
Win10 enterprise x64
Python 2.7.13 on win32
PyScripter 3.5.1.0 x86
and suddenly received a very strange error while trying to insert data into my database using pyodbc module.
My code -
class clsDataBaseWrapper():
def __init__(self,ServerAddress='WIGIG-703\SQLEXPRESS',DataBase='QCT_Python',dBEn=1):
if (dBEn) :
self.DataBase=DataBase
self.cnxn = pyodbc.connect("Driver={SQL Server}"+
"""
;Server={0};
Database={1};
Trusted_Connection=yes;""".format(ServerAddress,DataBase))
self.cursor = self.cnxn.cursor()
else : print "\n*** Be Aware - you are not writing data to the DataBase !!! ***"
def InsertData(self,Table,Data,dBEn):
if (dBEn) :
columns = self.cursor.columns(table=Table, schema='dbo').fetchall()
insertQuery = "insert into {0} values ({1})".format(Table, ','.join('?' * len(columns)))
try :
self.cursor.execute(insertQuery, Data)
self.cnxn.commit()
except Exception,e : print 'could not insert data into DB - ',e
else : pass
The error I receive -
C:\Python27\lib\threading.py:1151: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
return _active[_get_ident()]
Traceback (most recent call last):
File "C:\Qpython\IsonM_TC2\Services_IsonM_TC2.py", line 335, in <module>
test.InsertData('PLL_CL_PN_PLL',[],1)
File "C:\Qpython\IsonM_TC2\Services_IsonM_TC2.py", line 87, in InsertData
columns = columns.fetchall()
File "C:\Qpython\IsonM_TC2\Services_IsonM_TC2.py", line 87, in InsertData
columns = columns.fetchall()
File "C:\Python27\lib\bdb.py", line 49, in trace_dispatch
return self.dispatch_line(frame)
File "C:\Python27\lib\bdb.py", line 67, in dispatch_line
self.user_line(frame)
File "<string>", line 126, in user_line
File "C:\Python27\lib\threading.py", line 1151, in currentThread
return _active[_get_ident()]
TypeError: bad argument type for built-in operation
This is very strange because it is working perfectly on other old PC I have running the same environment.
What do I miss here ? tried re-installing python, pyscripter ...

Python 2.7 and Pyo, Server must be booted before creating any audio project

I am using Pyo (http://ajaxsoundstudio.com/software/pyo/) in Python 2.7.14 and I an trying to play an audio file with this code
from pyo import *
s = Server()
s.boot()
s.start()
sf = SfPlayer("C:\Users\name\Downloads\audio.mp3", speed=1, loop=True).out()
but I get this message:
Traceback (most recent call last):
File "C:\Python27\pyotest.py", line 5, in <module>
sf = SfPlayer("C:\Users\name\Downloads\audio.mp3", speed=1, loop=True).out()
File "C:\Python27\lib\site-packages\pyolib\players.py", line 98, in __init__
PyoObject.__init__(self, mul, add)
File "C:\Python27\lib\site-packages\pyolib\_core.py", line 967, in __init__
PyoObjectBase.__init__(self)
File "C:\Python27\lib\site-packages\pyolib\_core.py", line 809, in __init__
raise PyoServerStateException("The Server must be booted before "
PyoServerStateException: The Server must be booted before creating any audio object.
I had the same error once (on Windows). Started the E-Pyo Editor that comes with installing Pyo. After that it worked fine.
If that doesn't help you might want to set the server options and the output device manually.
Get the ID of the desired output device with
pa_list_devices()
If you found your ID you can set it with this command (change the 0 to your device ID) and then boot the server:
s.setOutputDevice(0)
s.boot()

PyHive is Hanging on Connection -- Thrift_sasl appears to be the issue

from pyhive import hive
import thrift_sasl
connection = hive.Connection(host='myhost', port=10000, database='local')
#hangs here
from sqlalchemy import create_engine
engine = create_engine('hive://myhost:10000/local')
logs = Table('mytable', MetaData(bind=engine), autoload=True)
#also hangs here
Both of these snippets will hang for me.
Hitting ctrl+c stops the execution here:
^CTraceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/apps/Python/lib/python2.7/site-packages/pyhive/hive.py", line 86, in __init__
self._transport.open()
File "thrift_sasl.py", line 74, in open
status, payload = self._recv_sasl_message()
File "thrift_sasl.py", line 92, in _recv_sasl_message
header = self._trans.readAll(5)
File "/apps/Python/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 58, in readAll
chunk = self.read(sz - have)
File "/apps/Python/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 105, in read
buff = self.handle.recv(sz)
KeyboardInterrupt
I am using Hive 0.12 and HiveServer2. I can connect to it using the python Hive library provided with Hadoop (.../hive/lib/py) but cannot do so with pyhive, which uses thrift_sasl.
Some people not using the thrift_sasl module suggest turning off SASL support in hive-site.xml via:
<property><name>hive.server2.authentication</name><value>NOSASL</value></property>
However after trying this the code still hanged with the same stack trace when I issued a KeyboardInterrupt.

Categories

Resources