Python connection with Hive database in HDInsight - python

I am trying to create connection to Hive hosted in HDInsight cluster through my python script and getting below error-
Traceback (most recent call last):
File "ClassLoader.java", line 357, in java.lang.ClassLoader.loadClass
File "Launcher.java", line 349, in sun.misc.Launcher$AppClassLoader.loadClass
File "ClassLoader.java", line 424, in java.lang.ClassLoader.loadClass
File "URLClassLoader.java", line 382, in java.net.URLClassLoader.findClass
java.lang.ClassNotFoundException: java.lang.ClassNotFoundException: org.apache.thrift.transport.TTransportException
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "org.jpype.JPypeContext.java", line 330, in org.jpype.JPypeContext.callMethod
File "Method.java", line 498, in java.lang.reflect.Method.invoke
File "DelegatingMethodAccessorImpl.java", line 43, in sun.reflect.DelegatingMethodAccessorImpl.invoke
File "NativeMethodAccessorImpl.java", line 62, in sun.reflect.NativeMethodAccessorImpl.invoke
File "NativeMethodAccessorImpl.java", line -2, in sun.reflect.NativeMethodAccessorImpl.invoke0
File "DriverManager.java", line 247, in java.sql.DriverManager.getConnection
File "DriverManager.java", line 664, in java.sql.DriverManager.getConnection
File "HiveDriver.java", line 105, in org.apache.hive.jdbc.HiveDriver.connect
Exception: Java Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "s.py", line 5, in <module>
"/root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar")
File "/usr/local/lib64/python3.6/site-packages/jaydebeapi/__init__.py", line 412, in connect
jconn = _jdbc_connect(jclassname, url, driver_args, jars, libs)
File "/usr/local/lib64/python3.6/site-packages/jaydebeapi/__init__.py", line 230, in _jdbc_connect_jpype
return jpype.java.sql.DriverManager.getConnection(url, *dargs)
java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: org/apache/thrift/transport/TTransportException
My Script is -
import jaydebeapi
conn = jaydebeapi.connect("org.apache.hive.jdbc.HiveDriver",
"jdbc:hive2://10.20.30.40:10001/default;transportMode=http;ssl=false;httpPath=/hive2",
["username", "password"],
"/root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar")
I have exported CLASSPATH wil all jar files.

The error is java.lang.ClassNotFoundException: java.lang.ClassNotFoundException which specifies that the execution is not able to find the jar /root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar.I believe it's only placed in a host from where you are executing the code. I would suggest placing the jar file in the same directory structure in all the nodes in the cluster and have a check on permissions so that the user executing the job has access to that path.

Related

When I need Initialize a new site in ERPNext

when I write the command line in terminal
bench new-site site1.local
This Error Will shown
> Traceback (most recent call last):
File "/home/ubuntu/.local/bin/bench", line 8, in <module>
sys.exit(cli())
File "/home/ubuntu/bench/bench/cli.py", line 71, in cli
cmd_from_sys = get_cmd_from_sysargv()
File "/home/ubuntu/bench/bench/utils/__init__.py", line 554, in get_cmd_from_sysargv
if sys_argv.index(arg) == 0 and arg in Bench(".").apps:
File "/home/ubuntu/bench/bench/bench.py", line 67, in __init__
self.apps = BenchApps(self)
File "/home/ubuntu/bench/bench/bench.py", line 174, in __init__
self.initialize_apps()
File "/home/ubuntu/bench/bench/bench.py", line 281, in initialize_apps
self.apps.remove("frappe")
ValueError: list.remove(x): x not in list
Note I use EC2 On AWS Ubunto server
Can Any One Help With This Problem Please.
I tried to install many lib but no help with this problem

Stuck with connecting to mongodb

I was about to start using mongodb as my database project. I'm always stuck when want to pushing some data to it. I use mongodb atlas. This code below always resulting errors. I've following and learning about mongodb from online (like youtube, w3, etc)
import pymongo
from pymongo import MongoClient
url = "mongodb+srv://<username>:<password>#cluster0.oecqh.mongodb.net/test?retryWrites=true&w=majority"
cluster = MongoClient(url)
db = cluster["discord"]
collection = db["discord"]
push = {"name": "test", "value": "test"}
try:
collection.insert_one(push)
except Exception as e:
print(e)
The errors
Exception ignored in: <function Client.__del__ at 0x00000213C5D84940>
Traceback (most recent call last):
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\httpx\_client.py", line 1139, in __del__
self.close()
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\httpx\_client.py", line 1111, in close
self._transport.close()
AttributeError: 'Client' object has no attribute '_transport'
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\threading.py", line 973, in _bootstrap_inner
self.run()
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\dns\win32util.py", line 48, in run
self.info.domain = _config_domain(interface.DNSDomain)
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\dns\win32util.py", line 26, in _config_domain
if domain.startswith('.'):
AttributeError: 'NoneType' object has no attribute 'startswith'
Traceback (most recent call last):
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\srv_resolver.py", line 89, in _resolve_uri
results = _resolve(
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\srv_resolver.py", line 43, in _resolve
return resolver.resolve(*args, **kwargs)
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\dns\resolver.py", line 1193, in resolve
return get_default_resolver().resolve(qname, rdtype, rdclass, tcp, source,
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\dns\resolver.py", line 1063, in resolve
(nameserver, port, tcp, backoff) = resolution.next_nameserver()
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\dns\resolver.py", line 646, in next_nameserver
raise NoNameservers(request=self.request, errors=self.errors)
dns.resolver.NoNameservers: All nameservers failed to answer the query _mongodb._tcp.cluster0.oecqh.mongodb.net. IN SRV:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\Users\chris\OneDrive\Documents\Programming\Python\tempCodeRunnerFile.py", line 5, in <module>
cluster = MongoClient(url)
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\mongo_client.py", line 704, in __init__
res = uri_parser.parse_uri(
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\uri_parser.py", line 542, in parse_uri
nodes = dns_resolver.get_hosts()
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\srv_resolver.py", line 121, in get_hosts
_, nodes = self._get_srv_response_and_hosts(True)
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\srv_resolver.py", line 101, in _get_srv_response_and_hosts
results = self._resolve_uri(encapsulate_errors)
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\site-packages\pymongo\srv_resolver.py", line 97, in _resolve_uri
raise ConfigurationError(str(exc))
pymongo.errors.ConfigurationError: All nameservers failed to answer the query _mongodb._tcp.cluster0.oecqh.mongodb.net. IN SRV:

tensorflow_hub throwing this error: 'SentencepieceOp' when loading the link

The following line of code I am trying to run in PyCharm and I have tensorflow_hub installed and imported.
use = hub.load("https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3")
Any suggestions for the below error? As I need this for my project.
Traceback (most recent call last):
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3820, in _get_op_def
return self._op_def_cache[type]
KeyError: 'SentencepieceOp'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/Jon10/OneDrive/Documents/Computer Science/Dissertation/PythonPractice/TFTest/test.py", line 28, in <module>
use = hub.load("https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3")
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_hub\module_v2.py", line 102, in load
obj = tf_v1.saved_model.load_v2(module_path, tags=tags)
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\saved_model\load.py", line 517, in load
return load_internal(export_dir, tags)
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\saved_model\load.py", line 541, in load_internal
export_dir)
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\saved_model\load.py", line 114, in __init__
meta_graph.graph_def.library))
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\saved_model\function_deserialization.py", line 312, in load_function_def_library
copy, copy_functions=False)
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\framework\function_def_to_graph.py", line 61, in function_def_to_graph
fdef, input_shapes, copy_functions)
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\framework\function_def_to_graph.py", line 214, in function_def_to_graph_def
op_def = ops.get_default_graph()._get_op_def(node_def.op) # pylint: disable=protected-access
File "C:\Users\Jon10\miniconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3824, in _get_op_def
c_api.TF_GraphGetOpDef(self._c_graph, compat.as_bytes(type), buf)
tensorflow.python.framework.errors_impl.NotFoundError: Op type not registered 'SentencepieceOp' in binary running on DESKTOP-..... Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
You need to install tensorflow_text, and import it before using hub.load

EOF error when I try to use 'expect' function from'Pexpect'

I'm trying to automate the process of switch between networks in "Cisco AnyConnect", but my code always is terminated with an 'EOF error'.
I've tried a search in API doc for some help and used different expressions like: "Network", "Network:" or "Network.*".
from pexpect import popen_spawn
import pexpect
vpnui = pexpect.popen_spawn.PopenSpawn(r'C:\Program Files (x86)\Cisco\Cisco AnyConnect Secure Mobility Client\vpnui.exe', maxread=2000)
vpnui.expect('Network:')
vpnui.sendline('SomeNetwork')
The error:
Traceback (most recent call last): File
"C:\Users\myUser\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\expect.py",
line 111, in expect_loop
incoming = spawn.read_nonblocking(spawn.maxread, timeout) File "C:\Users\myUser\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\popen_spawn.py",
line 75, in read_nonblocking
raise EOF('End Of File (EOF).') pexpect.exceptions.EOF: End Of File (EOF).
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File
"C:\Users\myUser\eclipse-workspace\teste\switchGuestWifi.py", line 15,
in
vpnui.expect('Network:') File "C:\Users\myUser\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\spawnbase.py",
line 341, in expect
timeout, searchwindowsize, async_) File "C:\Users\myUser\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\spawnbase.py",
line 369, in expect_list
return exp.expect_loop(timeout) File "C:\Users\myUser\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\expect.py",
line 117, in expect_loop
return self.eof(e) File "C:\Users\myUser\AppData\Local\Programs\Python\Python36\lib\site-packages\pexpect\expect.py",
line 63, in eof
raise EOF(msg) pexpect.exceptions.EOF: End Of File (EOF).
searcher: searcher_re:
0: re.compile(b'Network:')

Why I am getting file not found exception while copying file on HDFS using Pydoop?

My Python code using Pydoop API is show below. I have a text file /home/progen/test.txt. While running the script to copy this file to HDFS, I am getting an error.
import pydoop.hdfs as hdfs
local_path = '/home/progen/test.txt'
hdfs_path = '/spark_user/'
host = 'master'
port = 9000
hdfsobj = hdfs.hdfs(host, port, user='spark', groups=['supergroup'])
hdfsobj.copy(local_path, hdfsobj, hdfs_path)
I am getting the Error: "File does no exist"
hdfsCopyImpl(src=/home/progen/test.txt, dst=/spark_user/, deleteSource=0): FileUtil#copy error:
java.io.FileNotFoundException: File does not exist: /home/progen/test.txt
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1122)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
Traceback (most recent call last):
File "hdfs.py", line 8, in <module>
hdfsobj.copy(local_path, hdfsobj, hdfs_path)
File "/usr/local/lib/python2.7/dist-packages/pydoop/hdfs/fs.py", line 304, in copy
return self.fs.copy(from_path, to_hdfs, to_path)
hdfsCopyImpl(src=/home/progen/test.txt, dst=/spark_user/, deleteSource=0): FileUtil#copy error:
java.io.FileNotFoundException: File does not exist: /home/progen/test.txt
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1122)
at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
Traceback (most recent call last):
File "hdfs.py", line 8, in <module>
hdfsobj.copy(local_path, hdfsobj, hdfs_path)
File "/usr/local/lib/python2.7/dist-packages/pydoop/hdfs/fs.py", line 304, in copy
return self.fs.copy(from_path, to_hdfs, to_path)

Categories

Resources