Develop Reference

Python connection with Hive database in HDInsight

I am trying to create connection to Hive hosted in HDInsight cluster through my python script and getting below error-
Traceback (most recent call last):
File "ClassLoader.java", line 357, in java.lang.ClassLoader.loadClass
File "Launcher.java", line 349, in sun.misc.Launcher$AppClassLoader.loadClass
File "ClassLoader.java", line 424, in java.lang.ClassLoader.loadClass
File "URLClassLoader.java", line 382, in java.net.URLClassLoader.findClass
java.lang.ClassNotFoundException: java.lang.ClassNotFoundException: org.apache.thrift.transport.TTransportException
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "org.jpype.JPypeContext.java", line 330, in org.jpype.JPypeContext.callMethod
File "Method.java", line 498, in java.lang.reflect.Method.invoke
File "DelegatingMethodAccessorImpl.java", line 43, in sun.reflect.DelegatingMethodAccessorImpl.invoke
File "NativeMethodAccessorImpl.java", line 62, in sun.reflect.NativeMethodAccessorImpl.invoke
File "NativeMethodAccessorImpl.java", line -2, in sun.reflect.NativeMethodAccessorImpl.invoke0
File "DriverManager.java", line 247, in java.sql.DriverManager.getConnection
File "DriverManager.java", line 664, in java.sql.DriverManager.getConnection
File "HiveDriver.java", line 105, in org.apache.hive.jdbc.HiveDriver.connect
Exception: Java Exception
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "s.py", line 5, in <module>
"/root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar")
File "/usr/local/lib64/python3.6/site-packages/jaydebeapi/__init__.py", line 412, in connect
jconn = _jdbc_connect(jclassname, url, driver_args, jars, libs)
File "/usr/local/lib64/python3.6/site-packages/jaydebeapi/__init__.py", line 230, in _jdbc_connect_jpype
return jpype.java.sql.DriverManager.getConnection(url, *dargs)
java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: org/apache/thrift/transport/TTransportException
My Script is -
import jaydebeapi
conn = jaydebeapi.connect("org.apache.hive.jdbc.HiveDriver",
"jdbc:hive2://10.20.30.40:10001/default;transportMode=http;ssl=false;httpPath=/hive2",
["username", "password"],
"/root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar")
I have exported CLASSPATH wil all jar files.

The error is java.lang.ClassNotFoundException: java.lang.ClassNotFoundException which specifies that the execution is not able to find the jar /root/jdbc/hive-jdbc-1.2.1000.2.6.5.3009-43.jar.I believe it's only placed in a host from where you are executing the code. I would suggest placing the jar file in the same directory structure in all the nodes in the cluster and have a check on permissions so that the user executing the job has access to that path.

Namespace Secret not available on Raspberry Pi

I'm trying to retrieve a cookie on my Raspberry Pi 3B+ with pycookiecheat since pycookiecheat doesn't work on windows. The reason I want to do this is that I need to bypass a login that has multiple steps.
from pycookiecheat import chrome_cookies
import requests
url = 'google.com'
cookies = chrome_cookies(url)
r = requests.get(url, cookies=cookies)
print(r.text)
But there's an error:
Traceback (most recent call last):
File "/home/pi/Desktop/Homeworker.py", line 7, in <module>
cookies = chrome_cookies(url)
File "/home/pi/.local/lib/python3.7/site-packages/pycookiecheat/pycookiecheat.py", line 199, in chrome_cookies
config = get_linux_config(browser)
File "/home/pi/.local/lib/python3.7/site-packages/pycookiecheat/pycookiecheat.py", line 133, in get_linux_config
gi.require_version("Secret", "1")
File "/usr/lib/python3/dist-packages/gi/__init__.py", line 129, in require_version
raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace Secret not available
>>> %Run Homeworker.py
Traceback (most recent call last):
File "/home/pi/Desktop/Homeworker.py", line 7, in <module>
cookies = chrome_cookies(url)
File "/home/pi/.local/lib/python3.7/site-packages/pycookiecheat/pycookiecheat.py", line 199, in chrome_cookies
config = get_linux_config(browser)
File "/home/pi/.local/lib/python3.7/site-packages/pycookiecheat/pycookiecheat.py", line 133, in get_linux_config
gi.require_version("Secret", "1")
File "/usr/lib/python3/dist-packages/gi/__init__.py", line 129, in require_version
raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace Secret not available

Difficulty creating an XLSM file

I'm trying to create an xlsm file using xlwings. Or openpyxl if not possible with xlwings.
I'm on Mac so I can't use PyWin32.
I'm running the following python code:
import xlwings as xw
b = xw.Book()
b.save('test_book2.xlsm')
When I run that code I receive the message:
When I click 'yes' to that message I receive the following error:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/aeosa/appscript/reference.py", line 482, in __call__
return self.AS_appdata.target().event(self._code, params, atts, codecs=self.AS_appdata).send(timeout, sendflags)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/aeosa/aem/aemsend.py", line 92, in send
raise EventError(errornum, errormsg, eventresult)
aem.aemsend.EventError: Command failed: Parameter error. (-50)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/matthewbell/Desktop/test.py", line 4, in <module>
b.save('test_book2.xlsm')
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/xlwings/main.py", line 704, in save
return self.impl.save(path)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/xlwings/_xlmac.py", line 244, in save
self.xl.save_workbook_as(filename=hfs_path, overwrite=True)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/aeosa/appscript/reference.py", line 518, in __call__
raise CommandError(self, (args, kargs), e, self.AS_appdata) from e
appscript.reference.CommandError: Command failed:
OSERROR: -50
MESSAGE: Parameter error.
COMMAND: app(pid=6198).workbooks['Book1'].save_workbook_as(filename='Macintosh HD:Users:matthewbell:Desktop:test_book2.xlsm', overwrite=True)
What can I do to create an xlsm file?

Trying to connect to a SOAP / WSDL service using Python (SOAPPy) - getting ERROR

Im trying to connect to a SOAP/WSDL server through my Python script:
# based on the tutorial:
# http://www.diveintopython.net/soap_web_services/
import pprint
from SOAPpy import WSDL
WSDLFILE = "https://api.comscore.com/KeyMeasures.asmx?WSDL"
proxy = WSDL.Proxy(WSDLFILE)
proxy.soapserver.config.dumpSOAPIn=1
proxy.soapserver.config.dumpSOAPOut=1
When I run this script I get the following error:
Traceback (most recent call last):
File "/Users/XX/PycharmProjects/Test_Proj/Comscore_connector.py", line 9, in <module>
proxy = WSDL.Proxy(WSDLFILE)
File "/Users/XXpython_projects/lib/python2.7/site-packages/SOAPpy/WSDL.py", line 85, in __init__
self.wsdl = reader.loadFromString(str(wsdlsource))
File "/Users/XX/python_projects/lib/python2.7/site-packages/wstools/WSDLTools.py", line 47, in loadFromString
return self.loadFromStream(StringIO(data))
File "/Users/XX/python_projects/lib/python2.7/site-packages/wstools/WSDLTools.py", line 28, in loadFromStream
document = DOM.loadDocument(stream)
File "/Users/XX/python_projects/lib/python2.7/site-packages/wstools/Utility.py", line 645, in loadDocument
return xml.dom.minidom.parse(data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/dom/minidom.py", line 1918, in parse
return expatbuilder.parse(file)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/dom/expatbuilder.py", line 928, in parse
result = builder.parseFile(file)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/dom/expatbuilder.py", line 207, in parseFile
parser.Parse(buffer, 0)
xml.parsers.expat.ExpatError: not well-formed (invalid token): line 1, column 6
Process finished with exit code 1
Ive looked a the following links for SOAP examples:
http://users.skynet.be/pascalbotte/rcx-ws-doc/python.htm
http://code.activestate.com/recipes/502259-calling-a-web-service-using-soappy/
However, there isn't much I could find elsewhere....
Would really appreciate any advice to point me in the right direction.

DeleteResource from Google Docs with Python

I am trying to delete a spreadsheet in Google Docs with this function:
def f_DeleteResource(xls_name):
"""Delete a resource"""
client=Auth()
for e1 in client.GetResources().entry:
e2 = client.GetResource(e1)
if xls_name==e2.title.text:
client.DeleteResource(e2.resource_id.text,True)
And I obtain different errors when I change the first parameter of client.DeleteResource(p1,p2):
client.DeleteResource(e2.resource_id.text,True):
Traceback (most recent call last):
File "C:\xmp\D6GDocsDeleteUpload.py", line 164, in <module> main()
File "C:\xmp\D6GDocsDeleteUpload.py", line 157, in main f_DeleteResource(sys.argv[2])
File "C:\xmp\D6GDocsDeleteUpload.py", line 144, in f_DeleteResource client.DeleteResource(e2.resource_id.text,True)
File "C:\Python27\lib\site-packages\gdata\docs\client.py", line 540, in delete_resource uri = entry.GetEditLink().href
AttributeError: 'str' object has no attribute 'GetEditLink'
client.DeleteResource(e2,True):
Traceback (most recent call last):
File "C:\xmp\D6GDocsDeleteUpload.py", line 164, in <module> main()
File "C:\xmp\D6GDocsDeleteUpload.py", line 157, in main f_DeleteResource(sys.argv[2])
File "C:\xmp\D6GDocsDeleteUpload.py", line 144, in f_DeleteResource client.DeleteResource(e2,True)
File "C:\Python27\lib\site-packages\gdata\docs\client.py", line 543, in delete_resource return super(DocsClient, self).delete(uri, **kwargs)
File "C:\Python27\lib\site-packages\gdata\client.py", line 748, in delete **kwargs)
File "C:\Python27\lib\site-packages\gdata\docs\client.py", line 66, in request return super(DocsClient, self).request(method=method, uri=uri, **kwargs)
File "C:\Python27\lib\site-packages\gdata\client.py", line 319, in request RequestError)
gdata.client.RequestError: Server responded with: 403, <errors xmlns='http://schemas.google.com/g/2005'><error><domain>GData</domain><code>matchHeaderRequired</code><location type='header'>If-Match|If-None-Match</location><internalReason>If-Match or If-None-Match header or entry etag attribute required</internalReason></error></errors>
Anyone can help me?

It seems to be a bug in Google API Python library. I checked gdata-2.0.16 and noticed that DeleteResource() function uses only URL of the resource (gdata/docs/client.py lines 540-543), but later checks for hasattr(entry_or_uri, 'etag') (gdata/client.py lines 737-741) and of course string value (uri) doesn't have etag attribute.
You may work around it using force keyword argument:
import gdata.docs.data
import gdata.docs.client
client = gdata.docs.client.DocsClient()
client.ClientLogin('xxxxxx#gmail.com', 'xxxxxx', 'XxX')
for doc in client.GetAllResources():
if doc.title.text == 'qpqpqpqpqpqp':
client.DeleteResource(doc, force=True)
break
If you want you may report an error to library maintainers (if it isn't already reported).

This issue has been fixed in this revision:
http://code.google.com/p/gdata-python-client/source/detail?r=f98fff494fb89fca12deede00c3567dd589e5f97
If you sync you client to the repository, you should be able to delete a resource without having to specify 'force=True'.