Develop Reference

Write on HDFS using Python

I am trying to write on HDFS from Python.
Right now, I am using https://hdfscli.readthedocs.io/en/latest/quickstart.html
but for large file I get back:
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/hdfs/client.py", line 400, in write
consumer(data)
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/hdfs/client.py", line 394, in consumer
auth=False,
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/hdfs/client.py", line 179, in _request
**kwargs
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', gaierror(-2, 'Name or service not known'))
My code for writing is pretty simple:
client = InsecureClient('http://xxxxxxx.co:50070', user='hdfs')
client.write("/tmp/a",stringToWrite)
Anyone can suggest a decent package to write on HDFS?
Cheers

For the stacktrace, it seems to be security related. Are you sure you need to use the InsecureClient and not the Kerberos one?. Also, remember that library is just a binding for HttpFs, so doing a manual test with Postman or CURL would let you debug any issue cluster-side.

Python httplib2 "httplib2.SSLHandshakeError"

I have this piece of code that tries to get the page content from a given url.
import httplib2
start_url = "https://www.somesite.com"
http = httplib2.Http(disable_ssl_certificate_validation=True)
status, response = http.request(start_url)
However, when I run it, I get this error:
Traceback (most recent call last): File "C:\Documents and Settings\DD\Desktop\crawler.py", line 15, in <module>
resp, content = h.request(start_url, "GET") File "C:\Python27\lib\site-packages\httplib2-0.9-py2.7.egg\httplib2\__init__.py", line 1593, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "C:\Python27\lib\site-packages\httplib2-0.9-py2.7.egg\httplib2\__init__.py", line 1335, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers) File "C:\Python27\lib\site-packages\httplib2-0.9-py2.7.egg\httplib2\__init__.py", line 1257, in _conn_request
conn.connect() File "C:\Python27\lib\site-packages\httplib2-0.9-py2.7.egg\httplib2\__init__.py", line 1044, in connect
raise SSLHandshakeError(e) httplib2.SSLHandshakeError: [Errno 1] _ssl.c:510: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
As you may have seen, I tried to disable the ssl validation but with no succsess.
Any Help?
Thanks!

SSL3_GET_RECORD:decryption failed or bad record mac
This has nothing to do with SSL validation. It might be that the server simply does not talk SSL or that there are other SSL related problems, but validation is not one of them at this stage of communication.
If you provide the real URL or a full packet capture (file or at cloudshark.org) one might analyze the information in more detail.

SSL error using Python Requests to access Shibboleth authenticated server

I'm trying to access a journal article hosted by an academic service provider (SP), using a Python script.
The server authenticates using a Shibboleth login. I read Logging into SAML/Shibboleth authenticated server using python and tried to implement a login with Python Requests.
The script starts by querying the SP for the link leading to my IDP institution, and is supposed then to authenticate automatically with the IDP. The first part works, but when following the link to the IDP, it chokes on an SSL error.
Here is what I used:
import requests
import lxml.html
LOGINLINK = 'https://www.jsave.org/action/showLogin?redirectUri=%2F'
USERAGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0'
s = requests.session()
s.headers.update({'User-Agent' : USERAGENT})
# getting the page where you can search for your IDP
# need to get the cookies so we can continue
response = s.get(LOGINLINK)
rtext = response.text
print('Don\'t see your school?' in rtext) # prints True
# POSTing the name of my institution
data = {
'institutionName' : 'tubingen',
'submitForm' : 'Search',
'currUrl' : '%2Faction%2FshowBasicSearch',
'redirectUri' : '%2F',
'activity' : 'isearch'
}
response = s.post(BASEURL + '/action/showLogin', data=data)
rtext = response.text
print('university of tubingen' in rtext) # prints True
# get the link that leads to the IDP
tree = lxml.html.fromstring(rtext)
loginlinks = tree.cssselect('a.extLogin')
if (loginlinks):
loginlink = loginlinks[0].get('href')
else:
exit(1)
print('continuing to IDP')
response = s.get(loginlink)
rtext = response.text
print('zentrale Anmeldeseite' in rtext)
This yields:
continuing to IDP...
2014-04-04 10:04:06,010 - INFO - Starting new HTTPS connection (1): idp.uni-tuebingen.de
Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 480, in urlopen
body=body, headers=headers)
File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 285, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.4/http/client.py", line 1066, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.4/http/client.py", line 1104, in _send_request
self.endheaders(body)
File "/usr/lib/python3.4/http/client.py", line 1062, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.4/http/client.py", line 907, in _send_output
self.send(msg)
File "/usr/lib/python3.4/http/client.py", line 842, in send
self.connect()
File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/connection.py", line 164, in connect
ssl_version=resolved_ssl_version)
File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/util.py", line 639, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.4/ssl.py", line 344, in wrap_socket
_context=self)
File "/usr/lib/python3.4/ssl.py", line 540, in __init__
self.do_handshake()
File "/usr/lib/python3.4/ssl.py", line 767, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:598)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/requests/adapters.py", line 330, in send
timeout=timeout
File "/usr/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 504, in urlopen
raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:598)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./try.py", line 154, in <module>
response = s.get(loginlink)
File "/usr/lib/python3.4/site-packages/requests/sessions.py", line 395, in get
return self.request('GET', url, **kwargs)
File "/usr/lib/python3.4/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.4/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.4/site-packages/requests/adapters.py", line 385, in send
raise SSLError(e)
requests.exceptions.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:598)
Using s.get(loginlink, verify=False) yields exactly the same error. Simply using urllib.request.urlopen(loginlink) does so, too.
Printing and pasting the link into Firefox, on the other hand, works fine.

After trying with openssl s_client it looks like the destination idp.uni-tuebingen.de:443 is only support SSLv3 and misbehaving on anything newer. With forcing SSLv3 one gets:
$ openssl s_client -connect idp.uni-tuebingen.de:443 -ssl3
CONNECTED(00000003)
depth=3 C = DE, O = Deutsche Telekom AG, OU = T-TeleSec Trust Center, CN = Deutsche Telekom Root CA 2
...
But with default setup or forcing TLv1 (-tls1) it only returns an alert:
openssl s_client -connect idp.uni-tuebingen.de:443
CONNECTED(00000003)
140493591938752:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:741:
So you need to find a way to force SSLv3 for this connection. I'm not familiar with the python at this point but maybe http://docs.python-requests.org/en/latest/user/advanced/ chapter "Example: Specific SSL Version" helps.
And why it works with firefox: the browsers usually retry with a downgraded SSL version if the connects with the safer versions fail. E.g. everybody is trying to work around broken stuff so that the owner of the broken stuff has no intention to fix it :(

Ignore certificate validation with urllib3

I'm using urllib3 against private services that have self signed certificates. Is there any way to have urllib3 ignore the certificate errors and make the request anyways?
import urllib3
c = urllib3.HTTPSConnectionPool('10.0.3.168', port=9001)
c.request('GET', '/')
When using the following:
import urllib3
c = urllib3.HTTPSConnectionPool('10.0.3.168', port=9001, cert_reqs='CERT_NONE')
c.request('GET', '/')
The following error is raised:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3/dist-packages/urllib3/request.py", line 67, in request
**urlopen_kw)
File "/usr/lib/python3/dist-packages/urllib3/request.py", line 80, in request_encode_url
return self.urlopen(method, url, **urlopen_kw)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 415, in urlopen
body=body, headers=headers)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 267, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.3/http/client.py", line 1061, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.3/http/client.py", line 1099, in _send_request
self.endheaders(body)
File "/usr/lib/python3.3/http/client.py", line 1057, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.3/http/client.py", line 902, in _send_output
self.send(msg)
File "/usr/lib/python3.3/http/client.py", line 840, in send
self.connect()
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 103, in connect
match_hostname(self.sock.getpeercert(), self.host)
File "/usr/lib/python3/dist-packages/urllib3/packages/ssl_match_hostname/__init__.py", line 32, in match_hostname
raise ValueError("empty or no certificate")
ValueError: empty or no certificate
Using cURL I'm able to get the expected response from the service
$ curl -k https://10.0.3.168:9001/
Please read the documentation for API endpoints

Try following code:
import urllib3
c = urllib3.HTTPSConnectionPool('10.0.3.168', port=9001, cert_reqs='CERT_NONE',
assert_hostname=False)
c.request('GET', '/')
See Setting assert_hostname to False will disable SSL hostname verification

In this question I see many answers but, IMHO, too much unnecessary information that can lead to confusion.
Just add the cert_reqs='CERT_NONE' parameter
import urllib3
http = urllib3.PoolManager(cert_reqs='CERT_NONE')

I found the answer to my problem. The urllib3 documentation does not, in fact, completely explain how to suppress SSL certificate validation. What is missing is a reference to ssl.CERT_NONE.
My code has a boolean, ssl_verify, to indicate whether or not I want SSL validation. The code now looks like this:
import ssl
import urllib3
#
#
#
if (ssl_verify):
cert_reqs = ssl.CERT_REQUIRED
else:
cert_reqs = ssl.CERT_NONE
urllib3.disable_warnings()
http = urllib3.PoolManager(cert_reqs = cert_reqs)
auth_url = f'https://{fmc_ip}/api/fmc_platform/v1/auth/generatetoken'
type = {'Content-Type': 'application/json'}
auth = urllib3.make_headers(basic_auth=f'{username}:{password}')
headers = { **type, **auth }
resp = http.request('POST',
auth_url,
headers=headers,
timeout=10.0)

Try to instanciate your connection pool this way:
HTTPSConnectionPool(self.host, self.port, cert_reqs=ssl.CERT_NONE)
or this way:
HTTPSConnectionPool(self.host, self.port, cert_reqs='CERT_NONE')
Source: https://github.com/shazow/urllib3/blob/master/test/with_dummyserver/test_https.py
EDIT (after seeing your edit):
It looks like the remote host didn't send a certificate (is it possible?).
This is the code (from urllib3) which raised an exception:
def match_hostname(cert, hostname):
"""Verify that *cert* (in decoded format as returned by
SSLSocket.getpeercert()) matches the *hostname*. RFC 2818 rules
are mostly followed, but IP addresses are not accepted for *hostname*.
CertificateError is raised on failure. On success, the function
returns nothing.
"""
if not cert:
raise ValueError("empty or no certificate")
So it looks like cert is empty, which means that self.sock.getpeercert() returned an empty string.

python httplib Name or service not known

I'm trying to use httplib to send credit card information to authorize.net. When i try to post the request, I get the following traceback:
File "./lib/cgi_app.py", line 139, in run res = method()
File "/var/www/html/index.py", line 113, in ProcessRegistration conn.request("POST", "/gateway/transact.dll", mystring, headers)
File "/usr/local/lib/python2.7/httplib.py", line 946, in request self._send_request(method, url, body, headers)
File "/usr/local/lib/python2.7/httplib.py", line 987, in _send_request self.endheaders(body)
File "/usr/local/lib/python2.7/httplib.py", line 940, in endheaders self._send_output(message_body)
File "/usr/local/lib/python2.7/httplib.py", line 803, in _send_output self.send(msg)
File "/usr/local/lib/python2.7/httplib.py", line 755, in send self.connect()
File "/usr/local/lib/python2.7/httplib.py", line 1152, in connect self.timeout, self.source_address)
File "/usr/local/lib/python2.7/socket.py", line 567, in create_connection raise error, msg
gaierror: [Errno -2] Name or service not known
I build my request like so:
mystring = urllib.urlencode(cardHash)
headers = {"Content-Type": "text/xml", "Content-Length": str(len(mystring))}
conn = httplib.HTTPSConnection("secure.authorize.net:443", source_address=("myurl.com", 443))
conn.request("POST", "/gateway/transact.dll", mystring, headers)
to add another layer to this, it was working on our development server which has httplib 2.6 and without the source_address parameter in httplib.HTTPSConnection.
Any help is greatly appreciated.
===========================================================
EDIT:
I can run it from command line. Apparently this is some sort of permissions issue. Any ideas what permissions I would need to grant to which users to make this happen? Possibly Apache can't open the port?

As an (obvious) heads up, this same error can also be triggered by including the protocol in the host parameter. For example this code:
conn = httplib.HTTPConnection("http://secure.authorize.net", 80, ....)
will also cause the "gaierror: [Errno -2] Name or service not known" error, even if all your networking setup is correct.

gaierror: [Errno -2] Name or service not known
This error often indicates a failure of your DNS resolver. Does ping secure.authorize.net return successful replies from the same server that receives the gaierror? Does the hostname have a typo in it?

The problem ultimately came down to the fact that selinux was stopping apache from getting that port. Disabling selinux fixed the problems. I had an issue later where i didn't have /var/www/.python-eggs/, so MySQLdb was hosing on import. But after a mkdir, it was fixed.

pass the port separately from the host:
conn = httplib.HTTPSConnection("secure.authorize.net", 443, ....)