Develop Reference

Write on HDFS using Python

I am trying to write on HDFS from Python.
Right now, I am using https://hdfscli.readthedocs.io/en/latest/quickstart.html
but for large file I get back:
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/hdfs/client.py", line 400, in write
consumer(data)
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/hdfs/client.py", line 394, in consumer
auth=False,
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/hdfs/client.py", line 179, in _request
**kwargs
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', gaierror(-2, 'Name or service not known'))
My code for writing is pretty simple:
client = InsecureClient('http://xxxxxxx.co:50070', user='hdfs')
client.write("/tmp/a",stringToWrite)
Anyone can suggest a decent package to write on HDFS?
Cheers

For the stacktrace, it seems to be security related. Are you sure you need to use the InsecureClient and not the Kerberos one?. Also, remember that library is just a binding for HttpFs, so doing a manual test with Postman or CURL would let you debug any issue cluster-side.

Python httplib2 "httplib2.SSLHandshakeError"

I have this piece of code that tries to get the page content from a given url.
import httplib2
start_url = "https://www.somesite.com"
http = httplib2.Http(disable_ssl_certificate_validation=True)
status, response = http.request(start_url)
However, when I run it, I get this error:
Traceback (most recent call last): File "C:\Documents and Settings\DD\Desktop\crawler.py", line 15, in <module>
resp, content = h.request(start_url, "GET") File "C:\Python27\lib\site-packages\httplib2-0.9-py2.7.egg\httplib2\__init__.py", line 1593, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "C:\Python27\lib\site-packages\httplib2-0.9-py2.7.egg\httplib2\__init__.py", line 1335, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers) File "C:\Python27\lib\site-packages\httplib2-0.9-py2.7.egg\httplib2\__init__.py", line 1257, in _conn_request
conn.connect() File "C:\Python27\lib\site-packages\httplib2-0.9-py2.7.egg\httplib2\__init__.py", line 1044, in connect
raise SSLHandshakeError(e) httplib2.SSLHandshakeError: [Errno 1] _ssl.c:510: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
As you may have seen, I tried to disable the ssl validation but with no succsess.
Any Help?
Thanks!

SSL3_GET_RECORD:decryption failed or bad record mac
This has nothing to do with SSL validation. It might be that the server simply does not talk SSL or that there are other SSL related problems, but validation is not one of them at this stage of communication.
If you provide the real URL or a full packet capture (file or at cloudshark.org) one might analyze the information in more detail.

Automated Test Run - Python HTTPResponseNotReady()

I am trying to create an automated test using python for testing a youtube API request and response and all of those happening in random quick timely manner.
What I have been getting is an unstable HTTP response from the server.
I am not using the same object for every connection as I made it in separate methods, but when testing it, I test it all in same method (i.e. call create,edit, and delete consecutively)
Here is the Error that I got.
File "/var/lib/jenkins/shiningpanda/jobs/2a430f4f/virtualenvs/d41d8cd9/local/lib/python2.7/site-packages/oauth2client/client.py", line 490, in new_request
redirections, connection_type)
File "/var/lib/jenkins/shiningpanda/jobs/2a430f4f/virtualenvs/d41d8cd9/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1570, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/var/lib/jenkins/shiningpanda/jobs/2a430f4f/virtualenvs/d41d8cd9/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1317, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/var/lib/jenkins/shiningpanda/jobs/2a430f4f/virtualenvs/d41d8cd9/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1286, in _conn_request
response = conn.getresponse()
File "/usr/lib/python2.7/httplib.py", line 1018, in getresponse
raise ResponseNotReady()
ResponseNotReady
I was thinking that I should have a time.delay() between each HTTP request?
What do you guys think and suggest me to do on this case, since I am still learning on this matter?
Thank you for all the sugesstion and help ;)
Basic Idea of the Code I am using :
yt_service = gdata.youtube.service.YouTubeService()
yt_service.email = 'exampple#gmail.com'
yt_service.password = 'password'
def GetAndPrintUserUploads(username):
yt_service = gdata.youtube.service.YouTubeService()
uri = 'http://gdata.youtube.com/feeds/api/users/%s/uploads' % username
PrintVideoFeed(yt_service.GetYouTubeVideoFeed(uri))
and for testing it I authenticate the user and print user upload a few time after that consecutively

Python BadStatusLine error on certain DELETE requests

I am trying to use the python-rest-client ( http://code.google.com/p/python-rest-client/wiki/Using_Connection ) to perform testing of some RESTful webservices. Since I'm just learning, I've been pointing my tests at the sample services provided at http://www.predic8.com/rest-demo.htm.
I have no problems with creating entries, updating entries, or retrieving entries (POST and GET requests). When I try make a DELETE request, it fails. I can use the Firefox REST Client to perform DELETE requests and they work. I can also make DELETE requests on other services, but I've been driving myself crazy trying to figure out why it doesn't work in this case. I'm using Python 3 with updated Httplib2, but I also tried Python 2.5 so that I could use the python-rest-client with the included version of Httplib2. I see the same problem in either case.
The code is simple, matching the documented use:
from restful_lib import Connection
self.base_url = "http://www.thomas-bayer.com"
self.conn = Connection(self.base_url)
response = self.conn.request_delete('/sqlrest/CUSTOMER/85')
I've looked at the resulting HTTP requests from the browser tool and from my code and I can't see why one works and the other doesn't. This is the trace I receive:
Traceback (most recent call last):
File "/home/fmk/python/rest-client/src/TestExampleService.py", line 68, in test_CRUD
self.Delete()
File "/home/fmk/python/rest-client/src/TestExampleService.py", line 55, in Delete
response = self.conn.request_delete('/sqlrest/CUSTOMER/85')
File "/home/fmk/python/rest-client/src/restful_lib.py", line 64, in request_delete
return self.request(resource, "delete", args, headers=headers)
File "/home/fmk/python/rest-client/src/restful_lib.py", line 138, in request
resp, content = self.h.request("%s://%s%s" % (self.scheme, self.host, '/'.join(request_path)), method.upper(), body=body, headers=headers )
File "/home/fmk/python/rest-client/src/httplib2/__init__.py", line 1175, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/home/fmk/python/rest-client/src/httplib2/__init__.py", line 931, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/home/fmk/python/rest-client/src/httplib2/__init__.py", line 897, in _conn_request
response = conn.getresponse()
File "/usr/lib/python3.2/http/client.py", line 1046, in getresponse
response.begin()
File "/usr/lib/python3.2/http/client.py", line 346, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.2/http/client.py", line 316, in _read_status
raise BadStatusLine(line)
http.client.BadStatusLine: ''
What's breaking? What do I do about it? Actually, I'd settle for advice on debugging it. I've changed the domain in my script and pointed it at my own machine so I could view the request. I've viewed/modified the Firefox requests in BurpProxy to make them match my script requests. The modified Burp requests still work and the Python requests still don't.

Apparently the issue is that the server expects there to be some message body for DELETE requests. That's an unusual expectation for a DELETE, but by specifying Content-Length:0 in the headers, I'm able to successfully perform DELETEs.
Somewhere along the way (in python-rest-client or httplib2), the Content-Length header is wiped out if I try to do:
from restful_lib import Connection
self.base_url = "http://www.thomas-bayer.com"
self.conn = Connection(self.base_url)
response = self.conn.request_delete('/sqlrest/CUSTOMER/85', headers={'Content-Length':'0'})
Just to prove the concept, I went to the point in the stack trace where the request was happening:
File "/home/fmk/python/rest-client/src/httplib2/__init__.py", line 897, in _conn_request
response = conn.getresponse()
I printed the headers parameter there to confirm that the content length wasn't there, then I added:
if(method == 'DELETE'):
headers['Content-Length'] = '0'
before the request.
I think the real answer is that the service is wonky, but at least I got to know httplib2 a little better. I've seen some other confused people looking for help with REST and Python, so hopefully I'm not the only one who got something out of this.

The following script correctly produces 404 response from the server:
#!/usr/bin/env python3
import http.client
h = http.client.HTTPConnection('www.thomas-bayer.com', timeout=10)
h.request('DELETE', '/sqlrest/CUSTOMER/85', headers={'Content-Length': 0})
response = h.getresponse()
print(response.status, response.version)
print(response.info())
print(response.read()[:77])
python -V => 3.2
curl -X DELETE http://www.thomas-bayer.com/sqlrest/CUSTOMER/85
curl: (52) Empty reply from server
Status-Line is not optional; HTTP server must return it. Or at least send 411 Length Required response.
curl -H 'Content-length: 0' -X DELETE \
http://www.thomas-bayer.com/sqlrest/CUSTOMER/85
Returns correctly 404.

Python YQL package Error

Hi I am new to Python...
I am trying to use YQL using Python.
I installed httplib2-0.7.0, oauth2 and then installed yql package
For this sample code :
import yql
y = yql.Public()
query = 'select * from flickr.photos.search where text="panda" limit 3';
result = y.execute(query)
print result
I got the following error message.
Please help!!
Traceback (most recent call last):
File "test.py", line 4, in
result = y.execute(query)
File "C:\Python27\lib\site-packages\yql-0.7-py2.7.egg\yql__init__.py", line 306, in execute
resp, content = self.http.request(url, http_method)
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 1436, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey
)
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 1188, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 1123, in _conn_request
conn.connect()
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 890, in connect
self.disable_ssl_certificate_validation, self.ca_certs)
File "C:\Python27\lib\site-packages\httplib2__init__.py", line 76, in _ssl_wrap_socket
cert_reqs=cert_reqs, ca_certs=ca_certs)
File "C:\Python27\lib\ssl.py", line 344, in wrap_socket
ciphers=ciphers)
File "C:\Python27\lib\ssl.py", line 119, in init
ciphers)
ssl.SSLError: [Errno 185090050] _ssl.c:336: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
When I tried to use the *twitter python wrapper, I ended up getting the same SSL error.*
Please tell me what to do

I can see two likely issues:
SSL Certificate
I'm not familiar with Python or that library you're using, but the error sounds like it can't verify the SSL certificate. (Possibly because there is no suitable local SSL certificate bundle for authentication.) You may be able to configure it to skip the SSL certificate verification.
YQL Flickr Query
The YQL query is not correct and gives an error in the YQL console:
select * from flickr.photos.search where text="panda" limit 3
Actually, given the age of this question, it may have worked in June 2011. Now the Flickr tables require an API key as well, so the working query would look like:
select * from flickr.photos.search where text="panda" and api_key="insert-your-key-here" limit 3