Develop Reference

PUT Data on HDFS via HTTP WEB API

I'm trying to implement a PUT request on HDFS via the HDFS Web API.
So I looked up the Documentation on how to do that : https://hadoop.apache.org/docs/r1.0.4/webhdfs.html#CREATE
First do a PUT without redirect, to get a 307, catch the new URL and then PUT on that URL with DATA.
When I do my first put, I do get the 307, but the URL is the same has the first one. So I'm note sure if I'm already on the "good" datanode or not.
Any way, I get this URL and try to add DATA to it, but I get an error, from what I understand, it is a connection error. Host is cutting down the connection.
class HttpFS:
def __init__(self, url=settings.HTTPFS_URL):
self.httpfs = url
self.auth = HTTPKerberosAuth(mutual_authentication=OPTIONAL)
def put(self, local_file, hdfs_dir):
url = "{}{}".format(self.httpfs, hdfs_dir)
params = {"op": "CREATE", "overwrite": True}
print(url)
r = requests.put(url, auth=self.auth, params=params, stream=True, verify=settings.CA_ROOT_PATH, allow_redirects=False)
r = requests.put(r.headers['Location'], auth=self.auth, data=open(local_file, 'rb'), params=params, stream=True, verify=settings.CA_ROOT_PATH)
Here is the error given:
r = requests.put(r.headers['Location'], auth=self.auth, data=open(local_file, 'rb'), params=params, stream=True, verify=settings.CA_ROOT_PATH)
File "/home/cdsw/.local/lib/python3.6/site-packages/requests/api.py", line 134, in put
return request('put', url, data=data, **kwargs)
File "/home/cdsw/.local/lib/python3.6/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/cdsw/.local/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/cdsw/.local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/cdsw/.local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))
Edit 1:
I tried also with https://github.com/pywebhdfs/pywebhdfs repos. Since it is supposed to do exactly what i'm looking for. But I still have this Broken Pipe Error.
from requests_kerberos import OPTIONAL, HTTPKerberosAuth
from pywebhdfs.webhdfs import PyWebHdfsClient
from utils import settings
auth = HTTPKerberosAuth(mutual_authentication=OPTIONAL)
url = f"https://{settings.HDFS_HTTPFS_HOST}:{settings.HDFS_HTTPFS_PORT}/webhdfs/v1/"
hdfs_client = PyWebHdfsClient(base_uri_pattern=url, request_extra_opts={'auth':auth, 'verify': settings.CA_ROOT_PATH})
with open(data_dir + file_name, 'rb') as file_data:
hdfs_client.create_file(hdfs_path + file_name, file_data=file_data, overwrite=True)
Same error:
hdfs_client.create_file(hdfs_path + file_name, file_data=file_data, overwrite=True)
File "/home/cdsw/.local/lib/python3.6/site-packages/pywebhdfs/webhdfs.py", line 115, in create_file
**self.request_extra_opts)
File "/home/cdsw/.local/lib/python3.6/site-packages/requests/api.py", line 134, in put
return request('put', url, data=data, **kwargs)
File "/home/cdsw/.local/lib/python3.6/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/cdsw/.local/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/cdsw/.local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/cdsw/.local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))
Edit 2:
I found out I was sending too much data at once. So now I create a file on HDFS, then I append to it with chunk of data. But it is slow... And I still can get the same error as above randomly. The bigger are the are the chunk the more I have chances to get a Connection aborted. My files more in range of 200Mb, so it take ages comparing to the Hadoop binary "hdfs dfs -put"

OpenStack - KeyStone Requires Authentication 401

I'm trying to list the projects (but the same thing occurs whether I try to do that, or attempt to list the users, so let's just generalise the error to any API call). Every time I do this, I get the following HTTP 401 code:
File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond
response.body = self.handler()
File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/cherrypy/lib/jsontools.py", line 61, in json_handler
value = cherrypy.serving.request._json_inner_handler(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__
return self.callable(*self.args, **self.kwargs)
File "/var/www/frontend/controllers/api/user.py", line 63, in PUT
print keystoneClient.projects.list()
File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
return wrapped(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneclient/v3/projects.py", line 107, in list
**kwargs)
File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 75, in func
return f(*args, **new_kwargs)
File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 383, in list
self.collection_key)
File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 124, in _list
resp, body = self.client.get(url, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 173, in get
return self.request(url, 'GET', **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 331, in request
resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 98, in request
return self.session.request(url, method, **kwargs)
File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
return wrapped(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 387, in request
auth_headers = self.get_auth_headers(auth)
File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 647, in get_auth_headers
return auth.get_headers(self, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 84, in get_headers
token = self.get_token(session)
File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
return self.get_access(session).auth_token
File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
self.auth_ref = self.get_auth_ref(session)
File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref
authenticated=False, log=False, **rkwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 595, in post
return self.request(url, 'POST', **kwargs)
File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
return wrapped(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 484, in request
raise exceptions.from_response(resp, method, url)
Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-39d02130-6f47-4cae-bc30-0b645296752e)
Code:
import cherrypy
import ldap
from keystoneauth1 import loading
from keystoneauth1 import session as session
from keystoneclient.v3 import client as client
from keystoneauth1.identity import v3
import json
# Storing username and password in a cherrypy session
cherrypy.session['username'] = data.get("username")
cherrypy.session['password'] = data.get("password").replace(" ","%20")
# Setting up KeyStone Client
auth = v3.Password(
auth_url = KEYSTONE_URL,
username = cherrypy.session['username'],
password = cherrypy.session['password'],
user_domain_name="default",
domain_name = "default"
)
sess = session.Session(auth=auth, verify='/etc/ssl/certs/ca-bundle.crt')
keystoneClient = client.Client(session=sess)
# Getting list of projects
print keystoneClient.projects.list()
This occurs whilst using a normal user's credentials. If I login using admin credentials, there is no error and the projects are listed. I would like to know:
Why is the overall error occurring? (from what I've read, it seems like it has to reauthenticate the credentials and it doesn't like it?)
Why is it not happening when I use admin credentials but with a 'normal' users credentials?
How do I solve this?

To answer the first question, you need a project scoped token since the unscoped token doesn't have any role associated with it. You cannot do any operation with the unscoped token. Here is a way to get the project scoped token:
auth = v3.Password(auth_url=auth_url,
username=username,
password=password,
user_domain_name="default",
project_name=project_name,
project_domain_name="default")
sess = session.Session(auth=auth, verify='/etc/ssl/certs/ca-bundle.crt')
keystoneClient = client.Client(session=sess)
And for a normal user, you cannot get back the entire list of projects in the provider and this is why it works fine with using admin credentials but not for that of a normal user.
In order to solve it, you need to know the user_id of that normal user. One way is to use the admin credentials to get a keystone client and call
keystone.users.get(user="username")
Or just use the OpenStack Dashboard and go to the identity dashboard. There is a panel called User and you can see the user_id from there.
Once you have the user_id, you can do:
keystoneClient.projects.list(user=user_id)
HTH.

SSL Error occurs on one computer but not the other?

I can't figure out why all of a sudden the below code that uses Asana's API generates the below SSL error. Something must have changed on my laptop, since it runs perfectly on my other computer.
from asana import asana
class Login(object):
def __init__(self):
api = 'API'
self.asana_api = asana.AsanaAPI(api, debug=False)
self.user_id = 7359085011308L
class Test(Login):
def Test(self):
Id = 2467584555313L
print self.asana_api.list_tasks(Id,self.user_id)
Traceback (most recent call last):
File "/Users/Chris/Dropbox/AsanaPullPush.py", line 75, in <module>
if __name__ == "__main__": main()
File "/Users/Chris/Dropbox/AsanaPullPush.py", line 72, in main
print Test().Test()
File "/Users/Chris/Dropbox/AsanaPullPush.py", line 15, in Test
print self.asana_api.list_tasks(Id,self.user_id)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/asana/asana.py", line 174, in list_tasks
return self._asana(target)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/asana/asana.py", line 74, in _asana
r = requests.get(target, auth=(self.apikey, ""))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/adapters.py", line 389, in send
raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:507: error:0D0890A1:asn1 encoding routines:ASN1_verify:unknown message digest algorithm

We recently changed our SSL key in response to the Heartbleed bug you may have heard about. http://blog.asana.com/2014/04/heartbleed/
It looks like your laptop may not have the right SSL. See https://github.com/pypa/pip/issues/829 for discussion of a similar issue.
You should be able to check SSL version on the two machines with python -c "import ssl; print ssl.OPENSSL_VERSION". If indeed the laptop is behind, you'll need to update your python's SSL.

Fetching an jira issue using python i'm getting authentication error

>>> # Get all projects viewable by anonymous users.
... projects = jira.projects()
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "C:\Python27\lib\site-packages\jira_python-0.16-py2.7.egg\jira\client.py", line 941, in projects
r_json = self._get_json('project')
File "C:\Python27\lib\site-packages\jira_python-0.16-py2.7.egg\jira\client.py", line 1526, in _get_json
r = self._session.get(url, params=params, headers=self._options['headers'])
File "C:\Python27\lib\site-packages\requests-2.2.1-py2.7.egg\requests\sessions.py", line 395, in get
return self.request('GET', url, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.2.1-py2.7.egg\requests\sessions.py", line 383, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests-2.2.1-py2.7.egg\requests\sessions.py", line 486, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.2.1-py2.7.egg\requests\adapters.py", line 381, in send
raise ProxyError(e)
**requests.exceptions.ProxyError: Cannot connect to proxy. Socket error: Tunnel connection failed: 407 authenticationrequired.**
Can you please any one help me how to solve this issue.

Try This:
basic_auth = ('username', 'password')
options = {
'server': <server url>,
'proxies' : <proxy>,
'verify' : False,
}
jira = JIRA(options,basic_auth=basic_auth,validate=True)
should work.
I tried and it is working for me.
Make sure you have access to the project to acess the issue.

using POST with Angularjs and Google App Engine Python Server

I have a problem here.
I have been using $resource to do simple GET, but recently i have a large amount of data to send to the server, but gives a 414 Error URI too large. I know that one of the solution is to use POST instead of GET.
I tried to search for solutions such as $http.post and passing parameters to $http, but to no avail. Can someone give me a tutorial on how should i do it?
Edit:
Here is what i have done:
var data_2 = $.param({'method':"update_proj",
'proj_id': proj_id,
'ptitle': project.title,
'pdescription': p_desc,
'pexposure': tech,
'pcompany': project.company,
'pteamsize':project.teamsize,
'ppoc': project.poc,
'pemail': c_email,
'pcontact': project.contact,
'pimg':project.img,
'pvideo':project.video,
'prskill': rskills,
'poutcome': project.outcome});
$http({method:'update_proj',
url: "http://osmosisgal.appspot.com/update_proj",
data: data_2,
headers: {'Content-Type': 'application/x-www-form-urlencoded'}})
.success(function(data,status){
...
});
Here is my python code on GAE:
class ActionHandler(webapp.RequestHandler):
def update_project(self):
proj_id = self.request.POST["proj_id"]
.
.
.
But i still have this error
Traceback (most recent call last):
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~osmosisgal/1.366101252356646323/new_backend.py", line 1274, in update_project
proj_id = self.request.POST["proj_id"]
File "/python27_runtime/python27_lib/versions/third_party/webob-1.1.1/webob/multidict.py", line 312, in __getitem__
return self._decode_value(self.multi.__getitem__(self._encode_key(key)))
File "/python27_runtime/python27_lib/versions/third_party/webob-1.1.1/webob/multidict.py", line 568, in __getitem__
raise KeyError("No key %r: %s" % (key, self.reason))
KeyError: "No key 'proj_id': Not a form request"

From documentation you can see that method is getting the type of method and not the url, in first place.
$http({method: 'POST', url: '/someUrl', data: my_data}).
success(function(data, status, headers, config) {
// this callback will be called asynchronously
// when the response is available
}).
error(function(data, status, headers, config) {
// called asynchronously if an error occurs
// or server returns response with an error status.
});
For you data you can create a json object with pairs, key-value and pass it to the function like above.
Also do not forget to pass $http in your controller definition.
function MyController($scope, $http) {...};

I managed to solve it. Initially I did not host all my static files in GAE, after i hosted it, everything works.