Develop Reference

Basic Authentication returns 401 Client Error but works in postman

I have gone through number of similar posts related to firing GET requests with Basic Auth (eg: Python, HTTPS GET with basic authentication), still can't figure out the problem. I keep getting the error requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url
With the same credentials, headers tried the same in postman it works as expected. Verified that base64encoded value for the api_key, password is exactly same as the value used in postman, so I don't think its encoding or resource access permission problem.
python -V
Python 3.6.4 :: Anaconda, Inc.
Approach 1
api_key = 'some_api_key'
password = 'some_password'
headers = {'accept': 'application/json'}
url = 'https://test.access.com/this/url'
api_key_password = "%s:%s" % (api_key, password)
b64_encoded = b64encode(bytes(api_key_password, 'utf-8')).decode("ascii")
headers['authorization'] = 'Basic %s' % b64_encoded
response = requests.get(url,
headers=headers)
if (response.ok):
json_data = json.loads(response.content)
print (json_data)
else:
print (response)
response.raise_for_status()
Approach 2
api_key = 'some_api_key'
password = 'some_password'
url = 'https://test.access.com/this/url'
headers = {
'accept': 'application/json',
}
response = requests.get(url, headers=headers, auth=(api_key, password))
print (response.ok)
if (response.ok):
json_data = json.loads(response.content)
print (json_data)
else:
print (response)
response.raise_for_status()
Can you please provide some pointers?

I had a similar issue (although in .NET Framework).
In my case the reason was that I was using the url without a forward slash in the end and the API apparently does not support that.
So https://test.access.com/this/url
Throws 401 error Unauthorized
but
https://test.access.com/this/url/
Returns 200 OK.

Older post but I had a similar issue. Postman will cache your JSESSIONID. Be sure you are clearing out that cookie while testing. If you are hitting an API that requires a login API call to establish a session before you can make subsequent API calls, this Postman behavior can produce a false sense of security.
In this situation with Python requests, it can be handled with code similar to what I've provided below:
import requests,json
loginAPI = "https://myapi.myco.comv/someuri/someuri/users/login"
someHTTPGetAPI = "https://myapi.myco.com/someuri/someuri/someservice"
username = "myuser"
password = "mypass"
headers = {
"Content-Type": "application/json",
"login": username,
"password": password
}
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
verify=False
session = requests.Session()
sessionResponse = session.get(url=loginURL,headers=headers, verify=verify)
if sessionResponse.status_code == 200:
getResponse = session.get(url=someHTTPGetAPI)
if getResponse.status_code == 200:
responseJSON = agentStatus.json()

HTTP Basic Auth Failing or Incomplete

I'm trying to log into the vRealize Operations Rest API using basic auth method from this question:
HTTP Basic Authentication not working in Python 3.4
So I use the second code sample:
import urllib.parse
import urllib.request
import urllib.response
userName = "username"
passWord = "password"
top_level_url = "URL"
# create an authorization handler
p = urllib.request.HTTPPasswordMgrWithDefaultRealm()
p.add_password(None, top_level_url, userName, passWord);
auth_handler = urllib.request.HTTPBasicAuthHandler(p)
opener = urllib.request.build_opener(auth_handler)
urllib.request.install_opener(opener)
try:
result = opener.open(top_level_url)
messages = result.read()
print (messages)
except IOError as e:
print (e)
and I get a blue font: HTTP Error 401: Unauthorized
So I think this is an issue with trusting the certificate or it has something to do with the headers. How can I work around this? Any advice would be appreciated.

Python POST binary data

I am writing some code to interface with redmine and I need to upload some files as part of the process, but I am not sure how to do a POST request from python containing a binary file.
I am trying to mimic the commands here:
curl --data-binary "#image.png" -H "Content-Type: application/octet-stream" -X POST -u login:password http://redmine/uploads.xml
In python (below), but it does not seem to work. I am not sure if the problem is somehow related to encoding the file or if something is wrong with the headers.
import urllib2, os
FilePath = "C:\somefolder\somefile.7z"
FileData = open(FilePath, "rb")
length = os.path.getsize(FilePath)
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, 'http://redmine/', 'admin', 'admin')
auth_handler = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
request = urllib2.Request( r'http://redmine/uploads.xml', FileData)
request.add_header('Content-Length', '%d' % length)
request.add_header('Content-Type', 'application/octet-stream')
try:
response = urllib2.urlopen( request)
print response.read()
except urllib2.HTTPError as e:
error_message = e.read()
print error_message
I have access to the server and it looks like a encoding error:
...
invalid byte sequence in UTF-8
Line: 1
Position: 624
Last 80 unconsumed characters:
7z¼¯'ÅÐз2^Ôøë4g¸R<süðí6kĤª¶!»=}jcdjSPúá-º#»ÄAtD»H7Ê!æ½]j):
(further down)
Started POST "/uploads.xml" for 192.168.0.117 at 2013-01-16 09:57:49 -0800
Processing by AttachmentsController#upload as XML
WARNING: Can't verify CSRF token authenticity
Current user: anonymous
Filter chain halted as :authorize_global rendered or redirected
Completed 401 Unauthorized in 13ms (ActiveRecord: 3.1ms)

Basically what you do is correct. Looking at redmine docs you linked to, it seems that suffix after the dot in the url denotes type of posted data (.json for JSON, .xml for XML), which agrees with the response you get - Processing by AttachmentsController#upload as XML. I guess maybe there's a bug in docs and to post binary data you should try using http://redmine/uploads url instead of http://redmine/uploads.xml.
Btw, I highly recommend very good and very popular Requests library for http in Python. It's much better than what's in the standard lib (urllib2). It supports authentication as well but I skipped it for brevity here.
import requests
with open('./x.png', 'rb') as f:
data = f.read()
res = requests.post(url='http://httpbin.org/post',
data=data,
headers={'Content-Type': 'application/octet-stream'})
# let's check if what we sent is what we intended to send...
import json
import base64
assert base64.b64decode(res.json()['data'][len('data:application/octet-stream;base64,'):]) == data
UPDATE
To find out why this works with Requests but not with urllib2 we have to examine the difference in what's being sent. To see this I'm sending traffic to http proxy (Fiddler) running on port 8888:
Using Requests
import requests
data = 'test data'
res = requests.post(url='http://localhost:8888',
data=data,
headers={'Content-Type': 'application/octet-stream'})
we see
POST http://localhost:8888/ HTTP/1.1
Host: localhost:8888
Content-Length: 9
Content-Type: application/octet-stream
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.0.4 CPython/2.7.3 Windows/Vista
test data
and using urllib2
import urllib2
data = 'test data'
req = urllib2.Request('http://localhost:8888', data)
req.add_header('Content-Length', '%d' % len(data))
req.add_header('Content-Type', 'application/octet-stream')
res = urllib2.urlopen(req)
we get
POST http://localhost:8888/ HTTP/1.1
Accept-Encoding: identity
Content-Length: 9
Host: localhost:8888
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7
test data
I don't see any differences which would warrant different behavior you observe. Having said that it's not uncommon for http servers to inspect User-Agent header and vary behavior based on its value. Try to change headers sent by Requests one by one making them the same as those being sent by urllib2 and see when it stops working.

This has nothing to do with a malformed upload. The HTTP error clearly specifies 401 unauthorized, and tells you the CSRF token is invalid. Try sending a valid CSRF token with the upload.
More about csrf tokens here:
What is a CSRF token ? What is its importance and how does it work?

you need to add Content-Disposition header, smth like this (although I used mod-python here, but principle should be the same):
request.headers_out['Content-Disposition'] = 'attachment; filename=%s' % myfname

You can use unirest, It provides easy method to post request.
`
import unirest
def callback(response):
print "code:"+ str(response.code)
print "******************"
print "headers:"+ str(response.headers)
print "******************"
print "body:"+ str(response.body)
print "******************"
print "raw_body:"+ str(response.raw_body)
# consume async post request
def consumePOSTRequestASync():
params = {'test1':'param1','test2':'param2'}
# we need to pass a dummy variable which is open method
# actually unirest does not provide variable to shift between
# application-x-www-form-urlencoded and
# multipart/form-data
params['dummy'] = open('dummy.txt', 'r')
url = 'http://httpbin.org/post'
headers = {"Accept": "application/json"}
# call get service with headers and params
unirest.post(url, headers = headers,params = params, callback = callback)
# post async request multipart/form-data
consumePOSTRequestASync()

HTTP 400 response while sending POST request to UrbanAirship

I am using Web2Py to create a simple app which sends Push notifications through UrbanAirship. For some reason, I am getting a 400 response when I try to send it through my code. It UA API works fine using REST client. This is my code:
url = 'https://go.urbanairship.com/api/push/'
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
# this creates a password manager
passman.add_password(None, url, username, password)
# because we have put None at the start it will always
# use this username/password combination for urls
# for which `theurl` is a super-url
authhandler = urllib2.HTTPBasicAuthHandler(passman)
# create the AuthHandler
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
# All calls to urllib2.urlopen will now use our handler
# Make sure not to include the protocol in with the URL, or
# HTTPPasswordMgrWithDefaultRealm will be very confused.
# You must (of course) use it when fetching the page though.
values = {"device_tokens": ["<DEVICE TOKEN>"], "aps": {"alert": "Hello!"}}
data = urllib.urlencode(values)
headers = {'Content-Type': 'application/json'}
req = urllib2.Request(url, data, headers)
try:
response = urllib2.urlopen(req)
return response
except IOError, e:
if e.code == 200:
return "Push sent!"
else:
return 'The server couldn\'t fulfill the request. Error: %d' % e.code
As far as I can understand, the problem is in the format of data being sent. Where am I going wrong?

The urllib.urlencode function is for making a URL-encoded parameter body (Content-Type: application/x-www-form-urlencoded). For JSON, which is apparently what you want, use json.dumps instead.

does urllib2 support preemptive authentication authentication?

I am trying access a REST API.
I can get it working in Curl/REST Client (the UI tool), with preemptive authentication enabled.
But, using urllib2, it doesn't seem to support this by default and I can't find a way to turn it on.
Thanks :)

Here's a simple Preemptive HTTP basic auth handler, based on the code from urllib2.HTTPBasicAuthHandler. It can be used in the exact same manner, except an Authorization header will be added to every request with a matching URL. Note that this handler should be used with a HTTPPasswordMgrWithDefaultRealm. That's because there is no realm coming back in a WWW-Authenticate challenge since you're being preemptive.
class PreemptiveBasicAuthHandler(urllib2.BaseHandler):
def __init__(self, password_mgr=None):
if password_mgr is None:
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
self.passwd = password_mgr
self.add_password = self.passwd.add_password
def http_request(self,req):
uri = req.get_full_url()
user, pw = self.passwd.find_user_password(None,uri)
#logging.debug('ADDING REQUEST HEADER for uri (%s): %s:%s',uri,user,pw)
if pw is None: return req
raw = "%s:%s" % (user, pw)
auth = 'Basic %s' % base64.b64encode(raw).strip()
req.add_unredirected_header('Authorization', auth)
return req

similar to #thom-nichols's answer; but subclassing HTTPBasicAuthHandler also handling HTTPS requests.
import urllib2
import base64
class PreemptiveBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
'''Preemptive basic auth.
Instead of waiting for a 403 to then retry with the credentials,
send the credentials if the url is handled by the password manager.
Note: please use realm=None when calling add_password.'''
def http_request(self, req):
url = req.get_full_url()
realm = None
# this is very similar to the code from retry_http_basic_auth()
# but returns a request object.
user, pw = self.passwd.find_user_password(realm, url)
if pw:
raw = "%s:%s" % (user, pw)
auth = 'Basic %s' % base64.b64encode(raw).strip()
req.add_unredirected_header(self.auth_header, auth)
return req
https_request = http_request
here is an example for dealing with a jenkins server which does not send you 401 http errors (retry with auth). I'm using urllib2.install_opener to make things easy.
jenkins_url = "https://jenkins.example.com"
username = "johndoe"
api_token = "some-cryptic-value"
auth_handler = PreemptiveBasicAuthHandler()
auth_handler.add_password(
realm=None, # default realm.
uri=jenkins_url,
user=username,
passwd=api_token)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)

Depending on what kind of authentication is required, you can send the Authorization headers manually by adding them to your request before you send out a body.