Develop Reference

Python/Requests: Correct login returns 401 unauthorized

I have a python application logs in to a remote host via basic HTTP authentication.
Authentication is as follows:
def make_authenticated_request(host, username, password):
url = host
r = requests.get(url, auth=(username, password))
r.raise_for_status()
return r
test = Looter.make_authenticated_request("http://" + host + "/status/status_deviceinfo.htm", user, password)
This error is printed:
401 Client Error: Unauthorized for url
Strange thing is that this doesn't always happen. It randomly fails/succeeds, for the same host with the same credentials.
Login is however correct, and works flawlessly in my browser. I'm no python ninja. Any clues ?

I might rewrite it to look something like this.. change it around however you need or like. The point here is that i'm using a session and passing it around to make another requests. You can reuse that session object to make other requests. Now if you making lots of requests an authing in each time like your code suggests. The site could be locking you out, which is why a session works better because you don't have to continue to auth in.
import requests
class Looter(object):
def __init__(self):
self.s = None
def create_session(self, url, username, password):
# create a Session
s = requests.Session()
# auth in
res = s.get(url, auth=(username, password))
if res.status_code == 200:
self.s = s
def make_request(self, url):
self.s.get(url)
#do something with the requests
l = Looter()
l.create_session(url, username, password)
# print the session to see if it authed in
print l.s.status_code

Python: Use single login for multiple request

There is some random website, http://www.example.com. This website use HTTP basic authentication.
I would like to make multiple requests to this site. But I don't want to login for each of the requests.
I have written the following code:
def loginWeb():
global cookie
homeURL = "https://www.example.com"
request = urllib2.Request(homeURL)
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
response = urllib2.urlopen(request)
cookie = response.headers.get('Set-Cookie')
The aforementioned code fetches the cookie and uses it for subsequent requests.
The following code makes the subsequent request:
def getHTMLSourceCode(ID):
opener = urllib2.build_opener()
opener.addheaders.append(('Cookie', cookie))
response = opener.open('https://www.example.com/' + trID)
sourceCode = response.read()
However, opener.open throws urllib2.HTTPError: HTTP Error 401: Unauthorized.
I also tried the following code, but this also throws the same error:
def getHTMLSourceCode(trID):
request = urllib2.Request("https://www.example.com/" + trID)
request.add_header('cookie', cookie)
response = urllib2.urlopen(request)
sourceCode = response.read()
return sourceCode
urllib2.urlopen(request) throws urllib2.HTTPError: HTTP Error 401: Unauthorized.
By the way, I went through following answers, but the problem still persists.
Python urllib2, basic HTTP authentication, and tr.im
Keeping a session in python while making HTTP requests
python: urllib2 how to send cookie with urlopen request

You may want to try requests library and use it Session object
s = requests.Session()
s.auth = (username, password)
s.get(homeURL)
The cookie will be set to your session cookie s.cookies and you can use this session for other requests.

What is the cleanest way to do HTTP POST with basic auth in Python?

What is the cleanest way to do HTTP POST with Basic Auth in Python?
Using only the Python core libs.

Seriously, just use requests:
import requests
resp = requests.post(url, data={}, auth=('user', 'pass'))
It's a pure python library, installing is as easy as easy_install requests or pip install requests. It has an extremely simple and easy to use API, and it fixes bugs in urllib2 so you don't have to. Don't make your life harder because of silly self-imposed requirements.

Hackish workaround works:
urllib.urlopen("https://username:password#hostname/path", data)
A lot of people don't realize that the old syntax for specifying username and password in the URL works in urllib.urlopen. It doesn't appear the username or password require any encoding, except perhaps if the password includes an "#" symbol.

if you define a url, username, password, and some post-data, this should work in Python2...
import urllib2
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, username, password)
auth_handler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
content = urllib2.urlopen(url, post_data)
example from official Python docs showing Basic Auth in urllib2:
* http://docs.python.org/release/2.6/howto/urllib2.html
full tutorial on Basic Authentication using urllib2:
* http://www.voidspace.org.uk/python/articles/authentication.shtml

XML POST REST Request using Python

Does anyone have a simple example of sending an XML POST request to a RESTful API with Python? I am trying to use the urllib2 Python library to "create a new project" in the Harvest API, with no luck. The payload variable is a valid XML document that is a near copy/paste of their documentation (under the Create New Project heading) shown here:
http://www.getharvest.com/api/projects
Here is the code I am trying to execute.
def postRequest():
""" Makes POST request to url, and returns a response. """
url = 'http://subdomain.harvestapp.com/projects'
opener = urllib2.build_opener()
opener.addheaders = [('Accept', 'application/xml'),
('Content-Type', 'application/xml'),
('Authorization', 'Basic %s' % base64.encodestring('%s:%s' % (self.username, self.password))[:-1]),
('User-Agent', 'Python-urllib/2.6')]
req = urllib2.Request(url=url, data=payload)
assert req.get_method() == 'POST'
response = self.opener.open(req)
print response.code
return response
I receive a response code 200 (Status OK) instead of a response code 201 (Created)...is this a question for the Harvest Support guys?
Any hints anyone has would be greatly appreciated.
Thanks,
Jeff.

It's common to return a 200 response even when a 201 response would strictly be more appropriate. Are you sure that the request isn't correctly processed even if you are getting a 'correct' response?

You're using a local opener everywhere except on the line where you create the response, where you use self.opener, which looks like the problem.

Python urllib2, basic HTTP authentication, and tr.im

I'm playing around, trying to write some code to use the tr.im
APIs to shorten a URL.
After reading http://docs.python.org/library/urllib2.html, I tried:
TRIM_API_URL = 'http://api.tr.im/api'
auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password(realm='tr.im',
uri=TRIM_API_URL,
user=USERNAME,
passwd=PASSWORD)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
response = urllib2.urlopen('%s/trim_simple?url=%s'
% (TRIM_API_URL, url_to_trim))
url = response.read().strip()
response.code is 200 (I think it should be 202). url is valid, but
the basic HTTP authentication doesn't seem to have worked, because the
shortened URL isn't in my list of URLs (at http://tr.im/?page=1).
After reading http://www.voidspace.org.uk/python/articles/authentication.shtml#doing-it-properly
I also tried:
TRIM_API_URL = 'api.tr.im/api'
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, TRIM_API_URL, USERNAME, PASSWORD)
auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
response = urllib2.urlopen('http://%s/trim_simple?url=%s'
% (TRIM_API_URL, url_to_trim))
url = response.read().strip()
But I get the same results. (response.code is 200 and url is valid,
but not recorded in my account at http://tr.im/.)
If I use query string parameters instead of basic HTTP authentication,
like this:
TRIM_API_URL = 'http://api.tr.im/api'
response = urllib2.urlopen('%s/trim_simple?url=%s&username=%s&password=%s'
% (TRIM_API_URL,
url_to_trim,
USERNAME,
PASSWORD))
url = response.read().strip()
...then not only is url valid but it's recorded in my tr.im account.
(Though response.code is still 200.)
There must be something wrong with my code though (and not tr.im's API), because
$ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk
...returns:
{"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"200","message":"tr.im URL Added."},"date_time":"2009-03-11T10:15:35-04:00"}
...and the URL does appear in my list of URLs on http://tr.im/?page=1.
And if I run:
$ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk
...again, I get:
{"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"201","message":"tr.im URL Already Created [yacitus]."},"date_time":"2009-03-11T10:15:35-04:00"}
Note code is 201, and message is "tr.im URL Already Created [yacitus]."
I must not be doing the basic HTTP authentication correctly (in either attempt). Can you spot my problem? Perhaps I should look and see what's being sent over the wire? I've never done that before. Are there Python APIs I can use (perhaps in pdb)? Or is there another tool (preferably for Mac OS X) I can use?

This seems to work really well (taken from another thread)
import urllib2, base64
request = urllib2.Request("http://api.foursquare.com/v1/user")
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
result = urllib2.urlopen(request)

Really cheap solution:
urllib.urlopen('http://user:xxxx#api.tr.im/api')
(which you may decide is not suitable for a number of reasons, like security of the url)
Github API example:
>>> import urllib, json
>>> result = urllib.urlopen('https://personal-access-token:x-oauth-basic#api.github.com/repos/:owner/:repo')
>>> r = json.load(result.fp)
>>> result.close()

Take a look at this SO post answer and also look at this basic authentication tutorial from the urllib2 missing manual.
In order for urllib2 basic authentication to work, the http response must contain HTTP code 401 Unauthorized and a key "WWW-Authenticate" with the value "Basic" otherwise, Python won't send your login info, and you will need to either use Requests, or urllib.urlopen(url) with your login in the url, or add a the header like in #Flowpoke's answer.
You can view your error by putting your urlopen in a try block:
try:
urllib2.urlopen(urllib2.Request(url))
except urllib2.HTTPError, e:
print e.headers
print e.headers.has_key('WWW-Authenticate')

The recommended way is to use requests module:
#!/usr/bin/env python
import requests # $ python -m pip install requests
####from pip._vendor import requests # bundled with python
url = 'https://httpbin.org/hidden-basic-auth/user/passwd'
user, password = 'user', 'passwd'
r = requests.get(url, auth=(user, password)) # send auth unconditionally
r.raise_for_status() # raise an exception if the authentication fails
Here's a single source Python 2/3 compatible urllib2-based variant:
#!/usr/bin/env python
import base64
try:
from urllib.request import Request, urlopen
except ImportError: # Python 2
from urllib2 import Request, urlopen
credentials = '{user}:{password}'.format(**vars()).encode()
urlopen(Request(url, headers={'Authorization': # send auth unconditionally
b'Basic ' + base64.b64encode(credentials)})).close()
Python 3.5+ introduces HTTPPasswordMgrWithPriorAuth() that allows:
..to eliminate unnecessary 401 response handling, or to unconditionally send credentials on the first request in order to communicate with servers that return a 404 response instead of a 401 if the Authorization header is not sent..
#!/usr/bin/env python3
import urllib.request as urllib2
password_manager = urllib2.HTTPPasswordMgrWithPriorAuth()
password_manager.add_password(None, url, user, password,
is_authenticated=True) # to handle 404 variant
auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)
opener.open(url).close()
It is easy to replace HTTPBasicAuthHandler() with ProxyBasicAuthHandler() if necessary in this case.

I would suggest that the current solution is to use my package urllib2_prior_auth which solves this pretty nicely (I work on inclusion to the standard lib.

Same solutions as Python urllib2 Basic Auth Problem apply.
see https://stackoverflow.com/a/24048852/1733117; you can subclass urllib2.HTTPBasicAuthHandler to add the Authorization header to each request that matches the known url.
class PreemptiveBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
'''Preemptive basic auth.
Instead of waiting for a 403 to then retry with the credentials,
send the credentials if the url is handled by the password manager.
Note: please use realm=None when calling add_password.'''
def http_request(self, req):
url = req.get_full_url()
realm = None
# this is very similar to the code from retry_http_basic_auth()
# but returns a request object.
user, pw = self.passwd.find_user_password(realm, url)
if pw:
raw = "%s:%s" % (user, pw)
auth = 'Basic %s' % base64.b64encode(raw).strip()
req.add_unredirected_header(self.auth_header, auth)
return req
https_request = http_request

Try python-request or python-grab