I'm playing around, trying to write some code to use the tr.im
APIs to shorten a URL.
After reading http://docs.python.org/library/urllib2.html, I tried:
TRIM_API_URL = 'http://api.tr.im/api'
auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password(realm='tr.im',
uri=TRIM_API_URL,
user=USERNAME,
passwd=PASSWORD)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
response = urllib2.urlopen('%s/trim_simple?url=%s'
% (TRIM_API_URL, url_to_trim))
url = response.read().strip()
response.code is 200 (I think it should be 202). url is valid, but
the basic HTTP authentication doesn't seem to have worked, because the
shortened URL isn't in my list of URLs (at http://tr.im/?page=1).
After reading http://www.voidspace.org.uk/python/articles/authentication.shtml#doing-it-properly
I also tried:
TRIM_API_URL = 'api.tr.im/api'
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, TRIM_API_URL, USERNAME, PASSWORD)
auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
response = urllib2.urlopen('http://%s/trim_simple?url=%s'
% (TRIM_API_URL, url_to_trim))
url = response.read().strip()
But I get the same results. (response.code is 200 and url is valid,
but not recorded in my account at http://tr.im/.)
If I use query string parameters instead of basic HTTP authentication,
like this:
TRIM_API_URL = 'http://api.tr.im/api'
response = urllib2.urlopen('%s/trim_simple?url=%s&username=%s&password=%s'
% (TRIM_API_URL,
url_to_trim,
USERNAME,
PASSWORD))
url = response.read().strip()
...then not only is url valid but it's recorded in my tr.im account.
(Though response.code is still 200.)
There must be something wrong with my code though (and not tr.im's API), because
$ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk
...returns:
{"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"200","message":"tr.im URL Added."},"date_time":"2009-03-11T10:15:35-04:00"}
...and the URL does appear in my list of URLs on http://tr.im/?page=1.
And if I run:
$ curl -u yacitus:xxxx http://api.tr.im/api/trim_url.json?url=http://www.google.co.uk
...again, I get:
{"trimpath":"hfhb","reference":"nH45bftZDWOX0QpVojeDbOvPDnaRaJ","trimmed":"11\/03\/2009","destination":"http:\/\/www.google.co.uk\/","trim_path":"hfhb","domain":"google.co.uk","url":"http:\/\/tr.im\/hfhb","visits":0,"status":{"result":"OK","code":"201","message":"tr.im URL Already Created [yacitus]."},"date_time":"2009-03-11T10:15:35-04:00"}
Note code is 201, and message is "tr.im URL Already Created [yacitus]."
I must not be doing the basic HTTP authentication correctly (in either attempt). Can you spot my problem? Perhaps I should look and see what's being sent over the wire? I've never done that before. Are there Python APIs I can use (perhaps in pdb)? Or is there another tool (preferably for Mac OS X) I can use?
This seems to work really well (taken from another thread)
import urllib2, base64
request = urllib2.Request("http://api.foursquare.com/v1/user")
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
result = urllib2.urlopen(request)
Really cheap solution:
urllib.urlopen('http://user:xxxx#api.tr.im/api')
(which you may decide is not suitable for a number of reasons, like security of the url)
Github API example:
>>> import urllib, json
>>> result = urllib.urlopen('https://personal-access-token:x-oauth-basic#api.github.com/repos/:owner/:repo')
>>> r = json.load(result.fp)
>>> result.close()
Take a look at this SO post answer and also look at this basic authentication tutorial from the urllib2 missing manual.
In order for urllib2 basic authentication to work, the http response must contain HTTP code 401 Unauthorized and a key "WWW-Authenticate" with the value "Basic" otherwise, Python won't send your login info, and you will need to either use Requests, or urllib.urlopen(url) with your login in the url, or add a the header like in #Flowpoke's answer.
You can view your error by putting your urlopen in a try block:
try:
urllib2.urlopen(urllib2.Request(url))
except urllib2.HTTPError, e:
print e.headers
print e.headers.has_key('WWW-Authenticate')
The recommended way is to use requests module:
#!/usr/bin/env python
import requests # $ python -m pip install requests
####from pip._vendor import requests # bundled with python
url = 'https://httpbin.org/hidden-basic-auth/user/passwd'
user, password = 'user', 'passwd'
r = requests.get(url, auth=(user, password)) # send auth unconditionally
r.raise_for_status() # raise an exception if the authentication fails
Here's a single source Python 2/3 compatible urllib2-based variant:
#!/usr/bin/env python
import base64
try:
from urllib.request import Request, urlopen
except ImportError: # Python 2
from urllib2 import Request, urlopen
credentials = '{user}:{password}'.format(**vars()).encode()
urlopen(Request(url, headers={'Authorization': # send auth unconditionally
b'Basic ' + base64.b64encode(credentials)})).close()
Python 3.5+ introduces HTTPPasswordMgrWithPriorAuth() that allows:
..to eliminate unnecessary 401 response handling, or to unconditionally send credentials on the first request in order to communicate with servers that return a 404 response instead of a 401 if the Authorization header is not sent..
#!/usr/bin/env python3
import urllib.request as urllib2
password_manager = urllib2.HTTPPasswordMgrWithPriorAuth()
password_manager.add_password(None, url, user, password,
is_authenticated=True) # to handle 404 variant
auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)
opener.open(url).close()
It is easy to replace HTTPBasicAuthHandler() with ProxyBasicAuthHandler() if necessary in this case.
I would suggest that the current solution is to use my package urllib2_prior_auth which solves this pretty nicely (I work on inclusion to the standard lib.
Same solutions as Python urllib2 Basic Auth Problem apply.
see https://stackoverflow.com/a/24048852/1733117; you can subclass urllib2.HTTPBasicAuthHandler to add the Authorization header to each request that matches the known url.
class PreemptiveBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
'''Preemptive basic auth.
Instead of waiting for a 403 to then retry with the credentials,
send the credentials if the url is handled by the password manager.
Note: please use realm=None when calling add_password.'''
def http_request(self, req):
url = req.get_full_url()
realm = None
# this is very similar to the code from retry_http_basic_auth()
# but returns a request object.
user, pw = self.passwd.find_user_password(realm, url)
if pw:
raw = "%s:%s" % (user, pw)
auth = 'Basic %s' % base64.b64encode(raw).strip()
req.add_unredirected_header(self.auth_header, auth)
return req
https_request = http_request
Try python-request or python-grab
Related
I'm trying to download a zip artifact from teamcity using python 3 and i'm not having much luck.
From the browser I would normally do this:
http://USERNAME:PWD#SERVER/httpAuth/repository/downloadAll/dood_dad/latest.lastSuccessful
But if I try this with urllib.request.urlretrieve I get an exception about an invalid port - because It doesn't know about the username and password appended to the front of the url and is parsing after the ':' as a port - fair enough.
So I guess I need to use teamcitys httpAuth stuff and use the url
http://SERVERNAME/httpAuth/repository/downloadAll/dood_dad/latest.lastSuccessful
When I try that I get a 404 Unauthorized which I expected because I need to supply the username and password.
But I can't work out how.
I've added this:
auth_handler = urllib.request.HTTPBasicAuthHandler()
auth_handler.add_password(None,
uri=url_to_open,
user='userame',
passwd='password')
opener = urllib.request.build_opener(auth_handler)
urllib.request.install_opener(opener)
local_filename, headers = urllib.request.urlretrieve(url)
But I'm still getting HTTP Error 401: Unauthorized
TIA.
You can use a lib like requests that let's you put the basic auth as a param, see more here: http://docs.python-requests.org/en/latest/user/authentication/#basic-authentication
import requests
from requests.auth import HTTPBasicAuth
import shutil
response = requests.get('http://...', auth=HTTPBasicAuth('user', 'pass'), stream=True)
with open('filename.zip', 'wb') as out_file:
shutil.copyfileobj(response.raw, out_file)
this works ok:
import urllib
from urllib.request import HTTPPasswordMgrWithDefaultRealm
pwdmgr = HTTPPasswordMgrWithDefaultRealm()
pwdmgr.add_password(None, uri=url, user='XXXX', passwd='XXXX')
auth_handler = urllib.request.HTTPBasicAuthHandler(pwdmgr)
opener = urllib.request.build_opener(auth_handler)
urllib.request.install_opener(opener)
local_filename, headers = urllib.request.urlretrieve(url)
I'm not entirely sure why the newer code works over the older stuff.
FYI: the request code never worked either
response = requests.get('http://...', auth=HTTPBasicAuth('user', 'pass'), stream=True)
I kept getting an Unauthorized http errors
Obtaining Artifacts from a Build Script
import getpass
import subprocess
USERNAME = getpass.getuser()
PWD = getpass.getpass(prompt='PWD:', stream=None)
subprocess.run(['wget','http://'+USERNAME+':'+'PWD'+'#SERVER/httpAuth/repository/downloadAll/dood_dad/latest.lastSuccessful'])
I have a service, that exposes API for logging in. In my browser if I do
<host>:<port>/myservice/api/login?u=admin&pw=admin
The above url, returns a ticket, that I can pass along for my successive requests.
More details about the same, here.
Below is my python script.
import urllib
url = 'http://<host>:<port>/myservice/api/login?u=admin&pw=admin'
print 'Retrieving', url
uh = urllib.urlopen(url)
data = uh.read()
print 'Retrieved',len(data),'characters'
print data
When I run this I get
IOError: ('http error', 401, 'Authorization Required', <httplib.HTTPMessage instance at 0x<somenumber>>)
Now, I am not sure what am I supposed to do. So I went to my browser, and opened the developer's console.
Apparently, the url has moved to something else. I see two requests.
first one hits the url that I am hitting. Response Header has a Location:Parameter.
The second request hits the url that is returned as Location parameter. the Authorization header has 'Negotiation
It also has a setcookie in the response header.
Now, I am not sure what exactly to do with this information, but if someone can help. Thanks
I believe you problem is having the wrong URL for the login service
If I change you code to instead be:
import urllib, json
url = 'http://localhost:8080/alfresco/service/api/login?u=admin&pw=admin&format=json'
print "Retrieving %s" % url
uh = urllib.urlopen(url)
data = uh.read()
print "Retrieved %d characters" % len(data)
print "Data is %s" % data
ticket = json.loads(data)["data"]["ticket"]
print "Ticket is %s" % ticket
Then against a freshly installed Alfresco 4.2 server, I get back a login Ticket for the admin user.
Note the use of the json format of the login API - much easier to parse from JSON, and of the correct path to the login api - /alfresco/service/api/login
try this two small changes may be it will help:
1) use urllib.urlencode while passing parameters to request url
import urllib
params = urllib.urlencode({'u': 'admin', 'pw': 'admin'})
uh = urllib.urlopen("http://<host>:<port>/myservice/api/login?%s" % params')
2) Stimulate a web browser while making a request using urllib2
import urllib2
req = urllib2.Request('http://<host>:<port>/myservice/api/login?u=admin&pw=admin', headers={ 'User-Agent': 'Mozilla/5.0' })
uh = urllib2.urlopen(req)
401 is unauthorized error! it means you are not authorized to access API. Did you already signup for API keys and access tokens?
Check detailed description of 401 Error:
http://techproblems.org/http-error-401/
I am playing from Linux using curl inside bash but now I need to move my script to python in order to be more effective.
What is the best way to do something like the following line on python?
curl baseuri:port/resource -u user:pswd
I played already with urllib2 but I dont get a clue on how to send the "-u user:pswd"
You'll need to use a HTTPPasswordMgrWithDefaultRealm instance to handle the authentication, and add in a HTTPBasicAuthHandler handler to respond to the authentication challenge:
import urllib2
url = 'baseuri:port/resource'
username = 'user'
password = 'pswd'
pwmgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
pwmgr.add_password(None, url, username, password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
response = opener.open(theurl)
Yes, this is a handful.
If you can install a 3rd party library, then add use the requests library; it'll be so much easier:
import requests
url = 'baseuri:port/resource'
username = 'user'
password = 'pswd'
response = requests.get(url, auth=(username, password))
Actually, we use pycurl instead which also works pretty well. Here an example to get a json from the request:
import pycurl
import json
from io import BytesIO
data = BytesIO()
pyCurl = pycurl.Curl()
pyCurl.setopt(pyCurlClass.URL,string_http)
pyCurl.setopt(pycurl.USERPWD, 'user:password')
pycurl.setopt(c.WRITEFUNCTION, data.write)
pyCurl.perform()
dictionary = json.loads(data.getvalue())
What is the cleanest way to do HTTP POST with Basic Auth in Python?
Using only the Python core libs.
Seriously, just use requests:
import requests
resp = requests.post(url, data={}, auth=('user', 'pass'))
It's a pure python library, installing is as easy as easy_install requests or pip install requests. It has an extremely simple and easy to use API, and it fixes bugs in urllib2 so you don't have to. Don't make your life harder because of silly self-imposed requirements.
Hackish workaround works:
urllib.urlopen("https://username:password#hostname/path", data)
A lot of people don't realize that the old syntax for specifying username and password in the URL works in urllib.urlopen. It doesn't appear the username or password require any encoding, except perhaps if the password includes an "#" symbol.
if you define a url, username, password, and some post-data, this should work in Python2...
import urllib2
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, username, password)
auth_handler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
content = urllib2.urlopen(url, post_data)
example from official Python docs showing Basic Auth in urllib2:
* http://docs.python.org/release/2.6/howto/urllib2.html
full tutorial on Basic Authentication using urllib2:
* http://www.voidspace.org.uk/python/articles/authentication.shtml
I am trying to use http://developer.github.com/v3/ to retrieve project issues. This works:
curl -u "Littlemaple:mypassword" https://api.github.com/repos/MyClient/project/issues
It returns all private issues of my client's project. However, I am not able to find out how to implement this in Python. Both ways I have found (e.g. Python urllib2 Basic Auth Problem) doesn't work, they return 404 or 403 errors:
def fetch(url, username, password):
"""Wonderful method found on forums which does not work.""""
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, username, password)
urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman)))
req = urllib2.Request(url)
f = urllib2.urlopen(req)
return f.read()
...and:
def fetch(url, username, password):
"""Wonderful method found on forums which does not work neither.""""
request = urllib2.Request(url)
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
return urllib2.urlopen(request).read()
Any ideas? Thanks in advance!
r = requests.get('https://api.github.com', auth=('user', 'pass'))
Python requests is the way to go here. I've been using requests extensively at work and at home for various web service interactions. It is a joy to use compared to what came before it. Note: the auth keyword arg works on any call that requires auth. Thus, you can use it sparingly, i.e. you don't need it for every call against GitHub, only those that require logins. For instance:
r = requests.get('https://api.github.com/gists/starred', auth=('user', 'pass'))
The GitHub login is documented here:
http://pypi.python.org/pypi/requests/0.6.1
If it's 404, you probably just have the wrong URL. If it's 403, perhaps you have the realm wrong.
For starters, you're passing the URL to add_password, when in fact you should only be passing the base URL. Also, instead of install_opener, you should probably just create a new opener.
See this recipe for an example:
class NoOpHandler(urllib2.HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newUrl):
return None
passmanager = urllib2.HTTPPasswordMgrWithDefaultRealm()
passmanager.add_password(None, baseurl, username, password)
auth_handler = urllib2.HTTPBasicAuthHandler(passmanager)
opener = urllib2.build_opener(auth_handler, NoOpHandler())
You can also do it this way
r = requests.get('https://user:pass#api.github.com')
Use pycurl which is python interface to libcurl.