Develop Reference

TeamCity: urllib2.HTTPError: HTTP Error 401: Unauthorized

Wrote a script to cancel old builds running on TeamCity. I'm able to list the all running builds based as per my requirement(buildTypeId,branchName).I need to send a REST API call to cancel old build, this request is throwing 401 error. I guess misusing the POST method using urllib2. Anyone can help me on this please?
import urllib2
import re
from xml.dom import minidom
url = 'https://<TeamCity>/httpAuth/app/rest/builds?locator=running:true'
username = user_name
password = password
p = urllib2.HTTPPasswordMgrWithDefaultRealm()
p.add_password(None, url, username, password)
handler = urllib2.HTTPBasicAuthHandler(p)
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)
dom = minidom.parse(urllib2.urlopen(url))
build = dom.getElementsByTagName('build')
for i in range(0, len(build)):
for j in range(i+1, len(build)):
find_build_type = re.search(r'<Job_1>.*', build[i].getAttribute("buildTypeId"), re.M | re.I)
if build[i].getAttribute("branchName") != '<default>' and find_build_type:
if (build[i].getAttribute("buildTypeId") == build[j].getAttribute("buildTypeId") and build[i].getAttribute("branchName") == build[j].getAttribute("branchName")):
#Get the old build
old_run = "https://<TeamCity>/app/rest/buildQueue/id:" + str(min(int(build[j].getAttribute("id")),int(build[i].getAttribute("id"))))
#headers
headers={"Content-Type": "application/xml"}
#data
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, username, password)
opener = urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman))
urllib2.install_opener(opener)
r = urllib2.Request(old_run, data, headers)
#send API request to cancel the old build
u = urllib2.urlopen(r)

HTTP Basic Auth Failing or Incomplete

I'm trying to log into the vRealize Operations Rest API using basic auth method from this question:
HTTP Basic Authentication not working in Python 3.4
So I use the second code sample:
import urllib.parse
import urllib.request
import urllib.response
userName = "username"
passWord = "password"
top_level_url = "URL"
# create an authorization handler
p = urllib.request.HTTPPasswordMgrWithDefaultRealm()
p.add_password(None, top_level_url, userName, passWord);
auth_handler = urllib.request.HTTPBasicAuthHandler(p)
opener = urllib.request.build_opener(auth_handler)
urllib.request.install_opener(opener)
try:
result = opener.open(top_level_url)
messages = result.read()
print (messages)
except IOError as e:
print (e)
and I get a blue font: HTTP Error 401: Unauthorized
So I think this is an issue with trusting the certificate or it has something to do with the headers. How can I work around this? Any advice would be appreciated.

How do i implement cookielib to create a session

import urllib2
import urllib
url = "http://www.torn.com/authenticate.php"
username = raw_input("Your username; ")
password = raw_input("Your password: ")
query = {'player':username, 'password':password}
data_query = urllib.urlencode(query)
sending_data = urllib2.Request(url, data_query)
print sending_data()
response = urllib2.urlopen(sending_data)
print "You are currently logged unto to:", response.geturl()
print response.read()
How do i implement the cookielib to create a session and please explain line by line Thank you

from cookielib import CookieJar
cj = CookieJar()
login_data = urllib.urlencode({'j_username' : username, 'j_password' : password})
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
opener.open("http://www.torn.com/authenticate.php", login_data)
First, you want to initialize your CookieJar. Then you encode your credentials, as they need to be sent in a certain form to be read by the PHP server. Then you have to initialize an opener, which is basically an HTTP client, and configure it to use your CookieJar. Then submit your login info to the server to create the session and generate cookies. To continue using these cookies, use opener.open() instead of urllib2.urlopen (although you can still use urllib2.Request() to generate the requests.

You'll want to use an OpenerDirector for that.
From the documentation:
import cookielib, urllib2
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
r = opener.open("http://example.com/")
So in order to also pass data your code should look something like this:
import urllib2
import urllib
import cookielib
url = "http://www.torn.com/authenticate.php"
username = raw_input("Your username; ")
password = raw_input("Your password: ")
query = {'player':username, 'password':password}
data_query = urllib.urlencode(query)
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
response = opener.open(url, data_query)
print "You are currently logged unto to:", response.geturl()
print response.read()

Python client for multipart form with CAS

I am trying to write a Python script to POST a multipart form to a site that requires authentication through CAS.
There are two approaches that both solve part of the problem:
The Python requests library works well for submitting multipart forms.
There is caslib, with a login function. It returns an OpenerDirector that can presumably be used for further requests.
Unfortunately, I can't figure out how to get a complete solution out what I have so far.
There are just some ideas from a couple hours of research; I am open to just about any solution that works.
Thanks for the help.

I accepted J.F. Sebastian's answer because I think it was closest to what I'd asked, but I actually wound up getting it to work by using mechanize, Python library for web browser automation.
import argparse
import mechanize
import re
import sys
# (SENSITIVE!) Authentication info
username = r'username'
password = r'password'
# Command line arguments
parser = argparse.ArgumentParser(description='Submit lab to CS 235 site (Winter 2013)')
parser.add_argument('lab_num', help='Lab submission number')
parser.add_argument('file_name', help='Submission file (zip)')
args = parser.parse_args()
# Go to login site
br = mechanize.Browser()
br.open('https://cas.byu.edu/cas/login?service=https%3a%2f%2fbeta.cs.byu.edu%2f~sub235%2fsubmit.php')
# Login and forward to submission site
br.form = br.forms().next()
br['username'] = username
br['password'] = password
br.submit()
# Submit
br.form = br.forms().next()
br['labnum'] = list(args.lab_num)
br.add_file(open(args.file_name), 'application/zip', args.file_name)
r = br.submit()
for s in re.findall('<h4>(.+?)</?h4>', r.read()):
print s

You could use poster to prepare multipart/form-data. Try to pass poster's opener to the caslib and use caslib's opener to make requests (not tested):
import urllib2
import caslib
import poster.encode
import poster.streaminghttp
opener = poster.streaminghttp.register_openers()
r, opener = caslib.login_to_cas_service(login_url, username, password,
opener=opener)
params = {'file': open("test.txt", "rb"), 'name': 'upload test'}
datagen, headers = poster.encode.multipart_encode(params)
response = opener.open(urllib2.Request(upload_url, datagen, headers))
print response.read()

You could write a Authentication Handler for Requests using caslib. Then you could do something like:
auth = CasAuthentication("url", "login", "password")
response = requests.get("http://example.com/cas_service", auth=auth)
Or if you're making tons of requests against the website:
s = requests.session()
s.auth = auth
s.post('http://casservice.com/endpoint', data={'key', 'value'}, files={'filename': '/path/to/file'})

Python urllib2 Basic Auth Problem

Update: based on Lee's comment I decided to condense my code to a really simple script and run it from the command line:
import urllib2
import sys
username = sys.argv[1]
password = sys.argv[2]
url = sys.argv[3]
print("calling %s with %s:%s\n" % (url, username, password))
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, username, password)
urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman)))
req = urllib2.Request(url)
f = urllib2.urlopen(req)
data = f.read()
print(data)
Unfortunately it still won't generate the Authorization header (per Wireshark) :(
I'm having a problem sending basic AUTH over urllib2. I took a look at this article, and followed the example. My code:
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, "api.foursquare.com", username, password)
urllib2.install_opener(urllib2.build_opener(urllib2.HTTPBasicAuthHandler(passman)))
req = urllib2.Request("http://api.foursquare.com/v1/user")
f = urllib2.urlopen(req)
data = f.read()
I'm seeing the following on the Wire via wireshark:
GET /v1/user HTTP/1.1
Host: api.foursquare.com
Connection: close
Accept-Encoding: gzip
User-Agent: Python-urllib/2.5
You can see the Authorization is not sent, vs. when I send a request via curl: curl -u user:password http://api.foursquare.com/v1/user
GET /v1/user HTTP/1.1
Authorization: Basic =SNIP=
User-Agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4 OpenSSL/0.9.8k zlib/1.2.3
Host: api.foursquare.com
Accept: */*
For some reason my code seems to not send the authentication - anyone see what I'm missing?
thanks
-simon

The problem could be that the Python libraries, per HTTP-Standard, first send an unauthenticated request, and then only if it's answered with a 401 retry, are the correct credentials sent. If the Foursquare servers don't do "totally standard authentication" then the libraries won't work.
Try using headers to do authentication:
import urllib2, base64
request = urllib2.Request("http://api.foursquare.com/v1/user")
base64string = base64.b64encode('%s:%s' % (username, password))
request.add_header("Authorization", "Basic %s" % base64string)
result = urllib2.urlopen(request)
Had the same problem as you and found the solution from this thread: http://forums.shopify.com/categories/9/posts/27662

(copy-paste/adapted from https://stackoverflow.com/a/24048772/1733117).
First you can subclass urllib2.BaseHandler or urllib2.HTTPBasicAuthHandler, and implement http_request so that each request has the appropriate Authorization header.
import urllib2
import base64
class PreemptiveBasicAuthHandler(urllib2.HTTPBasicAuthHandler):
'''Preemptive basic auth.
Instead of waiting for a 403 to then retry with the credentials,
send the credentials if the url is handled by the password manager.
Note: please use realm=None when calling add_password.'''
def http_request(self, req):
url = req.get_full_url()
realm = None
# this is very similar to the code from retry_http_basic_auth()
# but returns a request object.
user, pw = self.passwd.find_user_password(realm, url)
if pw:
raw = "%s:%s" % (user, pw)
auth = 'Basic %s' % base64.b64encode(raw).strip()
req.add_unredirected_header(self.auth_header, auth)
return req
https_request = http_request
Then if you are lazy like me, install the handler globally
api_url = "http://api.foursquare.com/"
api_username = "johndoe"
api_password = "some-cryptic-value"
auth_handler = PreemptiveBasicAuthHandler()
auth_handler.add_password(
realm=None, # default realm.
uri=api_url,
user=api_username,
passwd=api_password)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)

Here's what I'm using to deal with a similar problem I encountered while trying to access MailChimp's API. This does the same thing, just formatted nicer.
import urllib2
import base64
chimpConfig = {
"headers" : {
"Content-Type": "application/json",
"Authorization": "Basic " + base64.encodestring("hayden:MYSECRETAPIKEY").replace('\n', '')
},
"url": 'https://us12.api.mailchimp.com/3.0/'}
#perform authentication
datas = None
request = urllib2.Request(chimpConfig["url"], datas, chimpConfig["headers"])
result = urllib2.urlopen(request)

The second parameter must be a URI, not a domain name. i.e.
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, "http://api.foursquare.com/", username, password)

I would suggest that the current solution is to use my package urllib2_prior_auth which solves this pretty nicely (I work on inclusion to the standard lib.