I need to HTTP Basic Auth for a REST call. In the username I have to provide a domain (which has a hyphen) and then a backslash to separate it from the username, like this: DOM-AIN\user_name. Then the password is pretty benign.
This works fine with curl:
curl 'https://DOM-AIN\user_name:password#myurl.com'
I need to put this into Python now, but I've tried with requests and urllib/2/3...they don't like the \ : or the #. Even when I URL encode to %40, etc., those get interpreted as an actual : and urllib thinks I'm trying to define a port and I get an error: Invalid socket, I think, I forgot.
So I tried passing the username and password in the header using urllib3, but I get an unauthorized access error and I suspect it's because I need to somehow encode the username in the header to account for the backslash (%5C), but that doesn't seem to be working either.
Here is some code that doesn't work:
# Attempt 1
http = urllib3.PoolManager()
url1 = https://ws.....
headers = urllib3.util.make_headers(basic_auth='DOM-AIN\user_name:password')
r1 = http.request('GET', url1, headers=headers)
response = r1.data
# Attempt 2
passwordManager = urllib2.HTTPPasswordMgrWithDefaultRealm()
passwordManager.add_password(None, url, 'DOM-AIN\user_name, password)
authenticationHandler = urllib2.HTTPBasicAuthHandler(passwordManager)
opener = urllib2.build_opener(authenticationHandler)
data = opener.open(url1)
There were some other attempts with request, but I don't have those anymore. I can get the errors of these if it would be useful, but if there is already a known thing I'm doing wrong that would be great...
Backslash should be escaped in Python string literals:
username = 'DOM-AIN\\user_name' # OR
username = r'DOM-AIN\user_name' # raw-string literal
Example:
import urllib2, base64
request = urllib2.Request('https://example.com')
credentials = base64.b64encode(username + b':' + password)
request.add_header('Authorization', b'Basic ' + credentials)
response = urllib2.urlopen(request)
Note: unlike HTTPBasicAuthHandler code; it always sends the credentials without waiting for 401 response with WWW-Authenticate header.
First convert your DOM-AIN\user_name into base64 string. Lets say its XXXXYYYYYYY. Now place this base64 string into the http header like below code with urllib2.
headers = { 'Authorization:' : 'Basic XXXXYYYYYYY' }
req = urllib2.Request(url, data, headers)
I found a way using urllib, with this post's mention of FancyURLopener sending me down the right path. This was the closest I could come to replicating the way it worked in curl, although looking at Sabuj's answer there might be a way to use headers properly, but I haven't tried his method.
import urllib
opener = urllib.FancyURLopener()
data = opener.open('https://DOM-AIN**%5C%user_name:password#url.com?whatever_parameters')
response = data.read()
It works when I only URL encoded the backslash. Didn't work when I encoded the other characters like : and #.
Related
I am writing a Python3 script to trigger an Axis camera via their HTTP API. The string they need in the url to trigger the Digital Input (with authorization) is:
"http://{ip}/axis-cgi/param.cgi?action=update&IOPort.I0.Input.Trig={open/closed}"
If I put this in a browser it works AKA response 200 - OK. When I run this on my Linux machine using the urllib request.
#encode user,pass
values = { 'username': username,'password': password }
data = urllib.urlencode(values)
#Axis param.cgi action
action = "update"
trig = "closed"
cgi_args = {"action": action, "IOPort.I0.Input.Trig": trig}
cgi_data = urllib.urlencode(cgi_args)
url = "http://{ip}/axis-cgi/param.cgi?{data}".format(ip=ip, data=cgi_data)
req = urllib2.Request(url, data, headers={'Content-type': 'text/html'})
print(req.get_full_url())
response = urllib2.urlopen(req)
result = response.read()
print (result)
The output is:
http://192.168.50.191/axis-cgi/param.cgi?action=update&IOPort.I0.Input.Trig=closed
action must be specified
I know that I am authenticated otherwise I get Unauthorized response from server.
As you can see for debugging I print the req.get_full_url() to make sure I've built the url string correctly, however the server responds with action must be specified which is what I get in the browser when I just have http://192.168.50.191/axis-cgi/param.cgi? in the address bar. So the action portion of the URL appears to not be making it to the server.
I've tried:
Using %3F as the ? character, but I get 404 error
Embedding the data in the data parameter does not work either
Anything I am missing here?
used pycurl with digest authorization for a GET request with auth:
def configCurl():
username = "user"
password = "pass"
auth_mode = pycurl.HTTPAUTH_DIGEST
curl = pycurl.Curl()
curl.setopt(pycurl.HTTPAUTH, auth_mode)
curl.setopt(pycurl.USERPWD, "{}:{}".format(username, password))
curl.setopt(pycurl.WRITEFUNCTION, lambda x: None)
return curl
curl = configCurl()
curl.setopt(curl.URL, url)
curl.perform()
curl.close()
I am trying to encode an auth token and pass it to a REST API, this works fine with powershell but applying the same method to python script throws 'unauthorized' exception.
I suspect there is a problem in encoded value. Not able to figure out the solution. Any ideas ?
The rest endpoint is IBM uDeploy.
Powershell
$tokenEncoded = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes( "PasswordIsAuthToken:{`"token`":`"$pass`"}" ))
$basicAuthValue = "Basic $tokenEncoded"
$headers = #{}
$headers.Add("Authorization", $basicAuthValue)
$response = Invoke-RestMethod -Method Put -Headers $headers -Uri $requestUri -Body $jsonRequest
Python
epass = base64.b64encode("PasswordIsAuthToken:{\"token\":\"$password\"}")
print 'base64 encoded: ' + epass
opener = urllib2.build_opener(urllib2.HTTPHandler)
req = urllib2.Request(reqUrl,json.dumps(json_data))
req.add_header('Authorization', 'Basic '+epass)
req.get_method = lambda: 'PUT'
resp = opener.open(req)
You are sending the literal string $password as the token, not the contents of a variable named password.
You only need to include the PasswordIsAuthToken and your token in a Basic Auth HTTP header (PasswordIsAuthToken forms the username, and token the password):
epass = base64.b64encode("PasswordIsAuthToken:" + password)
opener = urllib2.build_opener(urllib2.HTTPHandler)
req = urllib2.Request(reqUrl,json.dumps(json_data))
req.add_header('Authorization', 'Basic ' + epass)
If you reall need to wrap the token in a JSON-like structure then you'd need to use string formatting (which the Powershell code also does, but you omitted):
epass = base64.b64encode('PasswordIsAuthToken:{"token":"%s"}' % password)
or you could use the json module:
epass = base64.b64encode('PasswordIsAuthToken:' + json.dumps({'token': password}))
However, I believe the system should accept the token unwrapped.
I strongly recommend you use the requests library instead, which makes using a REST API vastly cleaner:
import requests
auth = ('PasswordIsAuthToken', password)
# or alternatively
# auth = ('PasswordIsAuthToken', '{"token":"%s"}' % password)
response = requests.put(json=json_data, auth=auth)
Note that there is no need to encode the JSON body yourself, nor do you have to encode the Basic Auth header.
I'm fairly new to Python programming and I don't know all the libraries needed for the following.
I would like to use Python to test some HTTP APIs. Mainly I want to use OAuth and make a few JSON calls. The APIs in question can be found on: https://developers.trustpilot.com/authentication and the generate product review link (I can only use one link)
I want to authenticate myself and then generate a product review link in one step. So far I've been using the Advanced REST client (ARC) to make these calls individually. I could also use .arc files if you think it's easier.
The idea would be make these calls successively in one go. So it would be something along the lines:
1) Make the authentication call.
The HTTP Method looks like this:
https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken
Method Post:
Header
Authorization: Basic Base64encode(APIkey:Secret)
Content-Type: application/x-www-form-urlencoded
Payload:
grant_type=password&username=user#mail.com&password=SomePass
Translate this bit into Python basically.
1.a) Add a header to the call
Header Authorization: base64encode hash Content-Type: application/x-www-form-urlencoded
1.b) Add a payload to the call
Payload: grant_type=password&username
4) Receive the token from call made in step 1) (Result is format)
"access token": Auth_token
5) Take the token and use it in creating a product review.
5.a) Add the token in the header
Header: Authorization: Bearer Auth_token
6.a) Add a JSON payload to the call made in step 5.
Here's the code I have so far:
Import requests
header = {'Authorization: Basic NnNrQUprTWRHTU5VSXJGYXBVRGxack1oT01oTUFRZHI6QTFvOGJjRUNDdUxBTmVqUQ==}','Content-Type: application/x-www-form-urlencoded'}
payload = {'grant_type=password&username=email#address.com&password=SomePassword'}
r = requests.post('https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken', headers=header, params=payload )
Ideally I want to create the requests.post(url, header, payload) and then return what the server answers in JSON format. I think that print r.text would do the last part.
So this is the code I have writtent (that works now):
import requests
import getpass
import json
from requests.auth import HTTPBasicAuth
header = {'grant_type':'password' , 'username':'mail#maildomain.com', 'password':'YourPassword'}
username= "YOURAPIKEY" #APIKey
password= "YOURSECRET" #Secret
res = requests.post(
'URL/v1/oauth/oauth-business-users-for-applications/accesstoken',
auth=HTTPBasicAuth(username, password), # basic authentication
data=header)
#print(res.content) #See content of the call result.
data = res.json() # get response as parsed json (will return a dict)
auth_token = data.get('access_token')
requests can do all what you ask without any work from your part.
See the doc for authentication, parameters, json output, json input
Make the authentication call.
import requests
import getpass
from requests.auth import HTTPBasicAuth
username = raw_input('Username: ')
password = getpass.getpass('Password: ')
res = requests.post(
'https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken',
auth=HTTPBasicAuth(username, password), # basic authentication
params={ # url parameters
'grant_type': 'password',
'username': 'email#address.com',
'password': 'SomePassword'
})
Receive the token from call made in step 1) (Result is format)
# res = requests.post.....
data = res.json() # get response as parsed json (will return a dict)
auth_token = data.get('access token')
Take the token and use it in creating a product review.
request.post(
'.../product_review',
headers={
'Authorization': 'Bearer ' + auth_token
},
json={'my': 'payload'}) # send data as json
I have the following code, which should perform the first part of creating a new download at github. It should send the json-data with POST.
jsonstring = '{"name": "test", "size": "4"}'
req = urllib2.Request("https://api.github.com/repos/<user>/<repo>/downloads")
req.add_header('Authorization', 'token ' + '<token>')
result = urllib2.urlopen(req, jsonstring)
If I remove the , jsonstring from the urlopen(), it does not fail, and gives me the list of available downloads. However, if I try to POST the json-string, I get 404 error.
The problem has to be with the json, or in the way I send it, but I can't figure out what the problem is.
The strings at <...> are right in the actual code, I just removed them from the post
I tried roughly the same with curl on the command-line, with slightly different method of authentication, and it worked.
Tested:
Works(returns the wanted json):
curl -u "user:password" --data "json..." https://api.github.com/repos/<user>/<repo>/downloads
Works:
curl -H 'Authorization: token <token>' https://api.github.com/
Does not work (returns "invalid credentials"):
curl -H 'Authorization: token <invalid_token>' https://api.github.com/
Does not work ("not found"):
curl -H 'Authorization: token <valid_token>' --data "json..." https://api.github.com/repos/<user>/<repo>/downloads
This does not seem to be an issue specific to the python code. The json POST data seems to be fine, and the OAuth token authorization seems to be (atleast partly) working. But when these are put together, it stops working.
I was finally able to work out, why it did not work.
I did not have the autorization scopes for the authorization token set correctly. The token I was using, was not "authorized" to do any modifications, and every action using it, that tried to modify something (add a download), failed.
I had to add the correct scopes to the authorization to make it work.
I have provided an answer as to how to POST JSON data to the V3 API without any authentication, but seen as you have identified that the original problem was with not setting up your OAUTH tokens correctly, I thought I would provide a programatic solution to getting one (this implementation gets a token every time the script is run, whereas in practice it would be done just once, and the token would be stored localy).
import urllib2
import json
import getpass
import base64
# Generate a token from the username and password.
# NOTE: this is a naive implementation. Store pre-retrieved tokens if possible.
username = 'pelson'
passwd = getpass.getpass() # <- this just puts a string in passwd (plaintext)
req = urllib2.Request("https://api.github.com/authorizations")
# add the username and password info to the request
base64string = base64.encodestring('%s:%s' % (username, passwd)).replace('\n', '')
req.add_header("Authorization", "Basic %s" % base64string)
data = json.dumps({"scopes":["repo"], "note":"Access to your repository."})
result = urllib2.urlopen(req, data)
result = json.loads('\n'.join(result.readlines()))
token = result['token']
Once you have this token, it can be used for any "repo" scope actions. Lets add a new issue to a repository:
# add an issue to the tracker using the new token
repo = 'name_of_repo'
data = json.dumps({'title': 'My automated issue.'})
req = urllib2.Request("https://api.github.com/repos/%s/%s/issues" % (username, repo))
req.add_header("Authorization", "token %s" % token)
result = urllib2.urlopen(req, data)
result = json.loads('\n'.join(result.readlines()))
print result['number']
Hope that helps somebody.
The v3 github api has a nice feature where it can convert markdown to html. No authentication is needed to request this information, hence it makes a nice example to try before delving into the more tricky world of authentication.
The API docs state:
Render an arbritrary Markdown document
POST /markdown Input
text Required string - The Markdown text to render
And even give an example of a markdown string to be converted:
{"text": "Hello world github/linguist#1 **cool**, and #1!"}
Given this information, lets build an urllib2 Request for the html-ified version of that markdown.
import urllib2
import json
data = {"text": "Hello world github/linguist#1 **cool**, and #1!"}
json_data = json.dumps(data)
req = urllib2.Request("https://api.github.com/markdown")
result = urllib2.urlopen(req, json_data)
print '\n'.join(result.readlines())
And the result is some html representing the markdown.
Given this example, it is not the JSON which is causing the 404 when posting your request, but more likely the authentication (as you have already answered).
Isn't the data meant to be URL-encoded, not JSON? Try:
data = {"name": "test", "size": "4"}
req = urllib2.Request("https://api.github.com/repos/<user>/<repo>/downloads")
req.add_header('Authorization', 'token ' + '<token>')
result = urllib2.urlopen(req, urllib.parse.urlencode(data))
I need to perform preemptive basic authentication against an HTTP server, i.e., authenticate right away without waiting on a 401 response. Can this be done with httplib2?
Edit:
I solved it by adding an Authorization header to the request, as suggested in the accepted answer:
headers["Authorization"] = "Basic {0}".format(
base64.b64encode("{0}:{1}".format(username, password)))
Add an appropriately formed 'Authorization' header to your initial request.
This also works with the built-in httplib (for anyone wishing to minimize 3rd-party libs/modules). I am using it to authenticate with our Jenkins server using the API Token that Jenkins can create for each user.
>>> import base64, httplib
>>> headers = {}
>>> headers["Authorization"] = "Basic {0}".format(
base64.b64encode("{0}:{1}".format('<username>', '<jenkins_API_token>')))
>>> ## Enable the job
>>> conn = httplib.HTTPConnection('jenkins.myserver.net')
>>> conn.request('POST', '/job/Foo-trunk/enable', None, headers)
>>> resp = conn.getresponse()
>>> resp.status
302
>>> ## Disable the job
>>> conn = httplib.HTTPConnection('jenkins.myserver.net')
>>> conn.request('POST', '/job/Foo-trunk/disable', None, headers)
>>> resp = conn.getresponse()
>>> resp.status
302
I realize this is old, but I figured I'd throw in the solution if you're using Python 3 with httplib2 since I haven't been able to find it anywhere else. I'm also authenticating against a Jenkins server using the API Token for each Jenkins user. If you're not concerned with Jenkins, simply substitute the actual user's password for the API Token.
b64encode is expecting an binary string of ASCII characters. With Python 3 a TypeError will be raised if a plain string is passed in. To get around this, the "user:api_token" portion of the header must be encoded using either 'ascii' or 'utf-8', passed to b64encode, then the resulting byte string must be decoded to a plain string before being placed in the header. The following code did what I needed:
import httplib2, base64
cred = base64.b64encode("{0}:{1}".format(
<user>, <api_token>).encode('utf-8')).decode()
headers = {'Authorization': "Basic %s" % cred}
h = httplib2.Http('.cache')
response, content = h.request("http://my.jenkins.server/job/my_job/enable",
"GET", headers=headers)