Develop Reference

API misinterpreting a password with special characters

I use a python code to connect to an API to retrieve information in json format. A password is used for the authentication to the API. The authentication was working fine so far, but since I changed my password into one with special characters, things went wrong. i guess the password is not well handled either by the API or the code I use.
This is the piece of code that seems to cause the errors:
# Create connection and request header.
# This class does not perform any verification of the server`s certificate.
conn = HTTPSConnection(ADDRESS)
auth_header = 'Basic %s' % (':'.join([USER_NAME, PASSWORD]).encode('Base64').strip('\r\n'))
request_header = {'Authorization':auth_header.decode(),
'Content-Type': content_type}
The error I get after executing the code is the following:
response = self.perform_request(log, 'GET', 'network', params=query)
File "/usr/local/lib/python2.7/query_code.py", line 103, in perform_request
auth_header = 'Basic %s' % str(":".join([USER_NAME, PASSWORD]).encode('Base64').strip('\r\n'))
TypeError: sequence item 1: expected string or Unicode, NoneType found
I should have had a successful connection to the API with code 200 and results returned in json format.
I have difficulties spotting the origin of all this, and would like to have your opinion on what I think it is, and how to solve it.

Try this instead:
def basic_token(key, secret):
return base64.b64encode(bytes(f'{key}:{secret}', 'utf-8')).decode('utf-8')
auth_header = f'basic {basic_token(USERNAME, PASSWORD)}'
If you are using a python version < 3.6, you will need to use .format instead of f-strings:
def basic_token(key, secret):
return base64.b64encode(bytes('{}:{}'.format(key, secret), 'utf-8')).decode('utf-8')
auth_header = 'basic {}'.format(basic_token(USERNAME, PASSWORD))

Using Python to test HTTP APIs

I'm fairly new to Python programming and I don't know all the libraries needed for the following.
I would like to use Python to test some HTTP APIs. Mainly I want to use OAuth and make a few JSON calls. The APIs in question can be found on: https://developers.trustpilot.com/authentication and the generate product review link (I can only use one link)
I want to authenticate myself and then generate a product review link in one step. So far I've been using the Advanced REST client (ARC) to make these calls individually. I could also use .arc files if you think it's easier.
The idea would be make these calls successively in one go. So it would be something along the lines:
1) Make the authentication call.
The HTTP Method looks like this:
https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken
Method Post:
Header
Authorization: Basic Base64encode(APIkey:Secret)
Content-Type: application/x-www-form-urlencoded
Payload:
grant_type=password&username=user#mail.com&password=SomePass
Translate this bit into Python basically.
1.a) Add a header to the call
Header Authorization: base64encode hash Content-Type: application/x-www-form-urlencoded
1.b) Add a payload to the call
Payload: grant_type=password&username
4) Receive the token from call made in step 1) (Result is format)
"access token": Auth_token
5) Take the token and use it in creating a product review.
5.a) Add the token in the header
Header: Authorization: Bearer Auth_token
6.a) Add a JSON payload to the call made in step 5.
Here's the code I have so far:
Import requests
header = {'Authorization: Basic NnNrQUprTWRHTU5VSXJGYXBVRGxack1oT01oTUFRZHI6QTFvOGJjRUNDdUxBTmVqUQ==}','Content-Type: application/x-www-form-urlencoded'}
payload = {'grant_type=password&username=email#address.com&password=SomePassword'}
r = requests.post('https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken', headers=header, params=payload )
Ideally I want to create the requests.post(url, header, payload) and then return what the server answers in JSON format. I think that print r.text would do the last part.
So this is the code I have writtent (that works now):
import requests
import getpass
import json
from requests.auth import HTTPBasicAuth
header = {'grant_type':'password' , 'username':'mail#maildomain.com', 'password':'YourPassword'}
username= "YOURAPIKEY" #APIKey
password= "YOURSECRET" #Secret
res = requests.post(
'URL/v1/oauth/oauth-business-users-for-applications/accesstoken',
auth=HTTPBasicAuth(username, password), # basic authentication
data=header)
#print(res.content) #See content of the call result.
data = res.json() # get response as parsed json (will return a dict)
auth_token = data.get('access_token')

requests can do all what you ask without any work from your part.
See the doc for authentication, parameters, json output, json input
Make the authentication call.
import requests
import getpass
from requests.auth import HTTPBasicAuth
username = raw_input('Username: ')
password = getpass.getpass('Password: ')
res = requests.post(
'https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken',
auth=HTTPBasicAuth(username, password), # basic authentication
params={ # url parameters
'grant_type': 'password',
'username': 'email#address.com',
'password': 'SomePassword'
})
Receive the token from call made in step 1) (Result is format)
# res = requests.post.....
data = res.json() # get response as parsed json (will return a dict)
auth_token = data.get('access token')
Take the token and use it in creating a product review.
request.post(
'.../product_review',
headers={
'Authorization': 'Bearer ' + auth_token
},
json={'my': 'payload'}) # send data as json

HTTP Authentication in URL with backslash in username

I need to HTTP Basic Auth for a REST call. In the username I have to provide a domain (which has a hyphen) and then a backslash to separate it from the username, like this: DOM-AIN\user_name. Then the password is pretty benign.
This works fine with curl:
curl 'https://DOM-AIN\user_name:password#myurl.com'
I need to put this into Python now, but I've tried with requests and urllib/2/3...they don't like the \ : or the #. Even when I URL encode to %40, etc., those get interpreted as an actual : and urllib thinks I'm trying to define a port and I get an error: Invalid socket, I think, I forgot.
So I tried passing the username and password in the header using urllib3, but I get an unauthorized access error and I suspect it's because I need to somehow encode the username in the header to account for the backslash (%5C), but that doesn't seem to be working either.
Here is some code that doesn't work:
# Attempt 1
http = urllib3.PoolManager()
url1 = https://ws.....
headers = urllib3.util.make_headers(basic_auth='DOM-AIN\user_name:password')
r1 = http.request('GET', url1, headers=headers)
response = r1.data
# Attempt 2
passwordManager = urllib2.HTTPPasswordMgrWithDefaultRealm()
passwordManager.add_password(None, url, 'DOM-AIN\user_name, password)
authenticationHandler = urllib2.HTTPBasicAuthHandler(passwordManager)
opener = urllib2.build_opener(authenticationHandler)
data = opener.open(url1)
There were some other attempts with request, but I don't have those anymore. I can get the errors of these if it would be useful, but if there is already a known thing I'm doing wrong that would be great...

Backslash should be escaped in Python string literals:
username = 'DOM-AIN\\user_name' # OR
username = r'DOM-AIN\user_name' # raw-string literal
Example:
import urllib2, base64
request = urllib2.Request('https://example.com')
credentials = base64.b64encode(username + b':' + password)
request.add_header('Authorization', b'Basic ' + credentials)
response = urllib2.urlopen(request)
Note: unlike HTTPBasicAuthHandler code; it always sends the credentials without waiting for 401 response with WWW-Authenticate header.

First convert your DOM-AIN\user_name into base64 string. Lets say its XXXXYYYYYYY. Now place this base64 string into the http header like below code with urllib2.
headers = { 'Authorization:' : 'Basic XXXXYYYYYYY' }
req = urllib2.Request(url, data, headers)

I found a way using urllib, with this post's mention of FancyURLopener sending me down the right path. This was the closest I could come to replicating the way it worked in curl, although looking at Sabuj's answer there might be a way to use headers properly, but I haven't tried his method.
import urllib
opener = urllib.FancyURLopener()
data = opener.open('https://DOM-AIN**%5C%user_name:password#url.com?whatever_parameters')
response = data.read()
It works when I only URL encoded the backslash. Didn't work when I encoded the other characters like : and #.

Python POST binary data

I am writing some code to interface with redmine and I need to upload some files as part of the process, but I am not sure how to do a POST request from python containing a binary file.
I am trying to mimic the commands here:
curl --data-binary "#image.png" -H "Content-Type: application/octet-stream" -X POST -u login:password http://redmine/uploads.xml
In python (below), but it does not seem to work. I am not sure if the problem is somehow related to encoding the file or if something is wrong with the headers.
import urllib2, os
FilePath = "C:\somefolder\somefile.7z"
FileData = open(FilePath, "rb")
length = os.path.getsize(FilePath)
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, 'http://redmine/', 'admin', 'admin')
auth_handler = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
request = urllib2.Request( r'http://redmine/uploads.xml', FileData)
request.add_header('Content-Length', '%d' % length)
request.add_header('Content-Type', 'application/octet-stream')
try:
response = urllib2.urlopen( request)
print response.read()
except urllib2.HTTPError as e:
error_message = e.read()
print error_message
I have access to the server and it looks like a encoding error:
...
invalid byte sequence in UTF-8
Line: 1
Position: 624
Last 80 unconsumed characters:
7z¼¯'ÅÐз2^Ôøë4g¸R<süðí6kĤª¶!»=}jcdjSPúá-º#»ÄAtD»H7Ê!æ½]j):
(further down)
Started POST "/uploads.xml" for 192.168.0.117 at 2013-01-16 09:57:49 -0800
Processing by AttachmentsController#upload as XML
WARNING: Can't verify CSRF token authenticity
Current user: anonymous
Filter chain halted as :authorize_global rendered or redirected
Completed 401 Unauthorized in 13ms (ActiveRecord: 3.1ms)

Basically what you do is correct. Looking at redmine docs you linked to, it seems that suffix after the dot in the url denotes type of posted data (.json for JSON, .xml for XML), which agrees with the response you get - Processing by AttachmentsController#upload as XML. I guess maybe there's a bug in docs and to post binary data you should try using http://redmine/uploads url instead of http://redmine/uploads.xml.
Btw, I highly recommend very good and very popular Requests library for http in Python. It's much better than what's in the standard lib (urllib2). It supports authentication as well but I skipped it for brevity here.
import requests
with open('./x.png', 'rb') as f:
data = f.read()
res = requests.post(url='http://httpbin.org/post',
data=data,
headers={'Content-Type': 'application/octet-stream'})
# let's check if what we sent is what we intended to send...
import json
import base64
assert base64.b64decode(res.json()['data'][len('data:application/octet-stream;base64,'):]) == data
UPDATE
To find out why this works with Requests but not with urllib2 we have to examine the difference in what's being sent. To see this I'm sending traffic to http proxy (Fiddler) running on port 8888:
Using Requests
import requests
data = 'test data'
res = requests.post(url='http://localhost:8888',
data=data,
headers={'Content-Type': 'application/octet-stream'})
we see
POST http://localhost:8888/ HTTP/1.1
Host: localhost:8888
Content-Length: 9
Content-Type: application/octet-stream
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.0.4 CPython/2.7.3 Windows/Vista
test data
and using urllib2
import urllib2
data = 'test data'
req = urllib2.Request('http://localhost:8888', data)
req.add_header('Content-Length', '%d' % len(data))
req.add_header('Content-Type', 'application/octet-stream')
res = urllib2.urlopen(req)
we get
POST http://localhost:8888/ HTTP/1.1
Accept-Encoding: identity
Content-Length: 9
Host: localhost:8888
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7
test data
I don't see any differences which would warrant different behavior you observe. Having said that it's not uncommon for http servers to inspect User-Agent header and vary behavior based on its value. Try to change headers sent by Requests one by one making them the same as those being sent by urllib2 and see when it stops working.

This has nothing to do with a malformed upload. The HTTP error clearly specifies 401 unauthorized, and tells you the CSRF token is invalid. Try sending a valid CSRF token with the upload.
More about csrf tokens here:
What is a CSRF token ? What is its importance and how does it work?

you need to add Content-Disposition header, smth like this (although I used mod-python here, but principle should be the same):
request.headers_out['Content-Disposition'] = 'attachment; filename=%s' % myfname

You can use unirest, It provides easy method to post request.
`
import unirest
def callback(response):
print "code:"+ str(response.code)
print "******************"
print "headers:"+ str(response.headers)
print "******************"
print "body:"+ str(response.body)
print "******************"
print "raw_body:"+ str(response.raw_body)
# consume async post request
def consumePOSTRequestASync():
params = {'test1':'param1','test2':'param2'}
# we need to pass a dummy variable which is open method
# actually unirest does not provide variable to shift between
# application-x-www-form-urlencoded and
# multipart/form-data
params['dummy'] = open('dummy.txt', 'r')
url = 'http://httpbin.org/post'
headers = {"Accept": "application/json"}
# call get service with headers and params
unirest.post(url, headers = headers,params = params, callback = callback)
# post async request multipart/form-data
consumePOSTRequestASync()

Github-api giving 404 when passing json-data with python + urllib2

I have the following code, which should perform the first part of creating a new download at github. It should send the json-data with POST.
jsonstring = '{"name": "test", "size": "4"}'
req = urllib2.Request("https://api.github.com/repos/<user>/<repo>/downloads")
req.add_header('Authorization', 'token ' + '<token>')
result = urllib2.urlopen(req, jsonstring)
If I remove the , jsonstring from the urlopen(), it does not fail, and gives me the list of available downloads. However, if I try to POST the json-string, I get 404 error.
The problem has to be with the json, or in the way I send it, but I can't figure out what the problem is.
The strings at <...> are right in the actual code, I just removed them from the post
I tried roughly the same with curl on the command-line, with slightly different method of authentication, and it worked.
Tested:
Works(returns the wanted json):
curl -u "user:password" --data "json..." https://api.github.com/repos/<user>/<repo>/downloads
Works:
curl -H 'Authorization: token <token>' https://api.github.com/
Does not work (returns "invalid credentials"):
curl -H 'Authorization: token <invalid_token>' https://api.github.com/
Does not work ("not found"):
curl -H 'Authorization: token <valid_token>' --data "json..." https://api.github.com/repos/<user>/<repo>/downloads
This does not seem to be an issue specific to the python code. The json POST data seems to be fine, and the OAuth token authorization seems to be (atleast partly) working. But when these are put together, it stops working.

I was finally able to work out, why it did not work.
I did not have the autorization scopes for the authorization token set correctly. The token I was using, was not "authorized" to do any modifications, and every action using it, that tried to modify something (add a download), failed.
I had to add the correct scopes to the authorization to make it work.

I have provided an answer as to how to POST JSON data to the V3 API without any authentication, but seen as you have identified that the original problem was with not setting up your OAUTH tokens correctly, I thought I would provide a programatic solution to getting one (this implementation gets a token every time the script is run, whereas in practice it would be done just once, and the token would be stored localy).
import urllib2
import json
import getpass
import base64
# Generate a token from the username and password.
# NOTE: this is a naive implementation. Store pre-retrieved tokens if possible.
username = 'pelson'
passwd = getpass.getpass() # <- this just puts a string in passwd (plaintext)
req = urllib2.Request("https://api.github.com/authorizations")
# add the username and password info to the request
base64string = base64.encodestring('%s:%s' % (username, passwd)).replace('\n', '')
req.add_header("Authorization", "Basic %s" % base64string)
data = json.dumps({"scopes":["repo"], "note":"Access to your repository."})
result = urllib2.urlopen(req, data)
result = json.loads('\n'.join(result.readlines()))
token = result['token']
Once you have this token, it can be used for any "repo" scope actions. Lets add a new issue to a repository:
# add an issue to the tracker using the new token
repo = 'name_of_repo'
data = json.dumps({'title': 'My automated issue.'})
req = urllib2.Request("https://api.github.com/repos/%s/%s/issues" % (username, repo))
req.add_header("Authorization", "token %s" % token)
result = urllib2.urlopen(req, data)
result = json.loads('\n'.join(result.readlines()))
print result['number']
Hope that helps somebody.

The v3 github api has a nice feature where it can convert markdown to html. No authentication is needed to request this information, hence it makes a nice example to try before delving into the more tricky world of authentication.
The API docs state:
Render an arbritrary Markdown document
POST /markdown Input
text Required string - The Markdown text to render
And even give an example of a markdown string to be converted:
{"text": "Hello world github/linguist#1 **cool**, and #1!"}
Given this information, lets build an urllib2 Request for the html-ified version of that markdown.
import urllib2
import json
data = {"text": "Hello world github/linguist#1 **cool**, and #1!"}
json_data = json.dumps(data)
req = urllib2.Request("https://api.github.com/markdown")
result = urllib2.urlopen(req, json_data)
print '\n'.join(result.readlines())
And the result is some html representing the markdown.
Given this example, it is not the JSON which is causing the 404 when posting your request, but more likely the authentication (as you have already answered).

Isn't the data meant to be URL-encoded, not JSON? Try:
data = {"name": "test", "size": "4"}
req = urllib2.Request("https://api.github.com/repos/<user>/<repo>/downloads")
req.add_header('Authorization', 'token ' + '<token>')
result = urllib2.urlopen(req, urllib.parse.urlencode(data))