I have the following code, which should perform the first part of creating a new download at github. It should send the json-data with POST.
jsonstring = '{"name": "test", "size": "4"}'
req = urllib2.Request("https://api.github.com/repos/<user>/<repo>/downloads")
req.add_header('Authorization', 'token ' + '<token>')
result = urllib2.urlopen(req, jsonstring)
If I remove the , jsonstring from the urlopen(), it does not fail, and gives me the list of available downloads. However, if I try to POST the json-string, I get 404 error.
The problem has to be with the json, or in the way I send it, but I can't figure out what the problem is.
The strings at <...> are right in the actual code, I just removed them from the post
I tried roughly the same with curl on the command-line, with slightly different method of authentication, and it worked.
Tested:
Works(returns the wanted json):
curl -u "user:password" --data "json..." https://api.github.com/repos/<user>/<repo>/downloads
Works:
curl -H 'Authorization: token <token>' https://api.github.com/
Does not work (returns "invalid credentials"):
curl -H 'Authorization: token <invalid_token>' https://api.github.com/
Does not work ("not found"):
curl -H 'Authorization: token <valid_token>' --data "json..." https://api.github.com/repos/<user>/<repo>/downloads
This does not seem to be an issue specific to the python code. The json POST data seems to be fine, and the OAuth token authorization seems to be (atleast partly) working. But when these are put together, it stops working.
I was finally able to work out, why it did not work.
I did not have the autorization scopes for the authorization token set correctly. The token I was using, was not "authorized" to do any modifications, and every action using it, that tried to modify something (add a download), failed.
I had to add the correct scopes to the authorization to make it work.
I have provided an answer as to how to POST JSON data to the V3 API without any authentication, but seen as you have identified that the original problem was with not setting up your OAUTH tokens correctly, I thought I would provide a programatic solution to getting one (this implementation gets a token every time the script is run, whereas in practice it would be done just once, and the token would be stored localy).
import urllib2
import json
import getpass
import base64
# Generate a token from the username and password.
# NOTE: this is a naive implementation. Store pre-retrieved tokens if possible.
username = 'pelson'
passwd = getpass.getpass() # <- this just puts a string in passwd (plaintext)
req = urllib2.Request("https://api.github.com/authorizations")
# add the username and password info to the request
base64string = base64.encodestring('%s:%s' % (username, passwd)).replace('\n', '')
req.add_header("Authorization", "Basic %s" % base64string)
data = json.dumps({"scopes":["repo"], "note":"Access to your repository."})
result = urllib2.urlopen(req, data)
result = json.loads('\n'.join(result.readlines()))
token = result['token']
Once you have this token, it can be used for any "repo" scope actions. Lets add a new issue to a repository:
# add an issue to the tracker using the new token
repo = 'name_of_repo'
data = json.dumps({'title': 'My automated issue.'})
req = urllib2.Request("https://api.github.com/repos/%s/%s/issues" % (username, repo))
req.add_header("Authorization", "token %s" % token)
result = urllib2.urlopen(req, data)
result = json.loads('\n'.join(result.readlines()))
print result['number']
Hope that helps somebody.
The v3 github api has a nice feature where it can convert markdown to html. No authentication is needed to request this information, hence it makes a nice example to try before delving into the more tricky world of authentication.
The API docs state:
Render an arbritrary Markdown document
POST /markdown Input
text Required string - The Markdown text to render
And even give an example of a markdown string to be converted:
{"text": "Hello world github/linguist#1 **cool**, and #1!"}
Given this information, lets build an urllib2 Request for the html-ified version of that markdown.
import urllib2
import json
data = {"text": "Hello world github/linguist#1 **cool**, and #1!"}
json_data = json.dumps(data)
req = urllib2.Request("https://api.github.com/markdown")
result = urllib2.urlopen(req, json_data)
print '\n'.join(result.readlines())
And the result is some html representing the markdown.
Given this example, it is not the JSON which is causing the 404 when posting your request, but more likely the authentication (as you have already answered).
Isn't the data meant to be URL-encoded, not JSON? Try:
data = {"name": "test", "size": "4"}
req = urllib2.Request("https://api.github.com/repos/<user>/<repo>/downloads")
req.add_header('Authorization', 'token ' + '<token>')
result = urllib2.urlopen(req, urllib.parse.urlencode(data))
Related
I'm trying to access GetDealItems API and i have a nightmare to get this working. Even though I use the valid client_id','client_secret','ruName' i keep getting
{'error': 'invalid_client', 'error_description': 'client authentication failed'}
below is the ebay doc
https://developer.ebay.com/api-docs/buy/deal/resources/deal_item/methods/getDealItems
I guess i need to use this scope and url in my request
scopes:'https://api.ebay.com/oauth/api_scope/buy.deal' and the
url='https://api.ebay.com/buy/deal/v1/deal_item?limit=1000'
Please see below my Python code.
import requests, urllib, base64
def getAuthToken():
AppSettings = {
'client_id':'xxxx7c8ec878c-c80c4c69',
'client_secret':'xxxx56db-4b4a-97b4-fad2',
'ruName':'xxxxx-gscrcsrtj'}
authHeaderData = AppSettings['client_id'] + ':' + AppSettings['client_secret']
encodedAuthHeader = base64.b64encode(str.encode(authHeaderData))
headers = {
"Content-Type" : "application/x-www-form-urlencoded",
"Authorization" : "Bearer " + str(encodedAuthHeader)
}
body= {
"grant_type" : "client_credentials",
"redirect_uri" : AppSettings['ruName'],
"scope" : "https://api.ebay.com/oauth/api_scope/buy.deal"
}
data = urllib.parse.urlencode(body)
tokenURL = "https://api.ebay.com/identity/v1/oauth2/token"
response = requests.post(tokenURL, headers=headers, data=data)
return response.json()
response = getAuthToken()
print(response)
response['access_token'] #access keys as required
response['error_description'] #if errors
The most obvious problem I see is that you are using Bearer when you should be using Basic in your Authorization header.
Also, You are urlencoding your redirect_url when you pass the entire dictionary into urlencode. The docs say you are supposed to urlencode the scope parameter, but honestly, I never encode the scope and it still works for me.
Here is your modified code, with a few formatting changes:
import requests, urllib, base64
client_id='xxxx7c8ec878c-c80c4c69'
client_secret='xxxx56db-4b4a-97b4-fad2'
ruName='xxxxx-gscrcsrtj'
scope = urllib.parse.quote('https://api.ebay.com/oauth/api_scope/buy.deal')
def basic_token(key, secret):
return 'Basic ' + base64.b64encode((key + ':' + secret).encode()).decode()
def getAuthToken():
headers = {
"Content-Type" : "application/x-www-form-urlencoded",
"Authorization" : basic_token(client_id, client_secret)
}
data = (
'grant_type=client_credentials&'
f'redirect_uri={ruName}&'
f'scope={scope}'
)
tokenURL = "https://api.ebay.com/identity/v1/oauth2/token"
response = requests.post(tokenURL, headers=headers, data=data)
return response.json()
Update:
I think you need to use the authorization_code grant instead of client_credentials.
To use the authorization_code grant, modify your body to look like this:
data = (
'grant_type=authorization_code&'
f'code={authorization_code}&'
f'redirect_uri={ruName}&'
f'scope={scope}'
)
Also, you will need to follow your "redirect url" to get the actual authorization code. Execute the following:
redirect_url = (
'https://auth.ebay.com/oauth2/authorize?'
f'client_id={client_id}&'
f'response_type=code&'
f'redirect_uri={ruName}&'
f'scope={scope}'
)
print(redirect_url)
Copy/paste the url from stdout, follow the link, and click "accept", then you will be redirected to a url that looks like this:
https://signin.ebay.com/ws/eBayISAPI.dll?ThirdPartyAuthSucessFailure&isAuthSuccessful=true&code=<authorization code here>&expires_in=299
Copy/paste the authorization code into your code, then see if it works.
Realistically, eBay expects you to automate this within your application using a server, but it doesn't make sense for you to go through the trouble if you are building an app for personal use.
GetDealItems API uses client_credentials grant as evident from the docs
The authorization should be using client_id and secret as described in getting access tokens
curl -X POST 'https://api.ebay.com/identity/v1/oauth2/token' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'Authorization: Basic UkVTVFRlc3...wZi1hOGZhLTI4MmY=' \
-d 'grant_type=client_credentials&scope=https%3A%2F%2Fapi.ebay.com%2Foauth%2Fapi_scope%2Fbuy.deal'
Note: if the error is client_authorization_failed, ensure that the correct Keyset for production is used for production. Also ensure that the keyset is also enabled for Oauth
Finally, you can use/refer to the official python SDK as well here
A simple way to check if the particular scope, in this case https://api.ebay.com/oauth/api_scope/buy.deal is even allowed for this app, is to navigate to the keyset page under Keys link and click on "Oauth scopes" under the keyset which details the scopes allowed and their purpose. If the application is once authorized for buy.deal, then the scope will appear there.
UPDATE
GetDeals API is restricted in Production for authorized applications only. Please reach out to the eBay developer program as provided in the link on the page below.
https://developer.ebay.com/api-docs/buy/deal/overview.html#API
I use a python code to connect to an API to retrieve information in json format. A password is used for the authentication to the API. The authentication was working fine so far, but since I changed my password into one with special characters, things went wrong. i guess the password is not well handled either by the API or the code I use.
This is the piece of code that seems to cause the errors:
# Create connection and request header.
# This class does not perform any verification of the server`s certificate.
conn = HTTPSConnection(ADDRESS)
auth_header = 'Basic %s' % (':'.join([USER_NAME, PASSWORD]).encode('Base64').strip('\r\n'))
request_header = {'Authorization':auth_header.decode(),
'Content-Type': content_type}
The error I get after executing the code is the following:
response = self.perform_request(log, 'GET', 'network', params=query)
File "/usr/local/lib/python2.7/query_code.py", line 103, in perform_request
auth_header = 'Basic %s' % str(":".join([USER_NAME, PASSWORD]).encode('Base64').strip('\r\n'))
TypeError: sequence item 1: expected string or Unicode, NoneType found
I should have had a successful connection to the API with code 200 and results returned in json format.
I have difficulties spotting the origin of all this, and would like to have your opinion on what I think it is, and how to solve it.
Try this instead:
def basic_token(key, secret):
return base64.b64encode(bytes(f'{key}:{secret}', 'utf-8')).decode('utf-8')
auth_header = f'basic {basic_token(USERNAME, PASSWORD)}'
If you are using a python version < 3.6, you will need to use .format instead of f-strings:
def basic_token(key, secret):
return base64.b64encode(bytes('{}:{}'.format(key, secret), 'utf-8')).decode('utf-8')
auth_header = 'basic {}'.format(basic_token(USERNAME, PASSWORD))
I'm fairly new to Python programming and I don't know all the libraries needed for the following.
I would like to use Python to test some HTTP APIs. Mainly I want to use OAuth and make a few JSON calls. The APIs in question can be found on: https://developers.trustpilot.com/authentication and the generate product review link (I can only use one link)
I want to authenticate myself and then generate a product review link in one step. So far I've been using the Advanced REST client (ARC) to make these calls individually. I could also use .arc files if you think it's easier.
The idea would be make these calls successively in one go. So it would be something along the lines:
1) Make the authentication call.
The HTTP Method looks like this:
https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken
Method Post:
Header
Authorization: Basic Base64encode(APIkey:Secret)
Content-Type: application/x-www-form-urlencoded
Payload:
grant_type=password&username=user#mail.com&password=SomePass
Translate this bit into Python basically.
1.a) Add a header to the call
Header Authorization: base64encode hash Content-Type: application/x-www-form-urlencoded
1.b) Add a payload to the call
Payload: grant_type=password&username
4) Receive the token from call made in step 1) (Result is format)
"access token": Auth_token
5) Take the token and use it in creating a product review.
5.a) Add the token in the header
Header: Authorization: Bearer Auth_token
6.a) Add a JSON payload to the call made in step 5.
Here's the code I have so far:
Import requests
header = {'Authorization: Basic NnNrQUprTWRHTU5VSXJGYXBVRGxack1oT01oTUFRZHI6QTFvOGJjRUNDdUxBTmVqUQ==}','Content-Type: application/x-www-form-urlencoded'}
payload = {'grant_type=password&username=email#address.com&password=SomePassword'}
r = requests.post('https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken', headers=header, params=payload )
Ideally I want to create the requests.post(url, header, payload) and then return what the server answers in JSON format. I think that print r.text would do the last part.
So this is the code I have writtent (that works now):
import requests
import getpass
import json
from requests.auth import HTTPBasicAuth
header = {'grant_type':'password' , 'username':'mail#maildomain.com', 'password':'YourPassword'}
username= "YOURAPIKEY" #APIKey
password= "YOURSECRET" #Secret
res = requests.post(
'URL/v1/oauth/oauth-business-users-for-applications/accesstoken',
auth=HTTPBasicAuth(username, password), # basic authentication
data=header)
#print(res.content) #See content of the call result.
data = res.json() # get response as parsed json (will return a dict)
auth_token = data.get('access_token')
requests can do all what you ask without any work from your part.
See the doc for authentication, parameters, json output, json input
Make the authentication call.
import requests
import getpass
from requests.auth import HTTPBasicAuth
username = raw_input('Username: ')
password = getpass.getpass('Password: ')
res = requests.post(
'https://api.trustpilot.com/v1/oauth/oauth-business-users-for-applications/accesstoken',
auth=HTTPBasicAuth(username, password), # basic authentication
params={ # url parameters
'grant_type': 'password',
'username': 'email#address.com',
'password': 'SomePassword'
})
Receive the token from call made in step 1) (Result is format)
# res = requests.post.....
data = res.json() # get response as parsed json (will return a dict)
auth_token = data.get('access token')
Take the token and use it in creating a product review.
request.post(
'.../product_review',
headers={
'Authorization': 'Bearer ' + auth_token
},
json={'my': 'payload'}) # send data as json
I need to HTTP Basic Auth for a REST call. In the username I have to provide a domain (which has a hyphen) and then a backslash to separate it from the username, like this: DOM-AIN\user_name. Then the password is pretty benign.
This works fine with curl:
curl 'https://DOM-AIN\user_name:password#myurl.com'
I need to put this into Python now, but I've tried with requests and urllib/2/3...they don't like the \ : or the #. Even when I URL encode to %40, etc., those get interpreted as an actual : and urllib thinks I'm trying to define a port and I get an error: Invalid socket, I think, I forgot.
So I tried passing the username and password in the header using urllib3, but I get an unauthorized access error and I suspect it's because I need to somehow encode the username in the header to account for the backslash (%5C), but that doesn't seem to be working either.
Here is some code that doesn't work:
# Attempt 1
http = urllib3.PoolManager()
url1 = https://ws.....
headers = urllib3.util.make_headers(basic_auth='DOM-AIN\user_name:password')
r1 = http.request('GET', url1, headers=headers)
response = r1.data
# Attempt 2
passwordManager = urllib2.HTTPPasswordMgrWithDefaultRealm()
passwordManager.add_password(None, url, 'DOM-AIN\user_name, password)
authenticationHandler = urllib2.HTTPBasicAuthHandler(passwordManager)
opener = urllib2.build_opener(authenticationHandler)
data = opener.open(url1)
There were some other attempts with request, but I don't have those anymore. I can get the errors of these if it would be useful, but if there is already a known thing I'm doing wrong that would be great...
Backslash should be escaped in Python string literals:
username = 'DOM-AIN\\user_name' # OR
username = r'DOM-AIN\user_name' # raw-string literal
Example:
import urllib2, base64
request = urllib2.Request('https://example.com')
credentials = base64.b64encode(username + b':' + password)
request.add_header('Authorization', b'Basic ' + credentials)
response = urllib2.urlopen(request)
Note: unlike HTTPBasicAuthHandler code; it always sends the credentials without waiting for 401 response with WWW-Authenticate header.
First convert your DOM-AIN\user_name into base64 string. Lets say its XXXXYYYYYYY. Now place this base64 string into the http header like below code with urllib2.
headers = { 'Authorization:' : 'Basic XXXXYYYYYYY' }
req = urllib2.Request(url, data, headers)
I found a way using urllib, with this post's mention of FancyURLopener sending me down the right path. This was the closest I could come to replicating the way it worked in curl, although looking at Sabuj's answer there might be a way to use headers properly, but I haven't tried his method.
import urllib
opener = urllib.FancyURLopener()
data = opener.open('https://DOM-AIN**%5C%user_name:password#url.com?whatever_parameters')
response = data.read()
It works when I only URL encoded the backslash. Didn't work when I encoded the other characters like : and #.
I need to perform preemptive basic authentication against an HTTP server, i.e., authenticate right away without waiting on a 401 response. Can this be done with httplib2?
Edit:
I solved it by adding an Authorization header to the request, as suggested in the accepted answer:
headers["Authorization"] = "Basic {0}".format(
base64.b64encode("{0}:{1}".format(username, password)))
Add an appropriately formed 'Authorization' header to your initial request.
This also works with the built-in httplib (for anyone wishing to minimize 3rd-party libs/modules). I am using it to authenticate with our Jenkins server using the API Token that Jenkins can create for each user.
>>> import base64, httplib
>>> headers = {}
>>> headers["Authorization"] = "Basic {0}".format(
base64.b64encode("{0}:{1}".format('<username>', '<jenkins_API_token>')))
>>> ## Enable the job
>>> conn = httplib.HTTPConnection('jenkins.myserver.net')
>>> conn.request('POST', '/job/Foo-trunk/enable', None, headers)
>>> resp = conn.getresponse()
>>> resp.status
302
>>> ## Disable the job
>>> conn = httplib.HTTPConnection('jenkins.myserver.net')
>>> conn.request('POST', '/job/Foo-trunk/disable', None, headers)
>>> resp = conn.getresponse()
>>> resp.status
302
I realize this is old, but I figured I'd throw in the solution if you're using Python 3 with httplib2 since I haven't been able to find it anywhere else. I'm also authenticating against a Jenkins server using the API Token for each Jenkins user. If you're not concerned with Jenkins, simply substitute the actual user's password for the API Token.
b64encode is expecting an binary string of ASCII characters. With Python 3 a TypeError will be raised if a plain string is passed in. To get around this, the "user:api_token" portion of the header must be encoded using either 'ascii' or 'utf-8', passed to b64encode, then the resulting byte string must be decoded to a plain string before being placed in the header. The following code did what I needed:
import httplib2, base64
cred = base64.b64encode("{0}:{1}".format(
<user>, <api_token>).encode('utf-8')).decode()
headers = {'Authorization': "Basic %s" % cred}
h = httplib2.Http('.cache')
response, content = h.request("http://my.jenkins.server/job/my_job/enable",
"GET", headers=headers)