Develop Reference

Replicate Python requests.post in R using httr POST

I am trying to adapt my HTTP request from running it in Python to R. This is the post request in Python:
import requests
import json
r = requests.post("https://feed-dev.ihsmarkit.com/apikey",
data={'username': 'markit/resellers/API_OPS/accounts/demo.dv', 'password':
'Example#N6'})
print("POST /apikey", r.status_code, r.reason)
apikey = r.text
print(apikey)
I did some research and found the httr package in R is best for dealing with API related requests. However I tried to use the POST() function in a few attempts but got the same error 400 ("MISSING_PARAMETER": Parameter username not provided.") responses. Here are a few attempts I used:
#attempt 1
response <- POST(url = "https://feed-dev.ihsmarkit.com/apikey",
add_headers(.headers = c("Content-Type"="application/x-www-form-urlencoded")),
authenticate('markit/resellers/API_OPS/accounts/demo.dv', 'Example#N6')
)
#attempt 2
request_body <- data.frame(
username = 'markit/resellers/API_OPS/accounts/demo.dv',
password = 'Example#N6'
)
request_body_json <- toJSON(list(data = request_body), auto_unbox = TRUE)
POST(url = "https://feed-dev.ihsmarkit.com/apikey",
add_headers(.headers = c("Content-Type"="application/x-www-form-urlencoded","Accept"="application/json"),
body = request_body_json))
#attempt 3
result <- POST(url = "https://feed-dev.ihsmarkit.com/apikey",
add_headers(.headers = c("Content-Type"="application/x-www-form-urlencoded","Accept"="application/json")),
body = '{"data":{"username":"markit/resellers/API_OPS/accounts/demo.dv", "password":"Example#N6}}',
encode = 'raw')
Do you know how should I properly convert my request?

Use
response <- POST(url = "https://feed-dev.ihsmarkit.com/apikey",
encode = "form",
body= list(
username ='markit/resellers/API_OPS/accounts/demo.dv',
password = 'Example#N6')
)
Just pass your data as a list and the POST will take care of formatting it as form data when you choose encode="form". Your python code doesn't seem to use JSON at all. You just have literal dictionary values where you are storing your data. Only use authenticate() when the HTTP endpoint requires basic HTTP authentication. For endpoints that require a username/password in the body of the message, that's not how basic HTTP authentication works.

Basic Authentication returns 401 Client Error but works in postman

I have gone through number of similar posts related to firing GET requests with Basic Auth (eg: Python, HTTPS GET with basic authentication), still can't figure out the problem. I keep getting the error requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url
With the same credentials, headers tried the same in postman it works as expected. Verified that base64encoded value for the api_key, password is exactly same as the value used in postman, so I don't think its encoding or resource access permission problem.
python -V
Python 3.6.4 :: Anaconda, Inc.
Approach 1
api_key = 'some_api_key'
password = 'some_password'
headers = {'accept': 'application/json'}
url = 'https://test.access.com/this/url'
api_key_password = "%s:%s" % (api_key, password)
b64_encoded = b64encode(bytes(api_key_password, 'utf-8')).decode("ascii")
headers['authorization'] = 'Basic %s' % b64_encoded
response = requests.get(url,
headers=headers)
if (response.ok):
json_data = json.loads(response.content)
print (json_data)
else:
print (response)
response.raise_for_status()
Approach 2
api_key = 'some_api_key'
password = 'some_password'
url = 'https://test.access.com/this/url'
headers = {
'accept': 'application/json',
}
response = requests.get(url, headers=headers, auth=(api_key, password))
print (response.ok)
if (response.ok):
json_data = json.loads(response.content)
print (json_data)
else:
print (response)
response.raise_for_status()
Can you please provide some pointers?

I had a similar issue (although in .NET Framework).
In my case the reason was that I was using the url without a forward slash in the end and the API apparently does not support that.
So https://test.access.com/this/url
Throws 401 error Unauthorized
but
https://test.access.com/this/url/
Returns 200 OK.

Older post but I had a similar issue. Postman will cache your JSESSIONID. Be sure you are clearing out that cookie while testing. If you are hitting an API that requires a login API call to establish a session before you can make subsequent API calls, this Postman behavior can produce a false sense of security.
In this situation with Python requests, it can be handled with code similar to what I've provided below:
import requests,json
loginAPI = "https://myapi.myco.comv/someuri/someuri/users/login"
someHTTPGetAPI = "https://myapi.myco.com/someuri/someuri/someservice"
username = "myuser"
password = "mypass"
headers = {
"Content-Type": "application/json",
"login": username,
"password": password
}
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
verify=False
session = requests.Session()
sessionResponse = session.get(url=loginURL,headers=headers, verify=verify)
if sessionResponse.status_code == 200:
getResponse = session.get(url=someHTTPGetAPI)
if getResponse.status_code == 200:
responseJSON = agentStatus.json()

Get request with Python 2.7

I'm trying to send a get request with python 2.7. The server I'm trying to access has basic authentication and the url I'm accessing is just displaying a string I want my script to print that string.
So basically my code is sending a get request to this server and it saves the string it receives from the server, which also has basic auth.
The problem is that it prints out the html page for the login page not the string.
username = 'username'
password = 'password'
server = "http://someserver/update"
def get_auth():
request = urllib2.Request(server)
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
result = urllib2.urlopen(request)
# print(result)
def get_string():
f = urllib2.urlopen(server)
print f.read()
def main():
get_auth()
get_string()
if __name__ == '__main__':
main()

They way you are using get_string() never actually sends the authentication header.
Rather than fiddling with urllib2, you might want to try the requests library instead::
import requests
requests.get('http://someserver/update', auth=('username', 'password'))
see: http://docs.python-requests.org/en/latest/user/authentication/

How to authenticate a site with Python using urllib2?

After much reading here on Stackoverflow as well as the web I'm still struggling with getting things to work.
My challenge: to get access to a restricted part of a website for which I'm a member using Python and urllib2.
From what I've read the code should be like this:
mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
url = 'http://www.domain.com'
mgr.add_password(None, url, 'username', 'password')
handler = urllib2.HTTPBasicAuthHandler(mgr)
opener = urllib2.build_opener(handler)
urllib2.install_opener(opener)
try:
response = urllib2.urlopen('http://www.domain.com/restrictedpage')
page = response.read()
print page.geturl()
except IOError, e:
print e
The print doesn't print "http://www.domain.com/restrictedpage", but shows "http://www.domain.com/login" so my credentials aren't stored/processed and I'm being redirected.
How can I get this to work? I've been trying for days and keep hitting the same dead ends. I've tried all the examples I could find to no avail.
My main question is: what's needed to authenticate to a website using Python and urllib2?
Quick question: what am I doing wrong?

Check first manually what is really happening when you are successfully authenticated (instructions with Chrome):
Open develper tools in Chrome (Ctrl + Shift + I)
Click Network tab
Go and do the authentication manually (go the the page, type user + passwd + submit)
check the POST method in the Network tab of the developer tools
check the Request Headers, Query String Parameters and Form Data. There you find all the information needed what you need to have in your own POST.
Then install "Advanced Rest Client (ARC)" Chrome extension
Use the ARC to construct a valid POST for authentication.
Now you know what to have in your headers and form data. Here's a sample code using Requests that worked for me for one particular site:
import requests
USERNAME = 'user' # put correct usename here
PASSWORD = 'password' # put correct password here
LOGINURL = 'https://login.example.com/'
DATAURL = 'https://data.example.com/secure_data.html'
session = requests.session()
req_headers = {
'Content-Type': 'application/x-www-form-urlencoded'
}
formdata = {
'UserName': USERNAME,
'Password': PASSWORD,
'LoginButton' : 'Login'
}
# Authenticate
r = session.post(LOGINURL, data=formdata, headers=req_headers, allow_redirects=False)
print r.headers
print r.status_code
print r.text
# Read data
r2 = session.get(DATAURL)
print "___________DATA____________"
print r2.headers
print r2.status_code
print r2.text

For HTTP Basic Auth you can refer this : http://www.voidspace.org.uk/python/articles/authentication.shtml

HTTP 400 response while sending POST request to UrbanAirship

I am using Web2Py to create a simple app which sends Push notifications through UrbanAirship. For some reason, I am getting a 400 response when I try to send it through my code. It UA API works fine using REST client. This is my code:
url = 'https://go.urbanairship.com/api/push/'
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
# this creates a password manager
passman.add_password(None, url, username, password)
# because we have put None at the start it will always
# use this username/password combination for urls
# for which `theurl` is a super-url
authhandler = urllib2.HTTPBasicAuthHandler(passman)
# create the AuthHandler
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
# All calls to urllib2.urlopen will now use our handler
# Make sure not to include the protocol in with the URL, or
# HTTPPasswordMgrWithDefaultRealm will be very confused.
# You must (of course) use it when fetching the page though.
values = {"device_tokens": ["<DEVICE TOKEN>"], "aps": {"alert": "Hello!"}}
data = urllib.urlencode(values)
headers = {'Content-Type': 'application/json'}
req = urllib2.Request(url, data, headers)
try:
response = urllib2.urlopen(req)
return response
except IOError, e:
if e.code == 200:
return "Push sent!"
else:
return 'The server couldn\'t fulfill the request. Error: %d' % e.code
As far as I can understand, the problem is in the format of data being sent. Where am I going wrong?

The urllib.urlencode function is for making a URL-encoded parameter body (Content-Type: application/x-www-form-urlencoded). For JSON, which is apparently what you want, use json.dumps instead.