I am trying to access custom headers sent back to my client by the server, but I cant find them in any of the header objects in the response object listed in the Requests documentation
From Requests documentation:
We can view the server’s response headers using a Python dictionary:
>>> r.headers
{
'content-encoding': 'gzip',
'transfer-encoding': 'chunked',
'connection': 'close',
'server': 'nginx/1.0.4',
'x-runtime': '148ms',
'etag': '"e1ca502697e5c9317743dc078f67693f"',
'content-type': 'application/json'
}
Access specific header like this:
>>> r.headers['Content-Type']
'application/json'
>>> r.headers.get('content-type')
'application/json'
Related
I have a Django restful API (using django-rest-framework) where the POST requests require prior authentication. I would like to populate the database by sending data to the API, however, I cannot figure out how to do the authentication programmatically. I tried requests, pycurl and httplib2 so far:
import httplib2
from urllib.parse import urlencode
h = httplib2.Http(".cache")
h.add_credentials(username, password)
headers = {'Content-type': 'application/x-www-form-urlencoded'}
data = {
"label": "SA2",
"flagged": "false",
"notes": ""
}
resp, content = h.request(
"https://example.com/api/data", "POST", urlencode(data), headers=headers
)
resp
>>>
{
'server': 'nginx/1.18.0 (Ubuntu)',
'date': 'Sat, 05 Mar 2022 00:06:32 GMT',
'content-type': 'text/html',
'transfer-encoding': 'chunked',
'connection': 'keep-alive',
'cross-origin-opener-policy': 'same-origin',
'referrer-policy': 'same-origin',
'vary': 'Origin',
'x-content-type-options': 'nosniff',
'x-frame-options': 'DENY',
'status': '403',
'content-length': '1867',
'-content-encoding': 'gzip'
}
content
>>>
b'{"detail":"Authentication credentials were not provided."}'
In the browser, I first have to visit the login page. Then the website sends a CRFT token.
Here is a solution using curl and bash. I prefer to use Python.
You need to provide the credentials in the header.
import base64
# ...
username="<username>"
password="<password>"
credentials=username + ":" + password
encoded_credentials = base64.b64encode(credentials.encode()).decode()
headers["Authorization"] = "Basic " + encoded_credentials
# ...
https://www.ibm.com/docs/en/ibm-mq/9.0?topic=security-using-http-basic-authentication-rest-api
I'm trying to implement a small function to verify possible phishing URL's and thought that using Google Safe Browsing API would be a good start.
After reading the API documentation I thought I had a handle on things and cobbled together the following code:
import requests
import json
url = "https://safebrowsing.googleapis.com/v4/threatMatches:find?key=<REDACTED>"
headers = {'content-type': 'application/json'}
payload = {'client': {'clientId': "mycompany", 'clientVersion': "0.1"},
'threatInfo': {'threatTypes': ["SOCIAL_ENGINEERING", "MALWARE"],
'platformTypes': ["ANY_PLATFORM"],
'threatEntryTypes': ["URL"],
'threatEntries:': [{'url': "http://www.urltocheck1.org"}]}}
print (json.dumps(payload, indent=4))
r = requests.post(url, headers=headers, json=payload)
If I do a
print (json.dumps(payload, indent=4)
it all looks ok. However, the reply I get back from Google doesn't agree.
{'error': {'message': 'Invalid JSON payload received. Unknown name
"threat_entries:" at \'threat_info\': Cannot find field.', 'status':
'INVALID_ARGUMENT', 'code': 400, 'details': [{'#type':
'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations':
[{'field': 'threat_info', 'description': 'Invalid JSON payload
received. Unknown name "threat_entries:" at \'threat_info\': Cannot
find field.'}]}]}} {'X-Frame-Options': 'SAMEORIGIN',
'Transfer-Encoding': 'chunked', 'Cache-Control': 'private', 'Date':
'Tue, 25 Oct 2016 07:55:30 GMT', 'Content-Type': 'application/json;
charset=UTF-8', 'Alt-Svc': 'quic=":443"; ma=2592000;
v="36,35,34,33,32"', 'X-Content-Type-Options': 'nosniff',
'Content-Encoding': 'gzip', 'X-XSS-Protection': '1; mode=block',
'Server': 'ESF'} application/json; charset=UTF-8
I can't - as usual - spot my mistake. Can someone else spot it and possibly put me on the correct track?
Just remove unnecessary colon after threatEntries
and it should work just fine.
Also if you are using requests version 2.4.2 or newer you do not need to insert content-type header to your code, instead you could move your key to the params section:
import requests
import json
api_key='your_key'
url = "https://safebrowsing.googleapis.com/v4/threatMatches:find"
payload = {'client': {'clientId': "mycompany", 'clientVersion': "0.1"},
'threatInfo': {'threatTypes': ["SOCIAL_ENGINEERING", "MALWARE"],
'platformTypes': ["ANY_PLATFORM"],
'threatEntryTypes': ["URL"],
'threatEntries': [{'url': "http://www.urltocheck1.org"}]}}
params = {'key': api_key}
r = requests.post(url, params=params, json=payload)
# Print response
print(r)
print(r.json())
I am using python-pipedrive to wrap Pipedrive's API though it doesn't quite work out of the box on python3 (which I'm using) so I modified it. I'm having trouble with just the Http requests portion.
This is what taught me how to use Httplib2: https://github.com/jcgregorio/httplib2/wiki/Examples-Python3
Basically, I just want to send a GET request to this:
https://api.pipedrive.com/v1/persons/123?api_token=1234abcd1234abcd
This works:
from httplib2 import Http
from urllib.parse import urlencode
PIPEDRIVE_API_URL = "https://api.pipedrive.com/v1/persons/123?api_token=1234abcd1234abcd"
response, data = http.request(PIPEDRIVE_API_URL, method='GET',
headers={'Content-Type': 'application/x-www-form-urlencoded'})
However, Pipedrive returns an error 401 with 'You need to be authorized to make this request.' if I do this:
PIPEDRIVE_API_URL = "https://api.pipedrive.com/v1/"
parameters = 'persons/123'
api_token = '1234abcd1234abcd'
response, data = http.request(PIPEDRIVE_API_URL + parameters,
method='GET', body=urlencode(api_token),
headers={'Content-Type': 'application/x-www-form-urlencoded'})
The actual response is:
response =
{'server': 'nginx',
'status': '401',
'connection': 'keep-alive',
'set-cookie': 'pipe-session=7b6ddadbc67abdadb6a67dbadcb; path=/; domain=.pipedrive.com; secure; httponly',
'date': 'Sat, 11 Jun 2016 06:50:13 GMT',
'transfer-encoding': 'chunked',
'x-frame-options': 'SAMEORIGIN',
'content-type': 'application/json, charset=UTF-8',
'x-xss-protection': '1; mode=block'}
data =
{'success': False,
'error': 'You need to be authorized to make this request.'}
How do I properly provide the api_token as a parameter (body) to the GET request? Anyone know what I'm doing wrong?
You need to provide the api_token as a query parameter. Concatenate the stings like this
PIPEDRIVE_API_URL = "https://api.pipedrive.com/v1/"
route = 'persons/123'
api_token = '1234abcd1234abcd'
response, data = http.request(PIPEDRIVE_API_URL + route + '?api_token=' + api_token,
method='GET',
headers={'Content-Type': 'application/x-www-form-urlencoded'})
I am trying to authenticate to Docushare with Python 3.4 using requests 2.7. I am relatively new to Python and to the requests module but I've done a lot of reading and am not able to make any more progress. My code doesn't give any errors and I receive a JSESSIONID cookie back from my post request, but I'm not getting the AmberUser authentication cookie. I don't know how to investigate this further to find out what the problem is.
The form of the request comes from http://docushare.xerox.com/en-us/Help/prog/prog5.htm#Authentication
Request:
POST /dscgi/ds.py/Login HTTP/1.1
Host: docushare.xerox.com
Content-Type: text/xml
Content-Length: xxxx
<?xml version="1.0" ?>
<authorization>
<username>msmith</username>
<password>mysecretstring</password>
</authorization>
My Python / requests code looks like:
import requests
url = "https://mydocusharesite/dsweb/Login"
xml="""<?xml version='1.0' ?>
<authorization>
<username>myusername</username>
<password><![CDATA[mypa$$word]]></password>
<domain>DocuShare</domain>
</authorization>"""
headers = {"DocuShare-Version":"5.0", 'Content-Type':'text/xml'}
s = requests.Session()
r = s.post(url,data=xml,headers=headers)
print('Status code:', r.status_code)
print('headers:\n', r.headers)
print('request headers:\n',r.request.headers)
c = s.cookies
print('Cookies:\n', c)
The output I get is
Status code: 200
headers:
{'set-cookie': 'JSESSIONID=21B7E5E0D83D1F1267371B9FD1B19BBC.tomcat1; Path=/docushare; Secure', 'transfer-encoding': 'chunked', 'connection': 'close', 'content-type': 'text/html;charset=UTF-8', 'cache-control': 'private', 'date': 'Sun, 07 Jun 2015 02:22:59 GMT', 'expires': '-1'}
request headers:
{'Connection': 'keep-alive', 'DocuShare-Version': '5.0', 'Accept': '*/*', 'Content-Type': 'text/xml', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '153', 'User-Agent': 'python-requests/2.7.0 CPython/3.4.3 Darwin/14.3.0'}
Cookies:
<RequestsCookieJar[<Cookie JSESSIONID=21B7E5E0D83D1F1267371B9FD1B19BBC.tomcat1 for mydocusharesite>]>
Your CDATA section should look like <![CDATA[mypa$$word]]>. Your code is currently sending ![CDATA[mypa$$word]] as the actual password.
Hi I am working on a simple program to get a token-id from a router using REST API. The problem that I am facing, is that I do not see the Authorization headers when I use HTTPDigestAuth. When I use the Google App POSTMAN, I can see the headers and it work. What I am missing in my code?
My code:
import requests
from requests.auth import HTTPBasicAuth, HTTPDigestAuth
user = 'pod1u1'
passwd = 'pass'
url = 'https://10.0.236.188/api/v1/auth/token-services'
auth = HTTPDigestAuth(user, passwd)
r = requests.post(url, auth=auth, verify=False)
print 'Request headers:', r.request.headers
print 'Status Code: ', r.status_code
print 'response Headers: ', r.headers
print '######################################'
auth = HTTPBasicAuth(user, passwd)
r = requests.post(url, auth=auth, verify=False)
print 'Request headers:', r.request.headers
print 'Status Code: ', r.status_code
print 'response Headers: ', r.headers
Shell commands w/ output:
My script --
$python digest.py
Request headers: CaseInsensitiveDict({'Content-Length': '0', 'Accept-Encoding': 'gzip, deflate, compress', 'Accept': '*/*', 'User-Agent': 'python-requests/2.2.0 CPython/2.7.5 Darwin/13.0.0'})
Status Code: 401
response Headers: CaseInsensitiveDict({'date': 'Tue, 14 Jan 2014 00:28:27 GMT', 'content-length': '83', 'content-type': 'application/json', 'connection': 'keep-alive', 'server': 'nginx/1.4.2'})
######################################
Request headers: CaseInsensitiveDict({'Accept': '*/*', 'Content-Length': '0', 'Accept- Encoding': 'gzip, deflate, compress', 'Authorization': u'Basic cG9kMXUxOkMxc2NvTDF2Mw==', 'User-Agent': 'python-requests/2.2.0 CPython/2.7.5 Darwin/13.0.0'})
Status Code: 401
response Headers: CaseInsensitiveDict({'date': 'Tue, 14 Jan 2014 00:28:27 GMT', 'content-length': '448', 'content-type': 'text/html', 'connection': 'keep-alive', 'server': 'nginx/1.4.2'})
POSTMAN
POST /api/v1/auth/token-services HTTP/1.1
Host: 10.0.236.188
Authorization: Digest username="pod1u1", realm="pod1u1#ecatsrtpdmz.cisco.com", nonce="", uri="/api/v1/auth/token-services", response="08ac88b7f5e0533986e9fc974f132258", opaque=""
Cache-Control: no-cache
{
"kind": "object#auth-token",
"expiry-time": "Tue Jan 14 00:09:27 2014",
"token-id": "Vj7mYUMTrsuljaiXEPoNJNiXLzf8UeDsRnEgh3DvQcU=",
"link": "https://10.0.236.188/api/v1/auth/token-services/9552418862"
}
You are doing a POST, so intuitively, you need to pass 'params' argument to requests.post method
You can use a sniffer to see exactly what POSTMAN send to the url and do the same...
Just for info, I did a requests.get with digest credentials (on another url), it worked and I saw auth headers.
Maybe you could first start with a GET to create a "session" then do your POST, just a guess :)
[ADDED]
I would also try to use "raw" headers as a workaround :
[...]
headers = {
"Host": "10.0.236.188",
"Authorization": '''Digest username="pod1u1", realm="pod1u1#ecatsrtpdmz.cisco.com", nonce="", uri="/api/v1/auth/token-services", response="08ac88b7f5e0533986e9fc974f132258", opaque=""''',
"Cache-Control": "no-cache"
}
r = requests.post(url, auth=auth, headers=headers, verify=False)
[/ADDED]
The Problem is in the server side: Lukasa # GitHUB help me. "That doesn't look like a service that requires Digest Auth. If Digest Auth is required, the 401 should contain a header like this: WWW-Authenticate: Digest qop="auth". This does not. Instead, you're being returned a JSON body that contains an error message.
Digest Auth should not send headers on the initial message, because the server needs to inform you how to generate the digest. I invite you to open up the section of code that generates the digest. We require the realm, nonce and qop from the server before we can correctly generate the header."