Develop Reference

How do I use Basic Auth and Bearer Token Auth with python requests module?

Question
How do I use Basic Auth and Bearer Token Auth with python requests module?
Background
I write curl script as same as I want to do with python requests module.
curl --location --request POST 'https://username#password:example.com/path/to/endpoint' \
--header 'Authorization: Bearer SOME_TOKEN' \
--header 'Content-Type: application/json' \
--data-raw '{
"example": "example"
}'
Below code is python edition for the above script.
import requests
import json
url = "https://username#password:example.com/path/to/endpoint"
payload = json.dumps({
"example": "example"
})
headers = {
'Authorization': 'Bearer SOME_TOKEN',
'Content-Type': 'application/json'
}
response = requests.request("POST", url, headers=headers, data=payload)
But this Python code does not work as intended.
I captured http request that this python code sends.
The result is below.
POST /path/to/endpoint HTTP/1.1
Host: example
Accept: */*
Accept-Encoding: gzip, deflate
Authorization: Basic xxxxx
Connection: keep-alive
Content-Length: 31
Content-Type: application/json
User-Agent: python-requests/2.28.1
{"example": "example"}
I don't know why Auhorization: Basic xxxx appears.
I intend that Authorization: Bearer SOME_TOKEN.
Please teach me how to use Basic auth and Bearer token auth with python requests module.
appendix
I think that I can't use both Basic auth and Bearer token auth with python requests module.
Because this module depends on python http.client module, and I think this module does not support basic auth with including username and password in url(https://username:password#example.com/ format).
I examined the http request that the above codes sends to http server.

Upload binary image file using linkedin create share api gives null response using python code but curl command works

Curl command below works perfectly but converted curl command into python code doesn't. However python code return status code 201 but response.text prints nothing.
FYI: I am following this documentation.
Here is Working Curl Command:
curl -i --upload-file "/home/abhay/download.jpeg" --header "Authorization: Bearer xxxxxxxxx" 'https://api.linkedin.com/mediaUpload/C5122AQE4GSagETyxQA/feedshare-uploadedImage/0?ca=vector_feedshare&cn=uploads&m=AQK_YGFfk_crEwAAAWoGPkMNETNZnhoeCKKmD_CKYeasF1NHooQ7pdeZWA&app=5969925&sync=0&v=beta&ut=1DL2-qd0JrroI1'
Converted python code
upload_url = 'https://api.linkedin.com/mediaUpload/C5122AQE4GSagETyxQA/feedshare-uploadedImage/0?ca=vector_feedshare&cn=uploads&m=AQK_YGFfk_crEwAAAWoGPkMNETNZnhoeCKKmD_CKYeasF1NHooQ7pdeZWA&app=5969925&sync=0&v=beta&ut=1DL2-qd0JrroI1'
headers = {
'accept' : 'application/json',
'X-Restli-Protocol-Version': '2.0.0',
'Authorization': 'Bearer xxxxxxxxx',
# 'Content-Type': 'multipart/form-data'
'Content-Type': 'image/jpeg',
}
image = open("/home/abhay/download.jpeg" ,"rb")
files = {'images_file': image }
response = requests.post(upload_url, headers=headers, files=files)
print response.status_code
print response.text
Curl gives correct response:
HTTP/2 201
date: Wed Apr 10 07:59:03 UTC 2019
server: Play
set-cookie: lang=v=2&lang=en-us; Path=/; Domain=api.linkedin.com
x-ambry-creation-time: Wed Apr 10 07:59:03 UTC 2019
access-control-allow-origin: https://www.linkedin.com
content-length: 0
x-li-fabric: prod-lsg1
x-li-pop: prod-tmu1
x-li-proto: http/2
x-li-uuid: FPDYGfsNlBXg+OFSZisAAA==
set-cookie: lidc="b=SB83:g=115:u=5:i=1554883143:t=1554953528:s=AQEVXPybFdnyKyPqSWbf_ax997ap-22S"
x-li-route-key: "b=SB83:g=115:u=5:i=1554883143:t=1554953528:s=AQEVXPybFdnyKyPqSWbf_ax997ap-22S"
Python code gives the following response
201 (status code)
None (response.text)
Can anybody help me out what I am missing out in python code? Thanks!

How send json's data via curl?

I have some simple code, like this:
import json
from bottle import route, request,run
#route('/process_json',methods='POST')
def data_process():
data = json.loads(request.data)
username = data['username']
password = data['password']
run(host='localhost', port=8080, debug=True)
I would like to send a data in json format, like this:
$ curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d
'{"username":"fooba","password":"1234"}' url
However, I have this problem:
File "/usr/lib/python2.7/dist-packages/bottle.py", line 862,
in _handle return route.call(**args)
File "/usr/lib/python2.7/dist-packages/bottle.py", line 1729,
in wrapper rv = callback(*a, **ka)
File "testtest.py", line 5,
in data_process data = json.loads(request.data)
File "/usr/lib/python2.7/dist-packages/bottle.py", line 1391,
in getattr raise AttributeError('Attribute %r not defined.' % name)
AttributeError: Attribute 'data' not defined
I also tried add at the beginning like here(http://bottlepy.org/docs/dev/tutorial.html#html-form-handling):
return '''
Username:
Password:
'''
But it also doesn't work.

It is good, you have provided the bottle application code. It was broken.
Following modification works:
import json
from bottle import route, request, run
#route('/process_json', method="POST")
def data_process():
data = json.load(request.body)
print "data", data
username = data['username']
password = data['password']
run(host='localhost', port=8080, debug=True)
Testing with curl
$ curl -i -H "Accept: application/json" -H "Content-Type: application/json" -X POST -d '{"username": "fooba", "password": "1234"}' http://localhost:8080/process_json
HTTP/1.0 200 OK
Date: Mon, 14 Jul 2014 16:18:25 GMT
Server: WSGIServer/0.1 Python/2.7.6
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Testing with HTTPie
$ http POST http://localhost:8080/process_json username=jan password=pswd
HTTP/1.0 200 OK
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Date: Mon, 14 Jul 2014 16:15:16 GMT
Server: WSGIServer/0.1 Python/2.7.6
Actually, I was playing with your example to find out, how it would look like in http command. It is really much simpler and using --verbose we may check, it really forms valid JSON payload:
$ http --verbose POST http://localhost:8080/process_json Accept:application/json
Content_type:application/json username=jan password=pswd
POST /process_json HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Authorization: Basic amFuOnZsY2luc2t5
Content-Length: 39
Content-Type: application/json; charset=utf-8
Content_type: application/json
Host: localhost:8080
User-Agent: HTTPie/0.8.0
{
"password": "pswd",
"username": "jan"
}
HTTP/1.0 200 OK
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Date: Mon, 14 Jul 2014 16:21:02 GMT
Server: WSGIServer/0.1 Python/2.7.6
The shortest form is:
$ http :8080/process_json username=jan password=pswd
HTTP/1.0 200 OK
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Date: Mon, 14 Jul 2014 16:25:12 GMT
Server: WSGIServer/0.1 Python/2.7.6
as http://localhost:8008 can be shortened to :8080 (the same applies to curl)
and if there is a payload, default method is POST.

python making POST request with JSON data

I'm trying to mimic a curl request I'm doing using python the call is. Note that
HTTPS Request
Ignoring SSL ceritification verification
My server is django
The curl command I used is
curl -k --dump-header - -H "Content-Type: application/json" -X POST --data '{"environment_name": "foo"}' https://localhost/api/v1/environment/
and the response from the server is successful
HTTP/1.1 201 CREATED
Date: Tue, 17 Jun 2014 00:59:59 GMT
Server: Server
Vary: Accept-Language,Cookie,User-Agent
Content-Language: en-us
Location: https://localhost/api/v1/environment/None/
Status: 201 CREATED
Content-Length: 0
Cneonction: close
Content-Type: text/html; charset=utf-8
However when I try to do a post request in python with 'requests' my script is
import json
data = {'enviornment_name' : 'foo'}
headers = {'Content-type' : 'application/json'}
response = requests.post("https://localhost/api/v1/environment", headers=headers, data=data, verify=False)
When running the script I get back a huge stack trace but the part in red is
E DecodeError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect data check',))
I'm not sure why I can't my script to work via python

Your server is claiming to return gzip'd content which it is not. The response is returning Content-Encoding: gzip in the headers. This could be because in requests, by default, it sends Accept-Encoding: gzip, compress. You should try setting "Accept-Encoding" to None in your headers dictionary in addition to #Fabricator's suggestion in the comments.
Your headers dictionary will look like:
headers = {'Content-Type': 'application/json', 'Accept-Encoding': None}
And your call to requests will look like
requests.post(url, headers=headers, data=json.dumps(data), verify=False)

#Fabricator I need the verify=False however I noticed the one thing in my code that was an issue for the server I was using I needed the trailing '/' at the end of the URI. In addition I also needed the json.dumps(data) not json.dump(data) in case others are looking. Thanks for the help

how to debug requests library?

I'm trying to convert my python script from issuing a curl command via os.system() to using requests. I thought I'd use pycurl, but this question convinced me otherwise. The problem is I'm getting an error returned from the server that I can see when using r.text (from this answer) but I need more information. Is there a better way to debug what's happening?
for what it's worth I think the issue revoles around converting my --data flag from curl/pycurl to requests. I've created a dictionary of the params i was passing to --data before. My guess is that one of those isn't valid but how can I get more info to know for sure?
example:
headers2 = {"Accept":"*/*", \
"Content-Type":"application/x-www-form-urlencoded", \
"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36", \
"Origin":"https://somedomain.com", \
"X-Requested-With":"XMLHttpRequest", \
"Connection":"keep-alive", \
"Accept-Language":"en-US,en;q=0.8", \
"Referer":"https://somedomain.com/release_cr_new.html?releaseid=%s&v=2&m=a&prev_release_id=%s" % (current_release_id, previous_release_id), \
"Host":"somedomain.com", \
"Accept-Encoding":"gzip,deflate,sdch", \
"Cookie":'cookie_val'}
for bug_id in ids:
print bug_id
data = {'dump_json':'1','releaseid':current_release_id, 'v':'2','m':'a','prev_release_id': previous_release_id,'bug_ids': bug_id, 'set_cols':'sqa_status&sqa_updates%5B0%5D%5Bbugid%5D=' + bug_id + '&sqa_updates%5B0%5D%5Bsqa_status%5D=6'}
print 'current_release_id' , data['releaseid']
print 'previous_release_id', data['prev_release_id']
r = requests.post(post_url, data=json.dumps(data), headers=headers2)
print r.text
The output I'm getting is a pretty generic html message that I've seen before when I've queried the server in the wrong way. So I know I'm reaching the right server at least.
I'm not really expecting any output. This should just post to the server and update a field in the DB.

Anatomy of an http response
Example (loading this page)
HTTP/1.1 200 OK
Cache-Control: public, max-age=60
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Expires: Fri, 27 Sep 2013 19:22:41 GMT
Last-Modified: Fri, 27 Sep 2013 19:21:41 GMT
Vary: *
X-Frame-Options: SAMEORIGIN
Date: Fri, 27 Sep 2013 19:21:41 GMT
Content-Length: 12706
<!DOCTYPE html>
<html>
... truncated rest of body ...
The first line is the status line and consists of the status code and status text.
Headers are key/value pairs. Headers are ended with an empty new line. The empty line denotes there are no more headers and the start of the payload / body follows.
body consumes the rest of the message.
The following explains how to extract the 3 parts:
Status Line
Use the following to get the status line sent back from the server
>>> bad_r = requests.get('http://httpbin.org/status/404')
>>> bad_r.status_code
404
>>> bad_r.raise_for_status()
Traceback (most recent call last):
File "requests/models.py", line 832, in raise_for_status
raise http_error
requests.exceptions.HTTPError: 404 Client Error
(source)
Headers:
r = requests.get('http://en.wikipedia.org/wiki/Monty_Python')
# response headers:
r.headers
# request headers:
r.request.headers
Body
Use r.text.
Post Request Encoding
The 'content-type' you send to the server in the request should match the content-type you're actually sending. In your case, you are sending json but telling the server you're sending form data (which is the default if you do not specify).
From the headers you show above:
"Content-Type":"application/x-www-form-urlencoded",
But your request.post call sets data=json.dumps(data) which is JSON. The headers should say:
"Content-type": "application/json",

The value returned from the request object contains the request information under .request.
Example:
r = requests.request("POST", url, ...)
print("Request headers:", r.request.headers)
print("Request body:", r.request.body)
print("Response status code:", r.status_code)
print("Response text:", r.text.encode('utf8'))