Develop Reference

Globally modify requests in a central place

I've been tasked to update an existing Python module that sends requests to an external API in many different places with requests. That is, in this module there are at least 50 usages of requests to send requests to said API.
There was a change in the API that requires a header to be added to all requests. Before manually adding the header to all 50 requests, I was wondering if it is possible to define some kind of "middleware" (as for example in Django) that could add the header for all requests at once.
Is something like this possible with requests?

You can monkey-patch the requests.request function with a wrapper that updates the dict specified by the headers arguments with additional entries.
Below is an example that would force all requests to have a header of User-Agent with the value My Browser:
import requests
import inspect
def override_headers(self, func, global_headers):
def wrapper(*args, **kwargs):
bound = sig.bind(*args, **kwargs)
bound.apply_defaults()
bound.arguments.setdefault('headers', {}).update(global_headers)
return func(*bound.args, **bound.kwargs)
sig = inspect.signature(func)
return wrapper
requests.request = override_headers(requests.request, {'User-Agent': 'My Browser'})

verify_oauth2_token uses object as function

I was doing google auth with use of backend from there:
https://developers.google.com/identity/sign-in/android/backend-auth
It seems a bit outdated and the most strange thing is that there is a line:
idinfo = id_token.verify_oauth2_token(token, requests.Request(), CLIENT_ID)
and in implementation you can see that in nested function calls, same request object lands there:
def _fetch_certs(request, certs_url):
"""Fetches certificates.
Google-style cerificate endpoints return JSON in the format of
``{'key id': 'x509 certificate'}``.
Args:
request (google.auth.transport.Request): The object used to make
HTTP requests.
certs_url (str): The certificate endpoint URL.
Returns:
Mapping[str, str]: A mapping of public key ID to x.509 certificate
data.
"""
response = request(certs_url, method='GET')
request is an object, even documentation claims so and it uses it as function. The error I get is:
TypeError: 'Request' object is not callable
What should be changed there?

Most likely you are calling the wrong python requests lib.
If you need to differentiate between the 2 available requests lib.
from google.auth.transport import requests as google_auth_request
import requests
req = google_auth_request.Request()
idinfo = id_token.verify_oauth2_token(token, req, CLIENT_ID)
See: https://google-auth.readthedocs.io/en/latest/reference/google.oauth2.id_token.html

Returning response of Tornado POST request

I have seen Tornado documentations and examples where self.write method is widely used to render some value on HTML, where the POST request was run in a handler. But I could not find much clarity on how to return the response back to client.
For example, I am calling a POST request on a Tornado server from my client. The code that accepts post request is:
class strest(tornado.web.RequestHandler):
def post(self):
value = self.get_argument('key')
cbtp = cbt.main(value)
With this, I can find the value of cbtp and with self.write(cbtp), I can get it printed in HTML. But instead, I want to return this value to the client in JSON format, like {'cbtp':cbtp}
I want to know how to modify my code so that this response is sent to the client, or give me some documentation where this this is fluently explained.
Doing something like
res = {cbtp: cbtp}
return cbtp
throws a BadYieldError: yielded unknown object

You just need to set the output type as JSON and json.dumps your output.
Normally I have the set_default_headers in a parent class called RESTRequestHandler. If you want just one request that is returning JSON you can set the headers in the post call.
class strest(tornado.web.RequestHandler):
def set_default_headers(self):
self.set_header("Content-Type", 'application/json')
def post(self):
value = self.get_argument('key')
cbtp = cbt.main(value)
r = json.dumps({'cbtp': cbtp})
self.write(r)

If the given chunk is a dictionary, we write it as JSON and set the Content-Type of the response to be application/json. (if you want to send JSON as a different Content-Type, call set_header after calling write()).
Using it should give you exactly what you want:
self.write(json.dumps({'cbtp': cbtp}))

HTTPS GET API call using base64 encoding

I am trying to make an HTTP GET API call to one of my server, which support HTTP basic authentication using an API key in base64 encoding. so basically I want to add my authorization header in base64 encoding to my request.
The one method of authorization I know is:
>>> import requests
>>> r = requests.get('https://test.com/test-API-Gateway/v0/deployments', auth=('user', 'password'), verify=False)).text
>>> print r
{"statusCode":401,"statusMsg":Unauthorized,"result":[]}
But my server does not return anything, since it does not take id and password for authentication, rather it needs the base64 encoding header. Can you please tell me how to achieve this?
Thanks in advance.

The Python Requests library does allow you to add custom headers. You should be able to create the appropriate header (with your base64 encoding) and pass it as a parameter, like so:
import requests
url = 'https://test.com/test-API-Gateway/v0/deployments'
myheaders = {'my-header-param': 'somedata'}
r = requests.get(url, headers=myheaders, verify=False)).text
The related documentation can be found here.

how to make a delete / put request in python

I can make get or post request using urllib, but how do I make DELETE- and PUT-requests?

The requests library can handle POST, PUT, DELETE, and all other HTTP methods, and is significantly less scary than urllib, httplib and their variants.

You can override get_method with something like this:
def _make_request(url, data, method):
request.urllib2.Request(url, data=data)
request.get_method = lambda: method
Then you pass "DELETE" as method.
This answer covers the details.

PUT request can be performed by httplib2
http://code.google.com/p/httplib2

http://twistedmatrix.com/documents/current/web/howto/client.html
If you're looking to work with HTTP in twisted using the client side I'd suggest checking that out. It demonstrates how you can really easily make a request using the agent class.

As far as I know, urllib and urllib2 only support GET and POST requests. You should probably take a look at httplib or httplib2.

The method is set implicitly in the urlopen call
When you provide the data parameter a POST will be used.
urllib.request.urlopen(url, data=None[, timeout])
I don't think it's possible to use a DELETE HTTP method with urlib because of this line:
Request.get_method()
Return a string
indicating the HTTP request method.
This is only meaningful for HTTP
requests, and currently always returns
'GET' or 'POST'.
Consider using httplib, httplib2, or Twisted instead .for better support of HTTP methods.

The default HTTP methods in urllib library are POST and GET:
def get_method(self):
"""Return a string indicating the HTTP request method."""
default_method = "POST" if self.data is not None else "GET"
return getattr(self, 'method', default_method)
But we can override this get_method() function to get DELETE request:
req = urllib.request.Request(new_url)
req.get_method = lambda: "DELETE"