context = ssl.create_default_context()
context.load_cert_chain(certificate, pkey)
opener = urllib.request.build_opener(urllib.request.HTTPSHandler(context=context))
response = opener.open(url, data=None)
print(response.read())
Executing the above code with data=None automatically sets the METHOD to GET, while setting data to anything else automatically sets the METHOD to POST.
Is there a way to override this behavior?
According to the docs you can use Request.method, but I'm not understanding how to reference that from 'opener'. https://docs.python.org/3/library/urllib.request.html
Request.method
The HTTP request method to use. By default its value is None, which means that get_method() will do its normal computation of the method to be used. Its value can be set (thus overriding the default computation in get_method()) either by providing a default value by setting it at the class level in a Request subclass, or by passing a value in to the Request constructor via the method argument.
New in version 3.3.
Changed in version 3.4: A default value can now be set in subclasses; >previously it could only be set via the constructor argument.
"method should be a string that indicates the HTTP request method that will be used (e.g. 'HEAD'). If provided, its value is stored in the method attribute and is used by get_method(). The default is 'GET' if data is None or 'POST' otherwise. Subclasses may indicate a different default method by setting the method attribute in the class itself."
If you use the urllib.request.Request you can use the method argument to set the specific method you want for this request:
class urllib.request.Request(url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None)
However - when you use the opener you can't provide the method:
OpenerDirector.open(url, data=None[, timeout])
There is no method argument for the open method.
What you can do - is create a Request object and use the opener to send that request:
req = urllib.request.Request(url, method='POST')
res = opener.open(req)
print(res.read())
Related
I'm learning Python requests through a book I purchased. From the book and from the websites I researched, it states this is the proper way to perform a GET request.
requests.get(url, params={key: value}, args)
In PyCharm it also shows the same thing.
However, when I use Postman to test and look at the sample code it shows:
requests.request("GET", params={key: value}, args
I'm not sure if I should use request.get or requests.request("GET",`
and why choose one over the other.
They are both correct and will work the same.
The best way to clear up this confusion is to look at the requests source code.
Here is the code for request.get (as of 2.25.1):
def get(url, params=None, **kwargs):
r"""Sends a GET request.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary, list of tuples or bytes to send
in the query string for the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
kwargs.setdefault('allow_redirects', True)
return request('get', url, params=params, **kwargs)
...which shows that requests.get just calls requests.request with a hardcoded 'get' for the 1st argument. All the other parameters (url, params, **kwargs) are all just passed through.
Basically, it is just a convenience method or a shorthand or a shortcut so you don't have to manually remember which string to pass for the method parameter. It's easier especially when using an IDE because your IDE's IntelliSense can help you select .get or .post or .delete, etc. but not the raw strings "GET", "POST", or "DELETE", etc.
The requests docs can also offer some clarity.
requests.request(method, url, **kwargs)
It says "Constructs and sends a Request.". So this one is for ANY type of request, and you need to pass in 2 required arguments: the method and the URL. All the possible kwargs are listed in the doc, including the params from your example.
requests.get(url, params=None, **kwargs)
It says "Sends a GET request.". So this one is more specific in that it is only for a GET request. It only has 1 required argument, the URL. No need to pass "GET". And then kwargs is "Optional arguments that request takes." which just points back to the main requests.request method.
I would say it's a matter of opinion and coding style which one to use. A use-case where it makes sense to prefer requests.request is when wrapping different API calls and providing a Python interface for them.
For example, I have these APIs:
GET /api/v1/user/[user-id]
PATCH /api/v1/user/[user-id]
DELETE /api/v1/user/[user-id]
When implementing get_user and update_user and delete_user, I could just call the .get, .post, .delete, etc. methods. But if calling these APIs required passing in many and/or complicated kwargs (headers, auth, timeout, etc.), then that would lead to a lot of duplicated code.
Instead, you can do it all in one method then use requests.request:
def get_user(user_id):
return call_api("GET", user_id)
def update_user(user_id, user):
return call_api("PATCH", user_id, user=user)
def delete_user(user_id):
return call_api("DELETE", user_id)
def call_api(method, user_id, user=None):
# Lots of preparation code to make a request
headers = {
"header1": "value1",
"header2": "value2",
# There can be lots of headers...
"headerN": "valueN",
}
timeout = (1, 30)
payload = user.json() if user else {}
url = f"/api/v1/user/{user_id}"
return requests.request(
method,
url,
headers=headers,
timeout=timeout,
json=payload,
)
There are probably other ways to refactor the code above to reduce duplication. That's just an example, where if you called .get, .patch, .delete directly, then you might end up repeating listing all those headers every time, setting up the URL, doing validations, etc.
I have seen Tornado documentations and examples where self.write method is widely used to render some value on HTML, where the POST request was run in a handler. But I could not find much clarity on how to return the response back to client.
For example, I am calling a POST request on a Tornado server from my client. The code that accepts post request is:
class strest(tornado.web.RequestHandler):
def post(self):
value = self.get_argument('key')
cbtp = cbt.main(value)
With this, I can find the value of cbtp and with self.write(cbtp), I can get it printed in HTML. But instead, I want to return this value to the client in JSON format, like {'cbtp':cbtp}
I want to know how to modify my code so that this response is sent to the client, or give me some documentation where this this is fluently explained.
Doing something like
res = {cbtp: cbtp}
return cbtp
throws a BadYieldError: yielded unknown object
You just need to set the output type as JSON and json.dumps your output.
Normally I have the set_default_headers in a parent class called RESTRequestHandler. If you want just one request that is returning JSON you can set the headers in the post call.
class strest(tornado.web.RequestHandler):
def set_default_headers(self):
self.set_header("Content-Type", 'application/json')
def post(self):
value = self.get_argument('key')
cbtp = cbt.main(value)
r = json.dumps({'cbtp': cbtp})
self.write(r)
If the given chunk is a dictionary, we write it as JSON and set the Content-Type of the response to be application/json. (if you want to send JSON as a different Content-Type, call set_header after calling write()).
Using it should give you exactly what you want:
self.write(json.dumps({'cbtp': cbtp}))
Related to Python 2.7
How would one go about building a request through a variable number of kwargs when using requests.
I am using the requests module to directly interact with a REST API which requires a variable number of keyword arguments in order to be successful.
Rather than re-writing the same GET/POST request code, I would like to maintain it within a single api class. However handling the variable number of arguments seems to boil down to a series of if-else statements which isn't particularly readable.
For example:
def request(self):
try:
if self.data:
request = requests.post(url=self.url, headers=self.headers,
data=self.data, timeout=self.timeout, verify=False)
else:
request = requests.get(url=self.url, headers=self.headers,
timeout=self.timeout, verify=False)
...
...
Preferably the request properties are build over time and them passed through a single GET or POST request (granted, the above code would still be require but that is minor).
If you make the attributes default to the same values as arguments to requests.post (basically, None), than you can safely pass all of them as keyword arguments:
def request(self):
try:
request = requests.post(url=self.url, headers=self.headers,
data=self.data, timeout=self.timeout,
verify=False)
...
The code called by my script is (this is code in api.py)
def post(url, data=None, **kwargs):
"""Sends a POST request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
"""
return request('post', url, data=data, **kwargs)
Now I need to POST a request which has no payload, as the info which I need to post is in url itself. I have tried following combinations but failed:
1) requests.post(url, auth, data=None)
Fails saying:
result = requests.post(api, auth, data=None)
TypeError: post() got multiple values for keyword argument 'data'
2) requests.post(api, auth, data=payload) where payload is empty json.
Please suggest..
Passed auth param is accepted by function as data one. Then you passed data again, as keyword argument.
result = requests.post(api, auth, data=None)
TypeError: post() got multiple values for keyword argument 'data'
Try this:
result = requests.post(api, auth=auth)
The issue is the incompatibility between the method signature and how you call it.
def post(url, data=None, **kwargs):
result = requests.post(api, auth, data=None)
First of all, based on the error, its save to assume requests is a library you've written (and not the Python requests module), and not a class because you'd get a completely different error otherwise.
Your post method has 2 arguments, url and data.
Your call has three arguments, which python has to unpack: the variables api and auth and a named argument of data=None.
Python assigns api to the url variable in the methods scope, and auth to the data variable. This leaves an extra named data variable in the methods scope, which now is attempting to be assigned again.
Hence the error:
post() got multiple values for keyword argument 'data'
I'm writing a pythonic web API wrapper with a class like this
import httplib2
import urllib
class apiWrapper:
def __init__(self):
self.http = httplib2.Http()
def _http(self, url, method, dict):
'''
Im using this wrapper arround the http object
all the time inside the class
'''
params = urllib.urlencode(dict)
response, content = self.http.request(url,params,method)
as you can see I'm using the _http() method to simplify the interaction with the httplib2.Http() object. This method is called quite often inside the class and I'm wondering what's the best way to interact with this object:
create the object in the __init__ and then reuse it when the _http() method is called (as shown in the code above)
or create the httplib2.Http() object inside the method for every call of the _http() method (as shown in the code sample below)
import httplib2
import urllib
class apiWrapper:
def __init__(self):
def _http(self, url, method, dict):
'''Im using this wrapper arround the http object
all the time inside the class'''
http = httplib2.Http()
params = urllib.urlencode(dict)
response, content = http.request(url,params,method)
Supplying 'connection': 'close' in your headers should according to the docs close the connection after a response is received.:
headers = {'connection': 'close'}
resp, content = h.request(url, headers=headers)
You should keep the Http object if you reuse connections. It seems httplib2 is capable of reusing connections the way you use it in your first code, so this looks like a good approach.
At the same time, from a shallow inspection of the httplib2 code, it seems that httplib2 has no support for cleaning up unused connections, or to even notice when a server has decided to close a connection it no longer wants. If that is indeed the case, it looks like a bug in httplib2 to me - so I would rather use the standard library (httplib) instead.